Dynamic data extraction method supporting multi-serialization protocol

Abstract

The invention discloses a dynamic data extraction method supporting a multi-serialization protocol, which comprises the following steps of: modeling a relationship between data used in data extraction service implementation, and solidifying the data in a database; the data extraction framework reads the service data extraction model in the database and drives the execution process of data query and serialization through the model, the data query and serialization process is standardized, a secondary development interface for data extraction is provided for upper-layer services, and the data extraction efficiency is improved. And a data query method and a data serialization format are supported to be modified when a program runs.

Description

Technical Field

[0001] The invention belongs to the technical field of software design and development, and in particular relates to a dynamic data extraction method supporting multiple serialization protocols. Background Art

[0002] Information systems often need to interact with external systems for data. The types of interactive data formats are reflected in two aspects: first, there are many data serialization protocols used: binary protocol may be used for interaction with a certain system, and XML protocol or Json protocol may be used for interaction with other systems; second, there are many business types of data: even if the same serialization protocol is used, in order to achieve different business functions, the interactive data content and data format may be different.

[0003] In order to meet the above requirements, developers not only need to care about the business logic itself when coding, but also spend a lot of work on repetitive work such as data encapsulation for different data types, which reduces work efficiency.

[0004] With the upgrade and transformation of the information system itself or external systems, the data format and content of the interaction between information systems often need to change. If each change requires modifying the code and recompiling and deploying, the scope of impact is too large and the cost of system use and maintenance is too high. Summary of the invention

[0005] In view of this, the present invention provides a dynamic data extraction method supporting multiple serialization protocols, which can support the program to modify the data query method and data serialization format during runtime.

[0006] The technical solution for implementing the present invention is as follows:

[0007] A dynamic data extraction method supporting multiple serialization protocols comprises the following steps:

[0008] Step 1: The upper-layer business calls the interface of the data extraction framework and passes the data type to be extracted and the query conditions necessary for extracting data into the data extraction framework;

[0009] Step 2: The data extraction framework calls MyBatis to call the data structure query method encapsulated in XML through the reflection mechanism of JAVA according to the class name and function name of the data structure query method configured in the data extraction business data model, and encapsulates the key-value pairs in the query result set in XML form to form a sub-data structure;

[0010] Step 3: The data extraction framework merges all XML-formatted sub-data structure query results into an XML result file, and uses the XML result file as the input of the data standardization method of this data type. The data standardization method calls the standardized XSLT file to convert the XML result file into a form that conforms to the standardized structure.

[0011] Step 4: The data extraction framework verifies the standardized XML according to the data standard format described in the XSD file of this data type. If it meets the standard, it starts to serialize the standardized XML document.

[0012] Furthermore, step one is specifically as follows: the data extraction framework provides a data extraction API interface, providing the upper-level business application with the ability of dynamic data query and dynamic protocol encapsulation; the upper-level business application, as the initiator of the extraction service, passes the parameters required for data query to the data extraction API interface, and after the data extraction framework service is executed, returns data that meets the format requirements to the upper-level business application.

[0013] Furthermore, the data query and serialization execution process of the data extraction framework is driven by the data extraction business data model; the data entity of the data extraction business data model includes two parts: database data and disk files.

[0014] Furthermore, specific data objects in the data extraction business data model include export data types, data results, data query methods, data conversion methods, and data verification methods; disk files include XML files used by the MyBatis framework and XSLT, proto files, and XSD files used in the serialization process.

[0015] Furthermore, the data model of the data extraction process is:

[0016] Split each data type into several sub-data structures in terms of structure. Each sub-data structure is a complete object in terms of business logic. Define a query method for the relevant data table in the business database corresponding to each sub-data structure. The query method definition should include the JAVA class name and method name that implement the query method. Write the SQL statement that implements the data structure query method in an XML file and place the file in a location that can be scanned and called by the MyBatis framework.

[0017] Define a data standardization conversion method for each data type. The method includes a standardized XSLT file, which describes the method of converting the execution result of the data structure query method expressed in XML form into the XML standardized structure corresponding to the data type; configure an XSD file for the standardized structure of each data type. The XSD file defines the format followed by the serialized XML document. The data extraction framework uses this file to check the format of the serialized XML data.

[0018] A serialization method is defined for each data type. The serialization method defines the serialization protocol that the output data should follow and the conversion file required for serialization. The conversion file defines the method of converting the XML standardized structure into the data structure required for output. Depending on the serialization protocol, the conversion file may have only one serialization XSLT file. If the data is exported in ProtocolBuffer format, a matching proto file is also required. If the data is exported in XML format, the serialization XSLT file implements the method of converting the standardized structure into the XML format required for output. If the data is exported in ProtocolBuffer format or JSON format, the serialization XSLT file implements the method of converting the standardized structure into the intermediate XML format.

[0019] Beneficial effects:

[0020] 1. The data extraction framework of the present invention models the data extraction and export process and stores it in a relational database. The purpose of modifying the data query method during program operation can be achieved by configuring certain parameters in the model. After the system is deployed, if there is a system upgrade requirement to update the output data format, it can be quickly implemented by upgrading the model configuration data, which improves the flexibility of the data extraction business.

[0021] 2. The data extraction framework of the present invention divides the data extraction process into two processes: data structure query and data structure serialization. The data structure query outputs a standardized XML file as the input of the data structure serialization process. For the same queried data structure, different serialization protocols can be used for conversion to achieve the reuse of the data structure query logic.

[0022] 3. The data extraction framework of the present invention provides a secondary development interface to the upper-level business, shielding the database query and data serialization process from the interface caller. The interface caller can focus on business-related tasks such as writing data query statements and extracting data format descriptions, thereby reducing the development burden of the interface caller and improving the development efficiency of the data extraction business. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1This is the working principle of the data extraction framework of the present invention.

[0024] Figure 2 This is a schematic diagram of the data extraction business data model of the present invention.

[0025] Figure 3 The software architecture of the data extraction framework of the present invention.

[0026] Figure 4 This is a data query and standardization workflow diagram for the data extraction framework of the present invention.

[0027] Figure 5 This is a schematic diagram of the principle of converting XML to JSON in an embodiment of the present invention. DETAILED DESCRIPTION

[0028] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0029] In view of the need for multiple data types and changing formats during information system interaction, the present invention designs a dynamic data extraction method that supports multiple serialization protocols after integrating and using the existing Java development framework. The method standardizes the process of data query and serialization, provides a secondary development interface for data extraction to upper-level services, and supports the program to modify the data query method and data serialization format at runtime.

[0030] How data extraction methods work

[0031] like Figure 1 As shown, by modeling the data used in the data extraction business implementation and the relationship between the data, it is solidified in the database. The data extraction framework reads the extraction business data model in the database and drives the execution process of data query and serialization with the model.

[0032] The data extraction framework provides a data extraction API interface, which provides the upper-level business application with the ability to dynamically query data and dynamically encapsulate protocols. As the initiator of the extraction service, the upper-level business application passes the parameters required for data query to the data extraction API interface. After the data extraction framework service is executed, it returns data that meets the format requirements to the upper-level business application.

[0033] The data query and serialization execution process of the data extraction framework is driven by the data extraction business data model. The data entities of the data extraction business data model include database data and disk files.

[0034] Based on object-oriented thinking, the data objects and the relationship between data objects in the data extraction and encapsulation process are identified, and this ER relationship is stored in a relational database. The specific data objects in the data extraction business data model include export data types, data results, data query methods, data conversion methods, and data verification methods. The disk files include XML files used by the MyBatis framework and XSLT, proto files, and XSD files used in the serialization process.

[0035] Data Modeling for the Data Extraction Process

[0036] The data extraction business data model is the core of realizing dynamic data query and dynamic data serialization. The most important work of building a data extraction business data model is to identify the data type extracted by the framework and configure the query and serialization methods for each data type. Figure 2 shown.

[0037] Each data type is structurally split into several sub-data structures, each of which is a complete object in terms of business logic. For each sub-data structure, define a query method for the relevant data table in the business database corresponding to the data structure. The query method definition should include the JAVA class name and method name that implements the query method. Write the SQL statement that implements the data structure query method in an XML file, and place the file in a location that can be scanned and called by the MyBatis framework.

[0038] A data standardization conversion method is defined for each data type. The method includes a standardized XSLT file, which describes the method of converting the execution result of the data structure query method represented in XML form into the XML standardized structure corresponding to the data type. An XSD file is configured for the standardized structure of each data type. The XSD file defines the format followed by the serialized XML document. The data extraction framework uses this file to check the format of the serialized XML data.

[0039] Define a serialization method for each data type. The serialization method defines the serialization protocol that the output data should follow and the conversion file required for serialization. The conversion file defines the method of converting the XML standardized structure into the data structure required for output. Depending on the serialization protocol, the conversion file may only have a serialization XSLT file. If the data is exported in ProtocolBuffer format, a matching proto file is also required. If the data is exported in XML format, the serialization XSLT file implements the method of converting the standardized structure into the XML format required for output; if the data is exported in ProtocolBuffer format or JSON format, the serialization XSLT file implements the method of converting the standardized structure into an intermediate XML format.

[0040] Software architecture of the data extraction framework

[0041] like Figure 3 As shown in the figure, the data extraction framework is developed in JAVA language, and the operation of the software depends on the JAVA runtime environment. The data extraction framework uses the Spring framework to integrate various frameworks or components. The Log4j component provides the data extraction framework with the function of diary recording; the Dom4j component provides the data extraction framework with the function of XML document operation. The database access of the data extraction framework is implemented through the MyBatis framework, and the database connection pool is managed by Hikari at the bottom layer.

[0042] The core business of the data extraction framework is divided into entity layer, data access layer, service layer and interface layer.

[0043] The entity layer adopts object-oriented thinking, and defines the objects and relationships between objects defined in the data extraction business data model as JAVA classes, corresponding to the data tables of the data model persisted in the relational database. The data type entity, data structure entity, data query method entity, data serialization method entity and data verification method entity correspond to the five objects in the data extraction business data model respectively; the data query method configuration entity corresponds to the correspondence between the data structure and the data query method; the data serialization method configuration corresponds to the correspondence between the data type and the data query method; the data verification method configuration entity corresponds to the correspondence between the data type and the data verification method.

[0044] The data access layer implements query methods for each entity in the entity layer.

[0045] The service layer modules implement the core functions of the data extraction framework. The data model loading module implements the function of calling the data access layer data query method from the database to query all entities of the data extraction business data model and load the model into the memory; the query result serialization module implements the function of calling the database access method of the business layer to query the business database according to the data query method configuration, and converts the query results into XML format; the query result standardization module implements the function of converting the query result XML into standardized XML format according to the data standardization method configuration; the standardized result verification module implements the function of verifying the standardized XML data according to the data verification method configuration; the standardized result serialization implements the function of serializing the standardized XML data that has passed the verification into the output required data format according to the configuration according to the data serialization method configuration.

[0046] The interface layer provides a data extraction interface for upper-layer business calls, with the extracted data type and business data query conditions expressed in the form of key-value pairs as input parameters, and serialized data as output parameters.

[0047] Data extraction framework Data query and standardization workflow such as Figure 4 As shown, the specific steps include:

[0048] a) The upper-layer business calls the interface of the data extraction framework and passes the data type to be extracted and the query conditions necessary for extracting data into the data extraction framework;

[0049] b) The data extraction framework calls MyBatis to call the data structure query method encapsulated in XML through the reflection mechanism of JAVA according to the class name and function name of the data structure query method configured in the data extraction business data model, and encapsulates the key-value pairs in the query result set in XML form to form a sub-data structure;

[0050] c) The data extraction framework merges all XML-formatted sub-data structure query results into one XML result file, and uses the XML result file as the input of the data standardization method of this data type. The data standardization method calls the standardized XSLT file to convert the XML result file into a form that conforms to the standardized structure;

[0051] d) The data extraction framework verifies the standardized XML according to the data standard format described in the XSD file of this data type. If it meets the standard, it starts serializing the standardized XML document.

[0052] Data extraction framework supports multiple serialization protocols

[0053] In order to be compatible with the data format requirements of different data consumers, the data extraction framework supports the conversion and output of extracted data through multiple serialization protocols.

[0054] The starting point of data extraction framework serialization is the standardized XML data that has been verified based on the data standard format description XSD file; the caller of the data extraction framework needs to agree with the data consumer to provide different types of files according to different serialization protocols: if you want to output unformatted XML, you need to provide an XSLT file that converts standardized XML to output format XML; if you want to output

[0055] ProtocolBuffer, you need to provide a proto file that describes the output data structure and an XSLT file that corresponds to the output structure; if you want to output in JSON format, you need to provide an XSLT file that corresponds to the JSON structure.

[0056] (1) Support for XML protocol format

[0057] Through XSLT technology, the data extraction framework performs secondary conversion on the standard formatted data and outputs it in the format required by the user.

[0058] (2) Support for Protobuf protocol

[0059] The data extraction framework outputs data in Protobuf protocol format in two steps:

[0060] The first step is to perform a secondary conversion on the standard formatted data through XSLT technology, converting the extracted data into an intermediate XML format. The tag name or attribute name of the intermediate XML is the same as the attribute name of the proto file, and the tag level of the intermediate XML is the same as the attribute level of the same name in the proto file.

[0061] The second step is to call the Protobuf command at runtime to generate the descriptor self-description file corresponding to proto; use the dynamic message mechanism of Protobuf to generate a message instance of the data type in the self-description file; obtain the attribute name and data type in the self-description file through the reflection mechanism of ProtocolBuffers, find the value of the tag with the same name as the attribute name in the self-description file from the intermediate XML, convert it into the required data type and assign it to the dynamic message instance object.

[0062] The third step is to call the Protobuf interface to serialize the dynamic message instance object and generate

[0063] Output data in Protobuf format.

[0064] (3) Support for Json protocol format

[0065] The data extraction framework outputs data in Json protocol format in two steps:

[0066] The first step is to perform a secondary conversion on the standard formatted data through XSLT technology, converting the extracted data into an intermediate XML format. The tag name or attribute name of the intermediate XML is the same as the key name of JSON, and the tag level of the intermediate XML is the same as the key-value pair level of the same name in JSON.

[0067] The second step is to convert the converted intermediate XML data into JSON data. Figure 5 As shown, the conversion rules are: use the tag of the intermediate XML data as the key name of the JSON data. When the tag contains other attributes, the value of the XML data is used as the value of the data with key as value in the JSON. The XML tag attribute name is used as the key name of the JSON data. The XML tag attribute value is used as the value of the JSON data. When the tag does not contain other attributes, the XML tag value is used as the value of the JSON data.

[0068] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A dynamic data extraction method supporting multiple serialization protocols, characterized in that: The following steps are involved: Step 1: The upper-layer business calls the interface of the data extraction framework and passes the data type to be extracted and the query conditions necessary for extracting data into the data extraction framework; Step 2: The data extraction framework calls MyBatis to call the data structure query method encapsulated in XML through the reflection mechanism of JAVA according to the class name and function name of the data structure query method configured in the data extraction business data model, and encapsulates the key-value pairs in the query result set in XML form to form a sub-data structure; Step 3: The data extraction framework merges all XML-formatted sub-data structure query results into an XML result file, and uses the XML result file as the input of the data standardization method of this data type. The data standardization method calls the standardized XSLT file to convert the XML result file into a form that conforms to the standardized structure. Step 4: The data extraction framework verifies the standardized XML according to the data standard format described in the XSD file of this data type. If it meets the standard, it starts to serialize the standardized XML document.

2. The dynamic data extraction method according to claim 1, characterized in that: Step one is as follows: the data extraction framework provides a data extraction API interface, and provides the upper-level business application with the ability to dynamically query data and dynamically encapsulate protocols; the upper-level business application, as the initiator of the extraction service, passes the parameters required for data query to the data extraction API interface, and after the data extraction framework service is executed, it returns data that meets the format requirements to the upper-level business application.

3. The dynamic data extraction method according to claim 1 or 2, characterized in that: The data query and serialization execution process of the data extraction framework is driven by the data extraction business data model; the data entities of the data extraction business data model include database data and disk files.

4. The dynamic data extraction method according to claim 1, characterized in that: The specific data objects in the data extraction business data model include exported data types, data results, data query methods, data conversion methods, and data verification methods; the disk files include XML files used by the MyBatis framework and XSLT, proto files, and XSD files used in the serialization process.

5. The dynamic data extraction method according to claim 4, characterized in that: The data model for the data extraction process is: Split each data type into several sub-data structures in terms of structure. Each sub-data structure is a complete object in terms of business logic. Define a query method for the relevant data table in the business database corresponding to each sub-data structure. The query method definition should include the JAVA class name and method name that implement the query method. Write the SQL statement that implements the data structure query method in an XML file and place the file in a location that can be scanned and called by the MyBatis framework. Define a data standardization conversion method for each data type. The method includes a standardized XSLT file, which describes the method of converting the execution result of the data structure query method expressed in XML form into the XML standardized structure corresponding to the data type; configure an XSD file for the standardized structure of each data type. The XSD file defines the format followed by the serialized XML document. The data extraction framework uses this file to check the format of the serialized XML data. A serialization method is defined for each data type. The serialization method defines the serialization protocol that the output data should follow and the conversion file required for serialization. The conversion file defines the method of converting the XML standardized structure into the data structure required for output. Depending on the serialization protocol, the conversion file may have only one serialization XSLT file. If the data is exported in ProtocolBuffer format, a matching proto file is also required. If the data is exported in XML format, the serialization XSLT file implements the method of converting the standardized structure into the XML format required for output. If the data is exported in ProtocolBuffer format or JSON format, the serialization XSLT file implements the method of converting the standardized structure into the intermediate XML format.

Images