A high-performance data conversion method and system based on non-blocking IO communication model
Through the data conversion method based on the non-blocking IO communication model, the problem of low versatility and scalability in heterogeneous data conversion is solved, high-performance data conversion is realized, adapted to different business scenarios and data formats, and data conversion is improved Flexibility and efficiency.
Patent Information
- Application Number
- CN202411074665.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-07
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-08-07
AI Technical Summary
When processing heterogeneous data conversion, the prior art has low versatility and reusability, low development efficiency, and poor scalability, making it difficult to flexibly realize the transformation of complex data conversion logic.
Adopt a high-performance data conversion method based on the non-blocking IO communication model, by receiving data conversion requests, performing source data preprocessing and configuration information analysis, establishing mapping rules between source data and target data, and realizing unified and flexible data formats. Field mapping combines protocol conversion, data preprocessing and data analysis to achieve high-performance data conversion.
It realizes heterogeneous data conversion that is adapted to different business scenarios, supports conversion of different data formats, and realizes data structure description and field mapping through visual configuration, improving the flexibility and performance of data conversion.
Smart Images

Figure CN118585574B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of heterogeneous data conversion, and in particular to a high-performance data conversion method and system based on a non-blocking IO communication model. Background Art
[0002] With the development of informatization, data exchange between enterprises, data applications and requirements within enterprises, and the diversification of technologies involved, data heterogeneity is widely present in enterprise information systems, which has brought some problems to data exchange and sharing. In order to achieve data exchange in different scenarios, it is necessary to convert heterogeneous data by analyzing the structure, structural differences, mapping relationship, conversion rules, etc. of source data and target data.
[0003] At present, the common approach to heterogeneous data conversion is to implement data conversion through hard coding through agreed data formats, or to use open source tools to implement data conversion with limited flexibility at the code level. However, these approaches have great limitations in many data application scenarios. On the one hand, the versatility and reusability are very low. They depend on the constraints and requirements of the data scenario, and many appropriate conversion programs need to be repeatedly written, which greatly reduces the development efficiency. On the other hand, the scalability is low. When the data conversion is complex and the data application scenarios are changing, the data conversion logic cannot be flexibly transformed. Summary of the invention
[0004] In view of the above problems, the present invention provides a high-performance data conversion method based on a non-blocking IO communication model, the method comprising:
[0005] Step S1, receiving a data conversion request from a client, the data conversion request including source data and a unique identifier of an associated data conversion rule;
[0006] Step S2: preprocessing the source data and the source data-associated data conversion rules based on the data conversion request to obtain preprocessed data;
[0007] Step S3: based on the configuration information of the data conversion request and the preprocessed data, the structure, type, dependency between fields and field path hierarchy of the configuration information are parsed to obtain the parsed configuration information;
[0008] Step S4: performing data conversion on the source data based on the parsed configuration information.
[0009] Optionally, in step S1, collecting the content of the data conversion request of each client specifically includes:
[0010] Based on responsive streams and non-blocking asynchronous IO, an asynchronous, non-blocking responsive Web service is established to collect data conversion requests in Web service information.
[0011] Optionally, in step S2, based on the data conversion request, the source data and the source data-associated data conversion rule are preprocessed to obtain the preprocessed data, specifically comprising:
[0012] Based on the data conversion request, the source data is converted into an internal unified JSON format to obtain preliminary data conversion results;
[0013] Parse the associated data conversion rules into clear data structure, field type, field dependency and field path hierarchy configuration information, obtain easy-to-use configuration information after parsing, and complete data preprocessing.
[0014] Optionally, in step S4, the content of performing data conversion on the source data based on the parsed configuration information specifically includes:
[0015] Determine the Json Schema of the source data and target data structure;
[0016] Based on the Json Schema, the Json Path of all fields is obtained by combining the field hierarchy, field type and field name;
[0017] Based on the Json Path, a mapping rule between the source data field and the target data field is established;
[0018] Based on the mapping rules, the field data corresponding to the source data is collected according to the source data field path, and then the source data is processed by taking values to obtain the conversion result and fill it into the target path.
[0019] The present invention also discloses a high-performance data conversion system based on a non-blocking IO communication model, the system comprising: a request collection module, a request preprocessing module, a configuration parsing module and a data conversion module;
[0020] The request collection module is used to receive a data conversion request from a client, the data conversion request including source data and an associated unique identifier of a data conversion rule;
[0021] The request preprocessing module is used to preprocess the source data based on the data conversion request;
[0022] The configuration parsing module is used to parse the structure, type, dependency between fields and field path hierarchy of the configuration information based on the configuration information of the data conversion request and the preprocessed source data information to obtain the parsed configuration information;
[0023] The data conversion module is used to perform data conversion on the source data based on the parsed configuration information.
[0024] Optionally, the workflow of the request collection module specifically includes:
[0025] Based on responsive streams and non-blocking asynchronous IO, an asynchronous, non-blocking responsive Web service is established to collect data conversion requests in Web service information.
[0026] Optionally, the workflow of the request preprocessing module specifically includes:
[0027] Based on the data conversion request, the source data is converted into an internal unified JSON format to obtain preliminary data conversion results;
[0028] Parse the associated data conversion rules into clear data structure, field type, field dependency and field path hierarchy configuration information, obtain easy-to-use configuration information after parsing, and complete data preprocessing.
[0029] Optionally, the workflow of the data conversion module specifically includes:
[0030] Determine the Json Schema of the source data and target data structure;
[0031] Based on the Json Schema, the Json Path of all fields is obtained by combining the field hierarchy, field type and field name;
[0032] Based on the Json Path, a mapping rule between the source data field and the target data field is established;
[0033] Based on the mapping rules, the field data corresponding to the source data is collected according to the source data field path, and then the source data is processed by taking values to obtain the conversion result and fill it into the target path.
[0034] Compared with the prior art, the present invention has the following beneficial effects:
[0035] The present invention is used to solve the existing heterogeneous data conversion problems. A universal and efficient heterogeneous data conversion system is designed and developed by combining technologies such as protocol conversion, data preprocessing, data analysis, data structure description, field mapping, formula calculation, etc. In heterogeneous data conversion for different business scenarios, data conversion adapted to different data formats is implemented flexibly through visual configuration, data structure description, field mapping, grouping and merging, etc., and high-performance data conversion is achieved by combining protocol conversion, data preprocessing, data analysis, data processing and data caching. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0037] Figure 1 A schematic diagram of a data conversion method according to an embodiment of the present invention;
[0038] Figure 2 The data processing flow diagram of the embodiment of the present invention. DETAILED DESCRIPTION
[0039] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.
[0040] Glossary:
[0041] 1.Json: (JavaScript Object Notation, JS object notation) is a lightweight data exchange format used to store and represent data.
[0042] 2. Json Schema: defines a set of vocabulary and rules to define Json metadata, and the metadata is also expressed in the form of Json data.
[0043] 3.Json Path: Represents the elements in Json data through expressions, making it easy to extract content from Json data structures.
[0044] 4. Structure: Use Json Schema to describe the data structure and add some attributes to expand the description of the data structure.
[0045] 5. Responsive: Declarative development based on change delivery.
[0046] 6. Responsive client API: supports asynchronous and synchronous usage, and can communicate with the server in two directions through data streams.
[0047] 7. Asynchronous: Multi-threaded asynchronous processing does not block the current thread to wait for processing to complete, but allows subsequent operations until other threads complete processing and call back to notify this thread.
[0048] 8. Source data: the original data to be converted.
[0049] 9. Target data: the result data after data conversion.
[0050] Embodiment 1
[0051] The present invention provides a high-performance data conversion method based on a non-blocking IO communication model. Figure 1-Figure 2 As shown, the method includes:
[0052] Step S1: receiving a data conversion request from a client, where the data conversion request includes source data and a unique identifier of an associated data conversion rule.
[0053] Based on responsive streams and non-blocking asynchronous IO, an asynchronous, non-blocking responsive Web service is established. At the same time, a responsive client API is also provided. The client can initiate requests and process response results asynchronously and non-blockingly in a functional way.
[0054] The data conversion service supports asynchronous non-blocking processing of requests and responses, and communicates with each other through responsive streams to achieve higher concurrency with fewer threads and faster responses. By providing a series of data format protocols, the conversion of different formats of data in different systems is realized. Based on protocol conversion, the internal data format is unified. At the same time, the request information and configuration information are forwarded to the corresponding data processor.
[0055] Step S2: Based on the data conversion request, preprocess the source data and the source data-associated data conversion rules to obtain preprocessed data.
[0056] The pre-configured protocol conversion template (the protocol template can be cached according to the usage to improve the efficiency of acquisition and parsing during use) can be used to convert the source and target data formats (such as XML to JSON, JSON to XML, etc.) to achieve data format unification, and forward the processed source data to the corresponding service for data processing through routing rules.
[0057] The unified method includes: 1. Convert the source data to JSON (if the original data format is XML); 2. Convert the target data to the desired data format (if the desired data format is XML, JSON needs to be converted to XML). Based on the data conversion request, convert the source data and the data conversion rules associated with the source data to obtain a preliminary conversion result; convert the preliminary conversion result to the target data format to complete data preprocessing.
[0058] Here, the unification of data formats can be understood as the first step of unifying the data format of the source data (for internal data processing) and the second step of unifying the data format of the desired target data. Specifically, in this embodiment, the corresponding configuration is found through the data conversion rule, the source data and the corresponding configuration are pre-processed to obtain the target data, and the target data is format-converted to obtain the final target data.
[0059] The generation of target data requires two format conversions: the first is data preprocessing, 1. converting the source data format (for example, if the source data format is XML, convert it to JSON); 2. obtaining the target data after processing (at this time, the target data format may not be the final desired data format. For example, if the final desired data format is XML, and it will become JSON data format after the previous step, then another format conversion is required to convert it to XML); the second is to convert the target data into the desired data format.
[0060] At the data access layer, data conversion requests are received and data is preprocessed. Based on the configured data preprocessing and data conversion configuration, it is possible to analyze whether the source data needs to be preprocessed, which data nodes, and what preprocessing needs to be done. If necessary, some nodes of the source data can be processed through relevant configuration information.
[0061] (1) By default, a unique identifier is inserted into each data node. This identifier is a string with a specified encoding rule, which contains the following information: the root node path, the parent node path, and the current node identifier. The purpose of inserting a unique identifier is to facilitate the tracing of data nodes in subsequent operations.
[0062] For example, when pushing up / down a node field, or depending on the field value of a parent node, the field value of the parent node or child node can be obtained and used.
[0063] (2) Operations such as pushing up / down node fields and caching source data are necessary at a certain stage of subsequent data conversion. For example, when mapping fields, it is necessary to rely on the difference in the parent node field value to determine the value of the current node field.
[0064] Based on the configured data preprocessing and data conversion configuration, you can flexibly preprocess the source data as needed, such as inserting specific identifiers in specified data nodes, pushing up or down certain attributes of nodes, etc. Through these preprocessing, you can locate data nodes or record the hierarchical relationship between nodes more conveniently and quickly in the subsequent data conversion process.
[0065] Step S3: Based on the configuration information of the data conversion request and the preprocessed source data information, the structure, type, dependency between fields and field path hierarchy of the configuration information are parsed to obtain the parsed configuration information.
[0066] The configuration and source data information are further combined to parse the data structure, type, dependencies between fields, and field path hierarchy. The parsed configuration information will be used in the subsequent data conversion layer, and the parsed configuration will also be cached to achieve efficient data conversion.
[0067] (1) Structure definition module, which is used to determine and store the structure of source and target data. The structure of the data is described through Json Schema, and supplementary descriptions of the structure can be made as needed.
[0068] (2) Field mapping module, which is used to describe the mapping rules between source fields and target fields, including field mapping relationships, value-taking methods, value-taking, and merging methods; Json Path is used to represent field information, and field names, field types, and field hierarchical relationships are included in it in a concise and unified manner. The value-taking method indicates how to process the source field data, such as direct assignment, fixed value, enumeration value mapping, dictionary value mapping, and formula calculation. Complex logical processing can be performed on the source data to respond to complex data conversion requirements in a flexible manner. The merging method indicates the target field value-taking strategy when merging multiple rows to achieve the conversion of collection data to a specific type of data.
[0069] The grouping and merging module is used to specify the grouping and merging rules of the corresponding relationship between the source data and the target data. The data of the source data set type field can be grouped by the specified grouping field; the grouped set data can be merged to generate a new set data by the specified merging field.
[0070] Step S4: performing data conversion on the source data based on the parsed configuration information.
[0071] (1) Determine the source and target data structure Json Schema. You can define the Json Schema based on the actual Json data or Json data format, including but not limited to the following field information: field name, field title, field type, field mandatory constraint, field length, field format, etc. Among them, the field name is unique at the same level, the field name, field title, and field type are required field information, and the field mandatory constraint, field length, field format, etc. can be selected according to the actual situation.
[0072] (2) Based on the Json Schema obtained in the previous step, the Json Path of all fields can be obtained by combining the field hierarchy, field type, and field name. The Json Path of a field can uniquely identify the path information of the field in the entire data structure. In the data conversion processing logic, the data of the corresponding path can be read or filled through the Json Path of the field.
[0073] (3) Based on the Json Path of the field, you can establish mapping rules between the source field and the target field. The source field path (corresponding to the Json Path of the source field) and the target field path (corresponding to the Json Path of the target field) are used as identifiers for a field mapping rule. Each field mapping rule includes but is not limited to the following: source field path, target field path, value, value-taking method, and merging method. Among them, the value is determined by the value-taking method; the value-taking method is a strategy for obtaining or generating the target field value. The final value can be determined by the value-taking method alone, or by the value-taking method and the merging method together. The value-taking methods include but are not limited to the following methods:
[0074] 1) Direct assignment: The target field value comes directly from the source field, regardless of direct mapping or indirect mapping
[0075] 2) Fixed value: The target field value is a given fixed value
[0076] 3) Enumeration value mapping: The target field value is the value of the source field after being converted using the enumeration value mapping rule.
[0077] 4) Formula calculation: Preset formulas are used to perform complex calculations on source fields. The preset formulas here can be understood as a series of functions that encapsulate specific functions. They can be general functions or functions that include specific business processing capabilities to simplify configuration and improve configuration flexibility and scalability.
[0078] 5) Merge method: When the field type is a non-set type and the actual value is a set, as a supplement to the value-taking method, methods such as concatenation, summing, and taking the first row are provided here. Select the corresponding merge method based on business needs to realize the data conversion from the set type to the non-set type.
[0079] (4) Based on the Json Path of the field, configure the grouping field and establish grouping rules. This step is optional. When we want to obtain classification information of a collection of data, we can use grouping. For example: when we want to distinguish a batch of orders by order status, we group the collection data under a certain field by all the values of the grouping field. Finally, the source data will be split into grouped data with consistent structure according to the grouping.
[0080] (5) Based on the Json Path of the field, configure the merge basis field and establish the merge rule. This step is optional. When we want to obtain the classified statistical information of a set of data, we can use the merge. For example: when we want to obtain the number of orders under the corresponding status of a batch of orders, we use the same value of the merge basis field as a condition to merge the set data under a certain field. During the merge process, in addition to the merge basis field, other fields need to specify the merge method to ensure that the value of the target field is semantically clear.
[0081] Convert the Json content of the source data to the Json content of the target data. Using the target data structure JsonSchema, the target data can be gradually constructed based on the field hierarchy, field name, field type and other information. Based on the field mapping rules, in the process of constructing the target data, the field data corresponding to the source data is collected according to the source field path, and then the source value can be processed accordingly through the value-taking method, and the result value obtained can be filled in the specified target path. The processing process includes: 1. Based on the mapping rules, the field data corresponding to the source data is collected (the data here can be understood as the raw materials to be processed next); 2. Processing is performed according to the value-taking method (the value-taking method here can be understood as the processing means or the value processing strategy), and there are the following methods: direct assignment, fixed value, enumeration value mapping, formula calculation, and merging method. Through these value-taking methods, the collected field data is processed. Each value-taking method has different processing methods, and different result data will be obtained (that is, the conversion result obtained after processing a certain field data).
[0082] Combining the above configuration information and source data, some parsing and analysis can be done, such as structure parsing, type parsing, field dependency analysis, and path hierarchy analysis. After completing data parsing, data conversion can be achieved based on the parsing results, and the conversion of source data to target data can be achieved through field mapping, group merging, formula calculation and other configuration information.
[0083] Embodiment 2
[0084] A high-performance data conversion system based on a non-blocking IO communication model, the system comprising:
[0085] The request collection module is used to receive a data conversion request from a client, where the data conversion request includes source data and a unique identifier of an associated data conversion rule.
[0086] Based on responsive streams and non-blocking asynchronous IO, an asynchronous, non-blocking responsive Web service is established. At the same time, a responsive client API is also provided. The client can initiate requests and process response results asynchronously and non-blockingly in a functional way.
[0087] The data conversion service supports asynchronous non-blocking processing of requests and responses, and communicates with each other through responsive streams to achieve higher concurrency with fewer threads and faster responses. By providing a series of data format protocols, the conversion of different formats of data in different systems is realized. Based on protocol conversion, the internal data format is unified. At the same time, the request information and configuration information are forwarded to the corresponding data processor.
[0088] The request preprocessing module is used to preprocess the source data based on the data conversion request.
[0089] The pre-configured protocol conversion template (the protocol template can be cached according to the usage to improve the efficiency of acquisition and parsing during use) can be used to convert the source and target data formats (such as XML to JSON, JSON to XML, etc.) to achieve data format unification, and forward the processed source data to the corresponding service for data processing through routing rules.
[0090] At the data access layer, data conversion requests are received and data is preprocessed. Based on the configured data preprocessing and data conversion configuration, it is possible to analyze whether the source data needs to be preprocessed, which data nodes, and what preprocessing needs to be done. If necessary, some nodes of the source data can be processed through relevant configuration information.
[0091] (1) By default, a unique identifier is inserted into each data node. This identifier is a string with specified encoding rules, which contains the following information: root node path, parent node path and current node identifier.
[0092] (2) Operations such as pushing up / down node fields and caching source data are necessary at a certain stage of subsequent data conversion. For example, when mapping fields, it is necessary to rely on the difference in the parent node field value to determine the value of the current node field.
[0093] Based on the configured data preprocessing and data conversion configuration, you can flexibly preprocess the source data as needed, such as inserting specific identifiers in specified data nodes, pushing up or down certain attributes of nodes, etc. Through these preprocessing, you can locate data nodes or record the hierarchical relationship between nodes more conveniently and quickly in the subsequent data conversion process.
[0094] The configuration parsing module is used to parse the structure, type, dependencies between fields, and field path hierarchy of the configuration information based on the configuration information of the data conversion request and the preprocessed source data information to obtain the parsed configuration information.
[0095] The configuration and source data information are further combined to parse the data structure, type, dependencies between fields, and field path hierarchy. The parsed configuration information will be used in the subsequent data conversion layer, and the parsed configuration will also be cached to achieve efficient data conversion.
[0096] (1) Structure definition module, which is used to determine and store the structure of source and target data. The structure of the data is described through Json Schema, and supplementary descriptions of the structure can be made as needed.
[0097] (2) Field mapping module, which is used to describe the mapping rules between source fields and target fields, including field mapping relationships, value-taking methods, value-taking, and merging methods; Json Path is used to represent field information, and field names, field types, and field hierarchical relationships are included in it in a concise and unified manner. The value-taking method indicates how to process the source field data, such as direct assignment, fixed value, enumeration value mapping, dictionary value mapping, and formula calculation. Complex logical processing can be performed on the source data to respond to complex data conversion requirements in a flexible manner. The merging method indicates the target field value-taking strategy when merging multiple rows to achieve the conversion of collection data to a specific type of data.
[0098] The grouping and merging module is used to specify the grouping and merging rules of the corresponding relationship between the source data and the target data. The data of the source data set type field can be grouped by the specified grouping field; the grouped set data can be merged to generate a new set data by the specified merging field.
[0099] The data conversion module is used to perform data conversion on the source data based on the parsed configuration information.
[0100] (1) Determine the source and target data structure Json Schema. You can define the Json Schema based on the actual Json data or Json data format, including but not limited to the following field information: field name, field title, field type, field mandatory constraint, field length, field format, etc. Among them, the field name is unique at the same level, the field name, field title, and field type are required field information, and the field mandatory constraint, field length, field format, etc. can be selected according to the actual situation.
[0101] (2) Based on the Json Schema obtained in the previous step, the Json Path of all fields can be obtained by combining the field hierarchy, field type, and field name. The Json Path of a field can uniquely identify the path information of the field in the entire data structure. In the data conversion processing logic, the data of the corresponding path can be read or filled through the Json Path of the field.
[0102] (3) Based on the Json Path of the field, you can establish mapping rules between the source field and the target field. The source field path (corresponding to the Json Path of the source field) and the target field path (corresponding to the Json Path of the target field) are used as identifiers for a field mapping rule. Each field mapping rule includes but is not limited to the following: source field path, target field path, value, value-taking method, and merging method. Among them, the value is determined by the value-taking method; the value-taking method is a strategy for obtaining or generating the target field value. The final value can be determined by the value-taking method alone, or by the value-taking method and the merging method together. The value-taking methods include but are not limited to the following methods:
[0103] 1) Direct assignment: The target field value comes directly from the source field, regardless of direct mapping or indirect mapping
[0104] 2) Fixed value: The target field value is a given fixed value
[0105] 3) Enumeration value mapping: The target field value is the value of the source field after being converted using the enumeration value mapping rule.
[0106] 4) Formula calculation: Preset formulas are used to perform complex calculations on source fields. The preset formulas here can be understood as a series of functions that encapsulate specific functions. They can be general functions or functions that include specific business processing capabilities to simplify configuration and improve configuration flexibility and scalability.
[0107] 5) Merge method: When the field type is a non-set type and the actual value is a set, as a supplement to the value-taking method, methods such as concatenation, summing, and taking the first row are provided here. Select the corresponding merge method based on business needs to realize the data conversion from the set type to the non-set type.
[0108] (4) Based on the Json Path of the field, configure the grouping field and establish grouping rules. This step is optional. When we want to obtain classification information of a collection of data, we can use grouping. For example: when we want to distinguish a batch of orders by order status, we group the collection data under a certain field by all the values of the grouping field. Finally, the source data will be split into grouped data with consistent structure according to the grouping.
[0109] (5) Based on the Json Path of the field, configure the merge basis field and establish the merge rule. This step is optional. When we want to obtain the classified statistical information of a set of data, we can use the merge. For example: when we want to obtain the number of orders under the corresponding status of a batch of orders, we use the same value of the merge basis field as a condition to merge the set data under a certain field. During the merge process, in addition to the merge basis field, other fields need to specify the merge method to ensure that the value of the target field is semantically clear.
[0110] Convert the Json content of the source data to the Json content of the target data. Using the target data structure JsonSchema, the target data can be gradually constructed based on the field hierarchy, field name, field type and other information. Based on the field mapping rules, in the process of constructing the target data, the field data corresponding to the source data is collected according to the source field path, and then the source value can be processed accordingly through the value acquisition method, and the result value can be filled in the specified target path.
[0111] Combining the above configuration information and source data, some parsing and analysis can be done, such as structure parsing, type parsing, field dependency analysis, and path hierarchy analysis. After completing data parsing, data conversion can be achieved based on the parsing results, and the conversion of source data to target data can be achieved through field mapping, group merging, formula calculation and other configuration information.
[0112] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.
Claims
1. A high-performance data conversion method based on a non-blocking IO communication model, characterized in that: The method comprises: Step S1, receiving a data conversion request from a client, the data conversion request including source data and a unique identifier of an associated data conversion rule; Step S2: preprocessing the source data and the unique identifier of the data conversion rule associated with the source data based on the data conversion request to obtain preprocessed data, specifically including: Based on the data conversion request, the source data is converted into an internal unified JSON format to obtain preliminary data conversion results; Parse the associated data conversion rule unique identifier into clear data structure, field type, field dependency and field path hierarchy configuration information, obtain easy-to-use configuration information after parsing, and complete data preprocessing; Step S3: based on the configuration information of the data conversion request and the preprocessed data, the structure, type, dependency between fields and field path hierarchy of the configuration information are parsed to obtain the parsed configuration information; Determine and store the structure of source and target data, describe the data structure through Json Schema, and make supplementary descriptions of the structure as needed; Describe the mapping rules between source fields and target fields, including field mapping relationships, value-taking methods, value-taking, and merging methods; use Json Path to represent field information, and include field names, field types, and field hierarchical relationships in a concise and unified manner; process source field data through value-taking methods, and perform logical processing on source data to cope with complex data conversion requirements; specify the target field value-taking strategy when merging multiple rows through merging methods, and implement the conversion of collection data to specific types of data; Specify the grouping and merging rules for the corresponding relationship between source data and target data; group the data of the source data set type field by the grouping basis field; merge the grouped set data to generate a new set data by the merging basis field; Step S4, performing data conversion on the source data based on the parsed configuration information, specifically comprising: Determine the Json Schema of the source data and target data structure; Based on the Json Schema, the Json Path of all fields is obtained by combining the field hierarchy, field type and field name; Based on the Json Path, a mapping rule between the source data field and the target data field is established; Based on the mapping rules, the field data corresponding to the source data is collected according to the source data field path, and then the source data is processed by taking values to obtain the conversion result and fill it into the target path.
2. The high-performance data conversion method based on the non-blocking IO communication model according to claim 1 is characterized in that: In step S1, receiving the data conversion request from the client specifically includes: Based on responsive streams and non-blocking asynchronous IO, an asynchronous, non-blocking responsive Web service is established to collect data conversion requests in Web service information.
3. A high-performance data conversion system based on a non-blocking IO communication model, the system is used to implement the high-performance data conversion method according to any one of claims 1-2, characterized in that: The system includes: a request collection module, a request preprocessing module, a configuration parsing module and a data conversion module; The request collection module is used to receive a data conversion request from a client, the data conversion request including source data and an associated unique identifier of a data conversion rule; The request preprocessing module is used to preprocess the source data and the source data-associated data conversion rules based on the data conversion request to obtain preprocessed data; The configuration parsing module is used to parse the structure, type, dependency between fields and field path hierarchy of the configuration information based on the configuration information of the data conversion request and the preprocessed data to obtain the parsed configuration information; The data conversion module is used to perform data conversion on the source data based on the parsed configuration information.
4. The high-performance data conversion system based on the non-blocking IO communication model according to claim 3 is characterized in that: The workflow of the request collection module specifically includes: Based on responsive streams and non-blocking asynchronous IO, an asynchronous, non-blocking responsive Web service is established to collect data conversion requests in Web service information.
5. The high-performance data conversion system based on the non-blocking IO communication model according to claim 3 is characterized in that: The workflow of the request preprocessing module specifically includes: Based on the data conversion request, the source data is converted into an internal unified JSON format to obtain preliminary data conversion results; Parse the associated data conversion rules into clear data structure, field type, field dependency and field path hierarchy configuration information, obtain easy-to-use configuration information after parsing, and complete data preprocessing.
6. The high-performance data conversion system based on the non-blocking IO communication model according to claim 3 is characterized in that: The workflow of the data conversion module specifically includes: Determine the Json Schema of the source data and target data structure; Based on the Json Schema, the Json Path of all fields is obtained by combining the field hierarchy, field type and field name; Based on the Json Path, a mapping rule between the source data field and the target data field is established; Based on the mapping rules, the field data corresponding to the source data is collected according to the source data field path, and then the source data is processed by taking values to obtain the conversion result and fill it into the target path.
