Configuration-based data format conversion system
Through the configuration-based data format conversion system, the configuration file is used to define data conversion rules, and interface adaptation between different systems is achieved, the problem of inconsistent interfaces between systems is solved, the flexibility and maintainability of the system is improved, and multiple data is supported. Format and conversion requirements.
Patent Information
- Application Number
- CN202410591208.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-05-13
- Publication Date
- 2024-11-19
AI Technical Summary
When data transmission between different systems, due to inconsistent interfaces and different protocols and formats, it is difficult to directly connect, and it is difficult for the existing technology to realize interface adaptation without code modification.
Design a configuration-based data format conversion system, including back-end module, front-end module, Tpli module, call module and adaptive module, define data conversion rules through configuration files, and use multi-threaded processing and state management to realize the data format and structure. Convert, supports json and xml formats, and adapts to different data sources through the adapter module.
It realizes flexible adaptation of inter-system interfaces, reduces maintenance costs, improves the reusability and scalability of the system, supports a variety of data formats and conversion requirements, and does not require modification of program code.
Smart Images

Figure CN118981490A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data exchange, and in particular to a data format conversion system based on configuration. Background Art
[0002] Data format conversion refers to converting data from one format to another so that it can be interacted and shared between different systems and applications. Common types of data format conversion include text format conversion, image format conversion, audio format conversion, video format conversion, etc. In data format conversion, common conversion methods include batch conversion and instant conversion. Batch conversion refers to converting a large amount of data into another format at one time, which is usually used in scenarios such as data migration and data processing; while instant conversion refers to converting a single data item into another format when needed, which is common in scenarios such as data display and data input on web pages.
[0003] When two systems are connected, it is generally not feasible for system A to directly call system B. The data transmission interfaces between the systems are inconsistent and cannot be directly connected. Usually, the interface of system A cannot be directly connected to the interface of system B because of different protocols / http, https, webservice, socket, etc., different formats, xml, json, etc., different structures, inconsistent hierarchical structures represented by xml or json, different meanings of field values, and inconsistent meanings of field values. Between system A and system B, the Tpli system is used for adaptation, so that the interfaces of the two systems can be opened up without modifying the code of either party. Summary of the invention
[0004] The purpose of the invention provided by the present invention is to provide a configuration-based data format conversion system to solve the problems in the above-mentioned background technology.
[0005] To achieve the above objectives, the present invention is implemented through the following technical solutions: A data format conversion system based on configuration, comprising:
[0006] Backend module, used for data transmission and multi-threaded processing.
[0007] The front-end module is responsible for providing an interactive interface between the user and the system, allowing the user to edit configuration information, start data conversion and view conversion results.
[0008] Tpli module is used to complete data information conversion.
[0009] Calling module, used for calling data.
[0010] The adapter module adapts different types of data sources to the specified framework.
[0011] The back-end module is connected to the front-end module by two-way signals, the back-end module is connected to the calling module by two-way signals, the back-end module is connected to the adapting module by two-way signals, and the back-end module is connected to the Tpli module by two-way signals.
[0012] Furthermore, the backend module includes a data source access module, a user interface management module, a data output module, a bottom layer support module and a multi-thread module.
[0013] Furthermore, the front-end module includes a UI component module, a data processing module, a state management module, a network communication module, a style library module, and a routing and page module.
[0014] Furthermore, the Tpli module includes a format conversion module, a structure conversion module and a data conversion module.
[0015] Furthermore, the calling module includes a SAPC WS caller module, a Tasly caller module, a Taobao Qimen caller module, a Sql caller module, an http caller module, a JAVA caller module, a webservice caller module, a data stream caller module and an HTTP REST caller module.
[0016] Furthermore, the adaptation module includes a SAPC WS adapter module, an HTTP REST adapter module, a Tasly adapter module, a Taobao Qimen adapter module and other extension modules.
[0017] Furthermore, the multithreading module first configures the thread pool, creates a task queue, submits tasks to the thread pool, the threads execute tasks, the threads complete tasks, and monitors and adjusts.
[0018] Furthermore, the state management module is responsible for managing the state information of the front end, including the editing state of the configuration information, the progress state of the conversion process, and the state of the conversion result. The state management is implemented using the Redux or Mobx state management library.
[0019] Furthermore, the data structures supported by the tpli module include tree, array and object, and the tpli module supports json and xml data formats.
[0020] Furthermore, the Taobao Qimen caller module adapts different types of data sources to the Taobao Qimen framework, and the Tasly adapter module is used for data connection of clinical equipment.
[0021] The present invention provides a data format conversion system based on configuration. It has the following beneficial effects:
[0022] (1) Flexibility: The configuration file can flexibly define data conversion rules and parameters to adapt to the requirements of different data sources and target systems. By modifying the configuration file, the data conversion method and logic can be easily adjusted.
[0023] (2) Maintainability: Configuration files are usually in text format, which is easy to read, modify, and understand. When data conversion requirements change, only the corresponding configuration files need to be modified without modifying the program code, which reduces maintenance costs.
[0024] (3) Reusability: The configuration-based data format conversion system can be applied to multiple different scenarios and requirements. By using a common configuration file, data conversion can be easily implemented between different systems and applications, improving the reusability of the system.
[0025] (4) Efficiency: Configuration-based data format conversion systems usually have higher processing speed and performance. Since the logic and rules of data conversion are separated into configuration files, the system can focus on the core functions of data processing and conversion without having to deal with complex business logic.
[0026] (5) Scalability: The configuration-based data format conversion system can be easily expanded and upgraded. By adding new configuration files or modifying existing configuration files, it can easily support new data formats and conversion requirements. At the same time, the application scope of the system can be expanded by integrating other modules and functions. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] Figure 1 It is a general system diagram of a data format conversion system based on configuration of the present invention;
[0028] Figure 2 A schematic diagram of a back-end module of a data format conversion system based on configuration according to the present invention;
[0029] Figure 3 A schematic diagram of a front-end module of a data format conversion system based on configuration according to the present invention;
[0030] Figure 4 A schematic diagram of a data format conversion system call module based on configuration in the present invention;
[0031] Figure 5 A schematic diagram of a configuration-based data format conversion system adapter module of the present invention;
[0032] Figure 6 A schematic diagram of a WIFI data acquisition module of a data format conversion system based on configuration of the present invention;
[0033] In the figure: 1. Back-end module; 2. Front-end module; 3. Tpli module; 4. Calling module; 5. Adaptation module; 9. WIFI data acquisition module. DETAILED DESCRIPTION
[0034] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.
[0035] Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.
[0036] The present invention will be further described below in conjunction with the accompanying drawings and embodiments:
[0037] See also Figure 1-6 The present invention provides a technical solution: a configuration-based data format conversion system, characterized in that it includes:
[0038] Backend module 1 is used for data transmission and multi-threaded processing.
[0039] The front-end module 2 is responsible for providing an interactive interface between the user and the system, allowing the user to edit configuration information, start data conversion and view conversion results.
[0040] Tpli module 3 is used to complete data information conversion.
[0041] The calling module 4 is used for calling data.
[0042] Adaptation module 5 adapts different types of data sources to the specified framework.
[0043] There is a bidirectional signal connection between the backend module 1 and the frontend module 2, a bidirectional signal connection between the backend module 1 and the calling module 4, a bidirectional signal connection between the backend module 1 and the adapter module 5, and a bidirectional signal connection between the backend module 1 and the Tpli module 3.
[0044] Specifically, the back-end module 1 includes a data source access module, a user interface management module, a data output module, a bottom-level support module and a multi-threaded module. The data source access module is responsible for acquiring data from various data sources and importing it into the system. This module can adopt different data acquisition methods according to different data source types. The user interface module is responsible for providing an interactive interface between the user and the system, allowing the user to edit configuration information, start data conversion and view conversion results. The data output module is responsible for outputting the converted data to the target data source or file. This module can adopt the corresponding data output method according to the type of the target data source. The bottom module support includes various libraries and tools to provide basic support for the entire system, such as data processing, file reading and writing, network communication, etc.
[0045] Specifically, the front-end module 2 includes a UI component module, a data processing module, a state management module, a network communication module, a style library module, and a routing and page module. The UI component module: This module is responsible for processing user interactions, including the input of configuration information, the visual display of the conversion process, and the display of the conversion results. The UI component module can be built using front-end frameworks such as React, Vue, etc. to achieve a user experience with a friendly interface and convenient operation. The data processing module: This module is mainly responsible for performing data conversion operations based on the configuration information. It realizes data reading, conversion, and output functions by calling the back-end interface or processing local data. The data processing module can use asynchronous operations to improve the efficiency of data conversion. Block: This module is responsible for managing the status information of the front end, including the editing status of the configuration information, the progress status of the conversion process, and the status of the conversion results. State management can be implemented using state management libraries such as Redux and Mobx to achieve unified management and maintenance of the state. Network communication module: This module is responsible for communicating with the back end, including operations such as uploading and downloading configuration information, reading and outputting data, etc. Network communication can be implemented using HTTP request libraries such as Axios and Fetch to achieve interaction with the back end interface and data transmission. Style library module: This module includes a combination of front-end styles and style libraries, which are used to define the appearance and style of UI components. The style library can be implemented using popular UI component libraries such as Bootstrap and Element UI to provide a rich selection of UI components and styles. Page module: This module is responsible for page routing management and page rendering, including loading corresponding pages and components according to the current route, as well as the function of rendering the page. Routing and page modules can be implemented using routing libraries such as React Router and Vue Router to improve the page response speed and user experience.
[0046] Specifically, the Tpli module 3 includes a format conversion module, a structure conversion module and a data conversion module. The format conversion module is used for data format conversion, the structure conversion module is used for implementing data structure conversion, and the data conversion module can implement data conversion.
[0047] Specifically, the calling module 4 includes SAPC WS caller module, Tasly caller module, Taobao Qimen caller module, Sql caller module, http caller module, JAVA caller module, webservice caller module, data flow caller module and HTTP REST caller module. SAP CWS caller is the Web service caller of SAP NetWeaver PI. It provides a simple way to create a SOAP client to call the Web service of SAP PI. The SQL call module refers to a module that can execute SQL statements, which is usually used for database access and operation. In different programming languages and frameworks, the usage of the SQL call module may vary. The HTTP call module can be used to construct and send HTTP requests, and process HTTP responses. It can access HTTP life cycle events, check incoming requests and take actions based on the request, and also allow outbound responses to be checked and modified. Java call refers to the use of methods and properties of other Java classes or interfaces in Java programs. Java calls can be implemented in the following ways: direct call: methods and properties of other classes can be directly called in the same class. Object call: by creating objects of other classes and then using the object Call its methods and properties, static call: directly call its static methods and properties by using the class name, reflection call: dynamically load classes and methods through the reflection mechanism, and call them. In Java, calling methods and properties of other classes requires understanding the API (application programming interface) of the class, including information such as method name, parameter list and return type. When calling, you need to pass the correct parameters and process the return value of the method. At the same time, you also need to consider issues such as access rights and security. Webservice call refers to the technology of realizing communication and data exchange through Web services between different platforms and different programming languages. Webservice calls usually use SOAP protocol and XML format for communication, so they have the advantages of cross-platform and cross-language. HTTP REST call is a way of remote calling (RPC) using HTTP protocol. It is based on REST (Representational State Transfer) architectural style. REST is a resource-based architectural style that treats data as resources and uses HTTP protocol to access and operate these resources.
[0048] Specifically, the adaptation module 5 includes the SAP CWS adapter module, the HTTP REST adapter module, the Tasly adapter module, the Taobao Qimen adapter module and other extension modules. The SAP CWS adapter is a tool for connecting SAP PI (Process Integration) with non-SAP systems. The CWS adapter is part of the SAP NetWeaver PI platform, which provides the possibility of integration with non-SAP systems. The working principle of the CWS adapter is to convert the data of the non-SAP system into a format compatible with SAP PI by defining the message format and communication protocol, thereby achieving seamless integration with the SAP system. Using the CWS adapter, data from the non-SAP system can be sent to SAP PI, or data can be received from SAP PI and converted into a format that the non-SAP system can understand. This adapter can help enterprises achieve cross-system data sharing and business process automation. In addition, the CWS adapter also provides flexible configuration options, which can be customized and optimized according to different business needs. It supports a variety of communication protocols and data formats, and can adapt to different non-SAP systems and application scenarios. In short, the SAP CWS adapter is a tool for connecting SAP The adapter between PI and non-SAP systems realizes integration and interaction with non-SAP systems by converting data formats and communication protocols. The HTTP REST adapter is an adapter used to convert non-HTTP protocol messages into HTTP protocol messages. It can convert messages from other protocols (such as JMS, MQ, etc.) into RESTful-style HTTP requests to interact with HTTP-based services. The role of the HTTP REST adapter is to convert between different messaging protocols so that messages from different protocols can be sent to HTTP-based services through the adapter. It supports the conversion of request and response messages into HTTP requests and responses, thereby realizing interoperability between different protocols. The remaining extension modules are used for efficient adapter expansion.
[0049] Specifically, the multi-threaded module first configures the thread pool, creates a task queue, submits tasks to the thread pool, threads execute tasks, threads complete tasks and monitor and adjust, and performs multi-threaded operations based on the configured data format conversion system. You can follow the following steps: Configure the thread pool: First, you need to configure a thread pool to manage multi-threaded operations. The thread pool can pre-set the number of threads and dynamically adjust the number of threads according to the system load. Create a task queue: Next, you need to create a task queue to store data conversion tasks. Each data conversion task can include data source information, target data format, conversion rules, etc. Submit tasks to the thread pool: Submit data conversion tasks to the thread pool, and the thread pool is responsible for dividing Allocation and management of threads. Threads execute tasks: When a thread in the thread pool is idle, it will get a task from the task queue and execute it. When the thread executes a task, it reads, converts and outputs data according to the configuration information. Threads complete tasks: When the thread completes the task, it submits the task result to the specified target data source or file. At the same time, the thread returns to the thread pool and waits for the next task to be assigned. Monitoring and adjustment: During multi-threaded operations, it is necessary to monitor the system's operating status and performance indicators in real time, such as CPU usage, memory usage, task completion progress, etc. According to the monitoring results, the size of the thread pool and the capacity of the task queue can be adjusted in time to optimize the system's performance and response speed.
[0050] Specifically, the state management module is responsible for managing the status information of the front end, including the editing status of the configuration information, the progress status of the conversion process, and the status of the conversion result. The state management is implemented using the Redux or Mobx state management library.
[0051] Specifically, the data structures supported by the tpli module include tree, array and object, and the tpli module supports json and xml data formats.
[0052] Specifically, the Taobao Qimen caller module adapts different types of data sources to the Taobao Qimen framework, and the Tasly adapter module is used for data connection of clinical equipment.
[0053] Working principle:
[0054] Utilize the adapter's SAPC WS adapter module, HTTP REST adapter module, Tasly adapter module, Taobao Qimen adapter module and other extension modules to match massive amounts of data. The workflow of the SAP CWS adapter can be summarized in the following steps: Define the interface: First, you need to define the interface to which the SAP CWS adapter will connect. This can be a Web service, a SOAP endpoint, or another type of interface. Create an adapter configuration: Based on the defined interface, create the configuration of the SAP CWS adapter, which includes specifying the URL of the interface, the format of the request and response messages, the communication protocol used, and other information. Implement data conversion logic: In the adapter, you need to implement data conversion logic, which includes converting data from non-SAP systems into a format compatible with SAP PI, and converting data from SAP PI into a format that non-SAP systems can understand. Send requests and receive responses: Based on the configuration, the adapter will send a request message to SAP PI and wait for the return of the response message. When sending a request, the adapter needs to convert the data into SAP PI expects the format and transmits it according to its communication protocol. After receiving the response message, the adapter needs to convert the response data back to the original format. Handle exceptions and errors: In the process of the adapter processing requests and responses, exceptions and errors may be encountered. The adapter needs to be able to handle these exceptions and errors and provide appropriate error messages or log information for debugging and troubleshooting. Maintenance and update: As business needs change and technology develops, the SAP CWS adapter may need maintenance and updates. Adapter developers need to regularly check and update the adapter's code, configuration, and logic to meet new business needs and technical requirements. HTTP The workflow of the REST adapter module sends HTTP requests: The client uses the HTTP protocol to send requests, including request methods (GET, POST, PUT, DELETE, etc.), request headers, request bodies, and other information. Processing HTTP requests: After receiving the request, the server determines the operation to be performed based on the request method and URL, and processes the request. Sending HTTP responses: After the server processes the request, it returns the result to the client in the form of an HTTP response. The response includes information such as the status code, response header, and response body. Processing HTTP responses: After receiving the response, the client parses the result based on the status code and response body, and processes the returned data. The workflow of the Tasly adapter module can be divided into the following steps: Initialization: After the adapter module is started, it performs initialization operations, including loading configuration information, establishing connections, etc. Receiving requests: The adapter module receives requests from other systems or services, which may use different protocols and data formats. Protocol conversion: The adapter module performs protocol conversion on the received requests, converting them into protocols and data formats that the target system or service can understand. Data mapping: After the protocol conversion is completed, the adapter module performs data mapping operations.Map the data fields in the source system to the data fields in the target system to ensure data consistency and accuracy, send request: the adapter module sends the converted and mapped request to the target system or service and waits for the response, receive response: the adapter module receives the response from the target system or service and performs corresponding processing, return result: the adapter module returns the processed response to the original requester to complete the entire request-response process. The workflow of the Taobao Qimen adapter module can be divided into the following steps: Initialization configuration: Before using the Taobao Qimen adapter module, initialization configuration is required, which includes setting the connection parameters of the adapter, configuring data format and interface specifications and other information, receiving request: once the initialization configuration is completed, the adapter The module starts to listen for requests. When a request arrives, the adapter module will receive the request and prepare to process it. Data parsing: After receiving the request, the adapter module will parse the request data, which includes parsing request parameters, processing data format conversion and other operations to ensure the accuracy and consistency of the request data. Adaptation processing: After the data parsing is completed, the adapter module will perform adaptation processing according to the interface specifications of Taobao Qimen, which includes mapping the data to the format required by Taobao Qimen, processing specific business logic and other operations. Sending request: The adapter module sends the adapted request to the interface of Taobao Qimen, which involves establishing a network connection, sending request data and other operations, and ensuring that the request can be accurately transmitted to the Taobao Qimen system. Receiving response: Adaptation The adapter module waits for the response from Taobao Qimen and processes the response after receiving it, which includes parsing the response data, processing possible error codes and other operations to ensure the accuracy and consistency of the response data, and returning the result: the adapter module returns the processed response to the original requester, which may include converting the response data into the format required by the original requester, adding necessary metadata and other operations. When the adapter completes the acquisition of the data, it sends the data to the data source access module of the back-end module 1, and uses the data output module to send the data to the Tpli module 3 to complete the conversion of the data format, structure and the data itself. Then, the data output module is used to send the processed data to the calling module 4. The front-end layer workflow of the data format conversion system based on the configuration can be summarized as follows: The following steps: receiving user requests: after the front-end layer receives the user request, it passes the request to the back-end layer for processing, parsing the configuration file: after the back-end layer receives the request, it first needs to parse the configuration file to determine the specific rules and parameters for data format conversion, data preprocessing: after parsing the configuration file, the back-end layer needs to preprocess the data, which includes cleaning, filtering, splitting and other operations on the data for subsequent data format conversion, data format conversion: after the data preprocessing is completed, the back-end layer converts the data format according to the rules and parameters in the configuration file, which may involve converting the original data into another data format, or performing specific processing and conversion on the data, data transmission and return: once the data format conversion is completed,The back-end layer passes the converted data to the front-end layer for display and return. The front-end layer needs to display the converted data to the user according to the user's needs and interface design. Exception handling and logging: During the entire data processing process, abnormal situations need to be handled and recorded, which includes processing abnormal data, recording and reporting abnormal situations, etc.
[0055] The above are only preferred embodiments of the present invention. It should be pointed out that a person skilled in the art can make several modifications and improvements without departing from the inventive concept of the present invention, and these all fall within the protection scope of the present invention.
Claims
1. A data format conversion system based on configuration, characterized in that: include: Backend module (1), used for data transmission and multi-threaded processing; The front-end module (2) is responsible for providing an interactive interface between the user and the system, allowing the user to edit configuration information, start data conversion, and view conversion results. Tpli module (3), used to complete data information conversion; Calling module (4), used for calling data; Adaptation module (5), adapting different types of data sources to the specified framework; A bidirectional signal connection is formed between the back-end module (1) and the front-end module (2), a bidirectional signal connection is formed between the back-end module (1) and the calling module (4), a bidirectional signal connection is formed between the back-end module (1) and the adapting module (5), and a bidirectional signal connection is formed between the back-end module (1) and the Tpli module (3).
2. The data format conversion system based on configuration according to claim 1, characterized in that: The back-end module (1) comprises a data source access module, a user interface management module, a data output module, a bottom layer support module and a multi-thread module.
3. The data format conversion system based on configuration according to claim 2, characterized in that: The front-end module (2) comprises a UI component module, a data processing module, a state management module, a network communication module, a style library module and a routing and page module.
4. The data format conversion system based on configuration according to claim 3, characterized in that: The Tpli module (3) comprises a format conversion module, a structure conversion module and a data conversion module.
5. The data format conversion system based on configuration according to claim 4, characterized in that: The calling module (4) includes a SAPC WS caller module, a Tasly caller module, a Taobao Qimen caller module, a Sql caller module, an http caller module, a JAVA caller module, a webservice caller module, a data stream caller module and an HTTP REST caller module.
6. A data format conversion system based on configuration according to claim 5, characterized in that: The adapter module (5) includes a SAPC WS adapter module, an HTTP REST adapter module, a Tasly adapter module, a Taobao Qimen adapter module and other expansion modules.
7. The configuration-based data format conversion system according to claim 6, characterized in that: The multithreading module first configures the thread pool, creates a task queue, submits tasks to the thread pool, threads execute tasks, threads complete tasks, and monitors and adjusts.
8. The configuration-based data format conversion system according to claim 7, characterized in that: The state management module is responsible for managing the state information of the front end, including the editing state of the configuration information, the progress state of the conversion process, and the state of the conversion result. The state management is implemented using the Redux or Mobx state management library.
9. The configuration-based data format conversion system according to claim 8, characterized in that: The data structures supported by the tpli module include tree, array and object, and the tpli module supports json and xml data formats.
10. The configuration-based data format conversion system according to claim 9, characterized in that: The Taobao Qimen caller module adapts different types of data sources to the Taobao Qimen framework, and the Tasly adapter module is used for data connection of clinical equipment.
Citation Information
Patent Citations
Adaptive method for data integration
CN102142027A
Data transmission service system based on pluggable component framework
CN110058900A
Real-time and offline big data processing system and method, storage medium and terminal
CN110659323A
Low-code data format conversion system and low-code data format conversion method
CN116149657A
Cited By
Data processing method and device, electronic equipment and storage medium
CN119248288A
