A data exchange method based on entity

Abstract

This invention discloses an entity-based data exchange method. This method abstracts data loading into queries, data push into interfaces, and data exchange tasks into multiple task nodes. Nodes interact with each other through unified, abstracted entities. The method specifically includes the following steps: S1. Define queries; S2. Define entities; S3. Define interfaces; S4. Configure tasks; and S5. Run tasks. This method can alleviate the complexity caused by varying data formats, encoding methods, and storage structures. It can also reduce the learning cost of writing and maintaining complex conversion rules, and offers more flexible scalability.

Description

Technical Field

[0001] The present invention relates to the technical field of data exchange, and in particular to an entity-based data exchange method. Background Art

[0002] In the public security industry, criminal investigation management information system development can lead to an overabundance of systems. This situation often stems from a lack of coordination and unified planning among various departments or units during system development. Alternatively, it can also stem from the development of separate systems tailored to specific specialized areas, such as public security traffic management or criminal investigation technology. This overabundance of systems can lead to information silos, preventing data sharing and interoperability between systems, impacting work efficiency and information security.

[0003] Data synchronization between different systems presents numerous challenges, and existing technologies have limitations in addressing these challenges. The most prominent issue is that different systems may use different data formats, encoding methods, and storage structures, resulting in data interoperability and sharing issues.

[0004] To address data synchronization issues caused by excessive systems and a lack of coordination, data warehouses and ETL tools are currently commonly used to integrate data from different systems onto a single platform, enabling data sharing and synchronization through standardized data models and interfaces. However, these data warehouses and ETL tools generally suffer from the following major issues:

[0005] Complexity: Data warehouses and ETL tools typically need to handle a large number of data sources and multiple data types with different formats, encoding methods, and storage structures, so their design and implementation are relatively complex.

[0006] High cost: This involves converting, cleaning, and integrating data, requiring significant time and effort to write and maintain complex conversion rules. To improve their applicability, these tools are often designed to be flexible, which in turn makes them difficult to learn.

[0007] Scalability: The scalability of data warehouses and ETL tools limits the data sources and data volumes they can support. If you need to process larger amounts of data or add new data sources, you will need to redesign and re-deploy the ETL process and data warehouse architecture. Summary of the Invention

[0008] The technical problem to be solved by the present invention is to provide an entity-based data exchange method, which can not only reduce the complexity caused by different data formats, encoding methods and storage structures, but also reduce the learning cost of writing and maintaining complex conversion rules, and has more flexible expansion capabilities.

[0009] In order to solve the above technical problems, the technical solutions adopted by the present invention are as follows.

[0010] An entity-based data exchange method abstracts data loading into queries, data push into interfaces, and data exchange tasks into multiple task nodes. Nodes interact with each other through unified abstract entities. The method specifically includes the following steps:

[0011] S1 definition query; specifically for the source target data that needs to be exchanged, define a query instance; the definition query process also performs step S2 definition entity;

[0012] S2. Define entities; Specifically, for data in different steps that require data exchange, define a standardized entity as a data exchange standard between different steps;

[0013] S3 define the interface; the interface is the data that needs to be exchanged, at different steps, or data channels exchanged between different systems; the process of defining the interface also continues with step S2 to define the entity;

[0014] S4 configuration tasks; specifically for the data to be exchanged, define the specific steps of the data exchange process, the steps and steps before the entity to transfer data;

[0015] S5. Run the task; specifically, define the task running mode for the defined data exchange task.

[0016] Preferably, step S1 includes the following steps:

[0017] S11. Define basic query information;

[0018] The basic information of the defined query includes the query name, query code and query category of the query; the query code is globally unique;

[0019] S12. Define data source;

[0020] The data source is the source of data, which is mainly a relational database;

[0021] S13. Define query input parameters;

[0022] The definition of query input parameters specifically refers to treating the query as a closed component and providing a call parameter entry to the outside. When the query component is used externally, the call parameters need to be passed in; the call parameters include but are not limited to query conditions, filtering conditions and sorting rules for specifying the specific content and method of the query, and the call parameters include four basic attributes: parameter code, parameter name, parameter type and parameter value.

[0023] S14. Define SQL query;

[0024] The definition of SQL query specifically uses SQL statements to load target exchange data from a data source, including compiling the SQL statements into SQL templates, embedding macro instructions of the template parsing engine in the SQL templates, and converting them into actual query SQL statements according to variable encoding when executing SQL.

[0025] S15. Define output set;

[0026] The output set is specifically defined as a target exchange data set loaded from a data source, which is converted into an entity set to eliminate differences in data types of different relational databases.

[0027] Preferably, step S2 includes the following steps:

[0028] S21. Define basic entity information;

[0029] The basic information of the defined entity includes the entity name, entity code and entity classification of the entity; the entity code is globally unique;

[0030] S22. Define entity attributes;

[0031] Each element of the entity attribute includes but is not limited to name, number, type, length and association.

[0032] Preferably, step S3 includes the following steps:

[0033] S31. Define basic interface information;

[0034] The basic information of the defined interface includes the interface name, interface code and interface classification of the defined interface; the interface code is globally unique;

[0035] S32. Define request parameters;

[0036] The definition of request parameters specifically refers to treating the interface as a closed component and providing a call parameter entry to the outside. When the interface component is used externally, the parameters to be passed are passed to the component in a specific format; the call parameters include four basic attributes: parameter code, parameter name, parameter type, and parameter value;

[0037] S33. Define interface type;

[0038] The interface types include but are not limited to RESTful interface and SOAP interface;

[0039] S34. Define receiving processing;

[0040] The definition of receiving and processing is specifically the way to process the returned message after calling the interface, including outputting it as is and allowing the use of custom functions to parse, convert the format, calculate and output the returned message;

[0041] S35. Define the receiving processing output set;

[0042] The definition of the receiving and processing output set specifically specifies the data type of the final output of this unit for the output result of the receiving and processing in step S34.

[0043] Preferably, step S4 includes the following steps:

[0044] S41. Define basic task information;

[0045] The basic information of the defined task includes the task name, task code and task classification of the defined task; the task code is globally unique;

[0046] S42. Define task input parameters;

[0047] Defining task input parameters specifically involves treating the task as a closed component and providing a call parameter entry to the outside. When the task component is used externally, the parameters to be passed are passed into the component in a specific format; the call parameters include four basic attributes: parameter code, parameter name, parameter type, and parameter value;

[0048] S43. Define task nodes;

[0049] The defined task nodes include a defined start node, an end node, and a data query node and an interface call node set between the start node and the end node, and the nodes run sequentially and exchange data through the entities defined in step S2;

[0050] S44. Define timing strategy;

[0051] The timing strategy is specifically defined by using a task scheduling framework to trigger the execution of data exchange tasks according to a specified time rule.

[0052] Preferably, the task nodes defined in step S43 specifically include the following nodes that are run in sequence:

[0053] S431. Start node;

[0054] S432. Query employee information node;

[0055] S433. Loop connection;

[0056] S434 identity authentication interface node;

[0057] S435. Push employee information interface node;

[0058] S436. End node.

[0059] Preferably, step S5 includes the following steps:

[0060] S51. Input operating parameters;

[0061] S52. Load task node;

[0062] S53. Loading timing strategy;

[0063] S54. Timing cycle operation.

[0064] Due to the adoption of the above technical solution, the technical progress achieved by the present invention is as follows.

[0065] The present invention abstracts data loading into queries, data push into interfaces, and data exchange tasks into multiple task nodes. The nodes interact with each other through a unified abstract entity, which can reduce the complexity caused by different data formats, encoding methods and storage structures, and can also reduce the learning cost of writing and maintaining complex conversion rules. It also has more flexible expansion capabilities and can adapt to new data sources without the need for secondary development of the existing system. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 is a flow chart of the present invention;

[0067] Figure 2 is a flow chart of step S1 of the present invention;

[0068] Figure 3 This is a flow chart of step S2 of the present invention;

[0069] Figure 4 This is a flow chart of step S3 of the present invention;

[0070] Figure 5 This is a flow chart of step S4 of the present invention;

[0071] Figure 6 is a flow chart of an embodiment of step S43 of the present invention;

[0072] Figure 7 This is a flow chart of step S5 of the present invention. DETAILED DESCRIPTION

[0073] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0074] This entity-based data exchange method addresses the high complexity, high cost, and poor scalability of data synchronization between different systems. It uses standardized data models and interfaces to achieve data sharing and synchronization, thereby reducing development and maintenance costs and improving system flexibility and scalability. By adopting a unified data model, data exchange and sharing between different systems can be more convenient, avoiding the difficulties caused by differences in data formats and structures between systems, while also reducing the workload of data conversion and mapping. Furthermore, through unified interface specifications, the complexity of system integration can be reduced, making integration between systems more stable and reliable.

[0075] Combine Figure 1 As shown, this method completes a complete data exchange process, including the following steps: S1. Define a query. During the query definition process, step S2. Define entities are also performed; then step S3. Define an interface. During the interface definition process, step S2. Define entities are continued; then step S4. Configure the task; and finally, step S5. Run the task. The following describes steps S1, S2, S3, S4, and S5 in detail.

[0076] S1. Define the query.

[0077] For the data exchange target, a query is first defined. In the process of defining the query, the input or output entity is defined, which is step S2.

[0078] like Figure 2 As shown, defining a query involves defining a query instance for the source and target data to be exchanged. This includes the following steps: S11. Define basic query information, S12. Define the data source, S13. Define query input parameters, S14. Define the SQL query, and S15. Define the output set. The following describes steps S11, S12, S13, S14, and S15 in detail.

[0079] S11. Define basic query information.

[0080] Defining basic query information involves defining the query name, query code, and query category. Query codes are globally unique so they can be referenced throughout the data exchange process. For example, if the query code is QUERY_USER_INFO, it can be referenced by this query code throughout the data exchange process. Query categories facilitate the management of large numbers of defined queries.

[0081] S12. Define the data source.

[0082] Data source refers to the source of data, mainly relational databases: such as MySQL, Oracle, SQL Server, etc.

[0083] S13. Define query input parameters.

[0084] Defining query input parameters treats the query as a closed component, providing external access to call parameters. External calls to this query component require the input of these parameters. These parameters can include query conditions, filter conditions, sorting rules, and other criteria, specifying the specific content and method of the query. Defining query input parameters makes the query component more flexible and versatile, adapting to diverse query requirements. Furthermore, input parameters can enhance query component security by preventing malicious calls and unauthorized access.

[0085] Specifically, the calling parameters include four basic attributes: parameter code, parameter name, parameter type and parameter value.

[0086] The parameter code is unique within this query, allowing you to reference the parameter within that query and throughout the entire data exchange task by adding the query code to the parameter code. For example, if the query code is QUERY_USER_INFO (query user information) and the parameter code is ORG_CODE (organization code), you can reference the parameter globally throughout the data exchange task by using QUERY_USER_INFO.ORG_CODE.

[0087] Parameter names are generally character types and are generally named to be easy to understand.

[0088] Parameter types include basic types, dictionary types, collection types, and entity types defined in step S2.

[0089] Basic types include integer, floating-point, Boolean, string, and date types. Integer represents whole numbers; floating-point represents values ​​with decimal points; Boolean represents true or false; string represents text; and date represents date and time.

[0090] A dictionary type consists of multiple key-value pairs, each consisting of a key and a value. Dictionary parameter types are used to pass a group of related parameters that can be accessed and manipulated using keys. In a dictionary parameter type, keys must be unique, and values ​​can be of any type, including integers, floating-point numbers, Booleans, strings, dates, collections, and entities. Dictionary parameter types allow parameter values ​​to be changed dynamically by adding, removing, or modifying key-value pairs.

[0091] A collection type consists of an unordered, non-repeating set of elements. Collections are typically used to store a group of related data, such as user names or organization names. Elements in a collection are not required to be unique, can be repeated, and are not ordered, allowing for fast search, insertion, and deletion operations. Collections can dynamically change their contents by adding and removing elements, enabling flexible data processing. The values ​​of elements in a collection can be of any type, including integers, floating-point numbers, Booleans, strings, dates, collection types, entity types, and more.

[0092] The entity type is to define a standardized entity as the data exchange standard between different steps for the data in different steps that need to be exchanged. The entity definition in step S2 will be described in detail later.

[0093] S14. Define SQL query.

[0094] Defining an SQL query means defining how to load target exchange data from a data source. SQL is a programming language used to manage relational databases. SQL statements are used to load target exchange data from relational databases. In the method provided by the present invention, SQL statements are compiled into SQL templates. Macros based on template parsing engines such as FreeMaker and Velocity can be embedded in the SQL templates. When executing SQL, these macros are converted into actual query SQL statements based on variable encoding. For example, the SQL query component defines one input parameter:

[0095] Parameter 1:

[0096] Parameter code: V_ORG_CODE

[0097] Parameter name: Organization code

[0098] Parameter type: character

[0099] Parameter value: 320300

[0100] The SQL query template defined at the same time queries all employees under the organization according to the organization code, such as:

[0101] select employee_id,employee_name,sex from t_employee where org_codelike'${V_ORG_CODE}%'

[0102] When executed in a relational database, the SQL is actually:

[0103] select employee_id,employee_name,sex from t_employee where org_codelike'320300%'

[0104] S15. Define an output set.

[0105] Defining an output set means loading the target exchange data set from the data source and converting it into an entity set to eliminate the differences in data types of different relational databases. For example, through the above SQL query, the query result set is obtained in the form of a table, such as:

[0106]

[0107] Convert the above table structure into an entity set. The attributes of the entity correspond to the fields in the above table. The output set definition is as follows:

[0108] Result code: ARRAY_EMPLOYEE

[0109] Result Type: Collection<Employee Entity>

[0110] Entity code: ENTITY_EMPLOYEE

[0111] Attribute codes: EMPLOYEE_ID, EMPLOYEE_NAME, SEX

[0112] Attribute name: employee number, name, gender

[0113] Attribute type: string, string, dictionary

[0114] To reference the result set based on the result code in all subsequent steps of the data exchange process, you need to define a unique code for the result set, such as ARRAY_EMPLOYEE. Each element of the result set ARRAY_EMPLOYEE is an instance of the entity ENTITY_EMPLOYEE. You can read each row of data by looping through the ARRAY_EMPLOYEE set.

[0115] S2. Define entities.

[0116] like Figure 3 As shown, defining an entity is to define a standardized entity for the data in different steps that need to be exchanged, as a data exchange standard between different steps, including the following steps: S21. Define basic entity information and S22. Define entity attributes. The above steps S21 and S22 are explained in detail below.

[0117] S21. Define basic entity information.

[0118] Defining basic entity information means defining the entity name, entity code, and entity classification. Entity codes are globally unique and can be referenced throughout the data exchange process. For example, if the entity code is USER_MODEL (user information entity), this entity code, USER_MODEL, can be used to reference the entity throughout the data exchange process. Entity classification makes it easier to manage large numbers of entities.

[0119] S22. Define entity attributes.

[0120] Each element of an entity attribute includes name, number, type, length, association, etc. Among them:

[0121] The name is the identifier of the attribute, which is used to distinguish different attributes;

[0122] The number is a unique identifier for the attribute, used to uniquely identify and reference it within the entity;

[0123] Type is the value type of the attribute, including basic type, dictionary type, collection type and entity type;

[0124] Length is the value length of the attribute, which is only meaningful for basic types, such as string length, number length, etc.

[0125] An association is a property that can be associated with another entity. For example, if the type of the P1 attribute of entity A is also the entity type, then the attribute P1 can be associated with entity B. For another example, if the P2 attribute of entity A is of string type, and the P1 attribute of entity B is also of string type, then the attribute P2 can be associated with the P1 attribute of entity B.

[0126] S3. Define the interface.

[0127] like Figure 4 As shown, an interface is a data channel for data exchange between different steps or systems. The interface definition process continues with defining the input or output entities, i.e., step S2. Interface definition includes the following steps: S31. Define basic interface information; S32. Define request parameters; S33. Define interface type; S34. Define receive processing; and S35. Define receive processing output set. The following describes these steps S31, S32, S33, S34, and S35 in detail.

[0128] S31. Define basic interface information.

[0129] Defining basic interface information means defining the interface name, interface code, and interface category. Interface codes are globally unique so they can be referenced throughout the data exchange process. For example, if the interface code is API_USER_INFO, the API_USER_INFO interface code can be used to reference the interface throughout the data exchange process. Interface categories facilitate the management of a large number of defined interfaces.

[0130] S32. Define request parameters.

[0131] Defining request parameters means using this interface as a closed component to provide an entry for external call parameters. When this interface component is used externally, the parameters to be passed will be passed into the component in a specific format.

[0132] Specifically, the calling parameters include four basic attributes: parameter code, parameter name, parameter type and parameter value, which are the same as those in step S13 and will not be repeated here.

[0133] S33. Define the interface type.

[0134] Defined interface types include but are not limited to RESTful interfaces and SOAP interfaces.

[0135] A RESTful interface is a web service interface design style based on the HTTP protocol. It uses HTTP methods such as GET, POST, PUT, and DELETE to operate on resources and locate resources through URLs. For example, there is a RESTful interface used to obtain a user's information. Its URL is: http: / / example.com / users / {userId}, where {userId} is the user's ID. If we want to obtain the information of user 123, we can use the HTTP GET method, replace {userId} in the URL with 123, and send the request. For example:

[0136] GET http: / / example.com / users / 123

[0137] In order to enable the interface instance component in step S3 to have the ability to call the RESTful interface, the method provided by the present invention defines four parameters: URL parameters, Header parameters, Body parameters and Cookie parameters. When the interface instance component in step S3 calls the RESTful interface, these types of parameters are passed as needed.

[0138] in:

[0139] The URL parameter is an array of key-value pairs, consisting of parameter name and parameter value. When calling the RESTful interface, it is automatically spliced ​​into the URL as a parameter. For example, there are two parameters as follows:

[0140] Parameter 1:

[0141] Parameter name: token

[0142] Parameter value: c8d7f9a9d5c8b6e7a4b3e1f2c5d6a7b8

[0143] Parameter 2:

[0144] Parameter name: appId

[0145] Parameter value: demo_app

[0146] When calling the http: / / example.com / users interface, these two parameters are automatically concatenated into the URL parameters as parameters in the URL. The final URLs for calling the interface are:

[0147] http: / / example.com / users?appId=demo_app

[0148] &token=c8d7f9a9d5c8b6e7a4b3e1f2c5d6a7b8

[0149] In the method provided by the present invention, placeholders can be used to obtain the values ​​of the above parameter names and parameter values. For example, the parameters of the above calling interface are defined as:

[0150] Parameter 1:

[0151] Parameter name: token

[0152] Parameter value: ${v_token}

[0153] Parameter 2:

[0154] Parameter name: appId

[0155] Parameter value: ${v_appid}

[0156] When calling the interface component, the request parameters are defined according to step S32. The parameters passed in are:

[0157] Parameter 1:

[0158] Parameter code: v_appid

[0159] Parameter name: Application ID

[0160] Parameter type: character

[0161] Parameter value: demo_app

[0162] Parameter 2:

[0163] Parameter encoding: v_token

[0164] Parameter name: token

[0165] Parameter type: character

[0166] Parameter value: c8d7f9a9d5c8b6e7a4b3e1f2c5d6a7b8

[0167] When calling the http: / / example.com / users interface, ${v_token} is automatically replaced with c8d7f9a9d5c8b6e7a4b3e1f2c5d6a7b8, and ${v_appid} is replaced with demo_app. These two parameters are then added to the URL parameter string. The final URL for calling the interface is:

[0168] http: / / example.com / users?appId=demo_app

[0169] &token=c8d7f9a9d5c8b6e7a4b3e1f2c5d6a7b8

[0170] The Header parameter is also an array of key-value pairs, consisting of parameter names and parameter values. When calling a RESTful interface, these parameters are added to the HTTP request header, such as the common authorization parameter for authentication based on OAUTH2.

[0171] The Body parameter includes a content type and a parameter template. The content type can be text, json, or xml. The parameter template is the content carried in the HTTP request message. The parameter template can use macros based on template parsing engines such as Freemaker and Velocity to convert variable encodings into actual request values ​​when making HTTP requests.

[0172] For example, in step S32, defining request parameters, two parameters are defined:

[0173] Parameter 1:

[0174] Parameter code: v_username

[0175] Parameter name: Username

[0176] Parameter type: character

[0177] Parameter value: admin

[0178] Parameter 2:

[0179] Parameter encoding: v_password

[0180] Parameter name: password

[0181] Parameter type: character

[0182] Parameter value: TuKXuPK52t

[0183] In the Body parameter definition area, define the Body type as JSON and the request content as follows:

[0184]

[0185] Then ${v_username} and ${v_password} correspond to the v_username and v_password parameters defined in step S32. When the HTTP request is actually made, the content in the actual request body is:

[0186]

[0187] The parameter to be referenced can also be a parameter defined in another component, and can be referenced using the "component code.parameter code" format, for example:

[0188] QUERY_USER_INFO.ORG_CODE corresponds to the ORG_CODE parameter in the query group QUERY_USER_INFO. Its value can be referenced using the placeholder ${QUERY_USER_INFO.ORG_CODE}. This parameter reference method applies to all components described below.

[0189] SOAP (Simple Object Access Protocol) is an XML-based communication protocol used to exchange data in distributed systems. A SOAP message consists of an envelope element and multiple header and body elements. The envelope element contains the description of the entire message.

[0190] In order to enable the interface instance component to call the SOAP interface, the method provided by the present invention defines three parameters: URL parameters, Header parameters, and Body parameters. When the interface instance component calls the RESTful interface, these parameters are passed as needed.

[0191] For example, there is a soap message template defined as follows:

[0192]

[0193]

[0194] In the above message, ${v_transid} and ${v_stockname} are two placeholders, which will be replaced by actual values ​​during the data exchange process. Accordingly, the following two parameters are required to define the request parameters of the interface instance component:

[0195] Parameter 1:

[0196] Parameter code: v_transid

[0197] Parameter name: serial number

[0198] Parameter type: Integer

[0199] Parameter value: 12345

[0200] Parameter 2:

[0201] Parameter code: v_stockname

[0202] Parameter name: store name

[0203] Parameter type: character

[0204] Parameter value: Sample store

[0205] In this way, the values ​​"12345" and "Sample Store" can be referenced through placeholders. In addition to the above parameters, the relevant message content in the SOAP interface can use the form of ${variable name} to reference the components and parameters defined in the entire data exchange process.

[0206] S34. Define receiving processing.

[0207] Defining reception processing refers to the way the returned message is processed after calling the HTTP or SOAP interface. This invention provides two methods for returning the message: outputting it as is; and allowing the use of custom functions to parse, convert, perform operations, and output the returned message. The output result type can be an entity, a dictionary, or basic types such as strings and numbers.

[0208] S35. Define a receiving processing output set.

[0209] Defining the receiving and processing output set means specifying the data type of the final output of this unit for the output result of the receiving and processing in step S34, whether it is a set or a single element to be output to the next step. The output result of this step can be passed to the next step of this step or all steps after this step in the data exchange process.

[0210] S4. Configure tasks.

[0211] like Figure 5 As shown, configuring a task defines the specific steps in the data exchange process for the data to be exchanged. Data is transferred between the steps through entities. The configuration task includes the following steps: S41. Define basic task information, S42. Define task input parameters, S43. Define task nodes, and S44. Define a timing strategy. The following describes these steps S41, S42, S43, and S44 in detail.

[0212] S41. Define basic task information.

[0213] Defining basic task information involves defining the task name, task code, and task category. Task codes are globally unique. For example, if a task code is JOB_SYNC_USER, it can be referenced by the JOB_SYNC_USER task code throughout the data exchange process. In some scenarios, tasks can be suspended or terminated at a specific step. Task categories facilitate the management of large numbers of defined tasks.

[0214] S42. Define task input parameters.

[0215] Defining task input parameters means treating the task as a closed component and providing an external parameter entry. When the task component is used externally, the parameters to be passed are passed into the component in a specific format.

[0216] Specifically, the calling parameters include four basic attributes: parameter code, parameter name, parameter type, and parameter value, which are the same as those in step S13 and will not be described in detail here.

[0217] S43. Define task nodes.

[0218] Task nodes are defined as the nodes that comprise a data exchange task. The starting and ending points of the task are a start node and an end node, respectively. Several task nodes can be set between the starting and ending points. Task nodes can be query nodes or interface nodes. Nodes can be connected by lines, with arrows always pointing from the previous node to the next, representing the order in which the nodes execute. These nodes can be instances of the query in step S1 or instances of the interface in step S3. Nodes exchange data through the entities defined in step S2.

[0219] The data query node is an instance of the query defined in step S1. The interface call node is an instance of the interface defined in step S3. Figure 6 The step S43. defining the task node is described with an example.

[0220] like Figure 6 As shown, in the embodiment of step S43, defining task nodes, the following nodes are executed in sequence: S431. Start node; S432. Query employee information node [QUERY_EMPLOYEE]; S433. Loop connection; S434. Identity authentication interface node [API_AUTH]; S435. Push employee information interface node [API_PUSH_EMPLOYEE]; S436. End node. The following describes steps S431, S432, S433, S434, S435, and S436 in detail.

[0221] S431. Start node.

[0222] The start node is the starting point of the data exchange task.

[0223] S432. Query employee information node [QUERY_EMPLOYEE].

[0224] The employee information query node [QUERY_EMPLOYEE] loads target data from the data source and exchanges it. The loaded data is assembled into an entity set as the output of this node, as defined in step S15. In the employee information query node, the query method defined is SQL.

[0225] The SQL query statement is as follows:

[0226] select * from t_employee

[0227] The output settings are as follows:

[0228] Result Type: Collection<Employee Entity>

[0229] Attribute encoding: ID, NAME, SEX

[0230] Attribute name: ID, name, gender

[0231] Type: integer, string, dictionary

[0232] S433. Loop connection.

[0233] A loop connection loops through the set output by the previous node and passes the entities in that set as parameters to the next node. Because the output of the previous node is a set type, and the input of the next node is an entity type, it is necessary to loop through the output set of the previous node and call the next node with each element in the set as an input parameter.

[0234] S434. Identity authentication interface node [API_AUTH].

[0235] The identity authentication interface node requires identity authentication before calling the push employee information interface node in step S435. After the authentication is passed, the push employee information interface node is allowed to be called.

[0236] The input parameters defined by the identity authentication interface node are as follows:

[0237] Parameter 1:

[0238] Parameter code: v_username

[0239] Parameter name: Username

[0240] Parameter type: character

[0241] Parameter value: myUserName

[0242] Parameter 2:

[0243] Parameter encoding: v_password

[0244] Parameter name: password

[0245] Parameter type: character

[0246] Parameter value: TuKXuPK52t

[0247] URL parameters: http: / / www.mydomain.com / auth

[0248] Request method: GET

[0249] The output settings of the identity authentication interface node definition are as follows:

[0250] Result code: ACCESS_TOEKN

[0251] Result type: string

[0252] After calling the identity authentication interface node, the string token obtained is as follows:

[0253] d41d8cd98f00b204e9800998ecf8427e. In subsequent nodes, you can use API_AUTH.ACCESS_TOEKN to use this token.

[0254] S435. Push employee information interface node [API_PUSH_EMPLOYEE].

[0255] Push employee information interface node, call this interface to push the target exchange data. The input parameters of the push employee information interface node are defined as follows:

[0256] Parameter 1:

[0257] Parameter encoding: ACCESS_TOKEN

[0258] Parameter name: token

[0259] Parameter type: String

[0260] Parameter 2:

[0261] Parameter code: ENTITY_EMPLOYEE

[0262] Parameter name: employee information

[0263] Parameter type: Entity

[0264] Parameter 1 is the token obtained after identity authentication. Only with the token can the push employee information interface node be called. Otherwise, it will be rejected. When calling a RESTful interface, the token is generally set in the header parameter. For example, set the following in the interface header parameter:

[0265] Parameter 1:

[0266] Parameter encoding: accessToken

[0267] Parameter type: String

[0268] Parameter value: ${ACCESS_TOKEN}

[0269] The ${ACCESS_TOKEN} above is a placeholder and will be replaced with the actual value each time a request is made to push the employee information interface. Accordingly, the definition of the employee entity in parameter 2 above is as follows:

[0270] Attribute encoding: ID, NAME, SEX

[0271] Attribute name: ID, name, gender

[0272] Type: integer, string, dictionary

[0273] The employee information interface node is a RESTful interface. The data to be transmitted will be placed in the body parameter in the post mode. Therefore, the following message template is defined in the body parameter:

[0274]

[0275]

[0276] When the HTTP request is actually submitted, the message will be replaced with the actual value according to the employee entity passed in. For example, the parameter value passed in to the employee entity is:

[0277] (No.: 113, Name: Zhang San, Gender: 1)

[0278] The message template in the body parameter of the final submitted request is converted into:

[0279]

[0280] S44. Define a timing strategy.

[0281] A timing strategy refers to using common task scheduling frameworks such as Quartz, Celery, and Timer to trigger the execution of data exchange tasks according to a specified time pattern. There are several methods for timing strategies:

[0282] Periodic timing: Execute operations at certain time intervals, such as performing data exchange operations every hour.

[0283] Scheduled execution: Execute the operation at a specified time, such as executing data exchange operations at dawn every day.

[0284] Delayed execution: Executes the operation after a specified time. For example, starting at 12:00 AM every Monday, data exchange is delayed until 30 minutes later.

[0285] For example, a cron expression can be used to schedule data exchange tasks. A cron expression is a string format used to represent the execution time of a scheduled task. It consists of six or seven fields, representing seconds, minutes, hours, day, month, week, and optional year. Each field can be a number, a group of numbers, a wildcard, or a range. For example, "0012**?" indicates that a task will be executed at 12:00 PM every day. Cron expressions are commonly used in scheduled task scheduling frameworks such as Quartz.

[0286] S5. Run the task.

[0287] like Figure 7 As shown, running a task is to define the running mode of a defined data exchange task. Running a task includes the following steps: S51. Inputting running parameters, S52. Loading task nodes, S53. Loading timing strategies, and S54. Running a scheduled loop. The following describes the above steps S51, S52, and S53 in detail.

[0288] S51. Input operating parameters.

[0289] Input running parameters are the parameters required to start the task. The most commonly used parameter is to pass in the current time as the startup parameter. In the next query node, the corresponding data will be queried based on this current time.

[0290] S52. Load the task node.

[0291] The loading task nodes are all the task nodes defined in step S43. Define task nodes, and are executed one by one in the order of the task nodes starting from the start node until the end node, thereby completing the operation of the entire data exchange task.

[0292] S53. Load the timing strategy.

[0293] Loading timing strategy refers to setting the time frequency of target task operation, and executing the entire data exchange operation according to rules such as seconds, minutes, hours, days, weeks, months, and years.

[0294] The present invention provides an entity-based data exchange method, abstracts data loading into queries, abstracts data push into interfaces, decomposes data exchange tasks into multiple task nodes, and allows nodes to interact with each other through unified abstract entities, thereby solving the problems of high complexity, high cost and poor scalability faced by data synchronization between different systems. By adopting standardized data models and interfaces to achieve data sharing and synchronization, the purpose of ultimately reducing development and maintenance costs and improving system flexibility and scalability can be achieved.

Claims

1. An entity-based data exchange method, characterized by: Data loading is abstracted as query, data push is abstracted as interface, and data exchange tasks are decomposed into multiple task nodes. Nodes interact with each other through a unified abstract entity. The specific steps include: S1 definition query; specifically for the source target data that needs to be exchanged, define a query instance; the definition query process also performs step S2 definition entity; S2. Define entities; Specifically, for data in different steps that require data exchange, define a standardized entity as a data exchange standard between different steps; S3 define the interface; the interface is the data that needs to be exchanged, at different steps, or data channels exchanged between different systems; the process of defining the interface also continues with step S2 to define the entity; S4 configuration tasks; specifically for the data to be exchanged, define the specific steps of the data exchange process, the steps and steps before the entity to transfer data; S5. Run the task; specifically, define the task running mode for the defined data exchange task.

2. The entity-based data exchange method according to claim 1, characterized in that: The step S1 comprises the following steps: S11. Define basic query information; The basic information of the defined query includes the query name, query code and query category of the query; the query code is globally unique; S12. Define data source; The data source is the source of data, which is mainly a relational database; S13. Define query input parameters; Defining query input parameters specifically involves treating the query as a closed component and providing an external call parameter entry. When the query component is used externally, call parameters need to be passed in. The call parameters include, but are not limited to, query conditions, filter conditions, and sorting rules for specifying the specific content and method of the query. Call parameters include four basic attributes: parameter code, parameter name, parameter type, and parameter value. S14. Define SQL query; Defining the SQL query specifically involves using an SQL statement to load target exchange data from a data source, including compiling the SQL statement into an SQL template, embedding macro instructions of a template parsing engine in the SQL template, and converting the macro instructions into actual query SQL statements according to variable encoding when executing the SQL; S15. Define output set; The output set is specifically defined as a target exchange data set loaded from a data source, which is converted into an entity set to eliminate differences in data types of different relational databases.

3. The entity-based data exchange method according to claim 1, characterized in that: The step S2 comprises the following steps: S21. Define basic entity information; The basic information of the defined entity includes the entity name, entity code and entity classification of the entity; the entity code is globally unique; S22. Define entity attributes; Each element of the entity attribute includes but is not limited to name, number, type, length and association.

4. The entity-based data exchange method according to claim 1, characterized in that: The step S3 comprises the following steps: S31. Define basic interface information; The basic information of the defined interface includes the interface name, interface code and interface classification of the defined interface; the interface code is globally unique; S32. Define request parameters; The definition of request parameters specifically refers to treating the interface as a closed component and providing a call parameter entry to the outside. When the interface component is used externally, the parameters to be passed are passed to the component in a specific format; the call parameters include four basic attributes: parameter code, parameter name, parameter type, and parameter value; S33. Define interface type; The interface types include but are not limited to RESTful interface and SOAP interface; S34. Define receiving process; The definition of receiving and processing is specifically the way to process the returned message after calling the interface, including outputting it as is and allowing the use of custom functions to parse, convert the format, calculate and output the returned message; S35. Define the receiving processing output set; The definition of the receiving and processing output set specifically specifies the data type of the final output of this unit for the output result of the receiving and processing in step S34.

5. The entity-based data exchange method according to claim 1, characterized in that: The step S4 comprises the following steps: S41. Define basic task information; The basic information of the defined task includes the task name, task code and task classification of the defined task; the task code is globally unique; S42. Define task input parameters; Defining task input parameters specifically involves treating the task as a closed component and providing a call parameter entry to the outside. When the task component is used externally, the parameters to be passed are passed into the component in a specific format; the call parameters include four basic attributes: parameter code, parameter name, parameter type, and parameter value; S43. Define task nodes; The defined task nodes include a defined start node, an end node, and a data query node and an interface call node set between the start node and the end node, and the nodes run sequentially and exchange data through the entities defined in step S2; S44. Define timing strategy; The timing strategy is specifically defined by using a task scheduling framework to trigger the execution of data exchange tasks according to a specified time rule.

6. The entity-based data exchange method according to claim 5, characterized in that: The task nodes defined in step S43 specifically include the following nodes that are run in sequence: S431. Start node; S432. Query employee information node; S433. Loop connection; S434 identity authentication interface node; S435. Push employee information interface node; S436. End node.

7. The entity-based data exchange method according to claim 1, characterized in that: The step S5 comprises the following steps: S51. Input operating parameters; S52. Load task node; S53. Loading timing strategy; S54. Timing cycle operation.

Images