SAAS platform multi-channel data docking method and system based on micro service

Abstract

The invention discloses a micro-service-based SAAS platform multi-channel data docking method and system, and the method comprises the steps: configuring the types of processes required to be executed by a request, and enabling the types of the processes to comprise a pushing process and a receiving process; selecting module nodes required for executing the process according to the type of the process; selecting a corresponding implementation strategy for each module node; and when a request enters a docking process, querying all module nodes and implementation strategies thereof, and executing the strategies according to a module node sequence to complete the docking process. According to the data docking method for the SAAS platform, a modularized docking process of the SAAS platform is constructed, so that the data docking method for the SAAS platform, which is lower in docking cost, higher in data transmission security and lighter and more efficient in data conversion, is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of data docking of an operation platform, and in particular to a multi-channel data docking method and system for a SAAS platform based on microservices. Background Art

[0002] During the development of a media production SaaS platform, numerous third-party systems must be connected. These systems include cloud-based clients, website systems, and local non-linear editing systems (NLEs) such as those from Dayang, Sobey, and Xin Ao Te. Each system has different connection methods and data signatures, different data formats for each interface, and different performance and monitoring requirements. This can lead to the following technical issues:

[0003] 1. Cost: Connecting to external systems may incur additional costs due to customization requirements, and the cost-benefit ratio needs to be evaluated.

[0004] 2. Security issues: Connecting to external systems may involve data security and privacy protection issues, and the security of data transmission needs to be ensured.

[0005] 3. Data format mismatch: Different systems may use different data formats, and each connection requires code development for data conversion and mapping. Summary of the Invention

[0006] The purpose of the present invention is to provide a multi-channel data docking method for SAAS platform based on microservices, so as to solve the problems that the data access method of the existing SaaS platform requires data conversion, has low security and high cost.

[0007] On the one hand, an embodiment of the present invention provides a multi-channel data docking method for a SAAS platform based on microservices, comprising the following steps:

[0008] Configure the process type that the request needs to execute, which includes push process and receive process.

[0009] Select the module node required to execute the process according to the process type;

[0010] Select the corresponding implementation strategy for each module node;

[0011] When the request enters the docking process, all module nodes and their implementation strategies are queried, and the strategies are executed in the order of module nodes to complete the docking process.

[0012] Furthermore, the push process includes the process of pushing data on the SAAS side to third-party channels, and the receiving process includes the process of the SAAS side receiving data uploaded by third-party channels and performing business processing.

[0013] Furthermore, the module nodes required for executing the push process include: push permission verification, push data rule filtering, push data body construction, push signature construction, push method selection, initiation of data push, and recording of push logs and results.

[0014] Furthermore, the module nodes required for executing the receiving process include: module nodes for interface authentication, recording access requests, access tenant authentication, input parameter conversion, business processing, and output parameter conversion.

[0015] Furthermore, it also includes: generating a signature for each request that requires the execution of a process and verifying the request;

[0016] Among them, when executing the push process, the signature is generated by the module node constructed by the push signature, and when executing the receiving process, the signature is generated by the module node of the interface authentication.

[0017] Furthermore, generating a signature also includes:

[0018] Allocate application IDs and keys;

[0019] The secret key is used as the encryption key. The encrypted content includes: request method, request interface path, current timestamp and MD5 value of request parameters;

[0020] The signature obtained by the above encryption is placed in the HTTP request header for signature verification of the present invention.

[0021] Furthermore, the request verification further includes:

[0022] After receiving the request, the interface parses the request body to obtain the request method, request interface path, timestamp, and request parameters, and performs at least the following three steps of verification:

[0023] Verify whether the timestamp of the interface request exceeds the preset time of the current moment. If not, the verification is successful;

[0024] Encrypt the request method, request interface path, current timestamp, and MD5 value of the request parameters to obtain a new signature, and compare it with the signature in the current HTTP request header to see if they are completely consistent. If they are consistent, the verification is successful;

[0025] Parse the tenant ID in the request parameter and verify whether the tenant ID is in the whitelist tenant of SecretKey. If so, the verification is successful;

[0026] If the verification of the above three steps is successful, the request can execute the subsequent business logic of the interface; otherwise, it returns the signature failure of the requester, and a new signature is generated to initiate a new call request.

[0027] Furthermore, it also includes:

[0028] The module nodes for pushing data structure, input parameter conversion, and output parameter conversion can process the requested data, including reducing the requested data conversion into a lightweight data configuration as an input parameter for calling a third-party interface or as a response to being called by a third-party interface.

[0029] Furthermore, a data conversion tool based on JSON serialization is built to reduce the requested data conversion to lightweight data configuration based on the data conversion protocol;

[0030] The data conversion tool processes source data in JSON format, including replacing field names, flattening data, and encapsulating objects. The data conversion protocol is designed as follows:

[0031] Stored in the database in JSON format, the fields are the fields of the target data, and the field values ​​are JSON objects.

[0032] On the other hand, an embodiment of the present invention further provides a SAAS platform multi-channel data docking system based on microservices, including:

[0033] The process configuration module is used to configure the process type required for the request, including push process and receive process;

[0034] The strategy selection module is used to select the module nodes required to execute the process according to the process type, and select the corresponding implementation strategy for each module node;

[0035] The process selection module is used to select the corresponding process type for the request entering the docking process;

[0036] The policy execution module is used to query all module nodes and their implementation policies, execute policies in the order of module nodes, and complete the docking process.

[0037] The beneficial effects of the present invention are:

[0038] 1. Lower docking costs: The modular docking process of the self-built SAAS platform supports custom orchestration, and the modules can be reused in multiple processes, reducing docking costs.

[0039] 2. Enhanced data transmission security: A self-built interface request signature verification algorithm is used to protect data transmission security and encrypt sensitive data to ensure that data cannot be stolen or tampered with during transmission.

[0040] 3. Lighter and more efficient data conversion: We develop our own data conversion protocols and tools to handle data conversion between different formats. The data conversion protocol supports visual configuration, reducing heavyweight custom code development to lightweight data configuration, resulting in more efficient docking. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0042] Figure 1 This is a schematic diagram of the overall process of the multi-channel data docking method of the SAAS platform based on microservices provided in an embodiment of the present invention;

[0043] Figure 2 This is a schematic diagram of the overall structure of the multi-channel data docking system of the SAAS platform based on microservices provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0044] To make the purpose, technical solutions and advantages of the embodiments of the present invention more clear, the embodiments of the present invention are further described in detail below with reference to the accompanying drawings. Here, the exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0045] In the description of this specification, the terms "include", "including", "have", "contain", etc. are all open terms, which mean including but not limited to. The descriptions with reference to the terms "one embodiment", "a specific embodiment", "some embodiments", "for example", etc. mean that the specific features, structures or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures or characteristics described can be combined in a suitable manner in any one or more embodiments or examples. The order of steps involved in each embodiment is used to schematically illustrate the implementation of the present application, and the order of steps therein is not limited and can be appropriately adjusted as needed.

[0046] The present invention provides a method for connecting data via multiple channels on a SAAS platform based on microservices. Figure 1 The following is a schematic diagram of the overall process of the multi-channel data docking method of the microservice-based SAAS platform, which includes:

[0047] S1, the process type to be executed by the configuration request, which includes push process and receive process;

[0048] In order to realize the personalized docking requirements of SAAS platform with multi-channel and multi-tenant, reduce development costs and reduce code entropy increase, the present invention builds a modular docking process of SAAS platform, namely push process and receiving process, which is composed of multiple modules.

[0049] The present invention uses the pipe-filter architecture style to connect the required modules into a business docking process according to business needs to achieve the purpose of business support. At the same time, the referenced modules can be used in any process, so it has strong reusability.

[0050] Common module types are defined for both the push and receive processes, and each specific business can customize the module type as needed.

[0051] Furthermore, the push process of the present invention includes the process of pushing data on the SAAS side to a third-party channel, and pushing the data to the docking system through the network; the receiving process includes the process of the SAAS side receiving data uploaded by the third-party channel and performing business processing, and receiving data transmitted from the docking system through the network.

[0052] When connecting to an external system, a new connection process will be created on the management page. You can select the connection process as a push process or a receive process based on actual needs.

[0053] S2, select the module node required to execute the process according to the process type;

[0054] The module nodes required to execute the push process include: push permission verification, push data rule filtering, push data body construction, push signature construction, push method selection, data push initiation, and push log and result recording. The functions of each module node are as follows:

[0055] Push authorization verification (PushAuthCheckerFlowComponent), used to verify the authority of pushed data and pushers;

[0056] PushDataFilterFlowComponent, used to filter the pushed data according to business rules;

[0057] PushDataBuilderFlowComponent is used to build the data body that is finally pushed out;

[0058] Push signature construction (PushSignBuilderFlowComponent), used to build the signature data required by the recipient;

[0059] Push method selection (PushMethodFlowComponent), used to select the method for the recipient to receive data;

[0060] Initiate data push (PushHandlerFlowComponent) to execute data push;

[0061] Record push logs and results (PushLoggerFlowComponent), which is used to record the input and output parameters of push requests, analyze the results, and store them in the database.

[0062] The module nodes required to execute the receiving process include: interface authentication, access request recording, access tenant authentication, input parameter conversion, business processing, and output parameter conversion. The functions of each module node are as follows:

[0063] Interface authentication (UpPathCheckerFlowComponent) is used to perform signature verification and permission verification on the access party's request, such as whether the application name passed in the request has the permission to request the interface;

[0064] Record access requests (UpLoggerFlowComponent), which stores all requests in the database to facilitate subsequent troubleshooting and traffic playback.

[0065] Access tenant authentication (UpTenantCheckerFlowComponent) to verify whether the application name in the request has the data permissions of the tenant requested;

[0066] Input parameter conversion (UpParamsParserFlowComponent) is used to format the input parameter data to facilitate subsequent process processing;

[0067] Business processing (UpHandlerFlowComponent), used for logical processing of specific business;

[0068] Output parameter conversion (UpResultParserFlowComponent) converts the logical processing results of the business and returns them to the access party as needed.

[0069] The business implementation strategies of all the above modules are registered in Spring IOC, and the BeanName (unique name of the Bean) and its description information of the strategy instance (Bean) are automatically maintained in the database.

[0070] S3, select and implement the corresponding implementation strategy for each module node;

[0071] Specifically, an implementation strategy is selected for each module node. For example, a default implementation strategy is selected for a given parameter conversion module node and a field mapping relationship is configured. A default implementation strategy is selected for an interface authentication module and an identifier and key are configured.

[0072] Furthermore, since the data in both processes pass through the network, in order to ensure the security and effectiveness of the data received by the present invention and the docking system, the docking system must generate a signature for each mutually called data transmission request, and then perform triple verification on the request with the third-party system.

[0073] Specifically, a signature is generated for each request that needs to execute a process, and the request is verified; when executing the push process, the signature is generated by the module node constructed by the push signature, and when executing the receiving process, the signature is generated by the module node of the interface authentication.

[0074] Furthermore, signature generation mainly consists of three steps:

[0075] 1. The present invention allocates an application identifier (AppCode) and a secret key (SecretKey) to each docking system;

[0076] 2. The docking system uses SecretKey as the HMACSHA1 encryption key. The encrypted content is: request method (such as POST / GET, etc.), request interface path (such as / openapi / pan), current timestamp, and MD5 value of request parameters.

[0077] 3. The docking system puts the encrypted signature into the HTTP request header (Authorization) for signature verification.

[0078] After receiving the request, the docking system parses the request header (Authorization) to obtain the signature, request method, path, timestamp, and request parameters, and performs the following verification steps:

[0079] 1. Check whether the timestamp of the interface request is more than 1 minute beyond the current time;

[0080] 2. Encrypt the request method, request interface path, current timestamp, and MD5 value of the request parameters to obtain a new signature, and compare it with the original signature to ensure that they are completely consistent;

[0081] 3. Parse the tenant ID in the request parameter and verify whether the tenant ID is in the whitelist tenant of SecretKey.

[0082] If the verification of the above three steps is successful, the request can execute the business logic of the interface S1 receiving process. If any step fails to pass the verification, the requester will be returned a signature failure, and a new signature must be generated to initiate a new call request. The same method can be used for the verification and processing of the docking system.

[0083] S4, when the request enters the docking process, all module nodes and their implementation strategies are queried, and the strategies are executed in the order of module nodes to complete the docking process.

[0084] When a request enters the docking process, all module nodes and their implementation strategy BeanName are queried from the database, and then the Bean instance is obtained through the Spring ApplicationContext.getBean() method. The process() method is then executed in sequence according to the module node order to complete the process docking.

[0085] All implementation strategies of module nodes can be used in all docking processes, which improves module reusability and reduces docking costs.

[0086] Further preferably, the module nodes for pushing data structure, input parameter conversion, and output parameter conversion can process the requested data, including reducing the requested data conversion to a lightweight data configuration as an input parameter for calling a third-party interface or as a response to being called by a third-party interface.

[0087] Specifically, data conversion usually requires heavyweight custom code development. To reduce development costs and improve docking efficiency, the present invention adopts an interpreter architecture style to build a data conversion tool based on JSON serialization, reducing data conversion to lightweight data configuration. The data conversion tool replaces field names, flattens data, and encapsulates objects on the source data in JSON format to obtain target data in JSON format. The target data is used as an input parameter for calling a third-party interface or as a response to a call from a third-party interface.

[0088] The data conversion tool is written in JAVA and uses FastJson internally for retrieval and assignment of JSON data. The field mapping relationship of the retrieval and assignment operations is carried out by a self-built data conversion protocol (hereinafter referred to as "the protocol"). The design idea of ​​the data conversion protocol in this invention is to first define a target data format, where the value of each field in the target data is derived from the field path or fixed value of the source data. The specific design of the protocol is as follows:

[0089] The protocol is stored in the database in JSON format, where the key (i.e., field) is the key of the target data and the value (the value of the field) is a JSON object (JSONObject). The value object contains three fields:

[0090] 1. type, used to declare the data type of the target key, including: data value type [Value], object type [Object], array type [Array];

[0091] 2. field is the key path of the source data. When type=Value, it represents the source data field path. If there are multiple levels in the path, each level is separated by a dot. If it is an array, [number] is used to represent the subscript. When type=Object, field is an object. Each key and value in the object represents the target key and source key path of a field. When type=Array, field is an object. Each key and value in the object represents the target key and source key path of a field of an array element.

[0092] 3. source: This field is only available when type=Array, and its value is the array object address in the source data json;

[0093] 4. fixedValue: When type=Value, a fixed value can be assigned to the current field

[0094] Based on the above protocol, and supplemented by FastJson for JSON parsing, the conversion tool JSONFormatUtil can be used to convert source data into target data.

[0095] To reduce the cost of managing field mapping relationships, a visual management page has been developed that supports adding, editing, deleting, list querying, and detail querying. Visual management operations include adding and editing. Developers simply upload complete examples of source and target data. The page formats the data fields and displays them on the left and right sides. Developers then connect the fields of the two data sets to generate field mapping relationships, which they then submit and save to the database.

[0096] like Figure 2 As shown, the present invention also proposes a SAAS platform multi-channel data docking system based on microservices, which can implement the above method. The system includes:

[0097] The process configuration module is used to configure the process type required for the request, including push process and receive process;

[0098] The strategy selection module is used to select the module nodes required to execute the process according to the process type, and select the corresponding implementation strategy for each module node;

[0099] The process selection module is used to select the corresponding process type for the request entering the docking process;

[0100] The policy execution module is used to query all module nodes and their implementation policies, execute policies in the order of module nodes, and complete the docking process.

[0101] The present invention is based on the docking process of a modular SAAS platform to reduce the cost of multi-channel data docking; improves the security of data transmission through encryption signatures; and constructs data conversion protocols and tools to improve data conversion efficiency. As the data docking center of the SAAS platform, the present invention can achieve low-cost, high-efficiency and secure multi-channel data docking.

[0102] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0103] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0104] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0105] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1A step that specifies a function in one or more boxes.

[0106] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A multi-channel data docking method for SAAS platform based on microservices, characterized in that: include: Configure the process type that the request needs to execute. The process types include push process and receive process. Select the module node required to execute the process according to the process type; Select the corresponding implementation strategy for each module node; When the request enters the docking process, all module nodes and their implementation strategies are queried, and the strategies are executed in the order of module nodes to complete the docking process.

2. The multi-channel data docking method of SAAS platform based on microservices according to claim 1, characterized in that: The push process includes the process of pushing data on the SAAS side to third-party channels, and the receiving process includes the process of the SAAS side receiving data uploaded by third-party channels and performing business processing.

3. The multi-channel data docking method of SAAS platform based on microservices according to claim 1, characterized in that: The module nodes required for executing the push process include: push permission verification, push data rule filtering, push data body construction, push signature construction, push method selection, initiation of data push, and module nodes for recording push logs and results.

4. The multi-channel data docking method of SAAS platform based on microservices according to claim 1, characterized in that: The module nodes required for executing the receiving process include: module nodes for interface authentication, recording access requests, access tenant authentication, input parameter conversion, business processing, and output parameter conversion.

5. The multi-channel data docking method of SAAS platform based on microservices according to claim 1, characterized in that: Also includes: Generate a signature for each request that needs to execute a process and verify the request; Among them, when executing the push process, the module node constructed by the push signature generates the signature, and when executing the receiving process, the module node of the interface authentication generates the signature.

6. The multi-channel data docking method of SAAS platform based on microservices according to claim 1, characterized in that: The generating of the signature further comprises: Allocate application IDs and keys; The secret key is used as the encryption key. The encrypted content includes: request method, request interface path, current timestamp and MD5 value of request parameters; The signature obtained by the above encryption is placed in the HTTP request header for signature verification of the present invention.

7. The multi-channel data docking method of SAAS platform based on microservices according to claim 6, characterized in that: The verifying of the request further comprises: After receiving the request, the interface parses the request body to obtain the request method, request interface path, timestamp, and request parameters, and performs at least the following three steps of verification: Verify whether the timestamp of the interface request exceeds the preset time of the current moment. If not, the verification is successful. Encrypt the request method, request interface path, current timestamp, and MD5 value of the request parameters to obtain a new signature, and compare it with the signature in the current HTTP request header to see if they are completely consistent. If they are consistent, the verification is successful; Parse the tenant ID in the request parameter and verify whether the tenant ID is in the whitelist tenant of SecretKey. If so, the verification is successful; If the verification of the above three steps is successful, the request can execute the subsequent business logic of the interface; otherwise, the requester signature fails and a new call request is initiated by regenerating the signature.

8. The multi-channel data docking method of SAAS platform based on microservices according to claim 1, characterized in that: Also includes: The module nodes for pushing data structure, input parameter conversion, and output parameter conversion can process the request data, including reducing the requested data conversion to a lightweight data configuration as an input parameter for calling a third-party interface or as a response to a call by a third-party interface.

9. The multi-channel data docking method of SAAS platform based on microservices according to claim 8, characterized in that: Build a data conversion tool based on JSON serialization, and reduce the requested data conversion to lightweight data configuration based on the data conversion protocol; The data conversion tool processes the source data in JSON format, including: replacing field names, flattening data, and encapsulating objects. The data conversion protocol is designed as follows: Stored in the database in JSON format, where the fields are the fields of the target data and the values ​​of the fields are JSON objects.

10. A SAAS platform multi-channel data docking system based on microservices, characterized in that: include: The process configuration module is used to configure the type of process that needs to be executed by the request, including the push process and the receive process; The strategy selection module is used to select the module nodes required for the execution process according to the process type, and select the corresponding implementation strategy for each module node; The process selection module is used to select the corresponding process type for the request entering the docking process; The policy execution module is used to query all module nodes and their implementation policies, execute the policies in the order of module nodes, and complete the docking process.

Images