Method and system for quickly constructing services and uniformly managing and controlling services based on multiple data sources

Abstract

The invention provides a method and a system for quickly constructing services and uniformly managing and controlling based on multiple data sources. The method comprises the following steps: constructing a request structural body of a multi-data-source service, and issuing the multi-data-source service to a service resource directory to set a security management and control strategy; in response to a service request sent by a user, the system queries and calls a target service in the service resource directory; the service request body parser parses the request structure body of the target service to obtain and verify an internal data structure, and the system dynamically controls the request of the target service according to the security control strategy; and the engine converter distributes the target service request to the service engine to construct a request structure body according to the attribute value of the request structure body of the target service, converts the request structure body into a data query grammar to perform data query, encapsulates a query result into a result request body and returns the result request body to the client. According to the invention, the service is flexibly managed, the construction efficiency of the service is improved, the calling process of the service is simplified, and the fine-grained control of the service is realized.

Description

Technical Field

[0001] The present invention relates to the field of data source management and unified service technology, and in particular to a method and system for rapidly constructing services based on multiple data sources and for unified management and control. Background Art

[0002] At present, data warehouse and data lake technologies can effectively integrate multiple heterogeneous data sources and support complex data analysis and decision support. The rise of NoSQL databases and NewSQL databases provides new solutions for handling large-scale, high-concurrency data storage and queries. Cloud database services provide convenient database hosting services and reduce the complexity of database management. However, database integration solutions are often targeted at specific data sources and scenarios, lack a unified integration framework, and are inefficient when handling multiple databases. Data warehouse and data lake technologies have certain delays when handling real-time data integration and queries.

[0003] RESTfulAPI is the mainstream framework of Web services and is widely used in service calls, but it lacks unified standards and tools in service management and monitoring. Although the microservice architecture is flexible, there are still challenges in service discovery, load balancing and fault tolerance.

[0004] Existing data encryption, access control and identity authentication technologies are becoming increasingly mature, but they lack flexibility and fine-grained control in practical applications.

[0005] Existing cache strategies such as Redis and Memcached are widely used to improve data access speed and system performance. Although cache strategies improve data access speed, there are still certain challenges in cache consistency and update strategies. Performance optimization technologies such as database index optimization and query optimization are often customized for specific scenarios and lack general optimization solutions.

[0006] In response to the above problems, this application proposes a method and system for quickly building services based on multiple data sources and unified management and control, aiming to achieve rapid construction and efficient calling of services by building a service resource directory and a unified service request parser, shielding the differences in the underlying data structures. Summary of the invention

[0007] The present invention proposes the following technical solutions to address one or more technical deficiencies in the above-mentioned prior art.

[0008] Based on the first aspect of the present invention, a system for rapidly constructing services based on multiple data sources and unified management and control is proposed, including a multiple data source service, a service resource directory, a service engine parser, a service request body parser and a service engine converter;

[0009] The multiple data source service is used to construct a request structure;

[0010] The service resource directory is used to store the multiple data source services and set security management and control policies;

[0011] The service engine parser is used to identify the service engine corresponding to the service request;

[0012] The service request body parser is used to parse and verify the request structure;

[0013] The service engine converter is used to distribute the service request to different service engines.

[0014] Furthermore, the multiple data source services include orchestration services, REST services, data resource services, database services, proxy services and model services.

[0015] Furthermore, the request structure includes a proxy service underlying data structure, an orchestration service underlying data structure, a model service underlying data structure, a model service underlying data structure, a database service underlying data structure, a REST service underlying data structure and a data resource service underlying data structure.

[0016] Through the service engine parser and engine converter, rapid response and query optimization of real-time data are achieved; the service request parser uniformly manages and monitors services, improving the maintainability and observability of services; the service dynamic management and control mechanism realizes fine-grained control of service calls and meets the requirements of privacy protection regulations; the engine converter and result converter realize universal optimization of service calls and improve the overall performance of the system.

[0017] Based on the second aspect of the present invention, a method for quickly building services and uniformly managing and controlling the services using the above system is also proposed, including:

[0018] S1: construct the request structure of the multi-data source service according to the service type and database type, and publish the multi-data source service to the service resource directory, and then set the security control strategy according to the service management requirements;

[0019] S2: In response to the service request sent by the user, the system searches the service resource directory for the target service, and calls the target service if the target service exists in the service resource directory, otherwise, the calling service is terminated;

[0020] At the same time, the service request body parser parses the request structure of the target service, obtains the relevant information of the request of the target service and parses it according to the request protocol and format, obtains the internal data structure and verifies it;

[0021] S3: The service engine parser identifies the service engine corresponding to the target service request and requests dynamic control, and the system dynamically controls the request for the target service according to the security control policy;

[0022] S4: the engine converter distributes the request of the target service to different service engines according to the attribute value in the request structure of the target service;

[0023] S5: The service engine constructs a request structure according to the request of the target service, converts the request structure into a corresponding data query syntax for data query, and encapsulates the query result into a result request body and returns it to the client.

[0024] A unified service resource directory can integrate multiple databases to achieve rapid construction and efficient calling of services.

[0025] Furthermore, the security control strategy includes interruption control, concurrency control, request control, traffic control and IP blacklist and whitelist control;

[0026] The request control is used to limit the number of times the multi-data source service is called within a specified time and to limit the total number of requests from each requester;

[0027] The flow control dynamically controls the flow according to the system load and resource utilization;

[0028] The interruption control interrupts the service request when an exception occurs in the call of the multi-data source service or the application end;

[0029] The concurrency control sets thresholds for the query rate per second, the number of threads, and the error percentage, and responds according to the effect of the control strategy when the query rate per second, the number of threads, and the error percentage exceed the thresholds, including:

[0030] Directly reject the traffic that exceeds the threshold;

[0031] The traffic exceeding the threshold is put into the waiting queue. If the traffic exceeding the threshold is not processed within the preset time, the traffic exceeding the threshold is rejected.

[0032] When the flow rate increases, slowly increase the flow rate so that the system flow rate reaches the threshold within a certain period of time to preheat the system;

[0033] The IP blacklist and whitelist control denies access when the service request belongs to an IP request in the blacklist, and allows access to the multi-data source service when the request belongs to an IP request in the whitelist.

[0034] Through the service dynamic management and control mechanism, fine-grained control of service calls is achieved, enhancing the stability and reliability of the service.

[0035] Furthermore, the obtaining of the internal data structure and verification includes using a verification library or a custom code to verify whether the parameters in the request of the target service are empty, whether they meet format requirements, and whether the parameters are within a valid range.

[0036] Furthermore, the relevant information of the request for the target service includes request path, request method, request header information, query parameters and request body content.

[0037] Furthermore, the interface call status, performance indicators and abnormal conditions of the multi-data source services are fully monitored through the multi-data source service access log, and a warning is issued when the monitoring indicator exceeds the set threshold.

[0038] Furthermore, an API service interface is selected according to the URL path, HTTP header and parameters of the multi-data source service request, the multi-data source service is connected to the system through the API service interface, and the service request processing logic is unified in the routing layer of the API service interface.

[0039] Based on the third aspect of the present invention, a computer program product is also proposed, which has one or more computer programs thereon, and when the computer program is executed by a computer processor, any of the methods described above is implemented.

[0040] The technical effect of the present invention is that: the present invention provides a unified service construction method and system based on multiple databases, which flexibly manages and controls services by constructing a service resource directory and a unified service request parser, effectively shielding the differences in the underlying data structures, realizing rapid construction and efficient calling of services, improving service construction efficiency, simplifying the service calling process, and enhancing service stability and security. It can be widely used in enterprise-level application integration, cloud computing service platform, big data analysis platform, Internet of Things application and mobile application development to meet ever-changing business needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Other features, objects and advantages of the present application will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings.

[0042] Figure 1 It is a system framework diagram for quickly building services based on multiple data sources and unified management and control according to an embodiment of the present invention.

[0043] Figure 2 It is a flowchart of a method for quickly building services based on multiple data sources and unified management and control provided according to an embodiment of the present invention.

[0044] Figure 3It is a service construction and calling flow chart provided according to an embodiment of the present invention.

[0045] Figure 4 It is a service underlying data structure diagram provided according to an embodiment of the present invention.

[0046] Figure 5 It is a service life cycle diagram provided according to an embodiment of the present invention.

[0047] Figure 6 It is a schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It should also be noted that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0049] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0050] Figure 1 The present invention shows a system for rapidly constructing services based on multiple data sources and unified management and control, including a multiple data source service a, a service resource directory b, a service engine parser c, a service request body parser d and a service engine converter e;

[0051] The multi-data source service a is used to construct a request structure;

[0052] The service resource directory b is used to store the multi-data source services and set security management and control policies;

[0053] The service engine parser c is used to identify the service engine corresponding to the service request;

[0054] The service request body parser d is used to parse and verify the request structure;

[0055] The service engine converter e is used to distribute service requests to different service engines.

[0056] It should be noted that the multiple data source services include orchestration services, REST services, data resource services, database services, proxy services and model services.

[0057] It should be noted that the request structure includes the proxy service underlying data structure, the orchestration service underlying data structure, the model service underlying data structure, the model service underlying data structure, the database service underlying data structure, the REST service underlying data structure and the data resource service underlying data structure.

[0058] It should be noted that the following method is implemented in the system for rapidly building services based on multiple data sources and unified management and control, the steps are as follows: Figure 2 As shown, including:

[0059] S1: construct the request structure of the multi-data source service according to the service type and database type, and publish the multi-data source service to the service resource directory, and then set the security control strategy according to the service management requirements;

[0060] S2: In response to the service request sent by the user, the system searches the service resource directory for the target service, and calls the target service if the target service exists in the service resource directory, otherwise, the calling service is terminated;

[0061] At the same time, the service request body parser parses the request structure of the target service, obtains the relevant information of the request of the target service and parses it according to the request protocol and format, obtains the internal data structure and verifies it;

[0062] S3: The service engine parser identifies the service engine corresponding to the target service request and requests dynamic control, and the system dynamically controls the request for the target service according to the security control policy;

[0063] S4: the engine converter distributes the request of the target service to different service engines according to the attribute value in the request structure of the target service;

[0064] S5: The service engine constructs a request structure according to the request of the target service, converts the request structure into a corresponding data query syntax for data query, and encapsulates the query result into a result request body and returns it to the client.

[0065] It should be noted that the security control strategy includes interruption control, concurrency control, request control, traffic control and IP blacklist and whitelist control;

[0066] The request control is used to limit the number of times the multi-data source service is called within a specified time and to limit the total number of requests from each requester;

[0067] The flow control dynamically controls the flow according to the system load and resource utilization;

[0068] The interruption control interrupts the service request when an exception occurs in the call of the multi-data source service or the application end;

[0069] The concurrency control sets thresholds for the query rate per second, the number of threads, and the error percentage, and responds according to the effect of the control strategy when the query rate per second, the number of threads, and the error percentage exceed the thresholds, including:

[0070] Directly reject the traffic that exceeds the threshold;

[0071] The traffic exceeding the threshold is put into the waiting queue. If the traffic exceeding the threshold is not processed within the preset time, the traffic exceeding the threshold is rejected.

[0072] When the flow rate increases, slowly increase the flow rate so that the system flow rate reaches the threshold within a certain period of time to preheat the system;

[0073] The IP blacklist and whitelist control denies access when the service request belongs to an IP request in the blacklist, and allows access to the multi-data source service when the request belongs to an IP request in the whitelist.

[0074] It should be noted that obtaining the internal data structure and performing verification includes using a verification library or custom code to verify whether the parameters in the request of the target service are empty, whether they meet the format requirements, and whether the parameters are within a valid range.

[0075] It should be noted that the relevant information of the request for the target service includes the request path, request method, request header information, query parameters and request body content.

[0076] It should be noted that the interface call status, performance indicators and abnormal conditions of the multi-data source services are fully monitored through the multi-data source service access log, and a warning is issued when the monitoring indicator exceeds the set threshold.

[0077] It should be noted that the API service interface is selected according to the URL path, HTTP header and parameters of the multi-data source service request, the multi-data source service is connected to the system through the API service interface, and the service request processing logic is unified in the routing layer of the API service interface.

[0078] In a specific embodiment, the service call and construction process is as follows: Figure 3As shown, the underlying data is constructed based on a variety of databases (including MySQL, MongoDB, Solr, PostgreSQL, Mongo, Dor is and El asticsearch) to form a database service, and an orchestration service, REST service, data resource service, database service, proxy service and model service are constructed. The corresponding request structure is constructed according to different database types and service types, and the constructed services are uniformly published to the service resource directory, and the API service interface is used to access the system, and full life cycle management (including publishing, editing, reviewing and monitoring) is performed;

[0079] After the service is built, the user sends a service request through the client. The system searches the service resource directory and applies for the target service. If the target service exists in the system, it calls the target service.

[0080] During the process of calling the target service, the system performs full-link monitoring;

[0081] At the same time, the service request body parser parses the request structure of the target service, and parses the relevant information of the target service request according to the request protocol and format, and further uses the verification library or custom code to verify whether the parameters in the request of the target service are empty, whether they meet the format requirements, whether the parameters are within the valid range, whether the legality, integrity and format of the service request parameters meet the expectations, and verifies the internal data structure of the service request body, whether the service request body has any required parameters missing, red list and IP restriction queries;

[0082] The service engine parser identifies the service engine corresponding to the target service request and requests dynamic control. The system dynamically controls the services that hit the policy during the call process according to the pre-set security control policy.

[0083] The request is distributed to different engine layers according to the type attributes (including rest and resource) in the target service request structure. The service engine constructs the request structure according to the target service and converts it into the corresponding data query syntax, calls the underlying data for query, and encapsulates the data results obtained from the query into a result request body and returns it to the client.

[0084] It should be noted that the database service extracts data from the database to construct the service;

[0085] The data resource service selects a governed data construction service from the data resource catalog;

[0086] The REST service registers the third-party interface to the service platform for unified management;

[0087] The model service selects a model building service from a model resource catalog;

[0088] The proxy service proxies the service of one service platform to another service platform for use, so as to achieve cascading and sharing of services;

[0089] The orchestration service orchestrates services and generates new orchestration services by visually dragging and dropping operators.

[0090] It should be noted that the structure of the request structure is as follows Figure 4 As shown, the service URL, service request parameters, and request results are recorded.

[0091] In a specific embodiment, multiple data source services are uniformly connected to the system through an API service interface, and service access logs are obtained to monitor the entire system, further simplifying development and maintenance work and improving overall availability and scalability. The specific steps are:

[0092] Define unified interface specifications for the API service interface, including request method, request path, request parameters, request body structure, and response body format;

[0093] In the API service, a web framework or API gateway is used to implement a unified routing layer, which routes all service requests to the corresponding processing logic, and verifies the request parameters and processing logic according to the interface specifications to generate corresponding results.

[0094] It should be noted that using a web framework or API gateway to implement a unified routing layer in the API service can ensure that all service requests go through the same entry point and centrally manage service requests.

[0095] It should be noted that the service resource directory shields the underlying data structure of the service request body. The caller does not care about the structural complexity of the underlying data structure. It only needs to obtain the service name and service address of the service resource directory and use a unified request structure to request the service to obtain the target data.

[0096] It should be noted that if Figure 5 As shown, the entire life cycle of the service resource directory includes service publishing, editing, review, management, online, deactivation, offline and deletion. When a service is offline or deactivated, the current service cannot be called and an application for service call must be made to the administrator.

[0097] It should be noted that the request control limits the request frequency by setting the maximum number of requests or the request rate within a time window, and sets an access quota for each requester to limit the total number of requests sent by each requester or the request sending frequency. When the request sent by the requester exceeds the access quota, the request is rejected or an error message is returned, which can prevent service abuse and denial of service attacks.

[0098] It should be noted that the interruption control immediately interrupts the service request when an exception occurs in the call of the multi-data source service or application end (including network failure or database connection failure), which can avoid system performance degradation caused by service unavailability or service delay.

[0099] It should be noted that the setting of IP blacklist and whitelist control can deny access services to certain IPs or open access rights to specific IPs, thereby improving the security and flexibility of the system.

[0100] It should be noted that the system periodically polls the service resource directory and caches the service request structure and detailed service information locally, which can improve the response speed and consistency of the service.

[0101] It should be noted that by managing the services that hit the strategy during the call process according to the service management and control strategy, it is possible to protect the services from abuse, prevent overload risks, and implement current limiting and concurrency control mechanisms to prevent the services from being overwhelmed by too many requests.

[0102] It should be noted that by obtaining service access logs for monitoring and recording logs of key operations and events, including the receipt of requests, the return of processing results and error information, it is possible to understand the interface call status, performance indicators and abnormal situations; through the alarm policy setting for monitoring warnings, when the monitoring indicators exceed the set threshold, the alarm mechanism is triggered, and the relevant personnel are notified through instant notification. It is possible to monitor the service access status, discover problems in a timely manner, and prevent service access failures.

[0103] It should be noted that this application simplifies the full life cycle management of services and speeds up the construction of services. The caller does not need to pay attention to the underlying data structure and can call the service through a unified query syntax and request structure, which reduces the difficulty and complexity of service calls. The service dynamic management mechanism can enhance system stability while ensuring the security of services, ensuring data security and user authority compliance, making service life cycle management more flexible and controllable.

[0104] It should be noted that the method of the present invention can realize unified management and calling of internal enterprise services, improve the efficiency and flexibility of enterprise application integration; it can also serve as the core technology of the cloud service platform, providing efficient and stable service construction and calling capabilities; it can also be used to build a unified data service layer to simplify the data analysis process; it can be applied to the construction of the Internet of Things data service platform to achieve unified management and efficient calling of data; it can be applied to provide a unified service calling interface to simplify the development and maintenance of mobile applications.

[0105] Reference below Figure 6 , which shows a schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application. Figure 6 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0106] like Figure 6 As shown, the computer system includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 into a random access memory (RAM) 603. Various programs and data required for system operation are also stored in the RAM 603. The CPU 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0107] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a liquid crystal display (LCD), etc. and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage section 608 as needed.

[0108] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the central processing unit (CPU) 601, the above functions defined in the method of the present application are executed. It should be noted that the computer-readable storage medium of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, - but not limited to - an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable storage medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wireless, wireline, optical cable, RF, etc., or any suitable combination of the foregoing.

[0109] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0110] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0111] The modules involved in the embodiments of the present application may be implemented by software or by hardware.

[0112] As another aspect, the present application further provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device. The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: constructs a request structure of the multi-data source service according to the service type and the database type, and publishes the multi-data source service to the service resource directory in a unified manner, and then sets a security control policy according to the service management requirements; in response to the service request sent by the user, the system queries the target service in the service resource directory, and calls the target service if the target service exists in the service resource directory, otherwise the call service is terminated; at the same time, the service request body parser parses the request structure of the target service, obtains the relevant information of the request of the target service, and parses it according to the request protocol and format, obtains the internal data structure and verifies it; the service engine parser identifies the service engine corresponding to the target service request and requests dynamic control, and the system dynamically controls the request of the target service according to the security control policy; the engine converter distributes the request of the target service to different service engines according to the attribute values ​​in the request structure of the target service; the service engine constructs a request structure according to the request of the target service, converts the request structure into a corresponding data query syntax for data query, and encapsulates the query result into a result request body and returns it to the client.

[0113] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above invention concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

[0114] Finally, it should be noted that the above embodiments are only intended to illustrate rather than limit the technical solutions of the present invention. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that the present invention can still be modified or replaced by equivalents. Any modification or partial replacement that does not depart from the spirit and scope of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A system for rapidly building services and unified management based on multiple data sources, characterized in that: Includes multi-data source services, service resource directory, service engine parser, service request body parser and service engine converter; The multiple data source service is used to construct a request structure; The service resource directory is used to store the multiple data source services and set security management and control policies; The service engine parser is used to identify the service engine corresponding to the service request; The service request body parser is used to parse and verify the request structure; The service engine converter is used to distribute the service request to different service engines.

2. The system according to claim 1, characterized in that The multiple data source services include orchestration services, REST services, data resource services, database services, proxy services and model services.

3. The system according to claim 1, characterized in that The request structure includes a proxy service underlying data structure, an orchestration service underlying data structure, a model service underlying data structure, a model service underlying data structure, a database service underlying data structure, a REST service underlying data structure and a data resource service underlying data structure.

4. A method for rapidly building services and unified management and control using the system according to any one of claims 1 to 3, characterized in that: include: S1: construct the request structure of the multi-data source service according to the service type and database type, and publish the multi-data source service to the service resource directory, and then set the security control strategy according to the service management requirements; S2: In response to the service request sent by the user, the system searches the service resource directory for the target service, and calls the target service if the target service exists in the service resource directory, otherwise, the calling service is terminated; At the same time, the service request body parser parses the request structure of the target service, obtains the relevant information of the request of the target service and parses it according to the request protocol and format, obtains the internal data structure and verifies it; S3: The service engine parser identifies the service engine corresponding to the target service request and requests dynamic control, and the system dynamically controls the request for the target service according to the security control policy; S4: the engine converter distributes the request of the target service to different service engines according to the attribute value in the request structure of the target service; S5: The service engine constructs a request structure according to the request of the target service, converts the request structure into a corresponding data query syntax for data query, and encapsulates the query result into a result request body and returns it to the client.

5. The method according to claim 4, characterized in that The security control strategies include interruption control, concurrency control, request control, traffic control, and IP blacklist and whitelist control; The request control is used to limit the number of times the multi-data source service is called within a specified time and to limit the total number of requests from each requester; The flow control dynamically controls the flow according to the system load and resource utilization; The interruption control interrupts the service request when an exception occurs in the call of the multi-data source service or the application end; The concurrency control sets thresholds for the query rate per second, the number of threads, and the error percentage, and responds according to the effect of the control strategy when the query rate per second, the number of threads, and the error percentage exceed the thresholds, including: Directly reject the traffic that exceeds the threshold; The traffic exceeding the threshold is put into the waiting queue. If the traffic exceeding the threshold is not processed within the preset time, the traffic exceeding the threshold is rejected. When the flow rate increases, slowly increase the flow rate so that the system flow rate reaches the threshold within a certain period of time to preheat the system; The IP blacklist and whitelist control denies access when the service request belongs to an IP request in the blacklist, and allows access to the multi-data source service when the request belongs to an IP request in the whitelist.

6. The method according to claim 4, characterized in that The obtaining and verifying of the internal data structure includes using a verification library or a custom code to verify whether the parameters in the request of the target service are empty, whether the format requirements are met, and whether the parameters are within a valid range.

7. The method according to claim 4, characterized in that The relevant information of the request for the target service includes the request path, request method, request header information, query parameters and request body content.

8. The method according to claim 4, characterized in that The interface call status, performance indicators and abnormal conditions of the multi-data source services are fully monitored through the multi-data source service access log, and a warning is issued when the monitoring indicators exceed the set thresholds.

9. The method according to claim 4, characterized in that An API service interface is selected according to the URL path, HTTP header and parameters of the multi-data source service request, the multi-data source service is connected to the system through the API service interface, and the service request processing logic is unified in the routing layer of the API service interface.

10. A computer program product having one or more computer programs thereon, characterized in that: When the computer program is executed by a computer processor, the method according to any one of claims 4 to 9 is implemented.

Images