Data integration service processing method, device and electronic equipment
By building offline tasks in SaaS services, obtaining and combining the output data of multiple data source services and saving them to the database, the problems of interface query latency and high QPS requests in SaaS services are solved, and overall efficiency and system performance are improved.
Patent Information
- Application Number
- CN202210271202.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-18
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2042-03-18
AI Technical Summary
In SaaS services, there are many tenants and many products under each tenant, resulting in a long overall delay time for the interface query service, and the downstream data source service cannot withstand high QPS requests, which affects efficiency.
By obtaining the dependencies between multiple data source services associated with the tenant, perform offline tasks to make remote calls, combine data and save it to the database on the SaaS service side, reducing direct calls to the data source services during online queries.
It improves the overall efficiency of SaaS services in providing interface query services, reduces the latency of database search, and improves the system's processing capabilities.
Smart Images

Figure CN114840585B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of data integration in software as a service, and in particular to a data integration service processing method, device and electronic device. Background Art
[0002] SaaS (Software-as-a-Service) services are usually based on a set of standard software systems to provide services to a large number of different customers (also known as tenants). This requires SaaS services to support the isolation of data and configurations between different tenants, thereby ensuring the security and privacy of each tenant's data, as well as the user's personalized needs for interfaces, processing logic, data structures, etc., among which data integration tasks are a very important aspect.
[0003] For example, in a SaaS service for corporate tenants (to B), the specific tenants may be some merchants (they may be merchants who have opened stores on some e-commerce platforms, or they may be offline merchants), and the main capability of this SaaS service is to provide tenants with replenishment strategy management services. That is to say, in the supply chain scenario, a specific merchant may need to replenish goods in a specific warehouse every day. Therefore, each time replenishment is performed, it is necessary to first calculate the replenishment quantity, and then put the goods into the specific warehouse. In this process, the merchant side needs to execute scheduled tasks related to replenishment, but the execution of the scheduled tasks requires the support of some strategies, and the aforementioned SaaS service can provide support in this regard. In other words, when using the aforementioned SaaS service, the formulation of strategies, including the quantity of replenishment required each time, etc., can be determined by the SaaS service, and the merchant side only needs to execute according to the strategy formulated by the SaaS service.
[0004] In the process of providing SaaS services, there are many tenants of the service, and each tenant has a large number of commodities (each commodity can correspond to an Item). The replenishment strategy corresponding to each commodity may be different, and it is necessary to request specific strategies from the SaaS service. Therefore, the SaaS service provider usually provides some interfaces for tenants, through which tenants can initiate queries to the SaaS service to obtain the required data.
[0005] In order to reduce the calling burden of tenants, the input parameter of the above interface is usually only "ItemId". However, it is usually necessary to pull more other data that can be obtained through "ItemId" according to "ItemId" inside the interface as parameters used in the actual processing process. When obtaining other data, it is usually necessary to call some other data source services (usually services on the specific tenant side). Calling these data source services usually involves RPC (Remote Procedure Call) access, and the call is made in a synchronous blocking manner, which causes the overall delay time of the above interface to be affected by the delay time of these data source services. In addition, since the number of tenants of SaaS services and the number of items under each tenant are very large, the QPS (Queries-per-second) of the interface provided by the SaaS service may be very high, but the downstream data source service may not be able to withstand such high QPS requests.
[0006] Therefore, how to improve the overall efficiency of SaaS services in the process of providing interface query services has become a technical problem that needs to be solved by technical personnel in this field. Summary of the invention
[0007] The present application provides a data integration service processing method, device and electronic device, which can improve the overall efficiency of SaaS services in the process of providing interface query services.
[0008] This application provides the following solutions:
[0009] A data integration service processing method, comprising:
[0010] Obtaining dependency relationships between multiple data source services associated with a tenant of Software as a Service (SaaS); the multiple data source services include a data source service deployed on the tenant side;
[0011] By executing an offline task, the input parameter acquisition logic in the service interface of the SaaS service is called offline with the commodity identifier associated with the tenant as a parameter, so as to remotely call the multiple data source services in the offline task;
[0012] After combining the output data returned by the multiple data source services, the corresponding relationship between the product identifier and the output data combination result is saved in the database of the SaaS service side;
[0013] After receiving an online query request initiated to the service interface of the SaaS service with the target product identifier as a parameter, the output data combination result corresponding to the target product identifier is queried from the database and used as an input parameter of the algorithm processing logic in the service interface to obtain the algorithm processing result and return it to the querying party.
[0014] A data integration service processing device, comprising:
[0015] A dependency acquisition unit, used to acquire the dependency between multiple data source services associated with a tenant of Software as a Service (SaaS); the multiple data source services include a data source service deployed on the tenant side;
[0016] An offline task execution unit, configured to perform an offline call to the input parameter acquisition logic in the service interface of the SaaS service by executing an offline task, taking the commodity identifier associated with the tenant as a parameter, so as to remotely call the multiple data source services in the offline task;
[0017] A storage unit, configured to store the corresponding relationship between the commodity identifier and the output data combination result in a database on the SaaS service side after combining the output data returned by the multiple data source services;
[0018] The database query unit is used to query the database for the output data combination result corresponding to the target product identifier after receiving an online query request initiated to the service interface of the SaaS service with the target product identifier as a parameter, and use it as an input parameter of the algorithm processing logic in the service interface to obtain the algorithm processing result and return it to the querying party.
[0019] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of any of the methods described above.
[0020] An electronic device, comprising:
[0021] one or more processors; and
[0022] A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of any of the methods described above.
[0023] According to the specific embodiments provided in this application, this application discloses the following technical effects:
[0024] Through the embodiment of the present application, the dependency relationship between multiple data source services associated with the tenant can be determined first, and then the input parameter acquisition logic in the service interface of the SaaS service can be called offline by executing an offline task, with the commodity identifier associated with the tenant as a parameter, and the multiple data source services can be remotely called in the offline task. After that, the output data of the multiple data source services can be combined, and the corresponding relationship between the commodity identifier and the output data combination result can be saved in the database on the SaaS service side. In this way, after receiving an online query request initiated to the service interface of the SaaS service with the target commodity identifier as a parameter, the output data combination result corresponding to the target commodity identifier can be directly queried from the database on the SaaS service side, and it can be used as an input parameter of the algorithm processing logic in the service interface to obtain the algorithm processing result and return it to the query party. In this way, since the process of initiating a remote call to the data source service on the tenant side has been completed in advance in the offline task, the delay corresponding to the database query operation is much lower than that of initiating a remote call to the data source service on the tenant side, so the overall efficiency of SaaS in the actual online query service process can be improved.
[0025] In an optional implementation, in order to support parallel processing between different data source services during the execution of offline tasks, a topological structure based on a directed graph is used to express the dependency relationship between data source services and perform node process orchestration, thereby improving the execution efficiency of offline tasks.
[0026] Of course, any product implementing the present application does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0028] Figure 1 It is a schematic diagram of the data integration service process in the prior art;
[0029] Figure 2 is a schematic diagram of a data integration service system provided by an embodiment of the present application;
[0030] Figure 3 It is a schematic diagram of the dependencies between services;
[0031] Figure 4 is a flow chart of the method provided in the embodiment of the present application;
[0032] Figure 5 is a schematic diagram of a configuration interface provided in an embodiment of the present application;
[0033] Figure 6 is a schematic diagram of a device provided in an embodiment of the present application;
[0034] Figure 7 It is a schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0035] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.
[0036] To facilitate understanding of the specific implementation scheme provided in the embodiments of the present application, it is first necessary to explain that in the example of providing a product replenishment strategy for tenants (usually merchants, etc.) through a SaaS service as described in the background technology section, it is assumed that a caller (a tenant, that is, a specific merchant) uses itemID (the ID of a product associated with the merchant) as an input parameter to request the service interface provided by the SaaS service to obtain the replenishment strategy for the product.
[0037] Among them, although the tenant only needs to provide ItemID as an input parameter when initiating a query request, the algorithm processing logic inside the specific interface usually includes very sophisticated logic. For example, for the same product, since it involves replenishing the product in multiple warehouses, and the replenishment strategies of the same product in different warehouses may be different, more data needs to be obtained as input parameters of the algorithm processing logic before executing the algorithm processing logic. When obtaining more parameters, the SaaS service interface usually needs to call the specific data source service. In most cases, this data source service is provided by a specific tenant and deployed in the relevant server on the tenant side. Therefore, when the SaaS service interface calls the data source service, it needs to use RPC to make a remote call.
[0038] In the prior art, the above process of obtaining more data as input parameters of specific algorithm logic is performed in real time after receiving an online query request initiated by a specific query party. The specific process can be as follows: Figure 1 As shown:
[0039] 1. The caller uses the itemID of a product as an input parameter to request the service interface of the SaaS service;
[0040] 2. A series of requests are made within the interface to retrieve more data that can be obtained using ItemId. These include:
[0041] a. Call ServiceA (a data source service, usually provided by a specific tenant, and sometimes multiple tenants share the same data source service) through itemId to obtain a list of storeCode (warehouse codes) (that is, to determine which warehouses the current product needs to be stored in). b. Call ServiceB through storeCode to obtain StoreInfo (warehouse information). c. Call ServiceC through storeCode to obtain StoreType (warehouse type);
[0042] 3. Merge the output data of the above data source services and provide the merged result as input parameter to the subsequent algorithm processing logic;
[0043] 4. Return the output results of the algorithm processing logic to the caller.
[0044] That is to say, after a specific service interface is called, it will include at least two processing stages. The first processing stage is to obtain more data based on the parameters input by the caller to obtain parameters; in the second stage, the more data obtained in the first stage is combined into input parameters of the specific algorithm processing logic to perform more refined processing and obtain the corresponding processing results. The problems with this processing method are:
[0045] 1. After the caller requests a specific service interface, it needs to call ServiceA, B, and C in real time. Since these services may be deployed on the tenant side, RPC access may be involved. This makes the overall delay time of the SaaS service interface depend on the delay of ServiceA, B, and C. In particular, when specific SaaS service interface calls are concentrated at certain specific times, this delay will be more obvious. In addition, in the process of calling these data source services, a synchronous blocking call method is usually used, which is not conducive to the overall IO (input / output)
[0046] In fact, computing resources are not fully utilized;
[0047] 2. Since multiple tenants may call the SaaS service interface at certain times, the QPS of the SaaS service interface may be very high. This will also make the QPS of Service A, B, and C very high. However, Service A, B, and C may not be able to withstand such a level of QPS.
[0048] In view of the above situation, the inventor of the present application found in the process of implementing the present application that: for SaaS services related to replenishment strategy formulation, the data source services (that is, the aforementioned Service A, B, C, etc.) accessed by specific tenants are generally related to master data, and the frequency of change is relatively low. Therefore, in most cases, offline data can be used directly. For example, the output data of a data source service is the warehouse code associated with a certain commodity, that is, the returned information is used to indicate in which warehouses a specific commodity is stored. However, since merchants specifically store goods in which warehouses, this situation is usually not easy to change, and there is actually no need to obtain the data through real-time query. On the other hand, the itemIDs of specific tenants and specific commodities associated with tenants can be enumerated. Therefore, in the embodiment of the present application, an offline request method can be used to call a specific SaaS service interface to obtain the output data returned by each remote data source service for each itemID, and the combined data is saved in the database on the SaaS service side. In this way, when a query request is received from a specific caller, the data corresponding to the specific product can be directly returned from the database on the SaaS service side, and then directly combined as input parameters for the subsequent algorithm processing logic, without the need to initiate a real-time remote call from the database service deployed on the tenant side.
[0049] In other words, if a tenant is associated with 1,000 products, the itemIDs of these products can be used to make offline calls to the service interface of a specific SaaS service. During the offline call, the interface will initiate calls to the data source services associated with the tenant during the parameter preparation phase. Correspondingly, the data source service returns output data. The output data returned by each data source service for the same itemID can be combined and saved in the database. In this way, a copy of the data can be saved in the database for each specific itemID. Later, when the tenant actually needs to call the above SaaS service interface for the product, Figure 2 As shown in the figure, the itemID of the product can also be used as an input parameter. However, after the interface receives the call request, it no longer needs to call the various data source services associated with the tenant in real time. Instead, it directly reads the data corresponding to the itemID from the database, and then uses the data as the input parameter of the subsequent algorithm processing logic. In this way, since the delay corresponding to the database query operation is much lower than that of the real-time request for a specific data source service, the overall efficiency of the SaaS service can be improved.
[0050] In order to achieve the above purpose, it is necessary to use a data middle layer to run offline tasks regularly (for example, once or several times a day) so that all data that can be associated with the itemId under the specific tenant (the data provided by the specific data source service for the itemId) are saved in the database in advance. However, there is no such data middle layer in the prior art, so the embodiment of the present application needs to build such an offline task. After the offline task is built, the input parameter acquisition logic in the service interface of the SaaS service can be called offline by executing this offline task regularly, with the commodity identifier associated with the tenant as a parameter, so as to remotely call the multiple data source services in the offline task. Then, the output data returned by the multiple data source services can be combined, and the correspondence between the commodity identifier and the output data combination result can be saved in the database on the SaaS service side.
[0051] In addition, when building offline tasks, another issue that needs to be considered is that different tenants are associated with different data source services, and there are different dependencies between the data source services. Moreover, such dependencies may be diverse and complex, so a data structure is needed to store such dependencies. In addition, since it is necessary to use a large number of items under multiple tenants to call specific SaaS service interfaces in a relatively concentrated period of time, it is necessary to support concurrent and non-blocking ID calls when calling the data source services under specific tenants, rather than serial blocking calls like real-time calls. Otherwise, offline tasks may take a long time to complete, and machine resources are not fully utilized.
[0052] For example, the four data source services associated with a tenant have dependencies such as Figure 3 As shown, since C has multiple dependencies, namely A and D, it is impossible to use a tree structure to express this dependency relationship. In addition, since A and D do not depend on any other data source services, ideally, A and D can both be used as the starting point of the offline task and executed in parallel; B only depends on A, so B can be executed after A is executed; and C depends on A and D, so it is necessary to wait until A and D are executed before C is executed. In other words, in the process of calling various data source services, some can be executed in parallel, while others need to be executed serially. Therefore, it is also necessary to create a specific node process orchestration plan for a specific tenant so that the various databases associated with the tenant can be called according to the plan, so that in the specific execution of offline tasks, the calling process can be executed in the above ideal way.
[0053] Taking the above into consideration, in the preferred embodiment of the present application, a directed graph (topological structure) can be used as a data structure for storing the dependency relationship between data source services, and combined with a tool that supports the operation of topological tasks, a specific offline task can be created. For example, Disruptor (a high-performance task distribution framework, an open source concurrent framework) can be used as the tool that supports the operation of topological tasks, and the after operation of the disruptor is used to build the topology, so as to meet the requirements of the order of IO requests and parallel requests during the execution of offline tasks. Experiments have shown that, under the same computing resources, if the serial execution method in the prior art is used directly, an offline task volume of about 230w will take about one hour and thirty to forty minutes. The topological structure-based solution provided by the embodiment of the present application only takes less than thirty to forty minutes. At the same time, the processing time burden of downstream that strongly depends on the results of such offline tasks is reduced (for example, most offline tasks can be completed in the early morning for algorithm analysis).
[0054] Among them, since the construction of directed graphs between different data source services is different and cumbersome for different tenants, this step can be simplified in a way. Specifically, according to the knowledge of graph theory in traditional discrete mathematics, each node only needs to know what node it must pass through in the previous step to know the dependency relationship between the nodes. Therefore, the data source service can be accessed and an additional upstream data source service information can be added during the configuration of the data source service. Under the preferred implementation, a data source service configuration interface can be provided to users such as industry developers on the tenant side, through which the upstream data source service information can be configured for the specific accessed data source service. In this way, after completing the configuration of each data source service, a directed graph (topological structure diagram) of the data source service can be generated for each tenant, including the data source services that each data source service depends on (of course, in the specific implementation, problems such as loops can also be verified).
[0055] In summary, in the above preferred embodiment, the specific processing flow may be that, in order to execute offline tasks, the dependency relationship between the data source services associated with each tenant is first obtained, and then, based on this dependency relationship, a specific node process orchestration scheme is created for the specific tenant. In this orchestration scheme, topological sorting can be performed according to the directed graph structure, so that the specific node process orchestration scheme can simultaneously support the parallel and serial execution of different data source services. When it is necessary to execute a specific offline task, the above node process orchestration scheme can be combined with a specific disruptor and other frameworks to generate a running topological task.
[0056] By executing the above offline tasks, more data related to multiple products of a specific tenant can be saved in the database. Therefore, when an online query request is received, it only needs to be read from the database to obtain more data about the specific product in addition to the itemID. These data can be combined into input parameters for subsequent calculation and processing logic.
[0057] Of course, in actual applications, there may be some data that changes frequently, or even data that needs to be obtained in real time. Among them, the former can be called "quasi-real-time data" and the latter is called "real-time data". For quasi-real-time data, in an embodiment of the present application, the tenant can provide the itemID of the specific data that has changed to the SaaS service side, and then re-execute the offline task for this itemID to update the data stored in the database. For items that need to apply for real-time data, the specific tenant can also add a real-time request tag when initiating a query request, so that the specific service interface can perform special processing after receiving the query request, that is, for the itemID, the parameters are no longer obtained by querying the database, but are obtained by real-time requesting the specific data source service.
[0058] The specific implementation scheme provided in the embodiments of the present application is described in detail below.
[0059] Specifically, the present application embodiment first provides a data integration service processing method, see Figure 4 , the method may specifically include:
[0060] S401: Determine the dependency relationship between multiple data source services associated with a tenant of Software as a Service (SaaS).
[0061] In specific implementation, the dependency relationship between multiple data source services associated with a specific tenant can be obtained in a variety of ways. For example, in one way, as described above, since the data source service provided by the tenant needs to be connected to the SaaS service, and the SaaS service can usually provide users such as industry developers on the tenant side with an interface for configuring the specific accessed data source service, in the embodiment of the present application, Figure 5 As shown at 51 in the figure, an operation option for specifying an upstream data source service for the current data source service can also be provided in the configuration interface, so that the tenant side can directly configure the upstream data source service of the specific data source service through the configuration interface, and the upstream data source service corresponding to the same data source service can be one or more. In this way, for industry developers on the tenant side, when using a specific SaaS service, they only need to be concerned about introducing a data source service that meets the specifications and adding simple configurations during the data integration stage.
[0062] For SaaS services, after receiving specific configuration information, the dependency relationship between each data source service can be determined based on the upstream data source service configuration information corresponding to each data source service associated with the same tenant. For example, assuming that a tenant has configured four data source services, A, B, C, and D, among which A and D have no upstream data source services, B's upstream data source service is A, and C's upstream data source services are A and D. Then it can be determined that A and D do not depend on other data source services, B depends on data source service A, and C needs to depend on two data source services, namely A and D, and so on.
[0063] Of course, as mentioned above, in order to improve the execution efficiency of offline tasks and make full use of service resources, after determining the dependencies between the various data source services associated with the same tenant, a directed graph structure between multiple data source services can be created to save and record such dependencies. In addition, a node process orchestration scheme that supports parallel and serial execution can be created for the tenant based on the topological sorting of the directed graph. In other words, since the dependencies between data source services are saved through a data structure such as a directed graph, a topological structure is formed between different data source services, and then a specific node process orchestration scheme can be generated according to the topological sorting between different data source services. In addition, since the orchestration is performed according to the topological structure between the data source services, multiple situations such as parallel execution and serial execution in sequence can exist between different nodes in the same process, so that parallel operation is supported during the execution of offline tasks to improve efficiency.
[0064] Among them, since the specific service interface involves calling multiple data source services in the process of parameter acquisition to form a calling process, the nodes described here specifically correspond to the data source services, that is, in a process, a data source service can correspond to a processing node. In this process, whether different nodes are called in parallel or in sequence can be executed according to the aforementioned node process arrangement scheme provided in the embodiment of this application.
[0065] It should be noted here that in the specific implementation, there may be multiple tenants. In the specific implementation, a corresponding process node orchestration plan can be established for each tenant. Similarly, multiple IO requests such as parallel and serial can be supported in the same plan.
[0066] S402: By executing an offline task, the input parameter acquisition logic in the service interface of the SaaS service is called offline with the commodity identifier associated with the tenant as a parameter, so as to remotely call the multiple data source services in the offline task.
[0067] After creating a process node orchestration plan for a specific tenant, you can execute offline tasks according to the specific orchestration plan. In specific implementation, offline tasks can be executed at a preset time, for example, twice a day, etc.
[0068] Specifically, when executing an offline task, the itemIDs of multiple commodities of a specific tenant can be used as initial input parameters to initiate a call to the SaaS service interface (the call is not initiated by the real tenant, etc., but by the offline task). Of course, since the offline task in the embodiment of the present application only needs to obtain the data provided by the multiple data source services of the specific tenant for the specific commodity to be used as the input parameter of the subsequent algorithm processing logic, when executing an offline task, only the input parameter acquisition logic in the specific service interface can be called without calling the specific algorithm processing logic.
[0069] When calling the specific input parameter acquisition logic, if the process nodes are pre-orchestrated according to the directed graph structure between the data source services, the aforementioned process node orchestration scheme can be combined with a tool that supports the running of topological tasks (such as Disruptor) to generate a running topological task, thereby calling the multiple data source services according to the node process orchestration scheme.
[0070] Specifically, there are multiple ways to combine the process node orchestration scheme with the tool that supports the running of topological tasks. For example, in one way, for each request corresponding to itemID, multiple rounds of loops can be performed after determining the specific tenant information and the corresponding node process orchestration scheme. In the first round of loops, the data source service that does not depend on other data source services in the process node orchestration scheme can be first determined as the starting point of the process, and this data source service is called and processed. After completing the first round of loops, the starting point data source service is added to the completed list. Then the second round of loops begins. At this time, for the data source services that have not been processed, it is determined whether the data source services they depend on are all in the completed list. If so, the data source service can be called for processing in this round of loops, otherwise wait for the next round, and so on, until all data source services in the process are processed.
[0071] S403: After combining the output data returned by the multiple data source services, the corresponding relationship between the product identification and the output data combination result is saved in the database of the SaaS service side.
[0072] After all data source services have been processed, the output data of each data source service has been known. Therefore, the output data generated by each data source service for the current itemID can be obtained, and then these output data can be combined and saved in the database on the SaaS service side. Of course, when saving, the correspondence between the specific itemID and the output data combination result can be saved. For example, in one way, it can be saved in the form of a key-value pair, and so on. In addition, for other items of the same tenant, as well as items of other tenants, the output data generated by multiple data source services can be obtained in the same way through offline tasks, and then combined and saved in the database.
[0073] In this way, multiple data entries can be saved in the database, and each data entry can correspond to a correspondence between an itemID and the combined result of the output data generated for it by multiple data source services (of course, in actual applications, in the specific correspondence, the key name part can include other information in addition to the itemID, which will be introduced later).
[0074] S404: After receiving an online query request initiated to the service interface of the SaaS service with the target product identifier as a parameter, the output data combination result corresponding to the target product identifier is queried from the database, and is used as an input parameter of the algorithm processing logic in the service interface to obtain the algorithm processing result and return it to the querying party.
[0075] Since the specific item is traversed through the offline task, and the data that can be passed by each data source service for the specific item is obtained in advance and saved in the database, after actually receiving the online query request initiated to the service interface with the target product identifier as a parameter (this request can be a real request initiated by an application on the specific tenant side, etc.), the output data combination result corresponding to the target product identifier can be queried from the database and used as the input parameter of the algorithm processing logic in the service interface, without the need to initiate a real-time call request to the specific data source service.
[0076] Of course, in actual applications, since the same tenant may add or remove some products every day, in the process of obtaining the above data through offline tasks, there may be incomplete enumeration, that is, the product actually requested by the tenant may not have been traversed when the offline task was previously executed. At this time, the data corresponding to the product may not exist in the database. In this case, when it is found that there is no data corresponding to the currently requested itemID in the database, a real-time call can be initiated to the specific data source service for the itemID. Of course, since the proportion of this situation in actual applications is not high, it will not have too much impact on the overall efficiency.
[0077] In addition, as mentioned above, although in most cases, the output data provided by each data source service for a specific product is stable, there may still be some special cases, so that the output data corresponding to some products may change temporarily. For example, a tenant previously distributed a certain product in warehouses S1 and S2. As a data source service B that provides warehouse code query services, after receiving a query request with the itemID of the product as a parameter, the returned result is the two storecodes S1 and S2 mentioned above. During the execution of offline tasks, the correspondence between itemID and S1 and S2 will be saved in the database. If the data does not change, before executing the next offline task, if the tenant initiates an online query request for the itemID, the corresponding storecode including S1, S2, etc. can be directly obtained from the database. However, suppose that before the next offline task starts, the tenant adds a new warehouse S3 for the product. At this time, if the data saved in the database is used directly, an error will occur. Therefore, in the embodiment of the present application, for this situation, the specific tenant can be required to promptly notify the offline task on the SaaS service side of the itemID whose data has changed (for example, the notification can be made by calling an interface or a message queue, etc.). After receiving the notification, the offline task can re-execute the offline task for the specified product, obtain the new output data corresponding to each data source service, and update the output data combination result corresponding to the specified product stored in the database. In this way, the correctness of the data read from the database can be maintained.
[0078] Furthermore, in actual applications, some products may need to obtain specific parameters by calling the data source service in real time. In this regard, in the embodiment of the present application, if the tenant has such a requirement, relevant tags can be added to the request when initiating a specific query request. In this way, after receiving the actual query request, if the specific SaaS service interface finds that the request contains a real-time query tag, it can obtain more parameters by initiating a real-time query to the specific data source service.
[0079] It should be noted here that in the process of implementing the solution provided by the embodiment of the present application, in addition to considering the overall performance of the service interface, it is also possible to consider whether the specific data source service side can tolerate the possible QPS. That is to say, in the process of executing offline tasks, it involves traversing multiple tenants and multiple itemIDs, and when executing a specific process node orchestration solution, there are also concurrency and other situations, so a very high QPS may be generated. However, the number of tasks that a specific data source service can process per second may be limited. Therefore, in the process of executing offline tasks, the specific data source service can also be current limited. In the specific implementation, a token bucket algorithm (which can also be implemented by other algorithms) can be used to limit the current for each connected data source service.
[0080] In addition, in actual applications, the following situations may also exist: among multiple data source services associated with the same tenant, there may be a data source service that splits one output data into multiple data for output. For example, the function of a data source service (serviceA) is to query the StoreCode of the warehouse associated with a specific product based on the input itemID. If a product is distributed in multiple warehouses, after the itemID of the product is input into the serviceA, the output data will include multiple StoreCodes. After obtaining multiple StoreCodes, they need to be processed through some other non-batch nodes (serviceB). For example, there will be problems with the abstraction of nodes at this time, because ServiceB does not support batch query (it cannot query the information of multiple warehouses at a time), so it can only be processed in a loop, but ServiceA cannot initiate such a loop.
[0081] In view of the above situation, the embodiment of the present application also provides a corresponding solution. Specifically, the concept of database service can be abstracted one more level. First, the data is standardized. For data, the observation perspective is the dimension of the data. For example, observing the data from the perspective of ItemId, we get data of one dimension; observing the data from the perspective of StoreCode, we get data of another dimension. This kind of data can be uniformly referred to as one-dimensional data. However, at this time, if the data of ItemId+Store is needed, then the observation dimension of the data becomes two, which can be called two-dimensional data. In addition, the data source service can be classified into "dimensional data source service" and "dimensional auxiliary data source service", among which the dimensional data source service is the data source service that converts one-dimensional data into two-dimensional data (that is, the input data and the output data belong to different dimensions, and the output data is multiple values, which can be understood as changing from a single perspective to two perspectives, such as ServiceA in the above example). The dimension-assisted data source service keeps the original dimension unchanged, but adds some observation information of the same dimension, which can be understood as a wider perspective of viewing data, such as the data source service that obtains ItemInfo through ItemId, and so on.
[0082] Under this premise, the implementation of the node is changed into two types, and different orchestration processing is performed on these two types of nodes to achieve the goal. In specific implementation, in order to distinguish between dimensional data source services and dimensional auxiliary data source services, the dimensional information to which the input data and output data corresponding to the multiple data source services respectively belong can also be determined (in one implementation method, corresponding operation options can be provided in the interface for database access configuration to configure the data dimension information). Then, when calling the multiple data source services according to the node process orchestration scheme, after an upstream node completes the call processing and generates output data, judgment logic can be added. Specifically, if the input data and output data of the upstream node belong to different dimensions, and the output data is multiple, multi-dimensional tag information can be added to the output data of the upstream node. In this way, when calling the downstream node, judgment logic and loop call logic can be added. The judgment logic can be used to judge whether the output data of the upstream node contains multi-dimensional tag information; the loop call logic can be used to: if it is judged that the output data of the upstream node contains multi-dimensional tag information, the output data of the upstream node can be split into multiple pieces, and after being combined with the input data information of the upstream node, multiple input data of the downstream node are generated, and a loop call is initiated to the downstream node, so that the downstream node generates output data for each input data. In this way, the data source service that does not support batch query is transformed into a data source service that supports batch query.
[0083] For example, assuming that the upstream node is ServiceA in the above example, according to its configuration information, the input data of the node is the itemID dimension, and the output data is the StoreCode dimension, which are different dimensions. After processing a specific itemID1, the output StoreCode is multiple, for example, StoreCode1, 2, and 3, then the upstream node belongs to the dimensional data source service, so a multi-dimensional tag can be added to the output data. Afterwards, when calling the downstream node ServiceB, since the output data of the upstream node ServiceA is added with a multi-dimensional tag, a specific loop call logic can be triggered. Specifically, the output data of ServiceA can be split into multiple pieces and combined with the input data of ServiceA to obtain the input data of ServiceB. For example, the multiple pieces of data split out are itemID1+StoreCode1, itemID1+StoreCode2, and itemID1+StoreCode3. The above three pieces of data can be input to ServiceB in three cycles, and ServiceB gives the corresponding output data. After ServiceB gives the output data, it is possible to continue to judge whether the input data and output data of ServiceB are of the same dimension. If they are different, the output data of ServiceB can be higher-dimensional data, and can also be split before being input into the downstream node for processing, and so on.
[0084] It should be noted here that the judgment logic, loop processing logic, etc. provided in the upstream and downstream nodes can all be provided by the specific SaaS service side, and the specific tenant side does not need to be aware of the existence of these logics. In addition, in an optional implementation method, after the above-mentioned dimension splitting, when saving the specific output data combination results in the database, in addition to reflecting the information on the itemID dimension in the key name, it can also reflect the information on other dimensions such as StoreCode. In this way, in actual online queries, in addition to supporting queries with itemID as the initial input parameter, it can also support queries with StoreCode as the initial input parameter, and so on.
[0085] In summary, through the embodiment of the present application, the dependency relationship between multiple data source services associated with the tenant can be determined first, and then the input parameter acquisition logic in the service interface of the SaaS service can be called offline by executing an offline task, with the commodity identifier associated with the tenant as a parameter, and the multiple data source services can be remotely called in the offline task. After that, the output data of the multiple data source services can be combined, and the corresponding relationship between the commodity identifier and the output data combination result can be saved in the database on the SaaS service side. In this way, after receiving an online query request initiated to the service interface of the SaaS service with the target commodity identifier as a parameter, the output data combination result corresponding to the target commodity identifier can be directly queried from the database on the SaaS service side, and it can be used as an input parameter of the algorithm processing logic in the service interface to obtain the algorithm processing result and return it to the query party. In this way, since the process of initiating a remote call to the data source service on the tenant side has been completed in advance in the offline task, the delay corresponding to the database query operation is much lower than that of initiating a remote call to the data source service on the tenant side, so the overall efficiency of SaaS in the actual online query service process can be improved.
[0086] In an optional implementation, in order to support parallel processing between different data source services during the execution of offline tasks, a topological structure based on a directed graph is used to express the dependency relationship between data source services and perform node process orchestration, thereby improving the execution efficiency of offline tasks.
[0087] It should be noted that the embodiments of the present application may involve the use of user data. In actual applications, user-specific personal data can be used in the scheme described herein within the scope permitted by applicable laws and regulations, subject to the requirements of applicable laws and regulations of the country where the user is located (for example, with the user's explicit consent, effective notification to the user, etc.).
[0088] Corresponding to the above method embodiment, the present application embodiment also provides a data integration service processing device, see Figure 6 , the device may include:
[0089] A dependency acquisition unit 601 is used to acquire the dependency between multiple data source services associated with a tenant of Software as a Service (SaaS); the multiple data source services include a data source service deployed on the tenant side;
[0090] The offline task execution unit 602 is used to perform an offline call to the input parameter acquisition logic in the service interface of the SaaS service by executing an offline task, taking the commodity identifier associated with the tenant as a parameter, so as to remotely call the multiple data source services in the offline task;
[0091] A storage unit 603 is used to store the corresponding relationship between the commodity identifier and the output data combination result in the database of the SaaS service side after combining the output data returned by the multiple data source services;
[0092] The database query unit 604 is used to query from the database and obtain the output data combination result corresponding to the target product identifier after receiving an online query request initiated to the service interface of the SaaS service with the target product identifier as a parameter, and use it as an input parameter of the algorithm processing logic in the service interface to obtain the algorithm processing result and return it to the querying party.
[0093] The dependency acquisition unit can be specifically used for:
[0094] Providing a data source service configuration interface for inputting configuration information for each data source service, wherein the configuration interface includes an operation option for specifying an upstream data source service for the current data source service, and the upstream data source service may be one or more;
[0095] Determine the dependency relationship between multiple data source services associated with the same tenant based on the received configuration information.
[0096] In a specific implementation, the device may further include:
[0097] A process orchestration scheme generating unit, configured to create a directed graph structure between multiple data source services according to the dependency relationship, and to create a node process orchestration scheme supporting parallel and serial execution for the tenant according to the topological sorting of the directed graph;
[0098] At this time, the offline task execution unit can be specifically used for:
[0099] The node process orchestration scheme is combined with a task distribution framework that supports the running of topological tasks to generate running topological tasks, and remote calls are made to the multiple data source services according to the node process orchestration scheme.
[0100] More specifically, the offline task execution unit may be used to:
[0101] Multiple loops are initiated based on the input product ID. In each loop, data source services that do not depend on other data source services and / or data source services that have completed processing on the dependent data source services are called and processed, and added to the completed collection after processing, until all calls to multiple data source services in the same process are completed.
[0102] In addition, the device may also include:
[0103] The first real-time calling unit is used to obtain output data of the data source service by real-time calling the data source service if the data corresponding to the target product identifier does not exist in the server, and combine the data into input parameters of the algorithm processing logic in the service interface.
[0104] Furthermore, the device may further include:
[0105] A notification information receiving unit, configured to receive notification information of changes in the output data corresponding to the designated product submitted by the tenant after the output data combination result is saved in the database;
[0106] The updating unit is used to obtain new output data corresponding to each data source service by re-executing the offline task for the designated commodity, and update the output data combination result corresponding to the designated commodity stored in the database.
[0107] Furthermore, the device may further include:
[0108] The second real-time calling unit is used to obtain the output data of the data source service by calling the data source service in real time if the online query request carries a real-time query flag, and combine the output data into input parameters of the algorithm processing logic in the service interface.
[0109] In addition, the device may also include:
[0110] The current limiting processing unit is used to limit the current of each data source service according to the token bucket algorithm during the offline call process.
[0111] In addition, the device may also include:
[0112] A dimension information determination unit, used to determine the dimension information to which the input data and output data respectively corresponding to the multiple data source services belong;
[0113] The offline task execution unit can be specifically used for:
[0114] If the input data and output data of the upstream node belong to different dimensions, and there are multiple output data, then add multi-dimensional tag information to the output data of the upstream node;
[0115] When calling a downstream node, the output data of the upstream node is split into multiple pieces according to the multi-dimensional labeling information, and are respectively combined with the input data information of the upstream node to generate multiple input data of the downstream node, and a loop call is initiated to the downstream node so that the downstream node generates output data for each input data.
[0116] In addition, an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the steps of any one of the methods in the aforementioned method embodiments are implemented.
[0117] And an electronic device, comprising:
[0118] one or more processors; and
[0119] A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of the method described in any one of the aforementioned method embodiments.
[0120] in, Figure 7 The architecture of the electronic device is shown as an example, which may include a processor 710, a video display adapter 711, a disk drive 712, an input / output interface 713, a network interface 714, and a memory 720. The processor 710, the video display adapter 711, the disk drive 712, the input / output interface 713, the network interface 714, and the memory 720 may be communicatively connected via a communication bus 730.
[0121] Among them, the processor 710 can be implemented by a general-purpose CPU (Central Processing Unit, processor), a microprocessor, an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, etc., to execute relevant programs to implement the technical solution provided in this application.
[0122] The memory 720 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 720 can store an operating system 721 for controlling the operation of the electronic device 700, and a basic input and output system (BIOS) for controlling the low-level operation of the electronic device 700. In addition, a web browser 723, a data storage management system 724, and a data integration service processing system 725, etc. can also be stored. The above-mentioned data integration service processing system 725 can be an application program that specifically implements the operations of the aforementioned steps in the embodiment of the present application. In short, when the technical solution provided by the present application is implemented by software or firmware, the relevant program code is stored in the memory 720 and is called and executed by the processor 710.
[0123] The input / output interface 713 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.
[0124] The network interface 714 is used to connect to a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).
[0125] The bus 730 comprises a pathway for transmitting information between the various components of the device (eg, the processor 710, the video display adapter 711, the disk drive 712, the input / output interface 713, the network interface 714, and the memory 720).
[0126] It should be noted that, although the above device only shows a processor 710, a video display adapter 711, a disk drive 712, an input / output interface 713, a network interface 714, a memory 720, a bus 730, etc., in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include components necessary for implementing the solution of the present application, and does not necessarily include all the components shown in the figure.
[0127] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly contributed to the prior art in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.
[0128] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.
[0129] The data integration service processing method, device and electronic device provided by the present application are introduced in detail above. The principle and implementation method of the present application are explained by using specific examples in this article. The description of the above embodiments is only used to help understand the method and core idea of the present application. At the same time, for those skilled in the art, according to the idea of the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as limiting the present application.
Claims
1. A data integration service processing method, characterized in that: include: Obtain the dependencies between multiple data source services associated with tenants of Software as a Service (SaaS); The multiple data source services include a data source service deployed on the tenant side; By executing an offline task, the input parameter acquisition logic in the service interface of the SaaS service is called offline with the commodity identifier associated with the tenant as a parameter, so as to remotely call the multiple data source services in the offline task; After combining the output data returned by the multiple data source services, the corresponding relationship between the commodity identifier and the output data combination result is saved in the database of the SaaS service side; After receiving an online query request initiated to the service interface of the SaaS service with the target product identifier as a parameter, the output data combination result corresponding to the target product identifier is queried from the database and used as an input parameter of the algorithm processing logic in the service interface to obtain the algorithm processing result and return it to the querying party.
2. The method according to claim 1, characterized in that The step of obtaining the dependency relationship between multiple data source services associated with a tenant of Software as a Service (SaaS) includes: Providing a data source service configuration interface for inputting configuration information for each data source service, wherein the configuration interface includes an operation option for specifying an upstream data source service for the current data source service, and the upstream data source service may be one or more; Determine the dependency relationship between multiple data source services associated with the same tenant based on the received configuration information.
3. The method according to claim 1, characterized in that Also includes: According to the dependency relationship, a directed graph structure is created between multiple data source services, and according to the topological sorting of the directed graph, a node process orchestration scheme supporting parallel and serial execution is created for the tenant; The remotely calling the multiple data source services in the offline task includes: The node process orchestration scheme is combined with a task distribution framework that supports the running of topological tasks to generate running topological tasks, and remote calls are made to the multiple data source services according to the node process orchestration scheme.
4. The method according to claim 3, characterized in that The step of combining the node process orchestration scheme with a task distribution framework that supports topology task execution includes: Multiple loops are initiated based on the input product ID. In each loop, data source services that do not depend on other data source services and / or data source services that have completed processing on the dependent data source services are called and processed, and added to the completed collection after processing, until all calls to multiple data source services in the same process are completed.
5. The method according to claim 1, characterized in that: Also includes: If the data corresponding to the target product identifier does not exist in the server, the output data of the data source service is obtained by calling the data source service in real time, and combined as input parameters of the algorithm processing logic in the service interface.
6. The method according to claim 1, characterized in that Also includes: After the output data combination result is saved in the database, receiving notification information of a change in the output data corresponding to the designated product submitted by the tenant; By re-executing the offline task for the designated commodity, new output data corresponding to each data source service is obtained, and the output data combination result corresponding to the designated commodity stored in the database is updated.
7. The method according to claim 1, characterized in that Also includes: If the online query request carries a real-time query flag, the output data of the data source service is obtained by calling the data source service in real time, and is combined as input parameters of the algorithm processing logic in the service interface.
8. The method according to claim 1, characterized in that Also includes: During the offline calling process, each data source service is limited in flow according to the token bucket algorithm.
9. The method according to claim 1, characterized in that: Also includes: Determine the dimension information to which the input data and output data respectively correspond to the multiple data source services; The remotely calling the multiple data source services in the offline task includes: If the input data and output data of the upstream node belong to different dimensions, and there are multiple output data, then add multi-dimensional tag information to the output data of the upstream node; When calling a downstream node, the output data of the upstream node is split into multiple pieces according to the multi-dimensional labeling information, and are respectively combined with the input data information of the upstream node to generate multiple input data of the downstream node, and a loop call is initiated to the downstream node so that the downstream node generates output data for each input data.
10. A data integration service processing device, characterized in that: include: A dependency acquisition unit, used to acquire the dependency between multiple data source services associated with a tenant of Software as a Service (SaaS); The multiple data source services include a data source service deployed on the tenant side; An offline task execution unit, configured to perform an offline call to the input parameter acquisition logic in the service interface of the SaaS service by executing an offline task, taking the commodity identifier associated with the tenant as a parameter, so as to remotely call the multiple data source services in the offline task; A storage unit, configured to store the corresponding relationship between the commodity identifier and the output data combination result in a database on the SaaS service side after combining the output data returned by the multiple data source services; The database query unit is used to query the database for the output data combination result corresponding to the target product identifier after receiving an online query request initiated to the service interface of the SaaS service with the target product identifier as a parameter, and use it as an input parameter of the algorithm processing logic in the service interface to obtain the algorithm processing result and return it to the querying party.
11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method described in any one of claims 1 to 9 are implemented.
12. An electronic device, characterized in that: include: one or more processors; as well as A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of the method described in any one of claims 1 to 9.
