A large model API aggregation calling method, device and system

Abstract

The embodiments of the present invention provide a large model API aggregation calling method, device and system, which relate to the field of artificial intelligence large model applications. The method includes: obtaining an original request; determining, based on user question information, a target large model product category applicable to the user question information; selecting a target large model channel from at least one large model channel corresponding to the target large model product category according to a preset channel selection rule configured for the target large model product category corresponding to the user client; converting the original request into a target request that complies with a target interface specification, and sending the target request to the large model corresponding to the target large model channel; receiving a first response from the large model corresponding to the target large model channel to the target request, and determining whether the status of the first response is successful; if the status of the first response is successful, obtaining answer information from the first response; encapsulating the answer information into a second response that complies with a preset unified interface specification and returning it to the user client.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence large model applications, and in particular to a large model API aggregation calling method, device and system. Background Art

[0002] With the rapid development of artificial intelligence technology, big model API (Application Programming Interface) has become a key tool for developers to realize intelligent applications. The input and output parameters of different third-party big model APIs are usually not unified, and the access protocols are also inconsistent. Some use the HTTP protocol, and some use the WebSocket protocol. The inventor found that there are interface differences between big model APIs of different manufacturers. When developers want to use interfaces of multiple manufacturers, they need to adapt each API separately when using it, which not only increases development costs, but also reduces development efficiency.

[0003] In the process of implementing the present invention, the applicant discovered that there are at least the following problems in the prior art:

[0004] There are interface differences between the large model APIs of different manufacturers, which requires developers to adapt separately when using interfaces of multiple manufacturers, increasing development costs and reducing development efficiency. Summary of the invention

[0005] The embodiments of the present invention provide a large model API aggregation calling method, device and system to solve the problem that there are interface differences in large model APIs of different manufacturers, which causes developers to need to adapt separately when using interfaces of multiple manufacturers, increasing development costs and reducing development efficiency.

[0006] To achieve the above objectives, in a first aspect, an embodiment of the present invention provides a large model API aggregation calling method, comprising:

[0007] Obtaining an original request; the original request includes: user question information from a user client; the original request complies with a preset unified interface specification;

[0008] According to the user question information, in at least one large model product category, the target large model product category applicable to the user question information is determined; wherein, for each user client, for each large model product category corresponding to the user client, at least one large model channel corresponding to the large model product category is pre-configured; each large model channel configured corresponding to each large model product category corresponds to large models applicable to the large model product category from different suppliers; at least one large model channel corresponding to each large model product category corresponding to each user client is configured with a preset channel selection rule;

[0009] According to the preset channel selection rule configured for the target large model product category corresponding to the user client, a first large model channel is selected as the target large model channel from at least one large model channel corresponding to the target large model product category;

[0010] Convert the original request into a target request that complies with the target interface specification, and send the target request to the large model corresponding to the target large model channel; the target request includes the user question information; the target interface specification is the interface specification of the large model corresponding to the target large model channel;

[0011] receiving a first response from the large model corresponding to the target large model channel to the target request, and determining whether the status of the first response is successful;

[0012] If the status of the first response is success, obtaining answer information corresponding to the user question information from the first response;

[0013] The answer information is encapsulated into a second response that complies with the preset unified interface specification, and the second response is returned to the user client.

[0014] In a second aspect, an embodiment of the present invention provides a large model API aggregation calling device, including:

[0015] The original request acquisition unit is used to acquire the original request; the original request includes: user question information from the user client; the original request complies with the preset unified interface specification;

[0016] A product type determination unit is used to determine, according to the user question information, in at least one large model product category, a target large model product category applicable to the user question information; wherein, for each user client, at least one large model channel corresponding to the large model product category corresponding to the user client is pre-configured; each large model channel configured corresponding to the large model product category corresponds to large models applicable to the large model product category from different suppliers; at least one large model channel corresponding to each large model product category corresponding to each user client is configured with a preset channel selection rule;

[0017] a channel selection unit, configured to select a first large model channel as a target large model channel from at least one large model channel corresponding to the target large model product category according to a preset channel selection rule configured for the target large model product category corresponding to the user client;

[0018] A request routing processing unit, configured to convert the original request into a target request that complies with a target interface specification, and send the target request to the large model corresponding to the target large model channel; the target request includes the user question information; the target interface specification is the interface specification of the large model corresponding to the target large model channel;

[0019] a response receiving unit, configured to receive a first response from the large model corresponding to the target large model channel to the target request, and determine whether the status of the first response is successful;

[0020] an answer information acquisition unit, configured to acquire answer information corresponding to the user question information from the first response if the status of the first response is success;

[0021] The answer information feedback unit is used to encapsulate the answer information into a second response that complies with the preset unified interface specification, and return the second response to the user client.

[0022] In a third aspect, an embodiment of the present invention provides a large model API aggregation calling system, including: a gateway, the large model API aggregation calling device as described above, and large models of multiple suppliers;

[0023] The gateway is connected to the large model API aggregation calling device, and is used to authenticate, limit, fuse and downgrade the original request from the user client, and submit the original request to the large model API aggregation calling device;

[0024] The big model API aggregation calling device is connected to the big models of multiple suppliers.

[0025] The above technical solution has the following beneficial effects: calling large model APIs of different manufacturers through a unified interface simplifies the developer's usage process and improves the stability and reliability of API calls. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0027] Figure 1 It is a flowchart of a large model API aggregation calling method according to one embodiment of the present invention;

[0028] Figure 2 This is an architectural diagram of a large model API aggregation calling device according to one embodiment of the present invention;

[0029] Figure 3 This is an architecture diagram of a large model API aggregation calling system according to one embodiment of the present invention;

[0030] Figure 4 This is another architecture diagram of a large model API aggregation calling system according to one embodiment of the present invention;

[0031] Figure 5 This is another flow chart of a large model API aggregation calling method according to one of the embodiments of the present invention. DETAILED DESCRIPTION

[0032] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0033] First, as Figure 1 As shown, an embodiment of the present invention provides a large model API aggregation calling method, including:

[0034] Step S10, obtaining an original request; the original request includes: user question information from the user client; the original request complies with a preset unified interface specification;

[0035] Step S11, according to the user question information, determine the target big model product category applicable to the user question information in at least one big model product category; wherein, for each user client, at least one big model channel corresponding to the big model product category corresponding to the user client is pre-configured; each big model channel configured corresponding to each big model product category corresponds to big models applicable to the big model product category from different suppliers; at least one big model channel corresponding to each big model product category corresponding to each user client is configured with a preset channel selection rule;

[0036] Step S12, selecting a first large model channel as a target large model channel from at least one large model channel corresponding to the target large model product category according to a preset channel selection rule configured for the target large model product category corresponding to the user client;

[0037] Step S12, converting the original request into a target request that complies with a target interface specification, and sending the target request to the large model corresponding to the target large model channel; the target request includes the user question information; the target interface specification is the interface specification of the large model corresponding to the target large model channel;

[0038] Step S13, receiving a first response from the large model corresponding to the target large model channel to the target request, and determining whether the status of the first response is successful;

[0039] Step S14: if the status of the first response is success, obtaining answer information corresponding to the user question information from the first response;

[0040] Step S15: encapsulate the answer information into a second response that complies with the preset unified interface specification, and return the second response to the user client.

[0041] In some embodiments, an embodiment of the present invention provides a large model API aggregation calling method adopted by a large model API aggregation calling device. The large model includes but is not limited to a large language model, and a large model for editing or generating pictures, videos, audios, etc. The user question information includes but is not limited to: composed of text information, videos, pictures, audios, flowcharts, mind maps, etc., or questions and other information for text information, videos, pictures, audios, etc., and the answer information corresponding to the user question information includes but is not limited to text information, videos, pictures, audios, flowcharts, tables, and / or mind maps, etc. The preset channel selection rule can be a preset priority, or a selection rule constructed by comprehensively considering the optimal cost and / or performance parameters such as request rate and request success rate, etc., and in time, each time the original request is received, the channel selection rule is dynamically determined according to the unit price change of the supplier and the dynamic change of the performance parameters of the supplier's large model, thereby dynamically selecting the target large model channel. The user submits the original request by calling the interface that conforms to the preset unified interface specification through the user client. According to the user question information in the original request, the applicable target big model product is determined; the embodiment of the present invention pre-classifies the big models of all suppliers to obtain at least one big model product category; the big model product category includes but is not limited to: text generates text, text generates pictures, text generates audio, text generates video, and video generates matching audio, etc., where the first type of information and the second type of information can be the same type of information or different types of information; for example, the first type of information includes but is not limited to text, audio, video, table, and / or mind map, etc., and the second type of information includes but is not limited to text, audio, video, table, and / or mind map, etc. The big model channel is defined and configured by supplier, product, model and client key. The big model channel is the minimum granularity of the big model interface. Supplier refers to the big model API manufacturer; product refers to which API of the big model API manufacturer, such as text generates text, text generates pictures; big model refers to a big model API that can be implemented using multiple different underlying models; client key refers to when the big model API aggregation calling device connects to the big model API manufacturer, each manufacturer can provide multiple keys for the big model API aggregation calling device to use. The combination of supplier, product, model and client key forms a big model channel. Each big model channel is the minimum granularity for the aggregation API platform to call multiple big model vendors' APIs. In the big model API aggregation calling device, each user client can configure the preset channel selection rules for each big model product category, or only configure the preset channel selection rules for the big model product category that the user client needs to use.According to the preset channel selection rule configured for the target big model product category corresponding to the user client, a first big model channel is selected as the target big model channel from at least one big model channel corresponding to the target big model product category; the original request is converted into a target request that conforms to the target interface specification, and the target request is sent to the big model corresponding to the target big model channel; the interface specifications of the big models of each supplier are usually different, the input and output parameters of the big model APIs of different third-party suppliers are not unified, and the access protocols are also inconsistent, some use the HTTP protocol, and some use the WebSocket protocol; under the prior art, if developers (users) want to use the APIs of multiple big model platforms (suppliers), they need to access each platform separately, which is costly and has low development efficiency; through the embodiments of the present invention, users can only use interfaces that conform to the preset unified interface specifications to indirectly call the big model APIs of different third-party suppliers, and users do not need to adapt the big model APIs of different third-party suppliers separately, thereby significantly improving the user's development efficiency. After the large model corresponding to the target large model channel processes the target request, it will return the response corresponding to the target request to the large model API aggregation calling device. If the response is successful, the answer information corresponding to the user's question information can be obtained from the response. When the answer information is provided to the user client that issued the original request, the answer information can be converted into an information format that complies with the unified interface specification, so that the answer information can be fed back to the user client according to the unified interface specification. After receiving the original request from the user client, authentication and authorization operations can also be performed to ensure that only legitimate clients can access the API through the OAuth2 authentication process.

[0042] The embodiments of the present invention have the following technical effects: calling the big model APIs of different manufacturers through a unified interface simplifies the developer's usage process and improves the stability and reliability of API calls. At the same time, when a new supplier's big model appears, it is only necessary to add support for the new supplier in the embodiments of the present invention. Since the embodiments of the present invention provide services to the user client through a unified interface specification, the big model of a specific supplier is transparent to the user client, and the user client does not need to add code adaptation for the big model of the newly added supplier, so that the user client can obtain support from the new supplier with minimal overhead.

[0043] Furthermore, the method further comprises:

[0044] If the status of the first response is failure, return to the preset channel selection rule configured according to the target large model product category corresponding to the user client, select the first large model channel as the target large model channel from at least one large model channel corresponding to the target large model product category, and reselect the target large model channel.

[0045] In some embodiments, by providing multiple large model channels, when an API call of the large model corresponding to the current target large model channel fails, it is automatically switched to the large model corresponding to other large model channels for retry, thereby ensuring the continuity and stability of the request. Administrators or users are allowed to set the number of retries, interval time and channel switching strategy according to business needs and system performance. During the retry process, technologies such as caching or context transfer are used to ensure the integrity and consistency of input parameters and avoid data loss or confusion. Cache the request parameters in memory or a distributed cache system (such as Redis), with the key being the user client ID + conversation ID and the value being the request information, so that even after the call fails, the request parameters can be restored from the cache. Use a context transfer mechanism (such as ThreadLocal or transactional messages in a message queue) to maintain the context information of the request to ensure that the same context can be used when retrying. The following is a specific example to illustrate the usage of the cache mechanism. Step 1: Receive the original request submitted by the user client, submit the original request to the message queue, and return the task submission success information to the user client; Step 2: The consumer of the message queue (large model API aggregation calling device) listens to the original request message, executes the selection of the target large model channel, converts the original request into a target request that complies with the interface specification of the large model corresponding to the target large model channel through the proxy class, and calls the large model interface operation corresponding to the target large model channel. If the large model interface is called successfully, the request result is output to the user client through asynchronous callback, or the request result is stored in the cache and database, and the user obtains the request result through polling. If the call to the large model interface fails, or the server crashes during the processing, this message (original request) returns to the queue to be consumed (message queue), and other normal nodes listen to this message and continue to execute the processing operation.

[0046] The embodiments of the present invention have the following technical effects: compared with directly using the big model system of a specific big model supplier, the embodiments of the present invention can obtain the support of multiple big model suppliers. When the big model system of the current big model supplier cannot provide services, the embodiments of the present invention can automatically switch to the systems of other available big model suppliers. This process is transparent to the user client and can provide the user client with continuous and stable big model services. The user client does not need to adapt to the big model interfaces of multiple suppliers separately, thereby reducing the development cost of the user client, improving development efficiency, and obtaining continuous and stable big model services.

[0047] Further, according to the preset channel selection rule configured for the target large model product category corresponding to the user client, selecting a first large model channel as the target large model channel from at least one large model channel corresponding to the target large model product category includes:

[0048] Acquire client information corresponding to the user client, and determine the balance corresponding to the user client according to the client information;

[0049] According to the order of preset channel priorities configured for the target large model product category corresponding to the user client, the usage fees of the available large model channels and the balance corresponding to the user client are compared in sequence; wherein the available large model channel is the remaining large model channels after excluding all target large model channels whose first response status is failed after obtaining the original request from at least one large model channel corresponding to the target large model product category;

[0050] The first available large model channel whose usage fee is less than or equal to the corresponding balance of the user client is used as the target large model channel;

[0051] The usage fee of the large model channel is calculated based on the user's question information and the preset unit value of the large model channel;

[0052] The preset channel priorities include: global default channel sorting, user-specified default channel sorting, or channel sorting based on scores determined by preset multiple performance parameters and corresponding weights; the performance parameters are used to score the performance of large models of each supplier.

[0053] The performance parameters include: response time, call success rate, load capacity, cost-effectiveness, service quality, historical performance, supplier reputation and / or user-defined indicators.

[0054] In some embodiments, the channel priority can be a predefined global default channel sorting (priority) that is the same for all user clients, or a user-defined default channel sorting (priority) defined by each user, or the channel priority can be determined by sorting the channels according to performance parameters. When an error occurs in the current channel call, it automatically switches to the next available channel to ensure uninterrupted service. When sorting the channels according to performance parameters to determine the channel priority, specifically, it is possible to monitor the performance parameters of each major model channel, calculate the score in real time, and update the channel sorting according to the score, so that the currently optimal channel can be selected to process the user client's request. Sorting channels according to performance parameters The key factors (performance parameters) involved in determining channel priority include: response time, which is used to measure the response speed of API calls, that is, the time required from sending a request to receiving a response. Channels with shorter response times will be given higher priority; success rate, which refers to the proportion of successful API calls. A channel with a high success rate indicates that it has higher stability and reliability, so it will be given priority in the sorting; load capacity, which indicates the number of requests that a channel can handle. A channel that can withstand high concurrent requests will be considered more advantageous in the sorting; cost-effectiveness, considering the relationship between the cost of API calls and their performance, a channel with a high cost-effectiveness ratio may be given higher priority. Quality of service (QoS), including indicators such as error rate, data accuracy, and service consistency. Channels with high service quality will be given priority; user-defined indicators, users may define other performance indicators based on their specific needs, such as support for specific functions; historical performance, evaluating the performance of the channel based on historical data, including past success rate, response time, and failure rate; supplier reputation, the reputation and historical performance of the supplier may also affect the priority of the channel. A channel provided by a reputable supplier may be given priority. The platform designs billing strategies based on the characteristics of different APIs, such as text generation is charged by token, image generation is charged by pixel specifications, etc. The preset unit value of the large model channel includes but is not limited to token billing, billing by quantity, billing by pixel specifications, and free within the preset free use conditions and scope according to specific business scenarios and user groups. Different billing methods are formulated for different large model service types (such as text generation, image generation, music generation, etc.), such as billing by token, billing by quantity, billing by pixel specifications, etc. Different large model suppliers, products and models are set with personalized billing unit prices and preferential strategies to meet diverse business needs. According to specific business scenarios and user groups, the conditions and scope of free use are set to promote business promotion and user experience. After the API call request passes the authentication and authorization, the user's fee balance, token quantity or number of uses are immediately checked to see if they meet the requirements of this call. If not, an error prompt is returned in a timely manner.

[0055] The embodiment of the present invention has the following technical effects: channels are sorted according to preset priorities or performance indicators. When the current channel call fails, the system can automatically switch to the next available channel to ensure the continuity and stability of the service. This dynamic channel management and switching mechanism improves the reliability of API calls.

[0056] Furthermore, the method further comprises:

[0057] If the status of the first response is failure, a failure error code and a failure error message of the first response are obtained, and the failure error code and the failure error message are mapped to an error code and an error message defined by the unified interface specification.

[0058] In some embodiments, different suppliers' big models have different error message definitions. The different error messages are mapped to the unified interface specification of the embodiment of the present invention. The user client can use the unified interface specification without having to understand the error definition of each supplier's big model in detail. It only needs to process it according to the unified interface specification of the embodiment of the present invention.

[0059] Further, obtaining answer information corresponding to the user question information from the first response includes:

[0060] Saving the user question information and the corresponding answer information in a history information list corresponding to the user client;

[0061] Sending the target request to the large model corresponding to the target large model channel includes:

[0062] The user question information and the corresponding answer information in the historical information list corresponding to the user client are provided as reference information of the target request to the big model corresponding to the target big model channel; the reference information is used as a context reference when the big model corresponding to the target big model channel infers the target request.

[0063] In some embodiments, in order to enable the newly selected target large model channel to understand the context of the current user question information when switching to a different large model channel to process user question information, the user question information and corresponding answer information that have been completed before the current user question information are saved and submitted to the newly selected target large model channel together with the current user question information.

[0064] Further, converting the original request into a target request that complies with the target interface specification includes:

[0065] Through a preset strategy factory method, a strategy class having the same supplier as the target large model channel is selected from multiple strategy classes as a target strategy class, and a strategy object corresponding to the target strategy class is created;

[0066] By using the policy object, the original request is subjected to parameter name mapping, parameter data structure conversion, parameter type conversion, and parameter encoding and formatting, so as to obtain a target request that complies with the target interface specification;

[0067] Among them, each strategy class is defined in advance using the strategy pattern method according to the interface specification of the big model of the supplier corresponding to the strategy class; each strategy class encapsulates the API interface of the big model of the supplier corresponding to the strategy class, and implements the adapter pattern in the strategy class to adapt the call of the API interface of the big model of the supplier corresponding to the strategy class to an interface that conforms to the preset unified interface specification.

[0068] In some embodiments, the API (interface) of the big model of different suppliers has the following characteristics: different parameters, different interface input parameters, output parameter names, types, structures; different data formats, different data formats, including json, xml, plain text; different request protocols, different request protocols, including http, websocket. Based on the above characteristics, the API interfaces of different suppliers are integrated into a unified strategy factory using the strategy and proxy mode to realize automatic routing and parameter conversion of requests. The strategy factory includes the strategy mode and factory mode in software design. The strategy mode is a behavioral design mode that allows algorithms or behaviors to be selected at runtime. First, a series of behaviors are defined through the interface, and the implementation class implements these interfaces, and these behaviors can be replaced with each other. It mainly solves the problem of a large number of ifelse in the program and facilitates the expansion of the program. The strategy factory uses the factory method to create management strategy objects. The factory method is a mode of creating objects that uses a common interface to create objects, but allows subclasses to decide which class to instantiate. The strategy factory can dynamically select and instantiate the corresponding strategy object according to the specific parameters or conditions of the request. This allows the system to flexibly handle different requests without modifying the existing code. First, different policy classes are designed according to the policy pattern, and each policy class corresponds to a specific supplier's big model API product. The policy factory is responsible for dynamically creating corresponding policy objects based on channel information. When an API request (original request) arrives, the original request is first parsed to determine the required big model channel based on the user client information of the original request. Based on the parsing results, the policy factory selects and instantiates the appropriate policy object. The policy object receives the request and routes it to the correct API supplier. According to the interface rules of different suppliers, the interface call is executed. When performing parameter conversion, by defining a unified input parameter format, all API requests (original requests) follow the same data structure and type before being sent to different suppliers. Each policy object contains a parameter mapping logic, which maps the unified input parameter format to the parameter format required by the API of the big model of a specific supplier. According to the requirements of the supplier's large model API, it may be necessary to convert the data structure, such as converting from a JSON object to a specific XML structure, and perform necessary type conversion to ensure that the parameter type matches the requirements of the supplier's large model API. For example, some parameters of the large model supplier API interface may have multiple types of parameters, such as a tool_choice parameter, which may be a string type or an object type. This will bring a lot of complexity to program development. The solution is to split it into two fields, tool_choice_type of string type and tool_choice of object type. Encode and format the parameters, such as URL encoding, date and time formatting, etc.

[0069] Another embodiment is used below for illustration: Define a unified interface: Define a unified interface specification, including the data structure, field type and naming specification of the request and response, which serves as the basis for all API conversions. Create a policy class: Create a policy class for each API vendor. These classes implement the unified interface. The interface may include methods / functions for obtaining channels, methods / functions for converting platform input parameters to vendor large model input parameters, methods / functions for converting large model output parameters to platform output parameters, methods / functions for obtaining request URLs, methods / functions for request execution, etc., and contain the logic required to convert the unified interface to a specific vendor API. Mapping rules: In each policy class, define specific mapping rules to map the parameters of the unified interface to the parameters of a specific API. These rules may include mapping of field names, conversion of data types, adjustment of structures, etc. Use adapters: Implement the adapter pattern in the policy class to adapt API calls from different vendors to a unified interface specification. Parameter parsing and construction: Before sending a request, the policy class parses the parameters of the unified interface and constructs the request parameters of a specific API according to the mapping rules. Response conversion: When receiving a response, the policy class converts the response of a specific API back to the response format of the unified interface. Ensure consistency and standardization through the following methods: Standardization, ensure that all vendors' API calls follow a unified interface specification (preset unified interface specification), including data structure, field naming, and type. Unified configuration management: Use a unified configuration management system to maintain mapping rules and parameter conversion logic to ensure that all policy classes use the same configuration. Error handling consistency: Implement a unified error handling mechanism to convert error responses from different vendors into a unified error code and message format. Code reuse: Reduce code duplication and improve code consistency by abstracting and encapsulating common conversion logic. Version control: Perform version control on unified interfaces and policy classes to ensure that all dependent systems and clients can smoothly transition when upgrading or modifying. Documentation and examples: Provide detailed API documentation and usage examples to help developers understand and use the unified interface.

[0070] The embodiments of the present invention have the following technical effects: through efficient algorithms and data structures, the unified input parameters of the platform are accurately converted into the input parameter formats specific to each supplier, and vice versa; at the same time, the consistency and standardization of the output parameters are ensured.

[0071] Furthermore, the method further comprises:

[0072] Receive the user's registration information through a preset registration interface process, and create a user account for the user after the registration information is successfully verified;

[0073] Communicate with the user's existing business system through the preset user management API interface to synchronize user accounts and permissions with the user's existing business system;

[0074] Alternatively, a single sign-on mechanism is established between the user's existing business system so that the user's login is completed through the single sign-on mechanism.

[0075] Furthermore, the method further comprises:

[0076] Through a preset client creation interface, a user's request to create a user client is received, and according to the request to create a user client, a user client is created under the user account of the user; the user account is pre-created for the user; each user account includes at least one user client; each user client includes client information corresponding to the user client; the client information includes: API key, access rights, quota limit and billing policy.

[0077] In some embodiments, users are allowed to create multiple clients based on their needs, and independently configure and assign permissions to them, and each client has an independent API key.

[0078] The embodiments of the present invention have the following technical effects: allowing users to create multiple user clients according to their needs, each user client can have independent configurations, such as API keys, access rights, quota limits, etc., which provides users with flexibility and customized services and realizes personalized configuration. Through the creation and management of user clients, users can set different permissions for different clients to ensure that each user client can only access its authorized resources and functions, thereby enhancing the security of the system. Each user client has an independent API key, which helps to achieve resource isolation and prevent resource conflicts or data leakage between different clients. The creation and management of user clients can be combined with billing strategies to set different billing rules and quotas for each user client, which is convenient for realizing refined billing management.

[0079] Furthermore, the method further comprises:

[0080] After encapsulating the answer information into a second response that complies with the preset unified interface specification and returning the second response to the user client, billing is performed based on the answer information, and deductions are made from the user client, the result of the deduction is notified to the user, and the relevant information of the deduction is stored in a billing details database; the billing details database is used for users to query billing details through a preset billing details query interface.

[0081] In some embodiments, after the API call is successful, the fee is deducted in real time through the billing gateway according to the user's token or number of times information. After the API call request passes the authentication and authorization, the user's fee balance, token quantity or number of times of use are immediately checked to see if they meet the requirements of this call. If not, an error prompt is returned in time to implement fee pre-check. After the call is successful, the Redis+Lua script is used to implement accurate fee deduction, ensure the atomicity and consistency of the fee deduction operation, and prevent over-deduction or under-deduction. In order to achieve atomicity and consistency, by using database transactions, it can be ensured that a series of operations are either all successful or all failed, thereby ensuring the atomicity of the operation. Use the Lua script function of Redis to perform the fee deduction operation. The Lua script can ensure that a series of commands executed inside Redis are atomic, because Redis will execute the Lua script as a separate operation and will not be interrupted by other commands. The fee deduction request is processed through the message queue to ensure that each fee deduction request is processed sequentially and that the execution of each request is idempotent. Before the actual fee deduction, a pre-check is performed to ensure that the user has enough balance or token for the operation. If the check passes, the fee deduction operation is performed. If the deduction operation fails, the system will record the failure and perform compensation operations when appropriate to ensure eventual consistency. Ensure that the deduction operation is idempotent, that is, repeating the same deduction operation multiple times will have the same result as executing it once. When the deduction operation fails, implement a retry mechanism, but ensure that the retry does not destroy the atomicity and consistency of the operation. Record detailed logs of all deduction operations, including operation time, operation results, etc., so that auditing and backtracking can be performed when problems occur. In a distributed system, consistent hashing is used to distribute deduction operations to ensure that deduction operations for the same user or the same request are always processed by the same node. Notify users of the results of billing deductions in a timely manner, and provide a detailed billing detail query interface to enhance billing transparency and user trust. Billing information is sent asynchronously to the billing service through the message queue, and stored in the database by the billing service to ensure the accuracy and completeness of the billing information. When each billing operation occurs, relevant billing information (such as call time, service type, usage, cost, etc.) is sent to the message queue in real time. Obtain billing data from the message queue through an independent billing service, and use batch writing or data partitioning to asynchronously store it in the database to improve the efficiency and performance of data storage. Data analysis and reporting: Based on the stored billing details, perform data analysis and mining to provide support for operational decisions, such as user behavior analysis, cost trend forecasting, cost-effectiveness evaluation, etc., and generate corresponding reports and visualization displays.

[0082] Second, as Figure 2 As shown, an embodiment of the present invention provides a large model API aggregation calling device, including:

[0083] The original request acquisition unit 200 is used to acquire the original request; the original request includes: user question information from the user client; the original request complies with the preset unified interface specification;

[0084] The product type determination unit 201 is used to determine, according to the user question information, in at least one large model product category, a target large model product category applicable to the user question information; wherein, for each user client, at least one large model channel corresponding to the large model product category corresponding to the user client is pre-configured; each large model channel configured corresponding to each large model product category corresponds to large models applicable to the large model product category from different suppliers; at least one large model channel corresponding to each large model product category corresponding to each user client is configured with a preset channel selection rule;

[0085] A channel selection unit 202 is configured to select a first large model channel as a target large model channel from at least one large model channel corresponding to the target large model product category according to a preset channel selection rule configured for the target large model product category corresponding to the user client;

[0086] The request routing processing unit 203 is used to convert the original request into a target request that complies with a target interface specification, and send the target request to the large model corresponding to the target large model channel; the target request includes the user question information; the target interface specification is the interface specification of the large model corresponding to the target large model channel;

[0087] A response receiving unit 204 is used to receive a first response from the large model corresponding to the target large model channel to the target request, and determine whether the status of the first response is successful;

[0088] an answer information acquisition unit 205, configured to acquire answer information corresponding to the user question information from the first response if the status of the first response is success;

[0089] The answer information feedback unit 206 is used to encapsulate the answer information into a second response that complies with the preset unified interface specification, and return the second response to the user client.

[0090] Furthermore, the device also includes:

[0091] A first response failure processing unit, configured to trigger the channel selection unit 202 to reselect a target large model channel if the status of the first response is failure;

[0092] Furthermore, the channel selection unit 202 includes:

[0093] An available balance acquisition module, used to acquire client information corresponding to the user client, and determine the balance corresponding to the user client according to the client information;

[0094] An available balance comparison module is used to compare the usage fees of the available large model channels and the balance corresponding to the user client in sequence according to the order of preset channel priorities configured for the target large model product category corresponding to the user client; wherein the available large model channel is the remaining large model channels among at least one large model channel corresponding to the target large model product category, excluding all target large model channels whose first response status is failed after obtaining the original request;

[0095] A channel determination module, configured to use a first available large model channel whose usage fee is less than or equal to the corresponding balance of the user client as a target large model channel;

[0096] The usage fee of the large model channel is calculated based on the user's question information and the preset unit value of the large model channel;

[0097] The preset channel priorities include: global default channel sorting, user-specified default channel sorting, or channel sorting based on scores determined by preset multiple performance parameters and corresponding weights; the performance parameters are used to score the performance of large models of each supplier.

[0098] The performance parameters include: response time, call success rate, load capacity, cost-effectiveness, service quality, historical performance, supplier reputation and / or user-defined indicators.

[0099] Furthermore, the device also includes:

[0100] The second response failure processing unit is used to obtain the failure error code and failure error message of the first response if the status of the first response is failure, and map the failure error code and the failure error message to the error code and error message defined by the unified interface specification.

[0101] Furthermore, the answer information acquisition unit 205 is specifically used to: save the user question information and the corresponding answer information into a history information list corresponding to the user client;

[0102] The request routing processing unit 203 is also used to: use the user question information and the corresponding answer information in the historical information list corresponding to the user client as reference information of the target request, and provide it to the big model corresponding to the target big model channel; the reference information is used as a context reference when the big model corresponding to the target big model channel infers the target request.

[0103] Furthermore, the request routing processing unit 203 further includes:

[0104] A policy object creation module, used to select a policy class with the same supplier as the target large model channel from multiple policy classes through a preset policy factory method as a target policy class, and create a policy object corresponding to the target policy class;

[0105] A parameter conversion module is used to perform parameter name mapping, parameter data structure conversion, parameter type conversion, and parameter encoding and formatting on the original request through the policy object to obtain a target request that complies with the target interface specification;

[0106] Among them, each strategy class is defined in advance using the strategy pattern method according to the interface specification of the big model of the supplier corresponding to the strategy class; each strategy class encapsulates the API interface of the big model of the supplier corresponding to the strategy class, and implements the adapter pattern in the strategy class to adapt the call of the API interface of the big model of the supplier corresponding to the strategy class to an interface that conforms to the preset unified interface specification.

[0107] Furthermore, the device also includes:

[0108] A user account creation unit, configured to receive the user's registration information through a preset registration interface process, and create a user account for the user after the registration information is successfully verified;

[0109] A user account and authority synchronization unit, used to communicate with the user's existing business system through a preset user management API interface to synchronize user accounts and authorities with the user's existing business system;

[0110] The user single sign-on unit is used to establish a single sign-on mechanism with the user's existing business system so as to complete the user's login through the single sign-on mechanism.

[0111] Furthermore, the device also includes:

[0112] A user client creation unit is used to receive a user's request to create a user client through a preset client creation interface, and create a user client under the user account of the user according to the request to create a user client; the user account is pre-created for the user; each user account includes at least one user client; each user client includes client information corresponding to the user client; the client information includes: API key, access rights, quota limit and billing policy.

[0113] Furthermore, the device also includes:

[0114] A billing unit is used to encapsulate the answer information into a second response that complies with the preset unified interface specification, return the second response to the user client, perform billing according to the answer information, deduct fees from the user client, notify the user of the result of the deduction, and store the relevant information of the deduction in a billing detail database; the billing detail database is used for the user to query the billing details through a preset billing detail query interface.

[0115] The embodiment of the present invention is a device class embodiment that corresponds one-to-one to the embodiment of the aforementioned large model API aggregation calling method. The embodiment of the present invention can be understood based on the embodiment of the aforementioned large model API aggregation calling method, and will not be repeated here.

[0116] Thirdly, Figure 3 As shown, an embodiment of the present invention provides a large model API aggregation calling system, including: a gateway 300, a large model API aggregation calling device 301 as described above, and a large model of at least one supplier;

[0117] The gateway 300 is connected to the large model API aggregation calling device 301, and is used to authenticate, limit, fuse and downgrade the original request from the user client, and submit the original request to the large model API aggregation calling device 301;

[0118] The big model API aggregation calling device 301 is connected to a big model 302 of at least one supplier.

[0119] The embodiment of the present invention is a system embodiment composed of a large model API aggregation calling device that adopts the aforementioned large model API aggregation calling method. The embodiment of the present invention can be understood based on the aforementioned embodiment of the large model API aggregation calling method, and will not be repeated here.

[0120] The above technical solution of the embodiment of the present invention is described in detail below in conjunction with specific application examples. For technical details not introduced during the implementation process, please refer to the relevant description in the previous text.

[0121] like Figure 4As shown, an embodiment of the present invention provides a large model API aggregation (call) platform (system), including: a gateway, a large model API aggregation call device, a large model of at least one supplier, a billing center, a security center, and a statistical analysis (system); the user client can communicate with the large model API aggregation call system through three request methods, namely synchronous call, streaming call and asynchronous call. The gateway has authentication / authorization, fuse, current limiting and downgrade functions. The gateway is connected to the large model API aggregation call device, and sends the legal request from the user client to the large model API aggregation call device. The large model API aggregation call device is connected to the large model of at least one supplier, and the large model of each supplier provides at least one large model product category. The large model API aggregation call device is also connected to the billing center to store the user's billing details in the billing center. The security center and the statistical analysis (system) can analyze the logs and data in the large model aggregation call device and the billing center to find security problems or analyze the user's fee details. The platform can uniformly call the large model APIs of different manufacturers, simplify the developer's use process, and improve the stability and reliability of API calls through a series of technical means. The platform has functions such as authentication, gateway management, user management, channel configuration, unified external API, multi-channel API integration, error retry, API call plan, billing strategy, billing execution and billing details.

[0122] The following is a description of each function:

[0123] Authentication mechanism: Implement the OAuth2 authentication process to ensure that only legitimate clients can access the API.

[0124] Gateway function: Authentication, circuit breaking, current limiting and downgrade are implemented through the gateway to ensure the stability of the platform.

[0125] User management: user registration process and integration with other application system user systems to achieve unified management of user identities.

[0126] Channel configuration: Configure channels through information such as suppliers, products, models, and client keys to achieve fine-grained management of APIs. It mainly achieves precise control and management of API calls. Specifically, fine-grained management is reflected in the following aspects: First, channel definition and configuration: By defining the components of the channel, such as suppliers, products, models, and client keys, the platform allows developers to flexibly configure and adjust API channels as needed. This configuration can be customized according to different suppliers and product characteristics, thereby achieving personalized management of API calls. Second, channel sorting and switching: The platform can sort channels according to preset priorities or performance indicators. When the current channel call fails, the system can automatically switch to the next available channel to ensure the continuity and stability of the service. This dynamic channel management and switching mechanism improves the reliability of API calls.

[0127] Unified external API: Design unified API input and output parameter formats, and support HTTP synchronous calls and SSE streaming calls.

[0128] Multi-channel API integration: Use the strategy pattern to integrate APIs from different channels and realize automatic conversion of input and output parameters.

[0129] Error retry mechanism: Configure multiple channels to achieve automatic switching and retry when an API call fails.

[0130] API call solutions: Provides three solutions: synchronous call, streaming synchronous call and asynchronous call to meet the needs of different scenarios.

[0131] Billing strategy: Design flexible billing strategies based on different products and models, and support billing methods such as token, quantity, pixel specifications, etc.

[0132] Billing execution: Real-time deduction of fees is achieved through the billing gateway to ensure the accuracy of billing.

[0133] Billing details: record billing information in real time and store it asynchronously in the database through the message queue.

[0134] The embodiments of the present invention have the following technical effects: The embodiments of the present invention simplify the developer's use process by providing an interface that complies with the unified interface specification. A multi-channel automatic switching and error retry mechanism are implemented to improve the stability of API calls. A flexible billing strategy is designed, which can be customized according to different products and models. The redis+lua method is adopted to ensure the atomicity of billing execution. The developer's work efficiency is significantly improved, the development cost is reduced, and the developer can focus more on the implementation of business logic rather than tedious API adaptation work. The security and stability of the system are enhanced, and through strict authentication and intelligent gateway control, data security is effectively protected, malicious attacks are resisted, and the availability of the system is improved. Resource allocation and utilization are optimized, and the service quality is dynamically adjusted according to the system load to ensure that services can still be provided under high pressure conditions, thereby improving the overall performance of the system and user experience. More choices and flexibility are provided to users, and multiple calling methods and billing strategies are supported to meet the needs of different users in different scenarios. Accurate billing management is achieved to protect the commercial interests of the platform and suppliers, while improving the transparency and fairness of billing and enhancing users' trust in the platform. Through real-time billing detail recording and analysis, it provides strong data support for the platform's operational decisions, helps optimize service strategies and improve economic benefits.

[0135] Another embodiment is used below to illustrate that a large model API aggregation calling platform (system) provided by the embodiment of the present invention includes but is not limited to authentication, gateway functions, user management, channel configuration, unified external API, multi-channel API integration, error retry mechanism, API calling scheme, billing strategy, billing execution and billing details implementation method.

[0136] 1. Authentication mechanism:

[0137] (1) Client management: Use a database table (such as the client table) to store client information, including fields such as client_id, client_secret, and api_key. Perform data legitimacy verification when registering and updating client information.

[0138] (2) Authentication: Use client_id and client_secret, plus the current timestamp, and use jwt to generate api_key using HMACSHA256. During authentication, first verify whether api_key exists, then verify the legitimacy of api_key. If the verification fails, an error message is returned.

[0139] (3) Interface authentication: Configure authentication rules in the API gateway to parse the user information and permissions in the access token and match them with the predefined interface permissions.

[0140] 2. Gateway function: The API gateway is responsible for processing all API requests, implementing request authentication, current limiting, circuit breaking, and degradation to ensure high availability of the platform.

[0141] (1) Authentication module: strictly examines the legitimacy of requests and prevents illegal access.

[0142] (2) Circuit breaker mechanism: When a large model API continuously encounters errors or the response timeout reaches a threshold, the channel is automatically cut off to prevent the fault from spreading.

[0143] (3) Current limiting strategy: Based on the system's load capacity and service level agreement, the number of concurrent requests is limited to ensure stable operation of the system.

[0144] (4) Downgrade plan: When the system faces high load or abnormal conditions, it prioritizes core functions and suspends or reduces the service quality of non-critical functions.

[0145] 3. User management:

[0146] (1) Registration process: guide users to provide necessary information to complete registration and conduct validity verification.

[0147] (2) System integration: Through interface adaptation or data synchronization, seamless connection with the user system of the existing business system is achieved, and user data and permissions are shared. Among them, interface adaptation: Develop an API interface so that the aggregation platform can communicate with the user management interface of the existing business system. This includes operations such as reading, updating, and deleting user information. Data synchronization: Through the data synchronization mechanism, ensure that user data between the aggregation platform and the existing business system remains consistent. This may involve scheduled synchronization or real-time synchronization to ensure the accuracy and latest user information. Permission mapping: Map user permissions in the existing business system to the aggregation platform to ensure that the user's permissions in the aggregation platform match their permissions in the original system. Single sign-on (SSO): Implement a single sign-on mechanism to allow users to log in to the aggregation platform using the account information of the existing business system without having to re-register or log in to the aggregation platform.

[0148] (3) Client creation and management: Allow users to create multiple clients according to their needs, and configure and assign permissions to them independently. Each client has an independent API key. Therefore, personalized configuration, permission control, resource isolation, and billing management can be performed independently for each client. For example, in terms of personalized configuration, users are allowed to create multiple clients according to their needs. Each client can have independent configuration, such as API keys, access permissions, quota limits, etc., which provides users with flexibility and customized services. In terms of permission control, through the creation and management of clients, users can set different permissions for different clients to ensure that each client can only access its authorized resources and functions, thereby enhancing the security of the system. In terms of resource isolation, each client has an independent API key, which helps to achieve resource isolation and prevent resource conflicts or data leakage between different clients. In terms of billing management, the creation and management of clients can be combined with billing policies to set different billing rules and quotas for each client, facilitating the implementation of refined billing management.

[0149] 4. Channel configuration: Define and configure channels according to the components of the channel, and support flexible configuration and adjustment. The components of the channel include suppliers, products, models, and client keys. The channel is the smallest granularity of the large model interface. Here, the supplier refers to the large model API manufacturer; the product refers to which API of the large model API manufacturer, such as text to generate text, text to generate pictures; the model refers to that a large model API can be implemented using multiple different underlying models; the client key refers to that when connecting to the large model API manufacturer, each manufacturer can provide multiple keys for the platform to use. The combination of suppliers, products, models, and client keys forms a channel, and each channel is the minimum granularity for the aggregation API platform to call multiple large model manufacturer APIs.

[0150] 5. Channel sorting and switching: Sort channels according to preset priorities or performance indicators. When an error occurs in the current channel call, automatically switch to the next available channel to ensure uninterrupted service. Channel sorting methods include but are not limited to the default sorting of large-model API aggregation call devices, user-defined sorting, or sorting based on scores determined by preset parameters and weights; wherein, preset parameters include: response time, which is used to measure the response speed of API calls, that is, the time required from sending a request to receiving a response. Usually, channels with shorter response times are given higher priorities; success rate, which refers to the proportion of successful API calls. A channel with a high success rate indicates that it has higher stability and reliability, so it will be given priority in sorting; load capacity, which is used to indicate the amount of requests that a channel can handle. A channel that can withstand high concurrent requests will be considered more advantageous in sorting; Cost-effectiveness is used to consider the relationship between the cost of API calls and their performance. A channel with a high cost-effectiveness ratio may be given a higher priority. Quality of Service (QoS) includes indicators such as error rate, data accuracy, and service consistency. Channels with high service quality will be given priority. User-defined indicators include other performance indicators defined by users based on their specific needs, such as support for specific functions. Historical performance means evaluating the performance of the channel based on historical data, including past success rate, response time, and failure rate. Supplier reputation: The supplier's reputation and historical performance may also affect the priority of the channel. A channel provided by a reputable supplier may be given priority.

[0151] 6. Unified external API: The platform has designed a unified API interface. Developers send requests through this interface. The platform selects the corresponding API supplier for processing according to the request type; (1) Input and output parameter specifications: Define unified, clear and easy-to-understand input and output parameter formats, covering data types, field meanings and constraints, etc.; through efficient algorithms and data structures, accurately convert the platform's unified input parameters into the input parameter format specific to each supplier, and vice versa; at the same time, ensure the consistency and standardization of output parameters. For example, the overall replication of parameters is achieved through the reflection mechanism. The data structure refers to the structural definition of the platform's input parameters and output parameters. At present, some parameters of the API interface of some large model suppliers have multiple types of parameters. For example, a tool_choice parameter may be of string type or object type, which will bring a lot of complexity to program development. We can split it into two fields, tool_choice_type of string type and tool_choice of object type. The following methods are used to ensure consistency and standardization: standardization, ensuring that all vendors' API calls follow unified interface specifications, including data structure, field naming and type; unified configuration management, using a unified configuration management system to maintain mapping rules and parameter conversion logic, ensuring that all policy classes use the same configuration; error handling consistency, implementing a unified error handling mechanism, converting error responses from different vendors into a unified error code and message format; code reuse, reducing code duplication and improving code consistency by abstracting and encapsulating common conversion logic; version control, versioning unified interfaces and policy classes to ensure that all dependent systems and clients can smoothly transition when upgrading or modifying; documentation and examples, providing detailed API documentation and usage examples to help developers understand and use the unified interface. (2) HTTP synchronous call: based on the standard HTTP protocol, implements a call method that waits for the real-time return of JSON format results after the request is sent; (3) SSE streaming call: using Server-Sent Events technology to achieve real-time streaming data output to meet the needs of real-time data transmission.

[0152] 7. Multi-channel API integration: The platform uses strategy and proxy patterns to integrate API interfaces from different suppliers into a unified strategy factory to achieve automatic routing and parameter conversion of requests. The strategy factory includes the strategy pattern and factory pattern in software design; Strategy pattern: It is a behavioral design pattern that allows algorithms or behaviors to be selected at runtime. First, through the interface, a series of behaviors are defined, and the implementation class implements these interfaces, and these behaviors can be replaced with each other. It mainly solves the problem of a large number of if-else in the program and facilitates the expansion of the program. Factory method: The strategy factory uses the factory method to create management strategy objects. The factory method is a pattern for creating objects that uses a common interface to create objects, but allows subclasses to decide which class to instantiate. The strategy factory can dynamically select and instantiate the corresponding strategy object based on the specific parameters or conditions of the request. This allows the system to flexibly handle different requests without modifying existing code. The automatic routing method for requests in multi-channel API integration includes: defining a unified interface: defining a unified interface specification, including the data structure, field type and naming specification of requests and responses, which serves as the basis for all API conversions; strategy pattern design: the system will design different strategy classes according to the strategy pattern, and each strategy class corresponds to the API product of a specific API supplier; according to the characteristics and requirements of different large-model APIs, customize the corresponding integration strategy to achieve dynamic selection and switching; different suppliers' APIs have the following characteristics: different parameters, with different interface input parameters, output parameter names, types, and structures; different data formats, with different data formats, including json, xml, and plain text; different request protocols, with different request protocols, including http and websocket;

[0153] Create a policy class for each API vendor. These classes implement a unified interface. The interface may include methods / functions for obtaining channels, methods / functions for converting platform input parameters into vendor large model input parameters, methods / functions for converting large model output parameters into platform output parameters, methods / functions for obtaining request URLs, methods / functions for request execution, etc., and include the logic required to convert the unified interface into a specific vendor API. In each policy class, define specific mapping rules to map the parameters of the unified interface to the parameters of a specific API. These rules may include mapping of field names, conversion of data types, adjustment of structures, etc.; implement the adapter pattern in the policy class to adapt API calls from different suppliers to a unified interface specification; policy factory: the system contains a policy factory, which is responsible for dynamically creating corresponding policy objects based on channel information; request parsing: when an API request arrives, based on the client information of the request, the system first parses the request to determine the required channel; policy selection: based on the parsing results, the policy factory selects and instantiates the appropriate policy object; request routing: the policy object receives the request and routes it to the correct API supplier; the policy class parses the parameters of the unified interface, and constructs the request parameters of the specific API according to the mapping rules, and executes the interface call according to the interface rules of different suppliers; parameter conversion method: input parameter specification, defines a unified input parameter format, so that all API requests follow the same data structure and type before being sent to different suppliers; parameter Data mapping, each policy object contains a parameter mapping logic, which maps the unified input parameter format to the parameter format required by the API of a specific vendor; data structure conversion, according to the requirements of the vendor API, the data structure may need to be converted, such as from a JSON object to a specific XML structure; type conversion, perform necessary type conversion to ensure that the parameter type matches the requirements of the vendor API; encoding and formatting, encode and format the parameters, such as URL encoding, date and time formatting, etc.; error mapping, establish an error mapping table, and convert the error response of the vendor API into a unified error code and description, so that developers can understand and handle it more easily; output parameter conversion, for the response received from the vendor API, the policy object also needs to convert the output parameter back to a unified format to maintain the consistency of the API aggregation platform; response conversion: when receiving the response, the policy class converts the response of the specific API back to the response format of the unified interface.

[0154] 8. Error retry mechanism

[0155] The platform is configured with multiple channels. When a channel call fails, it automatically switches to the next channel to ensure the continuity and stability of the request. The configurable retry mechanism allows administrators or users to set the number of retries, interval time and channel switching strategy according to business needs and system performance. The retry mechanism has data consistency guarantee. During the retry process, technologies such as caching or context transfer are used to ensure the integrity and consistency of input parameters and avoid data loss or confusion. Specifically, using caching, the request parameters are cached in memory or distributed cache systems (such as Redis). The key is the client ID + conversation ID, and the value is the request information. In this way, even after the call fails, the request parameters can be restored from the cache. Specifically, a context transfer mechanism (such as ThreadLocal or transactional messages in a message queue) is used to maintain the context information of the request to ensure that the same context can be used when retrying.

[0156] 9.API call scheme

[0157] The platform supports HTTP synchronous calls, SSE streaming calls and asynchronous calls to meet the needs of different scenarios.

[0158] (1) HTTP synchronous call implementation: Using mature Web development frameworks and technologies to ensure efficient and stable sending, receiving, and processing of requests.

[0159] (2) Streaming synchronous call optimization: For SSE streaming calls, optimize the network connection, data buffering and push mechanism to improve the real-time and smoothness of data transmission.

[0160] (3) Asynchronous call management: Build a reliable message queue and task scheduling system to implement the submission, processing and result feedback of asynchronous tasks, and ensure the system's responsiveness and resource utilization efficiency.

[0161] 10. Billing strategy

[0162] The platform designs billing strategies based on the characteristics of different APIs, such as charging text generation by token and charging image generation by pixel specifications.

[0163] (1) Billing model design: Differentiated billing methods are developed for different large model service types (such as text generation, image generation, music generation, etc.), such as billing by token, billing by quantity, billing by pixel specifications, etc.

[0164] (2) Customized billing rules: Supports setting personalized billing unit prices and preferential policies for different large model suppliers, products, and models to meet diverse business needs.

[0165] (3) Free strategy setting: Set the conditions and scope of free use based on specific business scenarios and user groups to promote business promotion and user experience.

[0166] 11. Billing Execution

[0167] After the API call is successful, the fee is deducted in real time through the billing gateway based on the user's token or number of times information.

[0168] (1) Fee pre-check: After the API call request passes authentication, the user's fee balance, token quantity, or usage times are immediately checked to see if they meet the requirements of this call. If not, an error prompt is returned in a timely manner.

[0169] (2) Real-time deduction implementation: After the call is successful, the Redis+Lua script is used to implement accurate deduction of fees, ensuring the atomicity and consistency of the deduction operation, and preventing over-deduction or under-deduction. Among them, atomicity and consistency are achieved through the following solutions: Use transactions. Database transactions can ensure that a series of operations are either all successful or all failed. This is a common method to ensure the atomicity of operations; Use Redis's Lua script function to perform deduction operations. Lua scripts can ensure that a series of commands executed inside Redis are atomic, because Redis will execute Lua scripts as a separate operation and will not be interrupted by other commands; Use message queues to process deduction requests to ensure that each deduction request is processed sequentially and that the execution of each request is idempotent; Perform a pre-deduction check. Before the actual deduction of fees, a pre-check is performed to ensure that the user has sufficient balance or tokens to perform the operation. If the check passes, the deduction operation is performed; Provide a compensation mechanism. If the deduction operation fails, the system will record the failure and perform compensation operations at an appropriate time to ensure final consistency. Adopt idempotent design to ensure that the deduction operation is idempotent, that is, repeating the same deduction operation multiple times will have the same result as executing it once; implement a deduction retry mechanism to implement a retry mechanism when a deduction operation fails, but ensure that the retry does not destroy the atomicity and consistency of the operation; record audit logs to record detailed logs of all deduction operations, including operation time, operation results, etc., so as to audit and backtrack when problems arise; adopt consistent hashing to distribute deduction operations in distributed systems to ensure that deduction operations for the same user or the same request are always processed by the same node; provide billing result notification to notify users of the results of billing deductions in a timely manner, and provide a detailed billing detail query interface to enhance billing transparency and user trust.

[0170] 12. Billing details

[0171] The billing information is sent asynchronously to the billing service through the message queue and stored in the database by the billing service to ensure the accuracy and completeness of the billing information.

[0172] (1) Real-time data collection: When each billing operation occurs, the relevant billing information (such as call time, service type, usage, cost, etc.) is sent to the message queue in real time.

[0173] (2) Asynchronous storage processing: The billing data is obtained from the message queue through an independent billing service, and is asynchronously stored in the database using technologies such as batch writing or data partitioning to improve the efficiency and performance of data storage.

[0174] (3) Data analysis and reporting: Based on the stored billing details, data analysis and mining are performed to provide support for operational decisions, such as user behavior analysis, cost trend forecasting, cost-benefit evaluation, etc., and corresponding reports and visual displays are generated.

[0175] like Figure 5 As shown, the steps of a large model API aggregation calling method are described below with another embodiment: Step S50, application request: receiving an original request initiated by a user;

[0176] Step S51, authentication and authorization: the gateway authenticates and authorizes the original request;

[0177] Step S52, determine whether the authentication is passed, if it is passed, continue to step S53, otherwise return to step S50 to wait for receiving a new original request;

[0178] Step S53, select a channel; obtain information of all channels (at least one large model channel); select a target large model channel from at least one large model channel according to channel priority; wherein the (large model) channel is a unique channel consisting of supplier+product+model;

[0179] Step S54, calling the channel interface: executing a specific big model calling interface operation, calling the big model interface of the supplier's big model corresponding to the target big model channel through the proxy class; calling the big model interface corresponding to the target big model channel through the proxy class; the proxy class includes synchronous calling and streaming calling methods, and obtains the specific big model calling processor (strategy object) through the dependency injection policy factory; the big model processing factory (strategy factory) creates a specific calling processor; for better scalability and loose coupling design, the big model calling processor adopts the strategy mode. The design is as follows: big model dialogue request parameter interface, big model dialogue response parameter interface, these two interfaces do not contain any methods, as empty interfaces, only used for program scalability design, different big model manufacturers' request parameter class and response parameter class implement these two interfaces respectively. Big model request error handling interface, the error handler class of different big model manufacturers implements this interface. The request strategies of big models of different manufacturers (suppliers) correspond to different implementation classes (strategy classes);

[0180] Step S55, determine the call result. If the call is successful, directly return the response result and return to step S50 to wait for receiving a new original request; if the call fails, return to step S53 and switch to the next channel to continue the request.

[0181] The operation steps in the asynchronous dialogue scenario are as follows:

[0182] 1. Submit request: submit request information to MQ (message queue); return task submission success information;

[0183] 2. The MQ consumer listens to the request message, selects the channel, and calls the large model interface operation through the proxy class. If the large model interface is successfully called, the request result is output to the user through asynchronous callback, or the request result is stored in the cache and database, and the user obtains the request result through polling. If the large model interface fails to be called, or the server crashes during the processing, the message returns to the MQ consumption queue, and other normal nodes listen to the message and continue to execute the processing operation.

[0184] In streaming calls, when a user initiates a request, the background will immediately establish an SSE connection. Since creating a connection and sending a message are two different operations, executing a large model request and sending a message in the thread that creates the connection will block the current thread, so the business processing here should be done in an asynchronous multi-threaded manner. In multi-threading, it is extremely important to obtain the context information of the main thread, from which user information, request information, etc. can be obtained. The context is passed in different ways depending on the language. Here, Java is taken as an example, and ThreadLocal is used to pass the context between threads.

[0185] It should be understood that the specific order or hierarchy of steps in the disclosed process is an example of an exemplary method. Based on design preferences, it should be understood that the specific order or hierarchy of steps in the process can be rearranged without departing from the scope of protection of the present disclosure. The attached method claims present the elements of the various steps in an exemplary order and are not intended to be limited to the specific order or hierarchy described.

[0186] In the above detailed description, various features are grouped together in a single embodiment to simplify the disclosure. This method of disclosure should not be interpreted as reflecting an intention that the embodiments of the claimed subject matter require more features than those set forth in each claim. On the contrary, as reflected in the appended claims, the invention is in a state of having less than all the features of the disclosed individual embodiments. Therefore, the appended claims are hereby expressly incorporated into the detailed description, with each claim standing on its own as a separate preferred embodiment of the invention.

[0187] The disclosed embodiments are described above to enable any person skilled in the art to implement or use the present invention. Various modifications of these embodiments are obvious to those skilled in the art, and the general principles defined herein may also be applied to other embodiments without departing from the spirit and scope of the present disclosure. Therefore, the present disclosure is not limited to the embodiments given herein, but is consistent with the broadest scope of the principles and novel features disclosed in this application.

[0188] The above description includes examples of one or more embodiments. Of course, it is impossible to describe all possible combinations of components or methods for the purpose of describing the above embodiments, but it should be recognized by those skilled in the art that the various embodiments may be further combined and arranged. Therefore, the embodiments described herein are intended to cover all such changes, modifications and variations that fall within the scope of protection of the appended claims. In addition, with respect to the term "comprising" used in the specification or claims, the word is covered in a manner similar to the term "including", just as "including," is explained as a transitional word in the claims. In addition, any term "or" used in the specification of the claims is intended to mean "non-exclusive or".

[0189] Those skilled in the art may also understand that the various illustrative logical blocks, units, and steps listed in the embodiments of the present invention may be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly demonstrate the interchangeability of hardware and software, the various illustrative components, units, and steps described above have generally described their functions. Whether such functions are implemented by hardware or software depends on the specific application and the design requirements of the entire system. Those skilled in the art may use various methods to implement the described functions for each specific application, but such implementation should not be understood as exceeding the scope of protection of the embodiments of the present invention.

[0190] The various illustrative logic blocks or units described in the embodiments of the present invention can be implemented or operated by a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, a discrete gate or transistor logic, a discrete hardware component, or any combination of the above. The general-purpose processor can be a microprocessor, and optionally, the general-purpose processor can also be any conventional processor, controller, microcontroller or state machine. The processor can also be implemented by a combination of computing devices, such as a digital signal processor and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a digital signal processor core, or any other similar configuration.

[0191] The steps of the method or algorithm described in the embodiments of the present invention can be directly embedded in hardware, a software module executed by a processor, or a combination of the two. The software module can be stored in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable disk, a CD-ROM, or other storage media of any form in the art. Exemplarily, the storage medium can be connected to the processor so that the processor can read information from the storage medium and can write information to the storage medium. Optionally, the storage medium can also be integrated into the processor. The processor and the storage medium can be arranged in an ASIC, and the ASIC can be arranged in a user terminal. Optionally, the processor and the storage medium can also be arranged in different components in the user terminal.

[0192] In one or more exemplary designs, the above functions described in the embodiments of the present invention can be implemented in hardware, software, firmware or any combination of the three. If implemented in software, these functions can be stored on a computer-readable medium, or transmitted in the form of one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media that facilitate the transfer of computer programs from one place to another. The storage medium can be any available medium that can be accessed by any general or special computer. For example, such computer-readable media can include but are not limited to RAM, ROM, EEPROM, CD-ROM or other optical disk storage, disk storage or other magnetic storage devices, or any other medium that can be used to carry or store program codes in the form of instructions or data structures and other forms that can be read by general or special computers, or general or special processors. In addition, any connection can be appropriately defined as a computer-readable medium, for example, if the software is transmitted from a website site, server or other remote resource through a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL) or wirelessly, such as infrared, wireless and microwave, it is also included in the defined computer-readable medium. The disk and disc include compact disk, laser disk, optical disk, DVD, floppy disk and blue-ray disk. Disks usually copy data magnetically, while discs usually copy data optically with lasers. The above combination can also be included in computer readable media.

[0193] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A large model API aggregation calling method, characterized in that: include: Get the original request; The original request includes: user question information from the user client; the original request complies with the preset unified interface specification; According to the user question information, in at least one large model product category, a target large model product category applicable to the user question information is determined; wherein, for each user client, at least one large model channel corresponding to the large model product category corresponding to the user client is pre-configured; each large model channel configured corresponding to each large model product category corresponds to large models applicable to the large model product category from different suppliers; at least one large model channel corresponding to each large model product category corresponding to each user client is configured with a preset channel selection rule; According to the preset channel selection rule configured for the target large model product category corresponding to the user client, a first large model channel is selected as the target large model channel from at least one large model channel corresponding to the target large model product category; Convert the original request into a target request that complies with the target interface specification, and send the target request to the large model corresponding to the target large model channel; the target request includes the user question information; the target interface specification is the interface specification of the large model corresponding to the target large model channel; receiving a first response from the large model corresponding to the target large model channel to the target request, and determining whether the status of the first response is successful; If the status of the first response is success, obtaining answer information corresponding to the user question information from the first response; The answer information is encapsulated into a second response that complies with the preset unified interface specification, and the second response is returned to the user client.

2. The large model API aggregation calling method according to claim 1, characterized in that: The method further comprises: If the status of the first response is failure, return to the preset channel selection rule configured according to the target large model product category corresponding to the user client, select the first large model channel as the target large model channel from at least one large model channel corresponding to the target large model product category, and reselect the target large model channel.

3. The large model API aggregation calling method according to claim 1 or 2, characterized in that: According to the preset channel selection rule configured for the target large model product category corresponding to the user client, selecting a first large model channel as the target large model channel from at least one large model channel corresponding to the target large model product category includes: Acquire client information corresponding to the user client, and determine the balance corresponding to the user client according to the client information; According to the order of preset channel priorities configured for the target large model product category corresponding to the user client, the usage fees of the available large model channels and the balance corresponding to the user client are compared in sequence; wherein the available large model channel is the remaining large model channels after excluding all target large model channels whose first response status is failed after obtaining the original request from at least one large model channel corresponding to the target large model product category; The first available large model channel whose usage fee is less than or equal to the corresponding balance of the user client is used as the target large model channel; The usage fee of the large model channel is calculated based on the user's question information and the preset unit value of the large model channel; The preset channel priorities include: global default channel sorting, user-specified default channel sorting, or channel sorting based on scores determined by preset multiple performance parameters and corresponding weights; the performance parameters are used to score the performance of large models of each supplier.

4. The large model API aggregation calling method according to claim 1, characterized in that: Obtaining answer information corresponding to the user question information from the first response, including: Saving the user question information and the corresponding answer information in a history information list corresponding to the user client; Sending the target request to the large model corresponding to the target large model channel includes: The user question information and the corresponding answer information in the historical information list corresponding to the user client are provided as reference information of the target request to the big model corresponding to the target big model channel; the reference information is used as a context reference when the big model corresponding to the target big model channel infers the target request.

5. The large model API aggregation calling method according to claim 1, characterized in that: Converting the original request into a target request that complies with the target interface specification includes: Through a preset strategy factory method, a strategy class having the same supplier as the target large model channel is selected from multiple strategy classes as a target strategy class, and a strategy object corresponding to the target strategy class is created; By using the policy object, the original request is subjected to parameter name mapping, parameter data structure conversion, parameter type conversion, and parameter encoding and formatting, so as to obtain a target request that complies with the target interface specification; Among them, each strategy class is defined in advance using the strategy pattern method according to the interface specification of the big model of the supplier corresponding to the strategy class; each strategy class encapsulates the API interface of the big model of the supplier corresponding to the strategy class, and implements the adapter pattern in the strategy class to adapt the call of the API interface of the big model of the supplier corresponding to the strategy class to an interface that conforms to the preset unified interface specification.

6. The large model API aggregation calling method according to claim 1, characterized in that: The method further comprises: Receive the user's registration information through a preset registration interface process, and create a user account for the user after the registration information is successfully verified; Communicate with the user's existing business system through the preset user management API interface to synchronize user accounts and permissions with the user's existing business system; Alternatively, a single sign-on mechanism is established between the user's existing business system so that the user's login is completed through the single sign-on mechanism.

7. The large model API aggregation calling method according to claim 1, characterized in that: The method further comprises: Through a preset client creation interface, a user's request to create a user client is received, and according to the request to create a user client, a user client is created under the user account of the user; the user account is pre-created for the user; each user account includes at least one user client; each user client includes client information corresponding to the user client; the client information includes: API key, access rights, quota limit and billing policy.

8. The large model API aggregation calling method according to claim 1, characterized in that: The method further comprises: After encapsulating the answer information into a second response that complies with the preset unified interface specification and returning the second response to the user client, billing is performed based on the answer information, and deductions are made from the user client, the result of the deduction is notified to the user, and the relevant information of the deduction is stored in a billing details database; the billing details database is used for users to query billing details through a preset billing details query interface.

9. A large model API aggregation calling device, characterized in that: include: An original request obtaining unit, used for obtaining an original request; The original request includes: user question information from the user client; the original request complies with the preset unified interface specification; A product type determination unit is used to determine, according to the user question information, in at least one large model product category, a target large model product category to which the user question information is applicable; wherein, for each user client, at least one large model channel corresponding to the large model product category corresponding to the user client is pre-configured; each large model channel configured corresponding to each large model product category corresponds to large models applicable to the large model product category from different suppliers; at least one large model channel corresponding to each large model product category corresponding to each user client is configured with a preset channel selection rule; a channel selection unit, configured to select a first large model channel as a target large model channel from at least one large model channel corresponding to the target large model product category according to a preset channel selection rule configured for the target large model product category corresponding to the user client; A request routing processing unit, configured to convert the original request into a target request that complies with a target interface specification, and send the target request to the large model corresponding to the target large model channel; the target request includes the user question information; the target interface specification is the interface specification of the large model corresponding to the target large model channel; a response receiving unit, configured to receive a first response from the large model corresponding to the target large model channel to the target request, and determine whether the status of the first response is successful; an answer information acquisition unit, configured to acquire answer information corresponding to the user question information from the first response if the status of the first response is success; The answer information feedback unit is used to encapsulate the answer information into a second response that complies with the preset unified interface specification, and return the second response to the user client.

10. A large model API aggregation calling system, characterized in that: include: A gateway, a large model API aggregation calling device as claimed in claim 9, and large models of multiple suppliers; The gateway is connected to the large model API aggregation calling device, and is used to authenticate, limit, fuse and downgrade the original request from the user client, and submit the original request to the large model API aggregation calling device; The big model API aggregation calling device is connected to the big models of multiple suppliers.

Images