Model request processing method and device, electronic equipment and storage medium

Abstract

The invention relates to a model request processing method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining the model request information sent by a client, carrying out the request distribution based on the model request information, obtaining the request distribution information, and carrying out the processing of a model request based on the request distribution information. Determining an allocated target model corresponding to the model request information, responding to the model request information through the target model to obtain request response information, and then generating a request processing result corresponding to the model request information based on the request response information; therefore, the model request information can be allocated to the accurate target model, the problem that different language models cannot be compatible and used concurrently in the prior art is solved, and the effect that the different language models are compatible and used concurrently can be achieved.

Description

Technical Field

[0001] The present application relates to the field of computers, and in particular to a model request processing method, device, electronic device and storage medium. Background Art

[0002] With the rapid development of artificial intelligence technology, large language models (LLM) have been widely used in natural language processing, text generation, dialogue systems and other fields. That is, language models can be used to enhance business capabilities in many scenarios. For example, language models can be used to complete various tasks in the business of applications. When applications use language models to meet business needs, they usually need to connect to multiple language models. However, there are large differences in different language model interfaces, request and response data structures, resource management, etc., which leads to the problem that different language models are incompatible and cannot be used concurrently. Summary of the invention

[0003] In order to solve the problem that different language models are incompatible and cannot be used concurrently, the present application provides a model request processing method, device, electronic device and storage medium.

[0004] In a first aspect, the present application provides a model request processing method, comprising:

[0005] Get the model request information sent by the client;

[0006] Request distribution is performed based on the model request information to obtain request allocation information;

[0007] Based on the request allocation information, determining a target model corresponding to the model request information;

[0008] Responding to the model request information through the target model to obtain request response information;

[0009] Based on the request response information, a request processing result corresponding to the model request information is generated.

[0010] Optionally, performing request diversion based on the model request information to obtain request allocation information includes:

[0011] Based on the model request information, obtain the allocation ratio information of the preset model;

[0012] The allocation ratio information is used to perform request diversion on the model request information to generate the request allocation information.

[0013] Optionally, obtaining allocation ratio information of a preset model based on the model request information includes:

[0014] Based on the model request information, obtaining preset ratio information and request queue length information of the preset model;

[0015] The request queue length information is used to proportionally adjust the preset ratio information to generate the allocation ratio information.

[0016] Optionally, the using the allocation ratio information to perform request diversion on the model request information to generate the request allocation information includes:

[0017] Determine the allocation ratio value corresponding to each of the preset models according to the allocation ratio information;

[0018] Generate a random number based on the model request information;

[0019] Determine the allocation ratio value corresponding to the random number as a target allocation ratio value;

[0020] Determine the preset model corresponding to the target allocation ratio value as the target model;

[0021] The request allocation information is generated according to the target model.

[0022] Optionally, generating a request processing result corresponding to the model request information based on the request response information includes:

[0023] Performing anomaly detection on the request response information to obtain an anomaly detection result;

[0024] When the abnormality check detection result is a request response abnormality result, obtaining request response adjustment information;

[0025] Using the request response adjustment information to adjust the request response information to obtain adjusted request response information;

[0026] The request processing result is generated based on the adjusted request response information.

[0027] Optionally, after obtaining the model request information sent by the client, the step further includes:

[0028] Obtaining cache request information corresponding to the client;

[0029] Determining whether the model request information matches the cache request information;

[0030] When the model request information matches the cache request information, obtaining cache response information corresponding to the cache request information, and generating a request processing result corresponding to the model request information based on the cache response information;

[0031] When the model request information does not match the cache request information, a step of performing request diversion based on the model request information to obtain request allocation information is performed.

[0032] Optionally, responding to the model request information through the target model to obtain request response information includes:

[0033] Determining the target priority of the model request information according to the queue priority information of the target model;

[0034] Based on the target priority, adding the model request information to a request queue of the target model;

[0035] According to the request queue, the request response information corresponding to the model request information is obtained.

[0036] In a second aspect, the present application provides a model request processing device, comprising:

[0037] The acquisition module is used to obtain the model request information sent by the client;

[0038] A request distribution module is used to distribute requests based on model request information and obtain request allocation information;

[0039] A determination module, configured to determine a target model to be assigned to the model request information based on the request assignment information;

[0040] A response module, used for responding to the model request information through the target model to obtain request response information;

[0041] A generation module is used to generate a request processing result corresponding to the model request information based on the request response information.

[0042] In a third aspect, an electronic device is provided, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0043] Memory, used to store computer programs;

[0044] The processor is used to implement the model request processing method described in any one of the first aspects when executing the program stored in the memory.

[0045] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the model request processing method as described in any one of the first aspects is implemented.

[0046] The embodiment of the present application obtains model request information sent by the client, and performs request diversion based on the model request information to obtain request allocation information, so as to determine the target model corresponding to the model request information based on the request allocation information, respond to the model request information through the target model, obtain request response information, and then generate a request processing result corresponding to the model request information based on the request response information; thereby, the model request information can be allocated to the accurate target model, which solves the problem in the existing related technology that different language models are not compatible and can be used concurrently, and can achieve the effect of compatible and concurrent use of different language models. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0048] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0049] Figure 1 A flowchart of a model request processing method provided in an embodiment of the present application;

[0050] Figure 2 A schematic diagram of the structure of a model request processing device provided in an embodiment of the present application;

[0051] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0053] Existing language models have significant differences in their model interfaces, request and response data structures, resource management, etc. In addition, language model resources are relatively scarce, such as graphics processing unit (GPU), account and other resources. However, when language models are applied to the business in the application, it is usually necessary to configure multiple different language models to meet business needs due to the scarcity of language model resources. However, the request and response data structures of different language models are inconsistent, which makes business docking complicated. In addition, the resource costs of different language models are also different. For example, some language models are charged by tokens and have high usage fees, while some language models are charged by time and date and have low usage costs, which leads to the problem that different language models added in the business are not compatible with concurrent use.

[0054] In order to solve the problem that different language models in the prior art are not compatible and used concurrently, the present application provides a model request processing method, device, electronic device and storage medium, which obtains model request information sent by a client and performs request diversion based on the model request information to obtain request allocation information, and determines the target model corresponding to the model request information based on the request allocation information, responds to the model request information through the target model to obtain request response information, and then generates a request processing result corresponding to the model request information based on the request response information; thereby, the model request information can be allocated to the accurate target model, which solves the problem that different language models in the prior related technology are not compatible and used concurrently, and can achieve the effect of compatible and concurrent use of different language models.

[0055] Figure 1 A flow chart of a method for determining a verification method provided in an embodiment of the present application. The method can be applied to one or more electronic devices such as a gateway, a client, and a server. In addition, the execution subject of the method can be hardware or software. When the above-mentioned execution subject is hardware, the execution subject can be one or more of the above-mentioned electronic devices. For example, a single electronic device can execute the method, or a plurality of electronic devices can cooperate with each other to execute the method. When the above-mentioned execution subject is software, the method can be implemented as multiple software or software modules, or as a single software or software module. No specific limitation is made here.

[0056] like Figure 1 As shown, the present application discloses an embodiment, and provides a model request processing method, which may specifically include the following steps:

[0057] Step S110: Obtain model request information sent by the client.

[0058] Among them, the client can refer to an application used to receive user requests, and the user request can include model request information, that is, when the client receives the model request information, it can output the model request information to obtain the model request information sent by the client, and the model request information can refer to information used to request a language model response.

[0059] Step S120: Perform request diversion based on the model request information to obtain request allocation information.

[0060] Specifically, after obtaining the model request information, request diversion can be performed based on the model request information to obtain request allocation information, wherein request diversion represents the process of diverting and allocating the model request information to different language models, and request allocation information may refer to allocation information for allocating the model request information to its corresponding language model, for example, the request allocation information may include a model identifier for allocating the model request information to its corresponding language model; the specific request diversion method may be allocation according to a pre-configured allocation strategy, and the allocation strategy may be a strategy for allocation in combination with dynamic information corresponding to each preset model, which is not specifically limited in this embodiment.

[0061] It should be noted that in this embodiment, multiple language models can be pre-configured. After obtaining the model request information, the request can be diverted for the model request information to generate request allocation information indicating the target model, and the target model can indicate that the model request information is allocated to its corresponding language model.

[0062] Step S130: Based on the request allocation information, determine the target model to which the model request information is allocated.

[0063] Specifically, after obtaining the request allocation information, the target model corresponding to the model request information can be determined based on the request allocation information. The specific determination method can be to extract the model identifier from the request allocation information and determine the language model corresponding to the model identifier as the target model.

[0064] Step S140: Respond to the model request information through the target model to obtain request response information.

[0065] Specifically, after determining the target model, the model request information can be sent to the target model, so that the target model responds to the model request information and outputs model response information, and based on the model response information output by the target model, generates request response information corresponding to the model request information, that is, the request response information can represent the response information output by the target model based on the model request information.

[0066] It should be noted that, in the process of sending the model request information to the target model, the preset format requirement information corresponding to the target model can be obtained, and the format requirement information can be used to convert the model request information to obtain the target request information, and the target request information can be sent to the target model, so that the target model responds to the target request information and outputs the model response information, and then generates the request response information corresponding to the model request information according to the model response information output by the target model; wherein the format requirement information indicates the format of the preset input request corresponding to the target model, and the format conversion can indicate the process of converting the request data contained in the model request information into the format data corresponding to the format requirement information, which can be text format conversion or data format conversion, etc., and the target request information indicates the request information that can be recognized and responded by the target model; therefore, when the client is connected to multiple preset language models, the model request information can also be converted into the target request information that can be recognized and responded by the target model through the preset format requirement information corresponding to each language model, thereby achieving the purpose of being compatible with different input formats required by different language models.

[0067] Step S150: Generate a request processing result corresponding to the model request information based on the request response information.

[0068] Specifically, after obtaining the request response information, a request processing result corresponding to the model request information can be generated based on the request response information. The request processing result can represent the result of the response output based on the request response information, such as generating an output text based on the request response information, and outputting the output text to the client, thereby obtaining the request processing result. Of course, the request processing result can also be in other forms, and this embodiment does not specifically limit this.

[0069] In addition, since the request response information can represent the response information output by the target model based on the model request information, and the response information formats output by different target models may be different, but the format of the client output is usually relatively fixed, the output format information corresponding to the client can be obtained at this time, and the request response information is formatted according to the output format information to generate output response information, so as to output the output response information and obtain the request processing result corresponding to the model request information. Therefore, when the client is connected to multiple preset language models and different preset language models are required for output, the output format information corresponding to the client can be used to uniformly convert the request response information output by each preset language model, and obtain the output format information that conforms to the client output format, thereby achieving the purpose of being compatible with the output of different language models and outputting in a unified format.

[0070] It can be seen that in this embodiment, by obtaining the model request information sent by the client and performing request diversion based on the model request information, request allocation information is obtained, and based on the request allocation information, the target model corresponding to the model request information is determined. This can play a role in adaptively allocating different target models to the model request information when docking multiple language models, thereby achieving the purpose of docking multiple language models at the same time, responding to the model request information through the target model, obtaining request response information, and then generating a request processing result corresponding to the model request information based on the request response information; thereby, the model request information can be allocated to the accurate target model, thereby solving the problem in the existing related technology that different language models cannot be compatible and used concurrently, and achieving the effect of compatible and concurrent use of different language models.

[0071] In an optional embodiment of the present application, step S120 performs request diversion based on the model request information to obtain request allocation information, which may specifically include the following sub-steps: based on the model request information, obtaining the allocation ratio information of the preset model; using the allocation ratio information to perform request diversion on the model request information and generate request allocation information.

[0072] In this embodiment, in the process of request diversion, after obtaining the model request information, the allocation ratio information of the preset model can be obtained. The preset model represents a pre-configured language model, and the allocation ratio information can refer to the allocation ratio corresponding to each preset model. Therefore, the allocation ratio information can be used to perform request diversion on the model request information and generate request allocation information. That is, when the model request information is diverted, the corresponding model can be allocated to the model request information according to the allocation ratio represented by the allocation ratio information; of course, in the process of request allocation, the real-time monitoring of the preset model can also be combined, so that the model request information can be allocated to the language model with lower load according to the configuration and real-time monitoring, thereby improving the utilization efficiency of system resources, reducing the request response time, and ensuring the stability and efficiency of the system.

[0073] It should be noted that, in this embodiment, the sum of the allocation ratios represented by the allocation ratio information of all preset models is 1. For example, when the preset model includes preset model A, preset model B, and preset model C, the allocation ratio information A of preset model A is 0.3, the allocation ratio information B of preset model B is 0.5, and the allocation ratio information C of preset model C is 0.2; in the process of request allocation, the corresponding model can be allocated to the model request information according to the allocation ratios of 0.3, 0.5, and 0.2, that is, the model request information has a ratio of 0.3 allocated to preset model A, a ratio of 0.5 allocated to preset model B, and a ratio of 0.2 allocated to preset model C. Of course, the above is only an example. In the specific implementation, it can be adaptively adjusted according to the scenario and needs, and this embodiment does not make specific limitations on this.

[0074] In an optional embodiment of the present application, based on the model request information, the allocation ratio information of the preset model is obtained, which may specifically include the following sub-steps: based on the model request information, the preset ratio information and request queue length information of the preset model are obtained; using the request queue length information, proportionally adjusting the preset ratio information to generate the allocation ratio information.

[0075] In this embodiment, in the process of obtaining the allocation ratio information of the preset model, when obtaining the model request information, the preset ratio information and request queue length information of each preset model can be obtained, wherein the preset ratio information indicates the ratio pre-configured for the preset model, and the request queue length information may refer to the number of requests in the request queue waiting for the preset model; thus, the request queue length information may be used to proportionally adjust the preset ratio information to generate the allocation ratio information; a specific proportional adjustment method may be to determine the quantity interval to which the number of requests represented by the request queue length information belongs, and to proportionally adjust the preset ratio information using the adjustment coefficient corresponding to the quantity interval to obtain the allocation ratio information; and then, according to the real-time request queue length information of each preset model, The preset ratio information is proportionally adjusted so that the allocation ratio information matches the real-time status of the preset model. For example, when the number of requests expressed by the request queue length information of the preset model is large, the ratio represented by the preset ratio information can be reduced to obtain the allocation ratio information, which can reduce the allocation ratio of the preset model with a higher load. Of course, when the number of requests expressed by the request queue length information of the preset model is small, the ratio represented by the preset ratio information can be increased to obtain the allocation ratio information, which can increase the allocation ratio of the preset model with a lower load. The effect of adaptively adjusting the allocation ratio of each preset model according to the real-time situation is achieved, which can effectively improve the utilization efficiency of system resources, avoid request congestion, reduce request response time, and ensure the stability and efficiency of the system.

[0076] In one example, it is determined that the number of requests represented by the request queue length information of the preset model is 30. At this time, it is determined that the number interval to which the number of requests 30 belongs is interval B, and the adjustment coefficient corresponding to interval B is -0.1. Then -0.1 can be used to proportionally adjust the preset ratio information. If the preset ratio information is 0.5, 0.5 and -0.1 can be combined to perform proportional adjustment, and the obtained allocation ratio information is 0.4. Therefore, when the number of requests of the preset model is large, the allocation ratio of the preset model can be reduced, so that the subsequent requests allocated to the preset model are reduced, and the requests are avoided from waiting for a long time in the request queue of the preset model, request congestion is avoided, the request response time is reduced, and the stability and efficiency of the system are ensured; the request waiting time of the preset model can also be dynamically monitored. When the request waiting time reaches the waiting time threshold, the preset ratio information can be proportionally adjusted to generate the allocation ratio information; of course, the specific proportional adjustment can also be other adjustment methods. The above is only an example for illustration, and this embodiment does not make specific limitations on this. It should be noted that, in combination with the scenario requirement that the sum of the allocation ratios represented by the allocation ratio information of all preset models in the aforementioned embodiment is 1, after the preset ratio information of a preset model is proportionally adjusted, the preset ratio information of other preset models can be adaptively adjusted according to the proportional adjustment result, so that the scenario requirement that the sum of the allocation ratios represented by the allocation ratio information of all preset models is 1 can be maintained.

[0077] In an optional embodiment of the present application, allocation ratio information is used to perform request diversion on model request information to generate request allocation information, which may specifically include the following sub-steps: determining the allocation ratio value corresponding to each preset model based on the allocation ratio information; generating a random number based on the model request information; determining the allocation ratio value corresponding to the random number as the target allocation ratio value; determining the preset model corresponding to the target allocation ratio value as the target model; and generating request allocation information based on the target model.

[0078] In this embodiment, in the process of request diversion, the allocation ratio value corresponding to each preset model can be determined based on the allocation ratio information of each preset model, and the allocation ratio value is the ratio value corresponding to the allocation ratio information. Then, in response to the model request information, a random number can be generated according to the preset rules, and according to the preset correspondence information, the allocation ratio value corresponding to the random number can be determined as the target allocation ratio value, and the preset model corresponding to the target allocation ratio value can be determined as the target model, so that the request allocation information can be generated based on the target model, and specifically, the model identifier corresponding to the target model can be determined as the request allocation information.

[0079] In one example, the preset model includes preset model A, preset model B, and preset model C. The allocation ratio value corresponding to preset model A is 0.3, the allocation ratio value corresponding to preset model B is 0.5, and the allocation ratio value corresponding to preset model C is 0.2. At this time, the preset corresponding relationship information is that the random number can be any one of 0-9, and the random numbers 0, 1, and 2 correspond to the allocation ratio value of 0.3, the random numbers 3, 4, 5, 6, and 7 correspond to the allocation ratio value of 0.5, and the random numbers 8 and 9 correspond to the allocation ratio value of 0.2. That is, when the random number is 8, the allocation ratio value 0.2 corresponding to the random number 8 can be determined as the target allocation ratio value, and the preset model C corresponding to the target allocation ratio value 0.2 can be determined as the target model; of course, the above is only an example description, and adaptive adjustments can be made according to the scenario and needs during specific implementation. This embodiment does not make specific limitations on this.

[0080] In an optional embodiment of the present application, step S150 generates a request processing result corresponding to the model request information based on the request response information, which may specifically include the following sub-steps: performing anomaly detection on the request response information to obtain anomaly detection results; when the anomaly detection result is an abnormal result of the request response, obtaining request response adjustment information; adjusting the request response information using the request response adjustment information to obtain adjusted request response information; and generating a request processing result based on the adjusted request response information.

[0081] In this embodiment, after obtaining the request response information output by the target model, an anomaly detection can be performed on the request response information to obtain an anomaly detection result, wherein the anomaly detection can include detection of whether the data format is abnormal or whether the data content is abnormal, and the anomaly detection result indicates whether the request response information has an abnormal result, so that it can be determined whether the anomaly check detection result is a request response abnormal result. The request response abnormal result indicates that the request response information has an abnormal result, that is, when the anomaly check detection result is not a request response abnormal result, it means that the request response information does not have an abnormality. At this time, the step of generating a request processing result corresponding to the model request information based on the request response information can be directly executed; and when the anomaly check detection result is a request response abnormal result, it means that the request response information has an abnormality. At this time, the request response adjustment information can be obtained. The request response adjustment information represents information used to adjust the request response information, and the request response adjustment information may be pre-configured or adjustment information input by the operation and maintenance personnel according to the abnormal result of the request response. This embodiment does not specifically limit this. The request response adjustment information may then be used to adjust the request response information to obtain the adjusted request response information, wherein the specific adjustment method may be an adjustment of the data format or an adjustment of the data content. Subsequently, the request processing result may be generated based on the adjusted request response information. This may avoid the problem that when an abnormality is received in the request response information, the client is unable to execute subsequent business steps based on the request response information. The request response adjustment information is used to adjust the request response information, and the correct request processing result may be returned to the client, thereby improving accuracy.

[0082] In one example, the model request information is: {"prompt": "Hello,world!", "model": "A"}. At this time, the target request information output to the target model is: {"prompt": "‘你好’,translated into English", "model": "A"}, and the request response information output by the target model is: {"answer": "Good day"}. At this time, when performing anomaly detection on the request response information, it can be determined that there is an error in the data content of the request response information. Thus, when the anomaly detection result is the request response anomaly result, the anomaly prompt information corresponding to the request response anomaly result can be requested, so that the operation and maintenance personnel can input the corresponding adjustment information according to the request response anomaly result, thereby obtaining the request response adjustment information, and using the request response adjustment information to adjust the request response information. For example, the operation and maintenance personnel input the request response adjustment information through the manual repair interface, and the adjusted request response information after adjusting the request response information is: {"answer": "Hello!"}, so as to achieve the purpose of adaptive adjustment when there is an anomaly in the request response information. Of course, the above is only an example, and specific implementation can be adaptively adjusted according to the scenario and requirements. This embodiment does not make specific limitations on this.

[0083] In an alternative embodiment of the present application, after step S110 of obtaining the model request information sent by the client, it may specifically further include the following sub-steps: obtaining the cached request information corresponding to the client; determining whether the model request information matches the cached request information; when the model request information matches the cached request information, obtaining the cached response information corresponding to the cached request information, and generating a request processing result corresponding to the model request information based on the cached response information; when the model request information does not match the cached request information, the step of performing request shunting based on the model request information to obtain request allocation information is executed.

[0084] In this embodiment, after obtaining the model request information, the cache request information corresponding to the client can be obtained. The cache request information can represent the request information cached in the database corresponding to the client, that is, the cache request information corresponding to the client can include multiple request information, and it is determined whether the model request information matches the cache request information. The specific determination method can be to convert the model request information into Message Digest Algorithm 5 (MD5) information, and compare the MD5 information with the MD5 information recorded in each cache request information. If the MD5 information is consistent, it can be considered that the model request information matches the cache request information. If the MD5 information is inconsistent, it can be considered that the model request information does not match the cache request information. Therefore, when the model request information does not match the cache request information, it means that the request information cached in the database corresponding to the client does not exist. The request information that matches the model request information does not exist. At this time, the content of the subsequent steps S120-S150 can be executed. That is, when the model request information does not match the cache request information, the request based on the model request information can continue to be executed. The steps of diverting the model request information and obtaining the request allocation information are as follows; when the model request information matches the cache request information, it indicates that the request information cached in the client's corresponding database exists and matches the model request information. At this time, the cache response information corresponding to the cache request information can be obtained from the client's corresponding database cache, and the request processing result corresponding to the model request information can be generated directly based on the cache response information, so that repeated request information and response information can be stored through the database cache, so that when the same model request information appears subsequently, the cache response information corresponding to the cache request information can be directly obtained from the database cache, which improves the response speed and saves language model resources.

[0085] In an optional embodiment of the present application, step S140 responds to the model request information through the target model to obtain request response information, which may specifically include the following sub-steps: determining the target priority of the model request information based on the queue priority information of the target model; adding the model request information to the request queue of the target model based on the target priority; and obtaining the request response information corresponding to the model request information based on the request queue.

[0086] In this embodiment, after determining the target model, the queue priority information of the target model can be obtained. The queue priority information represents the priority information of the request queue configuration corresponding to the target model, for example, the priority of a business request is greater than that of a test request, and the priority of a type A business request is greater than that of a type B business request, etc.; thereby, the target priority of the model request information can be determined based on the queue priority information of the target model, that is, the priority of the current model request information in the queue priority information can be determined, and then the model request information can be added to the request queue of the target model based on the target priority. Specifically, the adding method can determine the target position of the request queue according to the target priority, and add the model request information to the target position of the request queue, so that the target model can be processed in sequence according to the request queue, thereby obtaining the request response information output by the target model in response to the model request information.

[0087] In one example, since each language model has an independent priority queue, the target priority corresponding to the current model request information can be determined based on the queue priority information of the target model, so that the model request information can be added to the target position corresponding to the target priority in the request queue of the target model based on the target priority, that is, the model request information is sorted according to the priority queue of the language model; in addition, in the process of obtaining the request response information corresponding to the model request information according to the request queue, the number of concurrent requests corresponding to the target model can be obtained. The number of concurrent requests indicates the number of concurrent requests that can be processed simultaneously when the target model processes the requests. At this time, when the target model processes the requests in sequence according to the request queue In the process, requests within the concurrent number of requests can be processed simultaneously, and requests exceeding the concurrent number can return the queue status to the client, so that the client can poll to obtain the queue status until the request is processed; for example, the queue priority information corresponding to the target model includes priority A, priority B, and priority C in descending order, that is, the request queue of the target model is sorted from front to back in order of requests corresponding to priority A, requests corresponding to priority B, and requests corresponding to priority C. When the concurrent number of requests is 3, the first 3 requests in the request queue can be processed simultaneously, and the requests after the first 3 can return the queue status to the client, so that the client can poll to obtain the queue status until the request is processed.

[0088] like Figure 2 As shown, the present application also discloses an embodiment, providing a model request processing device, which may specifically include the following modules:

[0089] The acquisition module 210 is used to acquire the model request information sent by the client;

[0090] A request distribution module 220 is used to distribute requests based on the model request information to obtain request allocation information;

[0091] A determination module 230, configured to determine a target model to be assigned to the model request information based on the request assignment information;

[0092] A response module 240 is used to respond to the model request information through the target model to obtain request response information;

[0093] The generation module 250 is used to generate a request processing result corresponding to the model request information based on the request response information.

[0094] In an optional embodiment of the present application, the request distribution module 220 may include:

[0095] A first acquisition unit, configured to acquire allocation ratio information of a preset model based on the model request information;

[0096] The first request distribution unit is used to use the allocation ratio information to perform request distribution on the model request information to generate the request distribution information.

[0097] In an optional embodiment of the present application, the first acquiring unit may include:

[0098] An acquisition subunit, configured to acquire preset ratio information and request queue length information of the preset model based on the model request information;

[0099] The ratio adjustment subunit is used to adopt the request queue length information to adjust the preset ratio information to generate the allocation ratio information.

[0100] In an optional embodiment of the present application, the first request distribution unit may include:

[0101] A first determining subunit, configured to determine an allocation ratio value corresponding to each of the preset models according to the allocation ratio information;

[0102] A first generating subunit, configured to generate a random number based on the model request information;

[0103] A second determining subunit is used to determine the allocation ratio value corresponding to the random number as a target allocation ratio value;

[0104] A third determining subunit is used to determine the preset model corresponding to the target allocation ratio value as the target model;

[0105] The second generating subunit is used to generate the request allocation information according to the target model.

[0106] In an optional embodiment of the present application, the generating module 250 may include:

[0107] An anomaly detection unit, used to perform an anomaly detection on the request response information to obtain an anomaly detection result;

[0108] A second acquisition unit is used to acquire request response adjustment information when the abnormality check detection result is a request response abnormality result;

[0109] an adjusting unit, configured to adjust the request response information using the request response adjustment information to obtain adjusted request response information;

[0110] A generating unit is used to generate the request processing result based on the adjusted request response information.

[0111] In an optional embodiment of the present application, the model request processing device may further include:

[0112] A cache module, used to obtain cache request information corresponding to the client;

[0113] A matching determination module, used to determine whether the model request information matches the cache request information;

[0114] A cache response module, configured to obtain cache response information corresponding to the cache request information when the model request information matches the cache request information, and generate a request processing result corresponding to the model request information based on the cache response information;

[0115] The execution module is used to execute the step of performing request diversion based on the model request information to obtain request allocation information when the model request information does not match the cache request information.

[0116] In an optional embodiment of the present application, the response module 240 may include:

[0117] a determining unit, configured to determine a target priority of the model request information according to the queue priority information of the target model;

[0118] A queue unit, configured to add the model request information to a request queue of the target model based on the target priority;

[0119] A third acquisition unit is used to acquire the request response information corresponding to the model request information according to the request queue.

[0120] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, which will not be repeated here.

[0121] like Figure 3As shown, an embodiment of the present application provides an electronic device, including a processor 310, a communication interface 320, a memory 330 and a communication bus 340, wherein the processor 310, the communication interface 320, and the memory 330 communicate with each other through the communication bus 340;

[0122] Memory 330, for storing computer programs;

[0123] In one embodiment of the present application, the processor 310, when executing the program stored in the memory 330, implements the model request processing method provided by any of the aforementioned method embodiments, by obtaining the model request information sent by the client, and performing request diversion based on the model request information to obtain request allocation information, so as to determine the target model corresponding to the model request information based on the request allocation information, which can play a role in adaptively allocating different target models to the model request information when docking with multiple language models, thereby achieving the purpose of docking with multiple language models at the same time, responding to the model request information through the target model, obtaining request response information, and then generating a request processing result corresponding to the model request information based on the request response information; thereby, the model request information can be allocated to the accurate target model, solving the problem that different language models in the existing related technology are not compatible with concurrent use, and can achieve the effect of compatible and concurrent use of different language models.

[0124] The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the model request processing method provided by any of the aforementioned method embodiments is implemented, by obtaining the model request information sent by the client, and performing request diversion based on the model request information to obtain request allocation information, so as to determine the target model corresponding to the model request information based on the request allocation information. This can play a role in adaptively allocating different target models to the model request information when docking with multiple language models, thereby achieving the purpose of docking with multiple language models at the same time, responding to the model request information through the target model to obtain request response information, and then generating a request processing result corresponding to the model request information based on the request response information; thereby, the model request information can be allocated to an accurate target model, solving the problem that different language models in the existing related technology are not compatible with concurrent use, and can achieve the effect of compatible and concurrent use of different language models.

[0125] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0126] The above describes specific embodiments of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0127] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A model request processing method, characterized in that: include: Get the model request information sent by the client; Request distribution is performed based on the model request information to obtain request allocation information; Based on the request allocation information, determining a target model corresponding to the model request information; Responding to the model request information through the target model to obtain request response information; Based on the request response information, a request processing result corresponding to the model request information is generated.

2. The model request processing method according to claim 1, characterized in that: The request distribution based on the model request information to obtain the request allocation information includes: Based on the model request information, obtain the allocation ratio information of the preset model; The allocation ratio information is used to perform request diversion on the model request information to generate the request allocation information.

3. The model request processing method according to claim 2, characterized in that: The obtaining of the allocation ratio information of the preset model based on the model request information includes: Based on the model request information, obtaining preset ratio information and request queue length information of the preset model; The request queue length information is used to proportionally adjust the preset ratio information to generate the allocation ratio information.

4. The model request processing method according to claim 2, characterized in that: The step of using the allocation ratio information to perform request diversion on the model request information to generate the request allocation information includes: Determine the allocation ratio value corresponding to each of the preset models according to the allocation ratio information; Generate a random number based on the model request information; Determine the allocation ratio value corresponding to the random number as a target allocation ratio value; Determine the preset model corresponding to the target allocation ratio value as the target model; The request allocation information is generated according to the target model.

5. The model request processing method according to claim 1, characterized in that: The generating, based on the request response information, a request processing result corresponding to the model request information, comprises: Performing anomaly detection on the request response information to obtain an anomaly detection result; When the abnormality check detection result is a request response abnormality result, obtaining request response adjustment information; Using the request response adjustment information to adjust the request response information to obtain adjusted request response information; The request processing result is generated based on the adjusted request response information.

6. The model request processing method according to claim 1, characterized in that: After obtaining the model request information sent by the client, the method further includes: Obtaining cache request information corresponding to the client; Determining whether the model request information matches the cache request information; When the model request information matches the cache request information, obtaining cache response information corresponding to the cache request information, and generating a request processing result corresponding to the model request information based on the cache response information; When the model request information does not match the cache request information, a step of performing request diversion based on the model request information to obtain request allocation information is performed.

7. The model request processing method according to claim 1, characterized in that: The step of responding to the model request information through the target model to obtain request response information includes: Determining the target priority of the model request information according to the queue priority information of the target model; Based on the target priority, adding the model request information to a request queue of the target model; According to the request queue, the request response information corresponding to the model request information is obtained.

8. A model request processing device, characterized in that: include: The acquisition module is used to obtain the model request information sent by the client; A request distribution module is used to distribute requests based on model request information and obtain request allocation information; A determination module, configured to determine a target model to be assigned to the model request information based on the request assignment information; A response module, used for responding to the model request information through the target model to obtain request response information; A generation module is used to generate a request processing result corresponding to the model request information based on the request response information.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; The processor is used to implement the model request processing method described in any one of claims 1 to 7 when executing the program stored in the memory.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the model request processing method according to any one of claims 1 to 7 is implemented.

Images