Large model request traffic scheduling method based on container cluster management

Abstract

The invention provides a large model request traffic scheduling method, system and device based on container cluster management, and belongs to the technical field of traffic scheduling, and the method comprises the steps: obtaining the request content of a user service request, generating a request abstract, carrying out the vector coding processing according to the request abstract and user information, and generating a request identifier; monitoring a task processing condition of an internal execution unit in the large model through a container cluster monitoring interface, and obtaining a unit task quantity and a unit task identifier; matching is carried out according to the request identifier and the unit task identifier, the matched identifier is processed by the unit, and the user service request is transmitted to the unit for processing; and selecting a to-be-processed request unit according to the unit task load when the matched internal execution unit is not obtained, and transmitting the user service request to the to-be-processed request unit for request processing. According to the technical scheme, the utilization rate of resources is optimized, the dialogue efficiency of the large model service is improved, and then load balancing and dynamic scheduling of the large model service to the request traffic are achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of traffic scheduling, and in particular to a large model request traffic scheduling method, system and device based on container cluster management. Background Art

[0002] A large model refers to a large-parameter model trained using large-scale data and powerful computing power. It has a high degree of versatility and generalization ability and can be widely used in natural language processing, image recognition, speech recognition and other fields.

[0003] The GPU resource utilization and time consumption of the large model service are similar when processing a single task at a time and processing multiple tasks in parallel at a time. In traditional technology, the large model service processes a single task in sequence, resulting in a waste of GPU resources. In addition, due to the huge amount of calculation of the large model service, it takes a long time to process a single request, making the processing efficiency of user service requests low.

[0004] In a multi-round multi-conversation scenario, a large model service usually calculates the context of the conversation and caches the results. When a request for the same content is processed again, the cached information can be called without calculation. If the request is processed through different internal execution units in the large model, the cached information cannot be called, resulting in an increase in the amount of calculation for the large model service.

[0005] Therefore, a large model request traffic scheduling method, system and device based on container cluster management are proposed. Summary of the invention

[0006] In order to solve the above technical problems, the present invention provides a large model request traffic scheduling method, system and device based on container cluster management, which is used to solve the problem that the large model service in traditional technology has low efficiency in processing service requests and cannot fully utilize cache information.

[0007] The present invention provides a large model request flow scheduling method based on container cluster management, comprising:

[0008] Get user service request;

[0009] Parsing the request content of the user service request, generating a request summary, obtaining user information that initiates the user request, performing vector encoding processing according to the request summary and the user information, and generating a request identifier;

[0010] Monitor the task processing of the internal execution units in the large model through the container cluster monitoring interface to obtain the unit task volume and unit task identifier;

[0011] According to the request identifier, the unit task identifier corresponding to the internal execution unit is matched, the internal execution unit corresponding to the unit task identifier matched with the request identifier is obtained as an identifier processing unit, and the user service request is transmitted to the identifier processing unit for request processing;

[0012] When the internal execution unit corresponding to the unit task identifier matching the request identifier is not obtained, the internal execution unit with an idle unit task amount is selected as the pending request unit according to the unit task amount of the internal execution units in the large model, and the user service request is transmitted to the pending request unit for request processing.

[0013] Preferably, the present invention provides a large model request traffic scheduling method based on container cluster management, the steps of: monitoring the task processing status of the internal execution unit in the large model through the container cluster monitoring interface, obtaining the unit task amount and the unit task identifier; and then, further comprising:

[0014] When the container cluster monitoring interface monitors that the unit task volume of the internal execution unit in the large model is not idle, the user service request is temporarily suspended;

[0015] Monitor the task processing status of the internal execution unit in the large model according to the container cluster monitoring interface, detect whether the internal execution unit currently has a batch service request to process, and use the internal execution unit currently processing the batch service request as a combined processing unit;

[0016] Combining and processing several suspended user service requests according to the request combination strategy to obtain a combined service request;

[0017] When the unit task volume of the combined processing unit is in an idle state, the combined service request is transmitted to the combined processing unit for processing, a combined request processing result is obtained and returned to the user terminal device, and the unit task volume of the combined processing unit is updated;

[0018] When it is detected that the current internal execution unit has no batch service requests to process, the unit task volume of the internal execution unit in the large model is monitored in real time through the container cluster monitoring interface, the internal execution unit monitored to be idle is used as a pending combined request unit, and the combined service request is transmitted to the pending combined request unit for request processing.

[0019] Preferably, the present invention provides a large model request traffic scheduling method based on container cluster management, the steps of: parsing the request content of the user service request, generating a request summary, obtaining user information that initiates the user request, performing vector encoding processing according to the request summary and the user information, and generating a request identifier; comprising:

[0020] Build an initial summary generation model;

[0021] Obtain sample request content and sample standard summary in the sample set; the initial summary generation model performs vector encoding through an encoder according to the sample request content to obtain a sample request vector; perform feature attention adjustment on the sample request vector through an attention mechanism layer, and the decoder outputs a sample summary of a set feature length;

[0022] Calculating a model loss function according to the sample summary and the sample standard summary;

[0023]

[0024] Among them, L is the comprehensive loss function of the model, M is the sample summary length, z Si is the sample standard summary Z s The i-th standard summary vector in z i is the i-th summary vector in sample summary Z, z 1～i-1 is the 1st to i-1th summary vector in the sample summary Z, X is the sample request vector, μ is the loss balance parameter, P T (Z si |z i ) is the evaluation summary vector z i With the standard summary vector Z si The probability distribution value, p(z i |z 1～i-1 , X) is the i-th summary vector z in the sample summary obtained according to the 1st to i-1th summary vectors in the sample summary Z and the sample request vector X i The probability value of sim(Z,Z s ) is used to evaluate the sample summary Z and the sample standard summary Z s similarity;

[0025]

[0026] Among them, θ is the probability adjustment parameter, z Si is the i-th standard summary vector in the sample standard summary, and N is the number of vector tables;

[0027]

[0028] Where |X| is the length of the sample request vector, and cos is the cosine function;

[0029] Optimizing the control parameters of the decoder according to the model loss function until the model loss function meets the threshold requirement, thereby obtaining a summary generation model;

[0030] Obtaining the test request content and the test standard summary in the test set, processing the test request content through the summary generation model, and generating a test summary;

[0031] According to the test summary and the test standard summary, the summary generation model is evaluated based on a model evaluation function, and when the evaluation index meets the preset requirements, the summary generation model is used to generate a request summary of the user service request;

[0032]

[0033] Among them, R is the model evaluation function, σ is the evaluation balance parameter, PT is the common vector set of the test summary and the test standard summary, r is the vector length of the test standard summary, and v is the vector length of the test summary.

[0034] Preferably, the present invention provides a large model request traffic scheduling method based on container cluster management, the steps of: parsing the request content of the user service request, generating a request summary, obtaining user information that initiates the user request, performing vector encoding processing according to the request summary and the user information, and generating a request identifier; and further comprising:

[0035] Perform vector processing on the request summary and the user information to obtain a request vector and a user identification vector;

[0036] Construct a request identification template, set the symbol identification bit, timestamp identification bit, user identification bit, request identification bit and sequence number;

[0037] According to the request vector and the user identification vector, generating a request identification based on the request identification template; comprising:

[0038] A default symbol is used as the symbol identification bit, and vector encoding is performed according to the time when the request summary is generated to obtain a timestamp identification;

[0039] Obtaining the number of vector bits of the user identification vector, when the number of vector bits is less than the number of bits of the user identification bit, expanding the user identification vector by filling zero vector elements, and using the expanded user identification vector as the user identification bit; when the number of vector bits is greater than the number of bits of the user identification bit, encoding and compressing the user identification vector so that the number of vector bits of the user identification vector is the same as the number of bits of the user identification bit, and using the compressed user identification vector as the user identification bit;

[0040] According to the number of vector bits of the request vector, adjusting the number of bits and filling in the request identification bit;

[0041] When a serial number is generated for the first time, a first serial identifier is generated as the serial number through a pseudo-random number sequence, and a serial number for a subsequent request identifier is generated by performing serial number increment processing based on the first serial identifier.

[0042] Preferably, the present invention provides a large model request traffic scheduling method based on container cluster management, the steps of: monitoring the task processing status of the internal execution unit in the large model through the container cluster monitoring interface, obtaining the unit task amount and the unit task identifier; including:

[0043] According to the internal execution units in the large model, a replica component is created based on a container cluster management platform; the replica component corresponds to the internal execution unit one by one;

[0044] The task processing status of the internal execution unit is read through the replica component, and synchronously transmitted to the container cluster monitoring interface;

[0045] A filtering condition is set on the container cluster monitoring interface to filter the acquired task processing status to obtain a unit task amount and a unit task identifier.

[0046] Preferably, the present invention provides a large model request traffic scheduling method based on container cluster management, the steps of: matching the unit task identifier corresponding to the internal execution unit according to the request identifier, obtaining the internal execution unit corresponding to the unit task identifier matching the request identifier as an identifier processing unit, and transmitting the user service request to the identifier processing unit for request processing; including:

[0047] Call the distributed cache service to obtain the mapping relationship between the request identifier and the user service request;

[0048] Acquire an internal execution unit corresponding to a unit task identifier matching the request identifier as an identifier processing unit to process the user service request;

[0049] When a mapping relationship between the request identifier and the user service request is not obtained in the distributed cache service, a mapping relationship between the request identifier and the unit task identifier is created in the distributed cache service; including:

[0050] Obtaining an operation node of the distributed cache service, creating an "<request identifier, unit task identifier>" identifier pair in the operation node, and updating an operation log and a version number of the operation node;

[0051] Obtain all operation nodes that execute the distributed cache service, detect the consistency of the version numbers of all operation nodes, and when it is detected that the version numbers of the operation nodes are inconsistent, update all operation nodes that execute the distributed cache service with the identifier stored in the operation node corresponding to the latest version number, and synchronously update the operation logs and version numbers of other operation nodes.

[0052] Preferably, the present invention provides a large model request traffic scheduling method based on container cluster management, the steps of: when the internal execution unit corresponding to the unit task identifier matching the request identifier is not obtained, according to the unit task amount of the internal execution unit in the large model, selecting the internal execution unit with the unit task amount in the idle state as the request unit to be processed, and transmitting the user service request to the request unit to be processed for request processing; thereafter, also including:

[0053] When it is detected that the unit task amount of the internal execution unit in the large model is not in an idle state, the internal execution unit with the smallest task amount is selected according to the unit task amount to process the user service request, and the unit task amount of the internal execution unit is synchronously updated;

[0054] The internal execution unit updates the unit task identifier of the internal execution unit according to the request identifier of the user service request currently being processed;

[0055] After the internal execution unit completes the request processing for the user service request, the request processing result is obtained and returned to the user terminal device that initiated the user service request, and the unit task volume of the internal execution unit is synchronously updated.

[0056] The present invention provides a large model request flow scheduling system based on container cluster management, comprising:

[0057] Service acquisition module, used to obtain user service requests;

[0058] A service parsing module, used to parse the request content of the user service request, generate a request summary, obtain user information that initiates the user request, perform vector encoding processing according to the request summary and the user information, and generate a request identifier;

[0059] The model monitoring module is used to monitor the task processing of the internal execution units in the large model through the container cluster monitoring interface, and obtain the unit task volume and unit task identifier;

[0060] A request transmission module, used to match the unit task identifier corresponding to the internal execution unit according to the request identifier, obtain the internal execution unit corresponding to the unit task identifier matching the request identifier as an identifier processing unit, and transmit the user service request to the identifier processing unit for request processing;

[0061] The request transmission module is also used to select an internal execution unit with an idle unit task amount as a pending request unit according to the unit task amount of the internal execution units in the large model when the internal execution unit corresponding to the unit task identifier matching the request identifier is not obtained, and transmit the user service request to the pending request unit for request processing.

[0062] The present invention provides a large model request flow scheduling device based on container cluster management, comprising:

[0063] The signal processor executes the method according to any one of claims 1 to 7, and processes the acquired user service request through the internal execution unit in the large model.

[0064] Compared with traditional technologies, the beneficial effects of the present invention are: a large model request traffic scheduling method, system and device based on container cluster management, matching the request identifier of the user service request with the unit task identifier of the internal execution unit of the large model, transmitting the user service request to the matched internal execution unit for request processing, and realizing the transmission of user service requests with the same request content to the same internal execution unit for processing, so as to fully call the cache information and reduce the calculation amount of the large model service; when the identification processing unit is not obtained, selecting the idle internal execution unit for request processing to improve the processing efficiency of the user service request; the above technical scheme processes the user service request with the same request content by the internal execution unit, reduces the repeated calculation of the context in the multi-round multi-dialogue scenario, improves the utilization rate of the cache information, and selects the idle internal execution unit to perform multi-task parallel processing on the user service request, so as to optimize the utilization rate of GPU resources and improve the dialogue efficiency of the large model service, thereby realizing the load balancing and dynamic scheduling of the request traffic of the large model service.

[0065] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings.

[0066] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 A schematic diagram of a flow chart of a large model request traffic scheduling method based on container cluster management provided by the present invention;

[0068] Figure 2A schematic diagram of the structure of a large model request flow scheduling system based on container cluster management provided by the present invention. DETAILED DESCRIPTION

[0069] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0070] Embodiment 1:

[0071] The embodiment of the present invention provides a large model request flow scheduling method based on container cluster management, referring to Figure 1 ,include:

[0072] Get user service request;

[0073] Parse the request content of the user service request, generate a request summary, obtain the user information that initiated the user request, perform vector encoding processing based on the request summary and user information, and generate a request identifier;

[0074] Monitor the task processing of the internal execution units in the large model through the container cluster monitoring interface to obtain the unit task volume and unit task identifier;

[0075] According to the request identifier, the unit task identifier corresponding to the internal execution unit is matched, the internal execution unit corresponding to the unit task identifier matched with the request identifier is obtained as the identification processing unit, and the user service request is transmitted to the identification processing unit for request processing;

[0076] When the internal execution unit corresponding to the unit task identifier matching the request identifier is not obtained, the internal execution unit with idle unit task volume is selected as the pending request unit according to the unit task volume of the internal execution units in the large model, and the user service request is transmitted to the pending request unit for request processing.

[0077] In the above embodiment, the obtained user service request is parsed to generate a request summary, and the user information that initiates the user request is obtained, vector encoding processing is performed, and a request identifier is generated; the task processing status of the internal execution unit in the large model is monitored through the container cluster monitoring interface, the unit task amount and the unit task identifier are obtained, and the request identifier is matched with the unit task identifier, and the internal execution unit corresponding to the unit task identifier that matches the request identifier is used as an identifier processing unit, and the user service request is transmitted to the identifier processing unit for request processing;

[0078] When the internal execution unit corresponding to the unit task identifier matching the request identifier is not obtained, the internal execution unit with idle unit task volume is selected as the pending request unit according to the unit task volume of the internal execution unit, and the user service request is transmitted to the pending request unit for request processing.

[0079] In the above embodiments, the large model is provided with a plurality of internal execution units in the form of a container cluster to process user service requests.

[0080] In one embodiment, the request identifier is sent to a matching internal execution unit for request processing, thereby avoiding repeated calculations of the same content and making full use of the cache to improve cache retention and hit rates. Compared with the traditional technology that does not distinguish between request contents, user service requests are distributed to different internal execution units. There is no cache-related information in the internal execution unit, which increases the amount of calculation and results in a lower cache hit rate.

[0081] In one embodiment, the task processing status of the internal execution units in the large model is monitored through the container cluster monitoring interface, and user service requests are distributed to the internal execution units with the least tasks to improve the load utilization of the large model service, make full use of the resources of the large model service, and avoid the impact of the length of the conversation on the request.

[0082] The beneficial effects of the above technology are: according to the request identifier of the user service request and the unit task identifier of the internal execution unit of the large model, the user service request is transmitted to the matched internal execution unit for request processing, thereby realizing the transmission of user service requests with the same request content to the same internal execution unit for processing, so as to fully call the cache information and reduce the calculation amount of the large model service; when the identification processing unit is not obtained, the idle internal execution unit is selected for request processing to improve the processing efficiency of the user service request; compared with the traditional technology, the above method processes the user service request with the same request content by the internal execution unit, reduces the repeated calculation of the context in the multi-round and multi-dialogue scenario, improves the utilization rate of the cache information, and selects the idle internal execution unit to perform multi-task parallel processing on the user service request, so as to optimize the utilization rate of GPU resources and improve the dialogue efficiency of the large model service, thereby realizing the load balancing and dynamic scheduling of the request traffic of the large model service.

[0083] Embodiment 2:

[0084] The embodiment of the present invention provides a large model request flow scheduling method based on container cluster management, comprising the steps of: monitoring the task processing status of internal execution units in the large model through a container cluster monitoring interface, obtaining the unit task volume and unit task identifier; and then, further comprising:

[0085] When the container cluster monitoring interface detects that the unit task volume of the internal execution unit in the large model is not idle, the user service request is suspended;

[0086] Monitor the task processing status of the internal execution units in the large model according to the container cluster monitoring interface, detect whether the current internal execution unit has batch service requests to process, and use the internal execution unit currently processing the batch service requests as a combined processing unit;

[0087] Combining and processing several suspended user service requests according to the request combination strategy to obtain a combined service request;

[0088] When the unit task volume of the processing unit to be combined is in an idle state, the combined service request is transmitted to the combined processing unit for processing, the combined request processing result is obtained and returned to the user terminal device, and the unit task volume of the combined processing unit is updated;

[0089] When it is detected that the current internal execution unit has no batch service requests to process, the unit task volume of the internal execution units in the large model is monitored in real time through the container cluster monitoring interface, the internal execution units that are monitored to be idle are treated as pending combined request units, and the combined service requests are transmitted to the pending combined request units for request processing.

[0090] In the above embodiment, when the container cluster monitoring interface monitors that the unit task volume of the internal execution unit in the large model is not idle, the user service request is suspended, and several suspended user service requests are combined and processed according to the request combination strategy to obtain a combined service request;

[0091] Monitor the task processing status of the internal execution units in the large model according to the container cluster monitoring interface, detect whether the current internal execution unit has batch service requests to process, and use the internal execution unit currently processing batch service requests as the combined processing unit; when the unit task volume of the processing unit to be combined is idle, transmit the combined service request to the combined processing unit for processing, obtain the combined request processing result and return it to the user terminal device, and update the unit task volume of the combined processing unit;

[0092] When it is detected that the current internal execution unit has no batch service requests to process, the unit task volume of the internal execution units in the large model is monitored in real time through the container cluster monitoring interface, the internal execution units that are monitored to be idle are treated as pending combined request units, and the combined service requests are transmitted to the pending combined request units for request processing.

[0093] In one embodiment, the kubernetes container management platform provides a service-level load balancing and service discovery mechanism, which can only ensure the relative balance of the four-layer link level. In the normal http protocol, in order to reduce the four-layer link construction overhead, a long link mode is usually used, that is, a link will process multiple requests one by one. In this mode, the balance of the http request level cannot be ensured, and it is impossible to optimize the scheduling for specific scenarios (such as multi-round dialogues). Limited by the processing capacity of the large model service, when the large model service http request level is unbalanced, the GPU resources cannot be fully utilized because each request is independent, resulting in a long idle time of GPU resources, resulting in resource waste and low efficiency. The above-mentioned large model request traffic scheduling method based on container cluster management can effectively improve the utilization efficiency of GPU resources. Through the traffic scheduling service, the seven-layer traffic is directly distributed to the internal execution unit with the least ongoing tasks, which can avoid the problem of imbalance above four layers in the long link mode.

[0094] The beneficial effect of the above technology is that when it is monitored that the unit task volume of the internal execution unit in the large model is not idle, the user service requests that are suspended are merged and processed, and the combined service request is obtained. The request is processed by detecting the combined processing unit or the internal execution unit in the idle state, thereby realizing batch processing of user service requests, and giving priority to the combined processing unit to improve the processing efficiency of the combined service request.

[0095] Embodiment 3:

[0096] The embodiment of the present invention provides a large model request traffic scheduling method based on container cluster management, comprising the steps of: parsing the request content of a user service request, generating a request summary, obtaining user information that initiates the user request, performing vector encoding processing according to the request summary and the user information, and generating a request identifier; comprising:

[0097] Build an initial summary generation model;

[0098] The sample request content and sample standard summary in the sample set are obtained; the initial summary generation model performs vector encoding through the encoder according to the sample request content to obtain the sample request vector; the sample request vector is subjected to feature attention adjustment through the attention mechanism layer, and the decoder outputs the sample summary of the set feature length;

[0099] Calculate the model loss function based on the sample summary and the sample standard summary;

[0100]

[0101] Among them, L is the comprehensive loss function of the model, M is the sample summary length, z Si is the sample standard summary Z sThe i-th standard summary vector in z i is the i-th summary vector in sample summary Z, z 1～i-1 is the 1st to i-1th summary vector in the sample summary Z, X is the sample request vector, μ is the loss balance parameter, P T (Z si |z i ) is the evaluation summary vector z i With the standard summary vector Z si The probability distribution value, p(z i |z 1～i-1 , X) is the i-th summary vector z in the sample summary obtained according to the 1st to i-1th summary vectors in the sample summary Z and the sample request vector X i The probability value of sim(Z,Z s ) is used to evaluate the sample summary Z and the sample standard summary Z s similarity;

[0102]

[0103] Among them, θ is the probability adjustment parameter, z Si is the i-th standard summary vector in the sample standard summary, and N is the number of vector tables;

[0104]

[0105] Where |X| is the length of the sample request vector, and cos is the cosine function;

[0106] Optimize the control parameters of the decoder according to the model loss function until the model loss function meets the threshold requirement, and obtain the summary generation model;

[0107] Obtain the test request content and test standard summary in the test set, process the test request content through the summary generation model, and generate a test summary;

[0108] According to the test summary and the test standard summary, the summary generation model is evaluated based on the model evaluation function, and when the evaluation index meets the preset requirements, the summary generation model is used to generate the request summary of the user service request;

[0109]

[0110] Among them, R is the model evaluation function, σ is the evaluation balance parameter, PT is the common vector set of the test summary and the test standard summary, r is the vector length of the test standard summary, and v is the vector length of the test summary.

[0111] In the above embodiments, an initial summary generation model is constructed, a sample summary is generated according to the sample request content in the sample set, the deviation between the sample summary and the sample standard summary is calculated based on the model loss function, the initial summary generation model is optimized, and the summary generation model is obtained when the model loss function meets the threshold requirement; according to the test request content in the test set, a test summary is obtained based on the summary generation model, and the test summary and the test standard summary are analyzed through the model evaluation function to evaluate the summary generation model, and when the evaluation index meets the preset requirements, the summary generation model is used to generate the request summary of the user service request.

[0112] In the above embodiments, the initial summary generation model vector encodes the sample request content through the encoder to obtain the sample request vector; the sample request vector is feature-attentive adjusted through the attention mechanism layer, and the decoder outputs the sample summary of the set feature length.

[0113] In the above embodiments, the model loss function analyzes the deviation between the sample summary and the sample standard summary through the cross entropy loss function and the semantic regularization loss function.

[0114] In the above embodiments, the cross entropy loss function evaluates the accuracy of the sample summary by analyzing the deviation between the sample standard summary and each vector in the sample summary; the semantic regularization loss function evaluates the deviation between the sample summary and the sample standard summary by analyzing the semantic similarity.

[0115] In the above embodiment, the summary vector z is calculated based on the label smoothing technology. i With the standard summary vector Z si The probability distribution value of avoids the excessive bias of the calculated probability distribution value towards a certain vector, thereby improving the generalization ability of the model loss function.

[0116] In the above embodiment, the control parameters of the decoder are optimized according to the deviation obtained by the model loss function until the model loss function meets the threshold requirement, thereby obtaining the summary generation model.

[0117] In the above embodiments, the summary generation model is evaluated based on the model evaluation function, the overall coherence of the test summary is analyzed, and when the evaluation index meets the preset requirements, the summary generation model is used to generate the request summary of the user service request.

[0118] The beneficial effects of the above technology are: through the sample request content and sample standard summary in the sample set, the initial summary generation model is trained based on the model loss function, and the performance evaluation of the summary generation model is realized based on the model evaluation function through the test set. When the evaluation index meets the preset requirements, the summary generation model is used to generate the request summary of the user service request, thereby realizing the generation of the corresponding request summary according to the user service request.

[0119] Embodiment 4:

[0120] The embodiment of the present invention provides a large model request traffic scheduling method based on container cluster management, comprising the steps of: parsing the request content of a user service request, generating a request summary, obtaining user information initiating the user request, performing vector encoding processing according to the request summary and the user information, and generating a request identifier; and further comprising:

[0121] Perform vector processing on the request summary and user information to obtain a request vector and a user identification vector;

[0122] Construct a request identification template, set the symbol identification bit, timestamp identification bit, user identification bit, request identification bit and sequence number;

[0123] Generate a request identifier based on a request identifier template according to the request vector and the user identifier vector; including:

[0124] The default symbol is used as the symbol identification bit, and the vector encoding is performed according to the time when the request summary is generated to obtain the timestamp identification;

[0125] Obtain the number of vector digits of the user identification vector. When the number of vector digits is less than the number of user identification digits, fill the user identification vector with zero vector elements to expand it, and use the expanded user identification vector as the user identification digit; when the number of vector digits is greater than the number of user identification digits, perform encoding compression processing on the user identification vector so that the number of vector digits of the user identification vector is the same as the number of user identification digits, and use the compressed user identification vector as the user identification digit;

[0126] According to the number of bits of the request vector, adjust the number of bits and fill in the request identification bit;

[0127] When the serial number is generated for the first time, a first serial identifier is generated as the serial number through a pseudo-random number sequence, and a serial number increment process is performed based on the first serial identifier to generate a serial number for a subsequent request identifier.

[0128] In the above embodiment, the request summary and user information are vectorized to obtain a request vector and a user identification vector, and a request identification is generated based on a request identification template.

[0129] In the above embodiment, the request identification template includes a sign identification bit, a timestamp identification bit, a user identification bit, a request identification bit and a sequence number.

[0130] In the above embodiment, a default symbol is used as the symbol identification bit, vector encoding is performed according to the time when the request summary is generated to obtain a timestamp identification bit, the user identification vector is filled into the user identification bit, the request vector is filled into the request identification bit, and a first sequence identification is generated through a pseudo-random number sequence to generate a sequence number based on an incremental process.

[0131] In the above embodiments, when the number of vector bits of the user identification vector is less than the number of bits of the user identification bits, the user identification vector is expanded by filling zero vector elements, and the expanded user identification vector is used as the user identification bits; when the number of vector bits is greater than the number of bits of the user identification bits, the user identification vector is encoded and compressed so that the number of vector bits of the user identification vector is the same as the number of bits of the user identification bits, and the compressed user identification vector is used as the user identification bits.

[0132] In the above embodiment, when the serial number is generated for the first time, a first serial identifier is generated as the serial number through a pseudo-random number sequence, and in the subsequent request identifier generation process, a serial number is generated by performing an incremental process based on the previously generated serial number.

[0133] The beneficial effects of the above technology are: vector encoding processing is performed according to the request summary and user information, and the request identifier is generated based on the request identifier template; and when the number of vector bits of the user identifier vector is less than the number of bits of the user identifier bit, vector expansion is performed, and when the number of vector bits is greater than the number of bits of the user identifier bit, encoding compression processing is performed to make the number of vector bits of the user identifier vector the same as the number of bits of the user identifier bit, so that the number of request identifier bits obtained is consistent, and the first sequence identifier is generated by a pseudo-random number series for incremental processing, thereby ensuring the uniqueness of the serial number.

[0134] Embodiment 5:

[0135] The embodiment of the present invention provides a large model request flow scheduling method based on container cluster management, comprising the steps of: monitoring the task processing status of internal execution units in the large model through a container cluster monitoring interface, and obtaining unit task volume and unit task identifier; comprising:

[0136] According to the internal execution units in the large model, a replica component is created based on the container cluster management platform; the replica component corresponds to the internal execution unit one by one;

[0137] The task processing status of the internal execution unit is read through the replica component and synchronously transmitted to the container cluster monitoring interface;

[0138] Filter conditions are set on the container cluster monitoring interface to filter the acquired task processing status and obtain the unit task volume and unit task identifier.

[0139] In the above embodiments, the task processing status of the internal execution unit in the large model is read through the replica component and synchronously transmitted to the container cluster monitoring interface. Based on the filtering conditions set on the container cluster monitoring interface, the obtained task processing status is filtered to obtain the unit task volume and unit task identifier.

[0140] In the above embodiments, the replica component may be implemented as a List-Watch mechanism to read the task creation, task execution, and task termination of the internal execution unit.

[0141] The beneficial effect of the above technology is that through the replica component, the task processing status of the internal execution unit in the large model is monitored, and when the task processing status is obtained by the container cluster monitoring interface, it is processed based on the filtering rules, thereby realizing the analysis of the task processing status and obtaining the unit task volume and unit task identifier of the internal execution unit.

[0142] Embodiment 6:

[0143] The embodiment of the present invention provides a large model request traffic scheduling method based on container cluster management, the steps of: matching the unit task identifier corresponding to the internal execution unit according to the request identifier, using the internal execution unit corresponding to the unit task identifier matched with the request identifier as an identifier processing unit, and transmitting the user service request to the identifier processing unit for request processing; including:

[0144] Call the distributed cache service to obtain the mapping relationship between the request identifier and the user service request;

[0145] Obtaining the internal execution unit corresponding to the unit task identifier matching the request identifier as the identifier processing unit to process the user service request;

[0146] When a mapping relationship between the request identifier and the user service request is not obtained in the distributed cache service according to the request identifier, a mapping relationship between the request identifier and the unit task identifier is created in the distributed cache service; including:

[0147] Obtain the operation node of the distributed cache service, create an "<request identifier, unit task identifier>" identifier pair on the operation node, and update the operation log and version number of the operation node;

[0148] Get all the operation nodes that execute the distributed cache service, check the consistency of the version numbers of all the operation nodes, and when it is detected that the version numbers of the operation nodes are inconsistent, update all the operation nodes that execute the distributed cache service with the identifier stored in the operation node corresponding to the latest version number, and synchronously update the operation logs and version numbers of other operation nodes.

[0149] In the above embodiments, a mapping relationship between a request identifier and a unit task identifier is stored in the distributed cache service. The request identifier and the unit task identifier are matched by calling the distributed cache service, and the internal execution unit corresponding to the unit task identifier that matches the request identifier is used as an identifier processing unit to process the user service request.

[0150] In the above embodiments, when the mapping relationship with the user service request is not obtained in the distributed cache service, a mapping relationship between the request identifier and the unit task identifier is created in the distributed cache service, an identifier pair is created through the operation node of the distributed cache service, and the operation log and version number of the operation node are updated, all operation nodes of the distributed cache service are detected, and all operation nodes of the distributed cache service are updated and executed with the identifier pair stored in the operation node corresponding to the latest version number.

[0151] The beneficial effects of the above technology are: by calling the distributed cache service, the request identifier and the unit task identifier are matched, and when the mapping relationship is not obtained in the distributed cache service, an identifier pair is created, and all operation nodes in the distributed cache service are synchronously updated based on the operation log and version number to ensure the consistency of the mapping relationship stored in the distributed cache service.

[0152] Embodiment 7:

[0153] The embodiment of the present invention provides a large model request traffic scheduling method based on container cluster management, comprising the steps of: when an internal execution unit corresponding to a unit task identifier matching a request identifier is not obtained, an internal execution unit with an idle unit task amount is selected as a request unit to be processed according to the unit task amount of the internal execution unit in the large model, and the user service request is transmitted to the request unit to be processed for request processing; thereafter, the method further comprises:

[0154] When it is detected that the unit task volume of the internal execution unit in the large model is not in an idle state, the internal execution unit with the smallest task volume is selected according to the unit task volume to process the user service request, and the unit task volume of the internal execution unit is updated synchronously;

[0155] The internal execution unit updates the unit task identifier of the internal execution unit according to the request identifier of the user service request currently being processed;

[0156] After the internal execution unit completes the request processing for the user service request, it obtains the request processing result and returns it to the user terminal device that initiated the user service request, and simultaneously updates the unit task volume of the internal execution unit.

[0157] In the above embodiments, when the internal execution unit corresponding to the unit task identifier matching the request identifier is not obtained, the unit task volume of the internal execution unit in the large model is detected. When it is detected that the unit task volume of the internal execution unit in the large model is not idle, the internal execution unit with the smallest task volume is selected to process the user service request. The internal execution units process the user service requests in sequence according to the forwarding order of the user service requests, and synchronously update the unit task volume of the internal execution units.

[0158] In the above embodiment, after the internal execution unit completes the request processing for the user service request, the request processing result is returned to the user terminal device that initiated the user service request, and the unit task volume of the internal execution unit is synchronously updated.

[0159] In the above embodiment, the internal execution unit updates the unit task identifier of the internal execution unit according to the request identifier of the user service request currently being processed.

[0160] The beneficial effects of the above technology are: by giving priority to selecting the internal execution unit with the smallest task volume to process user service requests, the processing efficiency of the large model for user service requests is improved, and the unit task volume and unit task identifier of the internal execution unit are synchronously updated to facilitate the forwarding of user service requests in subsequent processes.

[0161] Embodiment 8:

[0162] The embodiment of the present invention provides a large model request flow scheduling system based on container cluster management, referring to Figure 2 ,include:

[0163] Service acquisition module, used to obtain user service requests;

[0164] The service parsing module is used to parse the request content of the user service request, generate a request summary, obtain the user information that initiated the user request, perform vector encoding processing based on the request summary and user information, and generate a request identifier;

[0165] The model monitoring module is used to monitor the task processing of the internal execution units in the large model through the container cluster monitoring interface, and obtain the unit task volume and unit task identifier;

[0166] A request transmission module is used to match the request identifier with the unit task identifier corresponding to the internal execution unit, obtain the internal execution unit corresponding to the unit task identifier matching the request identifier as an identifier processing unit, and transmit the user service request to the identifier processing unit for request processing;

[0167] The request transmission module is also used to select an internal execution unit with an idle unit task amount as a pending request unit according to the unit task amount of the internal execution units in the large model when the internal execution unit corresponding to the unit task identifier matching the request identifier is not obtained, and transmit the user service request to the pending request unit for request processing.

[0168] In the above embodiments, the service parsing module parses the user service request obtained by the service acquisition module to generate a request summary, obtains the user information that initiates the user request, performs vector encoding processing, and generates a request identifier; the model monitoring module monitors the task processing status of the internal execution unit in the large model through the container cluster monitoring interface, obtains the unit task volume and the unit task identifier, the request transmission module matches the unit task identifier with the request identifier, and uses the internal execution unit corresponding to the unit task identifier that matches the request identifier as the identifier processing unit, and transmits the user service request to the identifier processing unit for request processing; when the request transmission module fails to obtain the internal execution unit corresponding to the unit task identifier that matches the request identifier, the request transmission module selects the internal execution unit with an idle unit task volume as the request unit to be processed according to the unit task volume of the internal execution unit, and transmits the user service request to the request unit to be processed for request processing.

[0169] The beneficial effects of the above technology are: according to the request identifier of the user service request and the unit task identifier of the internal execution unit of the large model, the user service request is transmitted to the matched internal execution unit for request processing, thereby realizing the transmission of user service requests with the same request content to the same internal execution unit for processing, so as to fully call the cache information and reduce the calculation amount of the large model service; when the identification processing unit is not obtained, the idle internal execution unit is selected for request processing to improve the processing efficiency of the user service request; compared with the traditional technology, the above system processes the user service request with the same request content by the internal execution unit, reduces the repeated calculation of the context in the multi-round and multi-dialogue scenario, improves the utilization rate of the cache information, and selects the idle internal execution unit to perform multi-task parallel processing on the user service request, so as to optimize the utilization rate of GPU resources and improve the dialogue efficiency of the large model service, thereby realizing the load balancing and dynamic scheduling of the request traffic of the large model service.

[0170] Embodiment 9:

[0171] An embodiment of the present invention provides a large model request flow scheduling device based on container cluster management, comprising:

[0172] The signal processor executes any of the above methods to process the acquired user service request through the internal execution unit in the large model.

[0173] The signal processor in the large model request traffic scheduling device based on container cluster management implements the method of any of the above embodiments when executing the program, and forwards the acquired user service request to the internal execution unit of the large model for request processing.

[0174] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A large model request traffic scheduling method based on container cluster management, characterized in that: include: Get user service request; Parsing the request content of the user service request, generating a request summary, obtaining user information that initiates the user request, performing vector encoding processing according to the request summary and the user information, and generating a request identifier; Monitor the task processing of the internal execution units in the large model through the container cluster monitoring interface to obtain the unit task volume and unit task identifier; According to the request identifier, the unit task identifier corresponding to the internal execution unit is matched, the internal execution unit corresponding to the unit task identifier matched with the request identifier is obtained as an identifier processing unit, and the user service request is transmitted to the identifier processing unit for request processing; When the internal execution unit corresponding to the unit task identifier matching the request identifier is not obtained, the internal execution unit with an idle unit task amount is selected as the pending request unit according to the unit task amount of the internal execution units in the large model, and the user service request is transmitted to the pending request unit for request processing.

2. According to the large model request flow scheduling method based on container cluster management according to claim 1, it is characterized in that: The step: monitoring the task processing status of the internal execution unit in the large model through the container cluster monitoring interface, obtaining the unit task volume and unit task identifier; and then further comprising: When the container cluster monitoring interface monitors that the unit task volume of the internal execution unit in the large model is not idle, the user service request is temporarily suspended; Monitor the task processing status of the internal execution unit in the large model according to the container cluster monitoring interface, detect whether the internal execution unit currently has a batch service request to process, and use the internal execution unit currently processing the batch service request as a combined processing unit; Combining and processing several suspended user service requests according to the request combination strategy to obtain a combined service request; When the unit task volume of the combined processing unit is in an idle state, the combined service request is transmitted to the combined processing unit for processing, a combined request processing result is obtained and returned to the user terminal device, and the unit task volume of the combined processing unit is updated; When it is detected that the current internal execution unit has no batch service requests to process, the unit task volume of the internal execution unit in the large model is monitored in real time through the container cluster monitoring interface, the internal execution unit monitored to be idle is used as a pending combined request unit, and the combined service request is transmitted to the pending combined request unit for request processing.

3. According to the large model request flow scheduling method based on container cluster management according to claim 1, it is characterized in that: The step of parsing the request content of the user service request, generating a request summary, obtaining user information that initiates the user request, performing vector encoding processing according to the request summary and the user information, and generating a request identifier includes: Build an initial summary generation model; Obtain sample request content and sample standard summary in the sample set; the initial summary generation model performs vector encoding through an encoder according to the sample request content to obtain a sample request vector; perform feature attention adjustment on the sample request vector through an attention mechanism layer, and the decoder outputs a sample summary of a set feature length; Calculating a model loss function according to the sample summary and the sample standard summary; Among them, L is the comprehensive loss function of the model, M is the sample summary length, z Si is the sample standard summary Z s The i-th standard summary vector, z i is the i-th summary vector in sample summary Z, z 1～i-1 is the 1st to i-1th summary vector in the sample summary Z, X is the sample request vector, μ is the loss balance parameter, P T (Z si |z i ) is the evaluation summary vector z i With the standard summary vector Z si The probability distribution value, p(z i |z 1～i-1 , X) is the i-th summary vector z in the sample summary obtained based on the 1st to i-1th summary vectors in the sample summary Z and the sample request vector X i The probability value of sim(Z,Z s ) is used to evaluate the sample summary Z and the sample standard summary Z s similarity; Among them, θ is the probability adjustment parameter, z Si is the i-th standard summary vector in the sample standard summary, and N is the number of vector tables; Where |X| is the length of the sample request vector, and cos is the cosine function; Optimizing the control parameters of the decoder according to the model loss function until the model loss function meets the threshold requirement, thereby obtaining a summary generation model; Obtaining the test request content and the test standard summary in the test set, processing the test request content through the summary generation model, and generating a test summary; According to the test summary and the test standard summary, the summary generation model is evaluated based on a model evaluation function, and when the evaluation index meets the preset requirements, the summary generation model is used to generate a request summary of the user service request; Among them, R is the model evaluation function, σ is the evaluation balance parameter, PT is the common vector set of the test summary and the test standard summary, r is the vector length of the test standard summary, and v is the vector length of the test summary.

4. According to the large model request flow scheduling method based on container cluster management according to claim 1, it is characterized in that: The steps include: parsing the request content of the user service request, generating a request summary, obtaining user information that initiates the user request, performing vector encoding processing according to the request summary and the user information, and generating a request identifier; Also includes: Perform vector processing on the request summary and the user information to obtain a request vector and a user identification vector; Construct a request identification template, set the symbol identification bit, timestamp identification bit, user identification bit, request identification bit and sequence number; According to the request vector and the user identification vector, generating a request identification based on the request identification template; comprising: A default symbol is used as the symbol identification bit, and vector encoding is performed according to the time when the request summary is generated to obtain a timestamp identification; Obtaining the number of vector bits of the user identification vector, when the number of vector bits is less than the number of bits of the user identification bit, expanding the user identification vector by filling zero vector elements, and using the expanded user identification vector as the user identification bit; when the number of vector bits is greater than the number of bits of the user identification bit, encoding and compressing the user identification vector so that the number of vector bits of the user identification vector is the same as the number of bits of the user identification bit, and using the compressed user identification vector as the user identification bit; According to the number of vector bits of the request vector, adjusting the number of bits and filling in the request identification bit; When a serial number is generated for the first time, a first serial identifier is generated as the serial number through a pseudo-random number sequence, and a serial number for a subsequent request identifier is generated by performing serial number increment processing based on the first serial identifier.

5. The large model request flow scheduling method based on container cluster management according to claim 1 is characterized in that: The steps include: monitoring the task processing status of the internal execution units in the large model through the container cluster monitoring interface, and obtaining the unit task volume and unit task identifier; including: According to the internal execution units in the large model, a replica component is created based on a container cluster management platform; the replica component corresponds to the internal execution unit one by one; The task processing status of the internal execution unit is read through the replica component, and synchronously transmitted to the container cluster monitoring interface; A filtering condition is set on the container cluster monitoring interface to filter the acquired task processing status to obtain a unit task amount and a unit task identifier.

6. The large model request flow scheduling method based on container cluster management according to claim 1 is characterized in that: The step: matching the unit task identifier corresponding to the internal execution unit according to the request identifier, using the internal execution unit corresponding to the unit task identifier matched with the request identifier as an identifier processing unit, and transmitting the user service request to the identifier processing unit for request processing; include: Call the distributed cache service to obtain the mapping relationship between the request identifier and the user service request; Acquire an internal execution unit corresponding to a unit task identifier matching the request identifier as an identifier processing unit to process the user service request; When a mapping relationship between the request identifier and the user service request is not obtained in the distributed cache service, a mapping relationship between the request identifier and the unit task identifier is created in the distributed cache service; including: Obtain an operation node of the distributed cache service, create an "<request identifier, unit task identifier>" identifier pair in the operation node, and update an operation log and a version number of the operation node; Obtain all operation nodes that execute the distributed cache service, detect the consistency of the version numbers of all operation nodes, and when it is detected that the version numbers of the operation nodes are inconsistent, update all operation nodes that execute the distributed cache service with the identifier stored in the operation node corresponding to the latest version number, and synchronously update the operation logs and version numbers of other operation nodes.

7. The large model request flow scheduling method based on container cluster management according to claim 1 is characterized in that: The step: when the internal execution unit corresponding to the unit task identifier matching the request identifier is not obtained, according to the unit task amount of the internal execution unit in the large model, the internal execution unit with an idle unit task amount is selected as the request unit to be processed, and the user service request is transmitted to the request unit to be processed for request processing; thereafter, it also includes: When it is detected that the unit task amount of the internal execution unit in the large model is not in an idle state, the internal execution unit with the smallest task amount is selected according to the unit task amount to process the user service request, and the unit task amount of the internal execution unit is synchronously updated; The internal execution unit updates the unit task identifier of the internal execution unit according to the request identifier of the user service request currently being processed; After the internal execution unit completes the request processing for the user service request, the request processing result is obtained and returned to the user terminal device that initiated the user service request, and the unit task volume of the internal execution unit is synchronously updated.

8. A large model request flow scheduling system based on container cluster management, characterized in that: include: Service acquisition module, used to obtain user service requests; A service parsing module, used to parse the request content of the user service request, generate a request summary, obtain user information that initiates the user request, perform vector encoding processing according to the request summary and the user information, and generate a request identifier; The model monitoring module is used to monitor the task processing of the internal execution units in the large model through the container cluster monitoring interface, and obtain the unit task volume and unit task identifier; A request transmission module, used to match the unit task identifier corresponding to the internal execution unit according to the request identifier, obtain the internal execution unit corresponding to the unit task identifier matching the request identifier as an identifier processing unit, and transmit the user service request to the identifier processing unit for request processing; The request transmission module is also used to select an internal execution unit with an idle unit task amount as a pending request unit according to the unit task amount of the internal execution units in the large model when the internal execution unit corresponding to the unit task identifier matching the request identifier is not obtained, and transmit the user service request to the pending request unit for request processing.

9. A large model request flow scheduling device based on container cluster management, characterized in that: include: The signal processor executes the method according to any one of claims 1 to 7, and processes the acquired user service request through the internal execution unit in the large model.

Images