Data processing method, device and equipment and computer readable storage medium

Abstract

The embodiment of the invention discloses a data processing method and device, equipment and a computer readable storage medium, and the method comprises the steps: determining the number of requests corresponding to n request types, and determining model adjustment parameter groups corresponding to the n request types; n is a positive integer; obtaining average processing time corresponding to the n model adjustment parameter groups; according to the request quantity corresponding to the n request types and the n average processing time, respectively distributing processors in a processor set for the n model adjustment parameter groups; and the processors in the processor set are used for loading the distributed model adjustment parameter groups and processing the service requests belonging to the request types corresponding to the loaded model adjustment parameter groups through the loaded model adjustment parameter groups and the base model. By adopting the method, the memory resource utilization rate of the processor can be improved.

Description

Technical Field

[0001] The present application relates to the field of Internet technology, and in particular to a data processing method, apparatus, device, and computer-readable storage medium. Background Art

[0002] With the rapid development of artificial intelligence (AI) and computer Internet, intelligent services can be seen everywhere, such as text graphics, intelligent conversations, text generation, etc.

[0003] In the prior art, in order to support multiple intelligent businesses, the processor that performs model reasoning needs to deploy model adjustment parameter groups corresponding to each intelligent business, such as a text-based graph parameter group, an intelligent conversation parameter group, and a text generation parameter group. However, as the business parameters continue to increase, the processor's memory is continuously occupied, which leads to a reduction in the processor's memory resources used for model reasoning, thereby reducing the model reasoning efficiency and reducing the processor's memory resource utilization. Summary of the invention

[0004] The embodiments of the present application provide a data processing method, apparatus, device, and computer-readable storage medium, which can improve the memory resource utilization of a processor.

[0005] On the one hand, an embodiment of the present application provides a data processing method, which is executed by a computer device and includes:

[0006] Determine the number of requests corresponding to n request types, and determine the model adjustment parameter groups corresponding to the n request types; n is a positive integer;

[0007] Obtain the average processing times corresponding to n model adjustment parameter groups respectively; the average processing time corresponding to a model adjustment parameter group refers to the average time for processing business requests belonging to the request type corresponding to the model adjustment parameter group by using the base model to which the model adjustment parameter group is added;

[0008] According to the number of requests corresponding to the n request types and the n average processing times, processors in the processor set are respectively allocated to the n model adjustment parameter groups; the processors in the processor set are used to load the allocated model adjustment parameter groups, and to process business requests belonging to the request types corresponding to the loaded model adjustment parameter groups through the loaded model adjustment parameter groups and the base model.

[0009] On the one hand, an embodiment of the present application provides a data processing method, including:

[0010] Obtain a service request carrying a target parameter group identifier through a request router, and obtain a request routing table according to the service request; the request routing table is generated by the parameter manager according to n parameter group identifiers and n processor information after the parameter manager allocates processors in a processor set to n model adjustment parameter groups according to the number of requests corresponding to n request types and the average processing time corresponding to n model adjustment parameter groups; n is a positive integer; one model adjustment parameter group corresponds to one request type; one parameter group identifier is used to indicate one model adjustment parameter group; one processor information is used to indicate one processor;

[0011] Query the target parameter group identifier in the request routing table to obtain candidate processor information in the request routing table that has an index relationship with the target parameter group identifier;

[0012] In the processor set, a candidate processor indicated by the candidate processor information is determined; the candidate processor is loaded with a target model adjustment parameter group indicated by the target parameter group identifier;

[0013] A target processor is determined among candidate processors, and the business request is distributed to the target processor through a request router; the target processor is used to add the loaded target model adjustment parameter group to the base model according to the target parameter group identifier carried by the business request; the base model with the target model adjustment parameter group added is used to process the business request.

[0014] In one aspect, an embodiment of the present application provides a data processing device, which runs on a computer device and includes:

[0015] A determination module, used to determine the number of requests corresponding to n request types, and determine the model adjustment parameter groups corresponding to the n request types; n is a positive integer;

[0016] An acquisition module, used to acquire average processing times corresponding to n model adjustment parameter groups respectively; the average processing time corresponding to a model adjustment parameter group refers to the average time for processing business requests belonging to the request type corresponding to the model adjustment parameter group by using a base model to which the model adjustment parameter group is added;

[0017] An allocation module is used to allocate processors in a processor set to n model adjustment parameter groups respectively according to the number of requests corresponding to the n request types and n average processing times; the processors in the processor set are used to load the allocated model adjustment parameter groups, and to process business requests belonging to the request types corresponding to the loaded model adjustment parameter groups through the loaded model adjustment parameter groups and the base model.

[0018] In a possible implementation, the allocation module allocates processors in the processor set to the n model adjustment parameter groups respectively according to the number of requests corresponding to the n request types and the n average processing times, so as to perform the following operations:

[0019] According to the number of requests corresponding to the n request types and the n average processing times, the distribution numbers corresponding to the n model adjustment parameter groups are determined; the distribution numbers corresponding to the n model adjustment parameter groups are all less than the total number of processors in the processor set;

[0020] According to the distribution quantities corresponding to the n model adjustment parameter groups, processors in the processor set are respectively allocated to the n model adjustment parameter groups; the distribution quantity corresponding to a model adjustment parameter group is used to allocate processors in the processor set to the model adjustment parameter group; the number of processors allocated to a model adjustment parameter group is the same as the distribution quantity corresponding to the model adjustment parameter group.

[0021] In a possible implementation, the allocation module determines the distribution quantities corresponding to the n model adjustment parameter groups respectively according to the request quantities corresponding to the n request types and the n average processing times, and is used to perform the following operations:

[0022] Generate constraint conditions corresponding to n model adjustment parameter groups; the constraint conditions are used to indicate conditions that need to be satisfied when the n model adjustment parameter groups are loaded into the processor set;

[0023] Obtain the initial distribution quantities corresponding to the n model adjustment parameter groups, and determine the initial total request processing time according to the request quantities corresponding to the n request types, the n average processing times, and the n initial distribution quantities;

[0024] According to the initial total request processing time and the constraint conditions, the n initial distribution quantities are adjusted to obtain the distribution quantities corresponding to the n model adjustment parameter groups.

[0025] In a possible implementation, the n model adjustment parameter groups include a first model adjustment parameter group; the first model adjustment parameter group is any one of the n model adjustment parameter groups; the processor set includes a first processor; the first processor is any one of the processors in the processor set;

[0026] The allocation module generates constraints corresponding to n groups of model adjustment parameters, which are used to perform the following operations:

[0027] The first model adjustment parameter group is loaded into at least one processor in the processor set and determined as a first constraint condition;

[0028] Obtaining the memory of the first processor, and determining a memory utilization upper limit value corresponding to the memory of the first processor according to a memory utilization threshold;

[0029] The total occupied space of the model adjustment parameter group loaded on the first processor is less than or equal to the upper limit of memory utilization, and is determined as a second constraint condition;

[0030] The first constraint condition and the second constraint condition are determined as constraint conditions corresponding to the n model adjustment parameter groups.

[0031] In a possible implementation, the n model adjustment parameter groups include a first model adjustment parameter group; the first model adjustment parameter group is any one of the n model adjustment parameter groups;

[0032] The allocation module determines the initial total request processing time according to the number of requests corresponding to the n request types, the n average processing times, and the n initial distribution numbers, and is used to perform the following operations:

[0033] Obtain a time optimizer, and input the number of requests corresponding to the n request types, the n average processing times, and the n initial distribution numbers into the time optimizer; the time optimizer includes an objective function;

[0034] By using the objective function, the number of requests corresponding to the first model adjustment parameter group and the average processing time corresponding to the first model adjustment parameter group are multiplied to obtain the request processing time corresponding to the first model adjustment parameter group;

[0035] Performing ratio processing on the request processing time corresponding to the first model adjustment parameter group and the initial distribution quantity corresponding to the first model adjustment parameter group to obtain the initial request processing time corresponding to the first model adjustment parameter group;

[0036] The initial request processing times corresponding to the n model adjustment parameter groups are summed to obtain the initial total request processing time.

[0037] In a possible implementation, the allocation module adjusts the n initial distribution quantities according to the initial total request processing time and the constraint conditions to obtain the distribution quantities corresponding to the n model adjustment parameter groups, which are used to perform the following operations:

[0038] Input constraints into the time optimizer;

[0039] In the time optimizer, the n initial distribution quantities are adjusted according to the initial total request processing time and the constraint conditions to obtain n intermediate distribution quantities corresponding to the n model adjustment parameter groups; the n intermediate distribution quantities match the constraint conditions, and the total request processing time corresponding to the n intermediate distribution quantities is less than the initial total request processing time;

[0040] When the total request processing time corresponding to the n intermediate distribution quantities is the optimal solution of the objective function, the n intermediate distribution quantities are determined as the distribution quantities corresponding to the n model adjustment parameter groups respectively.

[0041] In a possible implementation, the n model adjustment parameter groups include a first model adjustment parameter group; the first model adjustment parameter group is any one of the n model adjustment parameter groups;

[0042] The allocation module allocates processors in the processor set to the n model adjustment parameter groups respectively according to the distribution quantities corresponding to the n model adjustment parameter groups, so as to perform the following operations:

[0043] Obtaining the historical distribution quantity corresponding to the first model adjustment parameter group, comparing the historical distribution quantity with the distribution quantity corresponding to the first model adjustment parameter group, and obtaining a quantity comparison result;

[0044] According to the quantity comparison result and the second processor corresponding to the first model adjustment parameter group, a processor in the processor set is allocated to the first model adjustment parameter group; the second processor is the processor allocated to the first model adjustment parameter group according to the historical distribution quantity, and the number of the second processors is the same as the historical distribution quantity.

[0045] In a possible implementation, the allocation module allocates a processor in the processor set to the first model adjustment parameter group according to the quantity comparison result and the second processor corresponding to the first model adjustment parameter group, so as to perform the following operations:

[0046] If the quantity comparison result is that the historical distribution quantity is the same as the distribution quantity corresponding to the first model adjustment parameter group, the second processor is allocated as the processor corresponding to the first model adjustment parameter group; the second processor belongs to the processor set;

[0047] If the historical distribution quantity is less than the distribution quantity corresponding to the first model adjustment parameter group, a third processor different from the second processor is determined in the processor set, and the third processor and the second processor are allocated as processors corresponding to the first model adjustment parameter group; the number of the third processors is the same as the difference between the first number; the first number difference refers to the difference between the historical distribution quantity and the distribution quantity corresponding to the first model adjustment parameter group;

[0048] If the number of historical distributions is greater than the number of distributions corresponding to the first model adjustment parameter group, a fourth processor is determined in the second processor and assigned as the processor corresponding to the first model adjustment parameter group; the number of fourth processors is the same as the number of distributions corresponding to the first model adjustment parameter group.

[0049] In a possible implementation, the allocation module is further configured to perform the following operations:

[0050] If the number of historical distributions is less than the number of distributions corresponding to the first model adjustment parameter group, generating a loading instruction for the first model adjustment parameter group in the parameter manager;

[0051] The loading instruction is sent to the third processor through the parameter manager; the third processor is used to load the first model adjustment parameter group according to the loading instruction;

[0052] If the number of historical distributions is greater than the number of distributions corresponding to the first model adjustment parameter group, then generating an uninstallation instruction for the first model adjustment parameter group in the parameter manager;

[0053] The unloading instruction is sent to the fifth processor through the parameter manager; the fifth processor is used to unload the loaded first model adjustment parameter group according to the unloading instruction; the fifth processor includes the processors in the second processor except the fourth processor.

[0054] In a possible implementation, the n model adjustment parameter groups include a second model adjustment parameter group; the second model adjustment parameter group is any one of the n model adjustment parameter groups;

[0055] The allocation module determines the distribution quantities corresponding to the n model adjustment parameter groups according to the number of requests corresponding to the n request types and the n average processing times, and is used to perform the following operations:

[0056] Performing product processing on the number of requests corresponding to the second model adjustment parameter group and the average processing time corresponding to the second model adjustment parameter group to obtain the request processing time corresponding to the second model adjustment parameter group;

[0057] A ratio processing is performed on the request processing time corresponding to the second model adjustment parameter group and the occupied space of the second model adjustment parameter group to obtain the distribution quantity corresponding to the second model adjustment parameter group.

[0058] In a possible implementation, the allocation module allocates processors in the processor set to the n model adjustment parameter groups respectively according to the distribution quantities corresponding to the n model adjustment parameter groups, so as to perform the following operations:

[0059] The distribution quantities corresponding to the n model adjustment parameter groups are sorted from large to small to obtain the sorted n distribution quantities;

[0060] According to the sorted n distribution quantities, processors in the processor set are allocated to the n model adjustment parameter groups in sequence; wherein the number of processors in the processor set allocated to the second model adjustment parameter group is the same as the distribution quantity corresponding to the second model adjustment parameter group; the order in which the second model adjustment parameter group is allocated to the processors is the same as the order of the distribution quantity corresponding to the second model adjustment parameter group in the sorted n distribution quantities; the total occupied space of the model adjustment parameter group loaded by the processor to which the second model adjustment parameter group is allocated is less than or equal to the available memory of the processor to which the second model adjustment parameter group is allocated.

[0061] In a possible implementation, the determination module determines the number of requests corresponding to the n request types, respectively, to perform the following operations:

[0062] Acquire at least two first service requests in the current distribution period, and divide the first service requests with the same request type among the at least two first service requests into the same service request set to obtain n service request sets; the request types corresponding to the n service request sets are different from each other;

[0063] The request quantities of the first service requests respectively included in the n service request sets are determined, and the request quantities respectively corresponding to the n service request sets are determined as the request quantities respectively corresponding to the n request types in the current distribution period.

[0064] In a possible implementation, the determination module determines the number of requests corresponding to the n request types, respectively, to perform the following operations:

[0065] When the system time reaches the distribution update timestamp corresponding to the first distribution period, determine the request type to which the historical business requests in the second distribution period associated with the first distribution period belong, and determine the number of historical requests corresponding to the request type in the second distribution period; the timestamp corresponding to the second distribution period is earlier than the distribution update timestamp; the distribution update timestamp is earlier than the timestamp corresponding to the first distribution period;

[0066] According to the historical request quantity corresponding to the request type in the second distribution period, the request quantity corresponding to the n request types in the first distribution period is predicted; the n request types belong to the request types in the second distribution period.

[0067] In a possible implementation, the determination module is further configured to perform the following operations:

[0068] In the parameter manager, determining processor information corresponding to n model adjustment parameter groups respectively; wherein one processor information is used to indicate a processor to which one model adjustment parameter group is allocated;

[0069] Obtain parameter group identifiers corresponding to n model adjustment parameter groups respectively; wherein one parameter group identifier is used to indicate one model adjustment parameter group;

[0070] According to the n parameter group identifiers and the n processor information, n index relationships are constructed; wherein one index relationship includes one parameter group identifier and one processor information;

[0071] Determine n index relationships as a request routing table;

[0072] The request routing table is sent to the request router through the parameter manager; the request router is used to forward the second business request to the target processor in the processor set according to the request routing table when determining the target model adjustment parameter group corresponding to the second business request; the target processor refers to the processor loaded with the target model adjustment parameter group; the target model adjustment parameter group refers to the model adjustment parameter group corresponding to the request type to which the second business request belongs.

[0073] In one possible implementation, the computer device includes a request router;

[0074] Identify modules, also used to do the following:

[0075] In the request router, determining the parameter group identifier carried in the second service request as the target parameter group identifier;

[0076] Querying the target parameter group identifier in the request routing table, obtaining processor information in the request routing table that has an index relationship with the target parameter group identifier, and determining the obtained processor information as candidate processor information;

[0077] In the processor set, a candidate processor indicated by the candidate processor information is determined; the candidate processor is loaded with a target model adjustment parameter group indicated by the target parameter group identifier;

[0078] Determine the target processor among the candidate processors, and distribute the second business request to the target processor through the request router; the target processor is used to add the loaded target model adjustment parameter group to the base model according to the parameter group identifier carried by the second business request; the base model with the target model adjustment parameter group added is used to process the second business request.

[0079] In a possible implementation, the determination module determines a target processor from among the candidate processors to perform the following operations:

[0080] Obtaining the memory utilization of the candidate processor, and determining the memory availability of the candidate processor according to the memory utilization of the candidate processor;

[0081] A target processor is determined from among the candidate processors according to memory availability, request queue lengths of the candidate processors, and network delays of the candidate processors.

[0082] In a possible implementation, the determination module determines a target processor from the candidate processors according to the memory availability, the request queue length of the candidate processors, and the network delay of the candidate processors, and is used to perform the following operations:

[0083] Determine a first reciprocal of a request queue length of a candidate processor, and determine a second reciprocal of a network delay of the candidate processor;

[0084] Perform weighted fusion processing on the memory availability, the first inverse and the second inverse to obtain the score of the candidate processor;

[0085] The maximum score among the scores of the candidate processors is obtained, and the processor with the maximum score among the candidate processors is determined as the target processor.

[0086] In a possible implementation, the processor set includes a sixth processor, and the sixth processor is any processor in the processor set;

[0087] Identify modules, also used to do the following:

[0088] In the request router, a request queue corresponding to the sixth processor is obtained; the request queue includes a service requests; a is a positive integer;

[0089] The service requests with the same request type among the a service requests are divided into the same service request set to obtain b service request sets; b is a positive integer, and b is less than or equal to a;

[0090] Count the number of business requests included in each of the b business request sets to obtain the number of requests corresponding to each of the b business request sets;

[0091] Determine the request types corresponding to the b business request sets respectively, and determine the processing priorities corresponding to the b request types respectively according to the number of the b requests; wherein there is a positive relationship between the processing priority corresponding to a request type and the number of requests corresponding to the request type;

[0092] The processing priorities respectively corresponding to the b request types are sent to the sixth processor through the request router; the sixth processor is used to process the a business requests according to the processing priorities respectively corresponding to the b request types.

[0093] In a possible implementation, the determination module is further configured to perform the following operations:

[0094] In the request router, determining a waiting processing time threshold of the second service request;

[0095] The waiting processing time threshold is sent to the target processor through the request router; the target processor is used to return the second service request to the request router when determining that the waiting processing time of the second service request exceeds the waiting processing time threshold, and delete the second service request in the request queue;

[0096] The second service request returned by the target processor is obtained by requesting the router, and the second service request is redistributed and processed.

[0097] On the one hand, an embodiment of the present application provides a data storage device, including:

[0098] The acquisition module is used to acquire a service request carrying a target parameter group identifier through a request router, and acquire a request routing table according to the service request; the request routing table is generated by the parameter manager according to n parameter group identifiers and n processor information after the parameter manager allocates processors in a processor set to n model adjustment parameter groups according to the number of requests corresponding to n request types and the average processing time corresponding to n model adjustment parameter groups; n is a positive integer; one model adjustment parameter group corresponds to one request type; one parameter group identifier is used to indicate one model adjustment parameter group; one processor information is used to indicate one processor;

[0099] The acquisition module is further used to query the target parameter group identifier in the request routing table, and obtain the candidate processor information in the request routing table that has an index relationship with the target parameter group identifier;

[0100] A determination module, configured to determine, in a processor set, a candidate processor indicated by the candidate processor information; the candidate processor is loaded with a target model adjustment parameter group indicated by the target parameter group identifier;

[0101] The distribution module is used to determine the target processor among the candidate processors and distribute the business request to the target processor through the request router; the target processor is used to add the loaded target model adjustment parameter group to the base model according to the target parameter group identifier carried by the business request; the base model with the target model adjustment parameter group added is used to process the business request.

[0102] In a possible implementation, the distribution module determines a target processor from the candidate processors to perform the following operations:

[0103] Obtaining the memory utilization of the candidate processor, and determining the memory availability of the candidate processor according to the memory utilization of the candidate processor;

[0104] A target processor is determined from among the candidate processors according to memory availability, request queue lengths of the candidate processors, and network delays of the candidate processors.

[0105] In a possible implementation, the distribution module determines a target processor from the candidate processors according to the memory availability, the request queue length of the candidate processors, and the network delay of the candidate processors, and performs the following operations:

[0106] Determine a first reciprocal of a request queue length of a candidate processor, and determine a second reciprocal of a network delay of the candidate processor;

[0107] Perform weighted fusion processing on the memory availability, the first inverse and the second inverse to obtain the score of the candidate processor;

[0108] The maximum score among the scores of the candidate processors is obtained, and the processor with the maximum score among the candidate processors is determined as the target processor.

[0109] In a possible implementation, the acquisition module is further used to perform the following operations:

[0110] In the request router, a request queue corresponding to the target processor is obtained; the request queue includes c business requests; c is a positive integer;

[0111] The service requests with the same request type among the c service requests are divided into the same service request set to obtain d service request sets; d is a positive integer, and d is less than or equal to c;

[0112] Counting the number of service requests respectively included in the d service request sets, and obtaining the number of requests respectively corresponding to the d service request sets;

[0113] Determine the request types corresponding to the d business request sets respectively, and determine the processing priorities corresponding to the d request types respectively according to the number of the d requests; wherein there is a positive relationship between the processing priority corresponding to a request type and the number of requests corresponding to the request type;

[0114] The processing priorities respectively corresponding to the d request types are sent to the target processor through the request router; the target processor is used to process the c business requests according to the processing priorities respectively corresponding to the d request types.

[0115] In a possible implementation, the determination module is further configured to perform the following operations:

[0116] In the request router, a threshold of the waiting time for processing the service request is determined;

[0117] The waiting processing time threshold is sent to the target processor through the request router; the target processor is used to return the service request to the request router and delete the service request in the request queue when determining that the waiting processing time of the service request exceeds the waiting processing time threshold;

[0118] By requesting the router, the service request returned by the target processor is obtained, and the service request is redistributed and processed.

[0119] On one hand, the present application provides a computer device, including: a processor, a memory, and a network interface;

[0120] The above-mentioned processor is connected to the above-mentioned memory and the above-mentioned network interface, wherein the above-mentioned network interface is used to provide a data communication function, the above-mentioned memory is used to store a computer program, and the above-mentioned processor is used to call the above-mentioned computer program so that the computer device executes the method in the embodiment of the present application.

[0121] On one hand, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. The computer program is suitable for being loaded by a processor and executing the method in the embodiment of the present application.

[0122] On the one hand, an embodiment of the present application provides a computer program product, which includes a computer program stored in a computer-readable storage medium; a processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the method in the embodiment of the present application.

[0123] In an embodiment of the present application, the computer device can allocate processors in a processor set to n model adjustment parameter groups according to the number of requests corresponding to the n request types and the average processing time corresponding to the n model adjustment parameter groups; the processors in the processor set are used to load the allocated model adjustment parameter groups, and through the loaded model adjustment parameter groups and the base model, the processors can process business requests belonging to the request types corresponding to the loaded model adjustment parameter groups. As can be seen from the above, the embodiment of the present application dynamically distributes multiple model adjustment parameter groups to different processors, which can reduce the memory usage of the processors, so the memory resources used by the processors for model reasoning are increased, thereby improving the memory resource utilization of the processors. BRIEF DESCRIPTION OF THE DRAWINGS

[0124] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0125] Figure 1 This is a schematic diagram of a network architecture provided by an embodiment of the present application;

[0126] Figure 2 It is a schematic diagram of a system architecture provided by an embodiment of the present application;

[0127] Figure 3 This is a flow diagram of a data processing method provided in an embodiment of the present application. Figure 1 ;

[0128] Figure 4 This is a data processing scenario provided by an embodiment of the present application. Figure 1 ;

[0129] Figure 5 This is a data processing scenario provided by an embodiment of the present application. Figure 2 ;

[0130] Figure 6 This is a flow diagram of a data processing method provided in an embodiment of the present application. Figure 2 ;

[0131] Figure 7 This is a flow diagram of a data processing method provided in an embodiment of the present application. Figure 3 ;

[0132] Figure 8 This is a data processing scenario provided by an embodiment of the present application. Figure 3 ;

[0133] Fig. 9 This is a flow diagram of a data processing method provided in an embodiment of the present application. Figure 4 ;

[0134] Fig.10 It is an interactive schematic diagram of a data processing method provided in an embodiment of the present application;

[0135] Fig.11 This is a schematic diagram of the structure of a data processing device provided in an embodiment of the present application. Figure 1 ;

[0136] Fig.12 This is a schematic diagram of the structure of a data processing device provided in an embodiment of the present application. Figure 2 ;

[0137] Fig.13 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0138] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0139] See also Figure 1 , Figure 1 Schematic diagram of a network architecture provided by an embodiment of the present application. Figure 1 As shown, the network architecture may include a business server 100 and a terminal device cluster. The terminal device cluster may include: terminal device 200a, terminal device 200b, terminal device 200c, ..., terminal device 200n. It can be understood that the above network architecture may include one or more terminal devices, and this application does not limit the number of terminal devices.

[0140] There may be communication connections between the terminal device clusters, for example, there is a communication connection between the terminal device 200a and the terminal device 200b, and there is a communication connection between the terminal device 200a and the terminal device 200c. At the same time, any terminal device in the terminal device cluster may have a communication connection with the business server 100, for example, there is a communication connection between the terminal device 200a and the business server 100, wherein the above communication connection does not limit the connection method, and may be directly or indirectly connected by wired communication, directly or indirectly connected by wireless communication, or by other methods, and this application does not limit it here.

[0141] It should be understood that Figure 1 Each terminal device in the terminal device cluster shown in FIG. 1 may be installed with an application client. When the application client runs in each terminal device, it may be respectively connected to the above-mentioned Figure 1 The business server 100 shown in the figure performs data interaction, that is, the above-mentioned communication connection. Among them, the application client can be an application client with intelligent interaction functions such as video application, convenient life application, payment application, financial application, game application, shopping application, novel application, camera application, browser, etc. Among them, the application client can be an independent client or an embedded sub-client integrated in a client (for example, a video client, an education client, and a multimedia client, etc.), which is not limited here.

[0142] Taking a browser as an example, the business server 100 may be a collection of multiple servers including a background server corresponding to the browser, a data processing server, etc. Therefore, each terminal device can transmit data with the business server 100 through the application client corresponding to the browser. For example, the terminal device 200a can upload the prompt text used to generate an image to the business server 100 through the application client of the browser, and then the business server 100 can generate an image based on the prompt text.

[0143] It is understandable that in the specific implementation of this application, related data such as user information (such as business requests) is involved. When the embodiments in this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data must comply with relevant local laws, regulations and standards.

[0144] Among them, the business server 100 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud database, cloud service, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, content delivery network (Content Delivery Network, CDN), as well as big data and artificial intelligence platform.

[0145] Figure 1 The terminal devices shown include but are not limited to mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle terminals, aircraft, etc. Among them, each terminal device and the service server can be directly or indirectly connected via wired or wireless means, and the embodiment of the present application does not limit this.

[0146] To facilitate understanding and description, two key words of this application are first described.

[0147] Model adjustment parameter group: The model adjustment parameter group can adjust the weight of the base model (also known as the pre-trained model), which can significantly reduce the number of parameters that need to be trained and stored, while maintaining performance comparable to fine-tuning of the full parameters of the base model. The base model with the model adjustment parameter group added can be used for specific business processing. The model adjustment parameter group involved in the embodiment of the present application can be generated by Low-Rank Adaptation (LoRA for short), so the model adjustment parameter group can also be called a LoRA module, specifically two low-rank matrices, such as matrix A and matrix B. Among them, the dimension of matrix A is r*m, and the dimension of matrix B is n*r, where n*m ​​is the dimension of the original matrix adjusted by the LoRA module in the base model, r is much smaller than n, and r is much smaller than m, so compared to fine-tuning the full amount of parameters in the base model, training an external LoRA module can significantly reduce the number of training parameters.

[0148] LoRA is a technique for fine-tuning large pre-trained models, which allows pre-trained models to be adjusted on specific tasks without retraining all the parameters of the pre-trained model. The core idea of ​​LoRA is to introduce two low-rank matrices in specific layers of the pre-trained model (such as the transformer layer). The two low-rank matrices are fused with the original attention matrix or feedforward network weight matrix to introduce new trainable parameters. The advantage of LoRA technology is that it can adapt the pre-trained model to a specific task or dataset while maintaining most of the knowledge of the pre-trained model. The number of parameters added by LoRA is much less than the total number of parameters of the original model, which greatly reduces the computational and storage requirements during fine-tuning.

[0149] Base models: Base models are large pre-trained models trained on large datasets through self-supervised learning. These models have strong general capabilities and can serve as a starting point for a variety of downstream tasks. Technologies such as LoRA are lightweight task-specific adaptations based on base models.

[0150] In one embodiment, after the business server trains the base model to obtain the model adjustment parameter groups corresponding to m request types, the base model and the m model adjustment parameter groups can be stored in the local memory of the business server respectively. Wherein, m is a positive integer, and the embodiment of the present application does not limit the request type, and can be set according to the actual application, such as text image type, intelligent conversation type, image recognition type, audio recognition type, etc. It can be understood that by adding the model adjustment parameter group corresponding to a request type to the base model, the business request corresponding to the request type can be processed.

[0151] The business server determines the number of requests corresponding to n request types, determines the model adjustment parameter groups corresponding to the n request types, where n is a positive integer and n is less than or equal to m; obtains the average processing time corresponding to the n model adjustment parameter groups, and allocates processors in a processor set to the n model adjustment parameter groups according to the number of requests corresponding to the n request types and the n average processing time. The processors in the processor set are used to load the allocated model adjustment parameter groups, and are used to process business requests belonging to the request types corresponding to the loaded model adjustment parameter groups through the loaded model adjustment parameter groups and the base model.

[0152] In one embodiment, after the business server trains the base model to obtain model adjustment parameter groups corresponding to m request types, the base model and the m model adjustment parameter groups can be sent to each terminal device respectively, and each terminal device can store the base model and the m model adjustment parameter groups in a local memory respectively. Therefore, the terminal device determines the number of requests corresponding to the n request types respectively, and determines the model adjustment parameter groups corresponding to the n request types respectively; obtains the average processing time corresponding to the n model adjustment parameter groups respectively, and allocates processors in the processor set to the n model adjustment parameter groups respectively according to the number of requests corresponding to the n request types and the n average processing times. The processors in the processor set are used to load the assigned model adjustment parameter groups, and are used to process business requests belonging to the request types corresponding to the loaded model adjustment parameter groups through the loaded model adjustment parameter groups and the base model.

[0153] In summary, the data processing method provided in this application can be Figure 1 It can be executed by any terminal device in the terminal device cluster in Figure 1 It can also be executed by the business server in Figure 1 The terminal devices and business servers in the terminal device cluster are executed collaboratively. The devices used to execute the data processing method in this application can be collectively referred to as computer devices.

[0154] Please also see Figure 2 , Figure 2 Schematic diagram of a system architecture provided by an embodiment of the present application. Figure 2 As shown in FIG. 1 , the system architecture may include the following components: a central controller 21a, a request router 22a, a parameter manager 23a, a processor cluster 24a, and a data detector 25a. The above five components may be deployed on the same device, for example, all deployed on Figure 1 The business server 100 shown in the figure is an independent physical server; alternatively, the above five components can be deployed on different devices respectively, for example, the central controller 21a is deployed on Figure 1The service server 100 shown in FIG. 1 (which may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers) requests the router 22a to be deployed at Figure 1 The terminal device 200a shown in FIG. 1 is a parameter manager 23a deployed in Figure 1 The terminal device 200b shown in FIG. 1 has a processor cluster 24a deployed on Figure 1 The terminal device 200c shown in the figure has a data detector 25a deployed at Figure 1 or, there are multiple components deployed on the same device among the above five components, such as the central controller 21a and the data detector 25a deployed on Figure 1 The service server 100 is shown.

[0155] In summary, Figure 2 The five components in the system architecture shown can be deployed according to actual application scenarios. The functions of each component in the system are described below.

[0156] The central controller 21a is the brain of the entire system and is responsible for coordinating the work of various components. Main functions:

[0157] 1. System initialization and configuration management.

[0158] 2. Coordinate the operations of parameter manager 23a and request router 22a.

[0159] 3. Handle system-level decisions, such as resource expansion, failure recovery, model adjustment parameter group adaptive caching.

[0160] It takes time to load the currently used model adjustment parameter group into the processor cluster. In order to improve the access efficiency of the model adjustment parameter group, in this application, the central controller 21a sets a multi-level cache strategy, which is as follows:

[0161] a) Graphics Processing Unit (GPU) video memory: caches the most commonly used or currently used model adjustment parameter groups, such as the n model adjustment parameter groups described below.

[0162] GPU is an electronic circuit specially designed to handle graphics and image rendering tasks. GPU was originally developed to improve the performance of computer graphics and video games, but over time, technicians found that GPU has significant advantages in handling parallel computing tasks, so GPU is also widely used in scientific computing, data analysis, machine learning and artificial intelligence.

[0163] GPUs are very important for training and reasoning large models (including the base model of this application) for the following reasons:

[0164] Parallel processing capabilities: GPUs have thousands of small cores, which enables GPUs to process a large number of small tasks at the same time. In deep learning, this means calculating the gradients of multiple data points at the same time, or processing the forward and backward propagation of multiple neural network layers in parallel.

[0165] High throughput: GPUs are designed to handle high-throughput computing tasks, which is especially important for training large neural networks, as these networks typically involve a large number of matrix and vector operations.

[0166] Floating-point computing power: Deep learning model training and reasoning require a large number of floating-point operations. GPUs usually have powerful floating-point computing power and can perform these operations quickly.

[0167] In summary, the processor cluster in this application mainly includes GPUs, because model reasoning is mainly performed on GPU nodes. GPU video memory is a memory specially designed for GPUs, which is used to store and manage the data required by the graphics processor when rendering graphics. The processor memory described in this application mainly refers to GPU video memory.

[0168] b) Central Processing Unit (CPU) memory: caches the less frequently used model adjustment parameter groups, or when the GPU memory cannot store all the currently used model adjustment parameter groups, caches part of the currently used model adjustment parameter groups that have not been cached to the GPU to the CPU.

[0169] The design of CPU focuses on serial processing and is suitable for processing complex logical operations and sequential tasks. Therefore, the processor cluster in this application may also include CPU, so the processor memory described in this application may also include CPU memory.

[0170] c) Solid State Disk (SSD): Caches model adjustment parameter groups that are not frequently used but may be needed.

[0171] In summary, the present application caches a full amount of model adjustment parameter groups (for example, m model adjustment parameter groups, m greater than or equal to n) to suitable memories according to a multi-level caching strategy, wherein the number of times the model adjustment parameter groups cached in the GPU video memory are used is greater than the number of times the model adjustment parameter groups cached in the CPU memory are used, and the number of times the model adjustment parameter groups cached in the CPU memory are used is greater than the number of times the model adjustment parameter groups cached in the SSD are used; because the speed of loading the model adjustment parameter groups from the CPU memory to the GPU video memory is fast, but the CPU memory space is limited, and the speed of loading the model adjustment parameter groups from the hard disk SSD to the GPU video memory is slow, but the SSD has low cost and large space.

[0172] The process of assigning n model adjustment parameter groups (belonging to the above-mentioned total m model adjustment parameter groups) to the processors in the processor cluster 24a is not described here for the time being. Figure 3 as well as Figure 6 The descriptions in the corresponding embodiments respectively.

[0173] The central controller 21a implements elastic resource management, supports dynamic addition or removal of processors (including GPU nodes), and realizes elastic scaling of the system.

[0174] A feasible resource elastic scaling method is as follows: the central controller 21a determines the memory utilization corresponding to all processors in the system, and determines the average memory utilization corresponding to the processor cluster 24a according to the memory utilization corresponding to all processors. The central controller 21a compares the average memory utilization with the memory utilization threshold (including the first memory utilization threshold and the second memory utilization threshold). If the average memory utilization is greater than or equal to the first memory utilization threshold, resource expansion is performed, that is, processors are added to the processor cluster 24a so that the average memory utilization is less than the first memory utilization threshold, and the average memory utilization is greater than or equal to the second memory utilization threshold, wherein the first memory utilization threshold is greater than the second memory utilization threshold. If the average memory utilization is less than the second memory utilization threshold, the central controller 21a performs resource contraction, that is, removes the processors in the processor cluster 24a so that the average memory utilization is greater than or equal to the second memory utilization threshold, and the average memory utilization is less than the first memory utilization threshold. The first memory utilization threshold and the second memory utilization threshold are both adjustable parameters. For example, the first memory utilization threshold is 0.85, and the second memory utilization threshold is 0.3.

[0175] Another feasible resource elastic scaling method is as follows: the central controller 21a determines the request queue lengths corresponding to all processors in the system, the request queue length refers to the number of business requests in the request queue, and determines the average request queue length corresponding to the processor cluster 24a according to the request queue lengths corresponding to all processors. The central controller 21a compares the average request queue length with the request queue length threshold (including the first request queue length threshold and the second request queue length threshold). If the average request queue length is greater than or equal to the first request queue length threshold, resource expansion is performed, that is, processors are added to the processor cluster 24a so that the average request queue length is less than the first request queue length threshold, and the average request queue length is greater than or equal to the second request queue length threshold, wherein the first request queue length threshold is greater than the second request queue length threshold. If the average request queue length is less than the second request queue length threshold, the central controller 21a performs resource contraction, that is, removes the processors in the processor cluster 24a so that the average request queue length is greater than or equal to the second request queue length threshold, and the average request queue length is less than the first request queue length threshold. The first request queue length threshold and the second request queue length threshold are both adjustable parameters. For example, the first request queue length threshold is 300, and the second request queue length threshold is 20.

[0176] Another feasible resource elastic scaling method is as follows: the central controller 21a compares the average memory utilization with the memory utilization threshold (including the first memory utilization threshold and the second memory utilization threshold), and compares the average request queue length with the request queue length threshold (including the first request queue length threshold and the second request queue length threshold). If the average memory utilization is greater than or equal to the first memory utilization threshold, and the average request queue length is greater than or equal to the first request queue length threshold, the central controller 21a performs resource expansion, that is, adds processors to the processor cluster 24a. If the average memory utilization is less than the second memory utilization threshold, and the average request queue length is less than the second request queue length threshold, the central controller 21a performs resource contraction, that is, removes processors from the processor cluster 24a.

[0177] In addition, when adding or removing processors, the central controller 21a implements smooth migration of business requests and model adjustment parameter groups to minimize service interruption.

[0178] The central controller 21a is designed with a fault-tolerant mechanism to ensure that the system can still operate normally when some processor nodes fail. This can be done by setting:

[0179] a) Heartbeat detection: Regularly check the status of each processor (GPU node).

[0180] b) Fault detection: If a processor fails the heartbeat detection for n consecutive times, the processor is marked as in a faulty state.

[0181] c) Automatic recovery: reroute the business requests of the processor in a faulty state (referred to as the faulty processor) to the processor in a healthy state (referred to as the healthy processor); reallocate the model adjustment parameter group loaded by the faulty processor on the healthy processor; try to restart or replace the faulty processor.

[0182] Please see again Figure 2 The parameter manager 23a is responsible for the allocation and management of the model adjustment parameter group and is one of the core components of the system. The request router 22a is responsible for distributing the service request to the processor adapted to the service request. The processor cluster 24a includes one or more processors, which are the computing units that actually perform the inference task. For the specific description of the above three components, please refer to the following Figure 3 , Figure 7 as well as Fig. 9 The descriptions in the corresponding embodiments respectively.

[0183] Data detector 25a is responsible for collecting system operation data to provide support for decision making. The key indicators collected by data detector 25a include:

[0184] 1. The number of requests and average processing time for each model adjustment parameter group;

[0185] 2. Processor memory utilization and memory usage;

[0186] 3. The request queue length and waiting time of the processor;

[0187] 4. Network latency and throughput;

[0188] The data detector 25a performs performance analysis and provides the following results:

[0189] a) Real-time dashboard: Visually display key system indicators and health status.

[0190] b) Performance bottleneck identification: Use statistical analysis methods to identify system bottlenecks, such as frequent switching of model adjustment parameter groups, unbalanced GPU load, and network congestion.

[0191] c) Trend analysis: Use time series analysis to predict future load trends and assist in resource planning.

[0192] Through the above system architecture, this application can efficiently manage and utilize large-scale model adjustment parameter groups to provide flexible and high-performance large-model inference services. The core innovation of the system lies in the dynamic distribution of model adjustment parameter groups, intelligent request routing, adaptive resource management, and comprehensive performance optimization strategies. The combination of these technologies not only solves the limitations of the large model service framework, but also provides strong technical support for the large-scale deployment and personalized application of AI services in the future. In addition, through continuous data detection, analysis, and optimization, the system can adapt to changing workloads and terminal requirements, while ensuring performance and maximizing resource utilization.

[0193] For further information, see Figure 3 , Figure 3 This is a flow diagram of a data processing method provided in an embodiment of the present application. Figure 1 The embodiments of the present application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, audio and video, etc. The embodiments of the present application can be applied to business scenarios such as image generation scenarios, image recommendation scenarios, text generation scenarios, and video creative scenarios. The specific business scenarios will not be listed one by one here.

[0194] The data processing method is executed by a computer device, wherein the computer device may be a service server, a terminal device, or a service server and a terminal device. Figure 3 As shown, the data processing method may at least include the following steps S101 to S103.

[0195] Step S101, determining the number of requests corresponding to n request types, and determining the model adjustment parameter groups corresponding to the n request types; n is a positive integer.

[0196] Specifically, when the system time reaches the distribution update timestamp corresponding to the first distribution period, determine the request type to which the historical business requests in the second distribution period associated with the first distribution period belong, and determine the number of historical requests corresponding to the request type in the second distribution period; the timestamp corresponding to the second distribution period is earlier than the distribution update timestamp; the distribution update timestamp is earlier than the timestamp corresponding to the first distribution period; based on the number of historical requests corresponding to the request type in the second distribution period, predict the number of requests corresponding to the n request types in the first distribution period; the n request types belong to the request types in the second distribution period.

[0197] The number of requests corresponding to a request type refers to the number of business requests belonging to the same request type. The embodiments of the present application do not limit the real-time nature of business requests. The first feasible implementation method is that the computer device determines the number of business requests (hereinafter referred to as real-time business requests) obtained in the current time period, determines the model adjustment parameter group assigned to each processor in the processor set according to the request type and number of the real-time business requests, and then processes the real-time business requests through the processor loaded with the model adjustment parameter group. The second feasible implementation method is that the computer device determines the number of business requests (hereinafter referred to as historical business requests) obtained in the historical time period associated with the current time period, determines the model adjustment parameter group assigned to each processor in the processor set according to the request type and number of the historical business requests, and then processes the real-time business requests through the processor loaded with the model adjustment parameter group.

[0198] This step does not describe the first feasible implementation method in detail. Please refer to the following Figure 6 Corresponding to the description of step S201 in the embodiment, this step describes in detail a second feasible implementation method.

[0199] In order to reduce the time waiting for the processor to load the model to adjust the parameter group, the parameter manager of the computer device executes an intelligent preloading strategy, which includes determining the request type to which the historical business requests obtained in the historical time period (i.e., the historical distribution period, such as the second distribution period in the embodiment of the present application) belong, and the number of requests corresponding to each request type, and using a time series to predict the number of requests of each request type in the next distribution period (such as the first distribution period in the embodiment of the present application).

[0200] The distribution period refers to the time period during which the model adjustment parameter group is assigned to the processor, that is, the parameter manager updates the model adjustment parameter group loaded by the processor according to the distribution period. The distribution period is an adjustable parameter that can be set according to the actual application scenario. For example, if every 5 seconds is a distribution period, the first distribution period can be 9:00 on January 1, 2024 to 9:05 on January 1, 2024; for example, if every 1 minute is a distribution period, the first distribution period can be 9:00 on January 1, 2024 to 9:01 on January 1, 2024.

[0201] Please also see Figure 4 , Figure 4 This is a data processing scenario provided by an embodiment of the present application. Figure 1 . System time refers to the real time of the computer device. For ease of understanding, Figure 4 The first distribution period 301a is exemplified as 2024.1.19:00-9:05, and the distribution update timestamp 302a is exemplified as 2024.1.18:59:58, that is, Figure 4 Assume that the duration of one distribution cycle is 5 minutes, and the request type and the number of requests in the next distribution cycle (ie, the first distribution cycle) are predicted 2 seconds in advance.

[0202] like Figure 4 As shown, when the system time reaches the distribution update timestamp 302a of example 2024.1.1 8:59:58, the parameter manager in the computer device predicts the request type and the number of requests in the first distribution period. It can be understood that the time corresponding to the current distribution period is 2024.1.1 8:55-9:00. The parameter manager determines a second distribution period 30b associated with the first distribution period 301a. The second distribution period 30b can refer to a time period in the historical date corresponding to the first distribution period 301a, such as Figure 4 The three time periods in the example are: the first time period is 2023.12.31 9:00-9:05, the second time period is 2023.12.30 9:00-9:05, and the third time period is 2023.12.29 9:00-9:05. Obviously, Figure 4 The first distribution period 301a and the second distribution period 301b in the example refer to the same time in different dates, namely 9:00-9:05. It is understandable that the number of the second distribution period 30b can be one or more, and the specific number can be set according to the actual application scenario.

[0203] Further, the parameter manager determines the request type to which the historical service request acquired in the second distribution period 30b belongs, such as Figure 4 In the example, 120 historical business requests were obtained in the distribution period corresponding to 2023.12.31 9:00-9:05, namely request 1z-request 120z, among which the request type of the 30 historical business requests from request 1z to request 30z is the question-and-answer type, that is, request 1z-request 30z are all question-and-answer requests; the request type of the 40 historical business requests from request 31z to request 70z is the video recommendation type, that is, request 31z-request 70z are all video recommendation requests; the request type of the 50 historical business requests from request 71z to request 120z is the text generation type, that is, request 71z-request 120z are all text generation requests. In summary, the 120 historical business requests obtained in the distribution period corresponding to 2023.12.31 9:00-9:05 are divided into three types of business requests, corresponding to three request types.

[0204] like Figure 4In the example, 120 historical business requests were obtained in the distribution period corresponding to 2023.12.30 9:00-9:05, which are request 1y-request 120y. Among them, the request type of 31 historical business requests from request 1y to request 31y is the question-and-answer type, that is, request 1y-request 31y are all question-and-answer requests; the request type of 40 historical business requests from request 32y to request 71y is the video recommendation type, that is, request 32y-request 71y are all video recommendation requests; the request type of 49 historical business requests from request 72y to request 120y is the text generation type, that is, request 72y request 120y are all text generation requests. In summary, the 120 historical business requests obtained in the distribution period corresponding to 2023.12.30 9:00-9:05 are divided into three types of business requests, corresponding to three request types.

[0205] like Figure 4 In the example, 116 historical business requests were obtained in the distribution period corresponding to 2023.12.29 9:00-9:05, which are requests 1x-request 116x. Among them, the request type of 29 historical business requests from request 1x to request 29x is the question-and-answer type, that is, requests 1x-request 29x are all question-and-answer requests; the request type of 42 historical business requests from request 30x to request 71x is the video recommendation type, that is, requests 30x-request 71x are all video recommendation requests; the request type of 45 historical business requests from request 72x to request 116x is the text generation type, that is, requests 72x and 116x are all text generation requests. In summary, the 116 historical business requests obtained in the distribution period corresponding to 2023.12.29 9:00-9:05 are divided into three types of business requests, corresponding to three request types.

[0206] It is understandable that if there are multiple second distribution periods 30b, the request types corresponding to the multiple second distribution periods may be the same or different, and should be determined according to the historical business requests actually obtained. Figure 4 The second distribution cycles of the three examples all involve question-answer type, video recommendation type, and text generation type.

[0207] The parameter manager predicts the number of requests corresponding to the n request types in the first distribution period based on the historical number of requests corresponding to the request types in the second distribution period, that is, uses time series analysis to predict the number of requests of each type in the first distribution period. The embodiment of the present application does not limit the algorithm for predicting the number of requests and can be set according to the actual application scenario.

[0208] Please see again Figure 4 , Figure 4The example parameter manager predicts that the first distribution period 301a includes three request types, namely question and answer type, video recommendation type and text generation type, and the predicted number of requests corresponding to the question and answer type is 30, the number of requests corresponding to the video recommendation type is 41, and the number of requests corresponding to the text generation type is 48.

[0209] This step mainly describes the prediction of the request type and number of requests in the next distribution period (i.e., the first distribution period) based on historical business requests. Therefore, by using the method described in this step, the model adjustment parameter group required to be loaded by the processor can be determined in advance, and then when the system time reaches the starting time of the first distribution period, Figure 4 At 2024.1.19:00:00, the processor quickly loads the assigned model adjustment parameter group, so the processor can quickly process the business requests obtained in the first distribution period through the loaded model adjustment parameter group and the base model.

[0210] Step S102, obtaining the average processing time corresponding to n model adjustment parameter groups respectively; the average processing time corresponding to a model adjustment parameter group refers to the average time for processing business requests belonging to the request type corresponding to the model adjustment parameter group by adding a base model with the model adjustment parameter group.

[0211] Specifically, the computer device is deployed with a parameter manager, which is responsible for the allocation and management of model adjustment parameter groups and is one of the core components of the computer device. The parameter manager contains a full set of model adjustment parameter groups, and the number m of the full set of model adjustment parameter groups is a positive integer greater than or equal to n. It can be understood that the parameter manager allocates the n model adjustment parameter groups to be used, and the n model adjustment parameter groups to be used may be part of the full set of model adjustment parameter groups, and the unused model adjustment parameter groups in the full set of model adjustment parameter groups do not need to be allocated to the processor, which can not only save the storage space of the processor, but also avoid the fan-in and fan-out of the unused model adjustment parameter groups. Fan-in and fan-out refer to the process of moving model adjustment parameter groups between different storage levels. Fan-in is to load the model adjustment parameter group from the CPU memory or hard disk to the GPU video memory, and fan-out is to load the model adjustment parameter group from the GPU video memory to the CPU memory or hard disk. Fan-in and fan-out have an important impact on the speed of base model switching and system efficiency.

[0212] For each model adjustment parameter group, the parameter manager is provided with a parameter group detection module, which is used to detect the average processing time, occupied space size, etc. corresponding to the corresponding model adjustment parameter group. Each parameter group detection module can be represented by the following structure:

[0213] LoRAMonitor

[0214] id: unique identifier

[0215] size: Model tuning parameter group size (MB)

[0216] type: model adjustment parameter group type (such as text generation, question answering, etc.)

[0217] performance_metrics performance indicators: {

[0218] avg_processing_time: average processing time

[0219] throughput: throughput

[0220] }

[0221] }

[0222] Among them, the id in the above structure refers to the unique identifier of the model adjustment parameter group, which is referred to as the parameter group identifier in this application. The model adjustment parameter group size size is also referred to as the occupied space of the model adjustment parameter group in this application. The model adjustment parameter group type type corresponds to the request type, that is, one type of model adjustment parameter group is used to process a business request of a request type; the average processing time refers to the average time for the model adjustment parameter group to process the corresponding business request.

[0223] After determining the model adjustment parameter groups corresponding to the n request types, the parameter manager may obtain the average processing time corresponding to each model adjustment parameter group from the parameter group detection modules corresponding to the n model adjustment parameter groups.

[0224] Step S103, according to the number of requests corresponding to the n request types and the n average processing times, respectively allocate processors in the processor set to the n model adjustment parameter groups; the processors in the processor set are used to load the allocated model adjustment parameter groups, and to process business requests belonging to the request types corresponding to the loaded model adjustment parameter groups through the loaded model adjustment parameter groups and the base model.

[0225] Specifically, according to the number of requests corresponding to n request types and n average processing times, the distribution quantities corresponding to n model adjustment parameter groups are determined; the distribution quantities corresponding to the n model adjustment parameter groups are all smaller than the total number of processors in the processor set; according to the distribution quantities corresponding to the n model adjustment parameter groups, processors in the processor set are allocated to the n model adjustment parameter groups; the distribution quantity corresponding to a model adjustment parameter group is used to allocate processors in the processor set to the model adjustment parameter group; the number of processors allocated to a model adjustment parameter group is the same as the distribution quantity corresponding to the model adjustment parameter group.

[0226] Among them, the n model adjustment parameter groups include the second model adjustment parameter group; the second model adjustment parameter group is any one of the n model adjustment parameter groups; according to the number of requests corresponding to the n request types and the n average processing times, the specific process of determining the distribution quantities corresponding to the n model adjustment parameter groups respectively may include: multiplying the number of requests corresponding to the second model adjustment parameter group and the average processing time corresponding to the second model adjustment parameter group to obtain the request processing time corresponding to the second model adjustment parameter group; performing ratio processing on the request processing time corresponding to the second model adjustment parameter group and the occupied space of the second model adjustment parameter group to obtain the distribution quantity corresponding to the second model adjustment parameter group.

[0227] Among them, the specific process of allocating processors in the processor set to n model adjustment parameter groups respectively according to the distribution quantities corresponding to the n model adjustment parameter groups may include: sorting the distribution quantities corresponding to the n model adjustment parameter groups from large to small to obtain n sorted distribution quantities; according to the sorted n distribution quantities, allocating processors in the processor set to the n model adjustment parameter groups in sequence; wherein the number of processors in the processor set allocated to the second model adjustment parameter group is the same as the distribution quantity corresponding to the second model adjustment parameter group; the order in which the processors are allocated to the second model adjustment parameter group is the same as the order of the distribution quantities corresponding to the second model adjustment parameter group in the sorted n distribution quantities; the total occupied space of the model adjustment parameter groups loaded by the processor to which the second model adjustment parameter group is allocated is less than or equal to the available memory of the processor to which the second model adjustment parameter group is allocated.

[0228] The parameter manager in the computer device obtains the number of requests corresponding to each of the n request types and the average processing time corresponding to each of the n model adjustment parameter groups. Figure 5 , Figure 5 This is a data processing scenario provided by an embodiment of the present application. Figure 2 . Figure 5In the example, n=3, the n request types are question-and-answer type, video recommendation type, and text generation type. The number of requests corresponding to the question-and-answer type is 30, the number of requests corresponding to the video recommendation type is 41, and the number of requests corresponding to the text generation type is 48.

[0229] in, Figure 5 The three request types in the example may be request types corresponding to historical distribution periods associated with the current distribution period. The request quantities corresponding to the three request types are predicted based on historical business requests obtained in the historical distribution period. For details, please refer to the description of step S101 above. Figure 5 The three request types in the example may also be request types corresponding to real-time service requests obtained in the current distribution period, and the sum of the request quantities corresponding to the three request types is the quantity of real-time service requests obtained in the current distribution period.

[0230] Figure 5 The model adjustment parameter group corresponding to the question-answer type is abbreviated as the question-answer type parameter group, the model adjustment parameter group corresponding to the video recommendation type is abbreviated as the video recommendation type parameter group, and the model adjustment parameter group corresponding to the text generation type is abbreviated as the text generation type parameter group. Among them, the average processing time corresponding to the question-answer type parameter group is 0.1 seconds per, the average processing time corresponding to the video recommendation type parameter group is 0.2 seconds per, and the average processing time corresponding to the text generation type parameter group is 0.1 seconds per.

[0231] The parameter manager can calculate the number of distributions corresponding to n model adjustment parameter groups using the following formula (1):

[0232] Score(LoRA_i) = requests_i * avg_processing_time_i / size_i (1)

[0233] Among them, the value range of i in formula (1) is 1-n, LoRA_i represents the i-th model adjustment parameter group, requests_i represents the number of requests corresponding to the i-th request type, avg_processing_time_i represents the average processing time corresponding to the i-th model adjustment parameter group, size_i represents the occupied space corresponding to the i-th model adjustment parameter group, that is, the size, Score(LoRA_i) represents the distribution quantity corresponding to the i-th model adjustment parameter group, which can also be called the importance score, meaning that the larger the importance score, the greater the priority of the allocated processor.

[0234] The following is Figure 5 The text generation type in is described as an example, that is, Figure 5The text generation type parameter group example in the second model adjustment parameter group. Figure 5 , the number of requests corresponding to the text generation type by the parameter manager ( Figure 5 48 as an example) and the average processing time corresponding to the text generation type parameter group ( Figure 5 Taking 0.1 seconds as an example, we multiply the request processing time corresponding to the text generation type parameter group, that is, Figure 5 48*0.1=4.8 seconds. Further, the parameter manager processes the request processing time corresponding to the text generation type parameter group and the occupied space of the text generation type parameter group ( Figure 5 Taking 1 megabyte as an example), we perform ratio processing to obtain the distribution quantity corresponding to the text generation type parameter group, that is, Figure 5 4.8 / 1=4.8, round 4.8 to the nearest integer, and the number of distributions corresponding to the text generation type parameter group is 5.

[0235] According to formula (1), the parameter manager can determine the distribution quantity corresponding to the question-answer type parameter group ( Figure 5 Example 3), and the distribution number corresponding to the video recommendation type parameter group ( Figure 5 The example is 4).

[0236] As mentioned above, the distribution number of the model adjustment parameter group determined by formula (1) can also reflect the importance of the model adjustment parameter group. Figure 5 By sorting the three distribution quantities in the example, it can be obtained that distribution quantity 5>distribution quantity 4>distribution quantity 3. Based on the sorting of the three distribution quantities, the parameter manager determines the priorities to which the three model adjustment parameter groups are assigned, such as Figure 5 As shown, the allocation priority is text generation type parameter group > video recommendation type parameter group > question and answer type parameter group. Therefore, the parameter manager preferentially allocates processors in the processor set to the text generation type parameter group, specifically 5 processors to the text generation type parameter group, and then allocates processors in the processor set to the video recommendation type parameter group, specifically 4 processors to the video recommendation type parameter group, and finally allocates processors in the processor set to the question and answer type parameter group, specifically 3 processors to the question and answer type parameter group.

[0237] Among them, a processor can load one or more model adjustment parameter groups, which is specifically determined according to the available memory of the processor, that is, the total occupied space corresponding to the model adjustment parameter groups loaded by a processor is smaller than the available memory of the processor.

[0238] In one feasible implementation, the parameter manager does not consider the processor where the text generation type parameter group is currently located, and reallocates the processor (which may be the same as the current processor) for the text generation type parameter group. Assume Figure 5 The number of historical distributions of the text generation type parameter group in is 4, and the allocated processors are processor 1, processor 2, processor 3, and processor 4. Figure 5 In the example distribution number 5, processor 1, processor 3, processor 6, processor 7 and processor 8 are reallocated.

[0239] In another feasible implementation, the parameter manager allocates the text generation type parameter group based on the processor where the text generation type parameter group is currently located. Figure 5 The number of historical distributions of the text generation type parameter groups in is 6, and the allocated processors are processor 1, processor 2, processor 3, processor 4, processor 5, and processor 6. Figure 5 In the example distribution number 5, the parameter manager determines 5 processors among processor 1, processor 2, processor 3, processor 4, processor 5 and processor 6, for example, processor 1, processor 2, processor 3, processor 4 and processor 5, so it sends an uninstall instruction for the text generation type parameter group to processor 6 to enable processor 6 to uninstall the text generation type parameter group.

[0240] The embodiment of the present application performs dynamic management of the model adjustment parameter group through a parameter manager, realizes on-demand loading and unloading of the model adjustment parameter group, does not need to load all the model adjustment parameter groups into each processor (mainly GPU video memory) at the same time, and breaks through the single processor memory limit by distributing the model adjustment parameter group across multiple processors. The present application can optimize the storage and transmission mode of the model adjustment parameter group, such as using compression technology and incremental update. By distributing the model adjustment parameter group, that is, not loading all the model adjustment parameter groups into each processor at the same time, the fan-in and fan-out frequency of a single processor can be reduced. Through the above-mentioned technology, the embodiment of the present application significantly improves the flexibility, efficiency and scalability of the multi-model adjustment parameter group system, and provides more powerful and efficient support for the deployment and application of large-scale language models.

[0241] See also Figure 6 , Figure 6 This is a flow diagram of a data processing method provided in an embodiment of the present application. Figure 2 The data processing method is executed by a computer device, such as Figure 6 As shown, the data processing method may at least include steps S201 to S204.

[0242] Step S201, determining the number of requests corresponding to n request types, and determining the model adjustment parameter groups corresponding to the n request types; n is a positive integer.

[0243] Specifically, at least two first business requests within the current distribution period are obtained, and the first business requests with the same request type in the at least two first business requests are divided into the same business request set to obtain n business request sets; the request types corresponding to the n business request sets are different from each other; the request numbers of the first business requests respectively included in the n business request sets are determined, and the request numbers corresponding to the n business request sets are determined as the request numbers corresponding to the n request types within the current distribution period.

[0244] The first feasible implementation method is to predict the request type and request quantity of the next distribution period through the request type and request quantity corresponding to the historical business requests in the historical distribution period, and intelligently implement the pre-loading strategy of the model adjustment parameter group, that is, to pre-load the model adjustment parameter group that may be needed based on historical data and prediction algorithms. For the specific implementation process, please refer to the above Figure 3 The corresponding embodiment is step S101.

[0245] In the second feasible implementation, the parameter manager in the computer device detects the request type and request quantity of the business request in real time, and adjusts the resource allocation through the real-time business request, which can improve the resource utilization of the overall system. It should be understood that the first feasible implementation is to predict the request type and request quantity of the next distribution cycle in advance, so that the processor preloads the model adjustment parameter group and processes the subsequent business requests. The second feasible implementation is to determine the model adjustment parameter group that the processor needs to load based on the request type and request quantity of the real-time business request obtained in the current distribution cycle, and process the real-time business request that has been obtained.

[0246] Step S202, obtaining the average processing time corresponding to n model adjustment parameter groups respectively; the average processing time corresponding to a model adjustment parameter group refers to the average time for processing business requests belonging to the request type corresponding to the model adjustment parameter group by adding a base model with the model adjustment parameter group.

[0247] Specifically, the embodiment of the present application does not limit the model structure and model type of the base model, and can be set according to the needs of the actual application scenario.

[0248] When the base model with the model adjustment parameter group added performs inference for a business request, it first fuses the model adjustment parameter group and the network layer to which the model adjustment parameter group is added to obtain the fused network layer (in fact, it adjusts the weight of the network layer), and processes the input features through the fused network layer.

[0249] Step S203, determining the distribution quantities corresponding to n model adjustment parameter groups respectively according to the request quantities corresponding to n request types and n average processing times; the distribution quantities corresponding to n model adjustment parameter groups respectively are all less than the total number of processors in the processor set.

[0250] Specifically, generate constraints corresponding to n model adjustment parameter groups; the constraints are used to indicate the conditions that need to be met for the n model adjustment parameter groups to be loaded into the processor set; obtain the initial distribution quantities corresponding to the n model adjustment parameter groups, and determine the initial total request processing time according to the request quantities corresponding to the n request types, the n average processing times and the n initial distribution quantities; adjust the n initial distribution quantities according to the initial total request processing time and the constraints to obtain the distribution quantities corresponding to the n model adjustment parameter groups.

[0251] Among them, the n model adjustment parameter groups include a first model adjustment parameter group; the first model adjustment parameter group is any one of the n model adjustment parameter groups; the processor set includes a first processor; the first processor is any one of the processors in the processor set; the specific process of generating constraints corresponding to the n model adjustment parameter groups may include: the first model adjustment parameter group is loaded to at least one processor in the processor set, and is determined as a first constraint; the memory of the first processor is obtained, and the memory utilization upper limit value corresponding to the memory of the first processor is determined according to the memory utilization threshold; the total occupied space of the model adjustment parameter group loaded on the first processor is less than or equal to the memory utilization upper limit value, and is determined as a second constraint; the first constraint and the second constraint are determined as constraints corresponding to the n model adjustment parameter groups.

[0252] Among them, the n model adjustment parameter groups include the first model adjustment parameter group; the first model adjustment parameter group is any one of the n model adjustment parameter groups; according to the number of requests corresponding to the n request types, the n average processing times and the n initial distribution numbers, the specific process of determining the initial total request processing time may include: obtaining a time optimizer, inputting the number of requests corresponding to the n request types, the n average processing times and the n initial distribution numbers into the time optimizer; the time optimizer includes an objective function; through the objective function, the number of requests corresponding to the first model adjustment parameter group and the average processing time corresponding to the first model adjustment parameter group are multiplied to obtain the request processing time corresponding to the first model adjustment parameter group; the request processing time corresponding to the first model adjustment parameter group and the initial distribution number corresponding to the first model adjustment parameter group are ratio processed to obtain the initial request processing time corresponding to the first model adjustment parameter group; the initial request processing times corresponding to the n model adjustment parameter groups are summed to obtain the initial total request processing time.

[0253] Among them, the specific process of adjusting the n initial distribution quantities according to the initial total request processing time and the constraints to obtain the distribution quantities corresponding to the n model adjustment parameter groups may include: inputting the constraints into the time optimizer; in the time optimizer, adjusting the n initial distribution quantities according to the initial total request processing time and the constraints to obtain the n intermediate distribution quantities corresponding to the n model adjustment parameter groups; the n intermediate distribution quantities match the constraints, and the total request processing time corresponding to the n intermediate distribution quantities is less than the initial total request processing time; when the total request processing time corresponding to the n intermediate distribution quantities is the optimal solution of the objective function, the n intermediate distribution quantities are determined as the distribution quantities corresponding to the n model adjustment parameter groups.

[0254] Above Figure 3 Step S103 in the corresponding embodiment describes a method for determining the number of distributions of the model adjustment parameter groups. This step describes another method for determining the number of distributions of the model adjustment parameter groups.

[0255] Each of the n model adjustment parameter groups is loaded on at least one processor, that is, the first constraint condition mentioned above. The first constraint condition can be expressed by the following formula (2):

[0256]

[0257] The value range of i in formula (2) is 1-n, and loraType_i_size represents the number of processors (mainly GPU nodes) that load the i-th model adjustment parameter group;

[0258] In addition, the total memory (i.e., the total occupied space) of the model adjustment parameter group loaded by each processor does not exceed the upper limit of the memory utilization of the processor, which is the second constraint condition mentioned above. The second constraint condition can be expressed by the following formula (3):

[0259]

[0260] Wherein, M in formula (3) is the GPU memory capacity, α is the upper limit of memory utilization (0<α≤1), that is, the memory utilization threshold described above. If the processor is a CPU, then M is the CPU memory capacity, and α is the upper limit of memory utilization.

[0261] The parameter manager in the computer device obtains the initial distribution quantities corresponding to n model adjustment parameter groups, which can be randomly generated or the historical distribution quantities of the model adjustment parameter groups. The embodiment of the present application transforms the problem of determining the distribution quantity of the model adjustment parameter groups into a mathematical optimization problem. In the mathematical model, the distribution quantity corresponding to each model adjustment parameter group needs to meet the above constraints, and at the same time, it is hoped that the time for processing all business requests is minimized.

[0262] The parameter manager obtains the time optimizer, and inputs the number of requests corresponding to the n request types, the n average processing times, and the n initial distribution numbers into the time optimizer. The time optimizer includes an objective function, which can be shown as formula (4):

[0263] f=∑(i=1to n){requestType_i_size / loraType_i_size*avg_processing_time_i}(4)

[0264] Wherein, requestType_i_size in formula (4) indicates the number of requests corresponding to the i-th request type, the goal of the time optimizer is to minimize f, and the meaning of the remaining expressions is explained in the above formula. Through formula (4), the parameter manager can determine the initial total request processing time.

[0265] The embodiment of the present application adopts the optimization concept to generate a time optimizer. The embodiment of the present application does not limit the optimization mathematical tool and can be any optimization tool, such as using a linear programming solver (such as the simplex method).

[0266] In the time optimizer, the parameter manager adjusts the n initial distribution quantities based on the initial total request processing time. The adjustment strategy is to move closer to the constraints, that is, the adjusted n initial distribution quantities (also called n intermediate distribution quantities) meet the constraints, and when the n intermediate distribution quantities meet the above constraints, it is hoped that the total request processing time corresponding to the n intermediate distribution quantities is minimized.

[0267] Step S204, according to the distribution quantities corresponding to the n model adjustment parameter groups, respectively allocate processors in the processor set to the n model adjustment parameter groups; the distribution quantity corresponding to a model adjustment parameter group is used to allocate processors in the processor set to the model adjustment parameter group; the number of processors allocated to a model adjustment parameter group is the same as the distribution quantity corresponding to the model adjustment parameter group; the processors in the processor set are used to load the allocated model adjustment parameter group, and are used to process business requests belonging to the request type corresponding to the loaded model adjustment parameter group through the loaded model adjustment parameter group and the base model.

[0268] Specifically, the n model adjustment parameter groups include a first model adjustment parameter group; the first model adjustment parameter group is any one of the n model adjustment parameter groups; the historical distribution quantity corresponding to the first model adjustment parameter group is obtained, and the historical distribution quantity is compared with the distribution quantity corresponding to the first model adjustment parameter group to obtain a quantity comparison result; according to the quantity comparison result and the second processor corresponding to the first model adjustment parameter group, a processor in the processor set is allocated to the first model adjustment parameter group; the second processor is a processor allocated to the first model adjustment parameter group according to the historical distribution quantity, and the number of the second processors is the same as the historical distribution quantity.

[0269] Among them, the specific process of allocating a processor in a processor set to the first model adjustment parameter group according to the quantity comparison result and the second processor corresponding to the first model adjustment parameter group may include: if the quantity comparison result is that the historical distribution quantity is the same as the distribution quantity corresponding to the first model adjustment parameter group, then the second processor is allocated as the processor corresponding to the first model adjustment parameter group; the second processor belongs to the processor set; if the historical distribution quantity is less than the distribution quantity corresponding to the first model adjustment parameter group, then a third processor different from the second processor is determined in the processor set, and the third processor and the second processor are allocated as processors corresponding to the first model adjustment parameter group; the number of the third processor is the same as the first quantity difference; the first quantity difference refers to the difference between the historical distribution quantity and the distribution quantity corresponding to the first model adjustment parameter group; if the historical distribution quantity is greater than the distribution quantity corresponding to the first model adjustment parameter group, then a fourth processor is determined in the second processor, and the fourth processor is allocated as the processor corresponding to the first model adjustment parameter group; the number of the fourth processor is the same as the distribution quantity corresponding to the first model adjustment parameter group.

[0270] Among them, this step may also include: if the number of historical distributions is less than the number of distributions corresponding to the first model adjustment parameter group, generating a loading instruction for the first model adjustment parameter group in the parameter manager; sending the loading instruction to the third processor through the parameter manager; the third processor is used to load the first model adjustment parameter group according to the loading instruction; if the number of historical distributions is greater than the number of distributions corresponding to the first model adjustment parameter group, generating an unloading instruction for the first model adjustment parameter group in the parameter manager; sending the unloading instruction to the fifth processor through the parameter manager; the fifth processor is used to unload the loaded first model adjustment parameter group according to the unloading instruction; the fifth processor includes the processor in the second processor except the fourth processor.

[0271] In summary, by adopting the embodiments of the present application, a dynamic load balancing mechanism based on real-time detection can be implemented, an adaptive resource allocation algorithm can be introduced, and the system configuration can be automatically adjusted according to historical data and current status. Active resource adjustment based on time periods and prediction models is supported, which can reduce the memory usage of the processor. Therefore, the memory resources of the processor used for model reasoning are increased, thereby improving the memory resource utilization of the processor.

[0272] See also Figure 7 , Figure 7 This is a flow diagram of a data processing method provided in an embodiment of the present application. Figure 3 The data processing method is executed by a computer device, such as Figure 7 As shown, the method may at least include the following steps S301-S311.

[0273] Step S301, determining the number of requests corresponding to n request types, and determining the model adjustment parameter groups corresponding to the n request types; n is a positive integer.

[0274] Step S302, obtaining the average processing time corresponding to n model adjustment parameter groups respectively; the average processing time corresponding to a model adjustment parameter group refers to the average time for processing business requests belonging to the request type corresponding to the model adjustment parameter group by adding a base model with the model adjustment parameter group.

[0275] Step S303, according to the number of requests corresponding to the n request types and the n average processing times, respectively allocate processors in the processor set to the n model adjustment parameter groups; the processors in the processor set are used to load the allocated model adjustment parameter groups, and to process business requests belonging to the request types corresponding to the loaded model adjustment parameter groups through the loaded model adjustment parameter groups and the base model.

[0276] For the specific implementation process of steps S301 to S303, please refer to the above Figure 3 Steps S101 to S103 in the corresponding embodiment are not described in detail here.

[0277] Step S304: In the parameter manager, determine the processor information corresponding to the n model adjustment parameter groups respectively; wherein one piece of processor information is used to indicate the processor to which one model adjustment parameter group is allocated.

[0278] For details, please refer to Figure 8 , Figure 8 This is a data processing scenario provided by an embodiment of the present application. Figure 3 . Figure 8In example n=3, the three model adjustment parameter groups are respectively an answer type parameter group, a video recommendation type parameter group and a text generation type parameter group. The answer type parameter group is used to process answer type business requests, the video recommendation type parameter group is used to process video recommendation type business requests, and the text generation type parameter group is used to process text generation type business requests.

[0279] The process of assigning processors to the 3 model tuning parameter groups by the parameter manager, see above Figure 3 The description of step S103 in the corresponding embodiment and the above Figure 6 In the description of step S203-step S204 in the corresponding embodiments, the above two embodiments respectively describe a method for respectively allocating processors to n model adjustment parameter groups. In actual application, one method can be selected from the above two methods according to needs.

[0280] Please see again Figure 8 , the parameter manager determines the processor information corresponding to the three model adjustment parameter groups, Figure 8 The processor information corresponding to the example answer type parameter group includes processor 1, processor 2, and processor 3, which means that the answer type parameter group is assigned to processor 1, processor 2, and processor 3, that is, only processor 1, processor 2, and processor 3 in the processor set load the answer type parameter group.

[0281] Figure 8 The processor information corresponding to the example video recommendation type parameter group includes processor 1 and processor 4, which indicates that the video recommendation type parameter group is allocated to processor 1 and processor 4, that is, only processor 1 and processor 4 in the processor set load the video recommendation type parameter group.

[0282] Figure 8 The processor information corresponding to the example text generation type parameter group includes processor 3, processor 5, and processor 6, which means that the text generation type parameter group is assigned to processor 3, processor 5, and processor 6, that is, only processor 3, processor 5, and processor 6 in the processor set load the text generation type parameter group.

[0283] Step S305, obtaining parameter group identifiers corresponding to n model adjustment parameter groups respectively; wherein one parameter group identifier is used to indicate one model adjustment parameter group.

[0284] For details, please refer to Figure 8 , Figure 8 The parameter group identification example corresponding to the answer type parameter group is 11...11, the video recommendation type parameter group example is 22...22, and the text generation type parameter group example is 33...33.

[0285] Step S306, constructing n index relationships according to the n parameter group identifiers and the n processor information; wherein one index relationship includes one parameter group identifier and one processor information.

[0286] Specifically, the parameter manager constructs an index relationship between a parameter group identifier and the processor information corresponding to the parameter group identifier, such as Figure 8 As shown, an index relationship is constructed between parameter group identifiers 11...11 for indicating an answer type parameter group and processor information (including processor 1, processor 2, and processor 3) for indicating a processor to which the answer type parameter group is assigned; an index relationship is constructed between parameter group identifiers 22...22 for indicating a video recommendation type parameter group and processor information (including processor 1 and processor 4) for indicating a processor to which the video recommendation type parameter group is assigned; and an index relationship is constructed between parameter group identifiers 33...33 for indicating a text generation type parameter group and processor information (including processor 3, processor 5, and processor 6) for indicating a processor to which the text generation type parameter group is assigned.

[0287] Step S307, determine the n index relationships as a request routing table, and send the request routing table to the request router through the parameter manager; the request router is used to forward the second business request to the target processor in the processor set according to the request routing table when determining the target model adjustment parameter group corresponding to the second business request; the target processor refers to the processor loaded with the target model adjustment parameter group; the target model adjustment parameter group refers to the model adjustment parameter group corresponding to the request type to which the second business request belongs.

[0288] Step S308: In the request router, the parameter group identifier carried in the second service request is determined as the target parameter group identifier.

[0289] Specifically, the request router is responsible for distributing pending service requests to appropriate processors (mainly GPU nodes). The request router and the parameter manager can be deployed on the same device or on different devices, which is not limited in this application. In the example of the embodiment of this application, the request router and the parameter manager are deployed on the same device.

[0290] Among them, if the n request types are determined based on the service requests obtained in the current distribution period, the second service request belongs to the service request obtained in the current distribution period (also referred to as the first service request in this application). If the n request types are determined based on the historical service requests obtained in the second distribution period associated with the first distribution period (referring to the next distribution period of the current distribution period), the second service request refers to the service request obtained in the first distribution period.

[0291] Step S309, querying the target parameter group identifier in the request routing table, obtaining processor information having an index relationship with the target parameter group identifier in the request routing table, and determining the obtained processor information as candidate processor information.

[0292] Specifically, for ease of understanding and description, the parameter group identifier carried by the second service request in the embodiment of the present application is Figure 8 11…11 in , so the request type of the second service request is question-answer type. Figure 8 , requesting the router to Figure 8 The target parameter group identifier is queried in the example request routing table 80a. This step example is 11...11, and the processor information having an index relationship with the target parameter group identifier in the request routing table 80a is obtained, that is, Figure 8 The processor 1, processor 2, and processor 3 in the example request the router to determine the acquired processor information as candidate processor information.

[0293] Step S310: Determine a candidate processor indicated by the candidate processor information in the processor set; the candidate processor is loaded with a target model adjustment parameter group indicated by the target parameter group identifier.

[0294] Step S311, determine the target processor among the candidate processors, and distribute the second business request to the target processor by requesting the router; the target processor is used to add the loaded target model adjustment parameter group to the base model according to the parameter group identifier carried by the second business request; the base model with the added target model adjustment parameter group is used to process the second business request.

[0295] Specifically, the memory utilization of the candidate processors is obtained, and the memory availability of the candidate processors is determined according to the memory utilization of the candidate processors; and the target processor is determined among the candidate processors according to the memory availability, the request queue length of the candidate processors, and the network delay of the candidate processors.

[0296] Among them, according to the memory availability, the request queue length of the candidate processors and the network delay of the candidate processors, the specific process of determining the target processor among the candidate processors may include: determining the first inverse of the request queue length of the candidate processor, and determining the second inverse of the network delay of the candidate processor; performing weighted fusion processing on the memory availability, the first inverse and the second inverse to obtain the score of the candidate processor; obtaining the maximum score among the scores of the candidate processors, and determining the processor with the maximum score among the candidate processors as the target processor.

[0297] The processor set includes a sixth processor, and the sixth processor is any processor in the processor set; this step may also include: in a request router, obtaining a request queue corresponding to the sixth processor; the request queue includes a business requests; a is a positive integer; the business requests with the same request type in the a business requests are divided into the same business request set to obtain b business request sets; b is a positive integer, and b is less than or equal to a; counting the number of business requests respectively included in the b business request sets, and obtaining the number of requests respectively corresponding to the b business request sets; determining the request types respectively corresponding to the b business request sets, and determining the processing priorities respectively corresponding to the b request types according to the b request numbers; wherein there is a positive relationship between the processing priority corresponding to a request type and the number of requests corresponding to the request type; sending the processing priorities respectively corresponding to the b request types to the sixth processor through the request router; the sixth processor is used to process the a business requests according to the processing priorities respectively corresponding to the b request types.

[0298] This step may also include: determining a waiting processing time threshold of the second service request in the request router; sending the waiting processing time threshold to the target processor through the request router; when the target processor determines that the waiting processing time of the second service request exceeds the waiting processing time threshold, the target processor returns the second service request to the request router and deletes the second service request in the request queue; obtaining the second service request returned by the target processor through the request router, and redistributing the second service request.

[0299] The request router intelligently forwards the second service request to the most appropriate GPU machine based on the request type, current system load, and distribution of the model adjustment parameter group. Specifically, the optimal processor is selected among the processors loaded with the target model adjustment parameter group, taking into account the following factors: 1. Current node load, 2. Request queue length, 3. Network latency.

[0300] For the second service request, the request router calculates the score (i.e., rating) of each candidate processor. The score calculation method can be expressed by the following formula (5):

[0301] Score=w1*(1-load_j)+w2*(1 / queue_length_j)+w3*(1 / network_latency_j)(5)

[0302] Where w1, w2, and w3 are weight factors that can be adjusted according to system characteristics. The value range of J is 1 to k, where k is a positive integer and refers to the number of candidate processors. load_j represents the memory utilization of the jth candidate processor, 1-load_j represents the memory availability of the jth candidate processor, queue_length_j represents the request queue length of the jth candidate processor, and network_latency_j represents the network latency of the jth candidate processor.

[0303] The key point of the embodiment of the present application is to solve the limitations of the multi-model adjustment parameter group framework, while providing a more flexible and efficient resource management and request processing mechanism. The core is to introduce a parameter manager to realize dynamic model adjustment parameter group distribution and intelligent request routing. The embodiment of the present application avoids loading all model adjustment parameter groups on each processor (especially GPU machine), thereby solving the problems of high video memory pressure, frequent fan-in and fan-out time-consuming, and low concurrency efficiency. The parameter manager dynamically adjusts the distribution of the model adjustment parameter group in the GPU cluster according to the real-time request mode and load conditions to ensure the optimal use of resources.

[0304] The intelligent routing mechanism accurately forwards the service request to the appropriate GPU machine to improve processing efficiency. The embodiment of the present application supports multiple base models, breaking through the limitation of a single model.

[0305] Through these innovations, this application achieves more efficient resource utilization, more flexible model support, and more optimized request processing, providing strong technical support for large-scale AI services.

[0306] See also Fig. 9 , Fig. 9 This is a flow diagram of a data processing method provided in an embodiment of the present application. Figure 4 The method is executed by a computer device. Fig. 9 As shown, the data processing method may at least include the following steps S401 to S404.

[0307] Step S401, obtain a business request carrying a target parameter group identifier through a request router, and obtain a request routing table according to the business request; the request routing table is generated by the parameter manager according to n parameter group identifiers and n processor information after the parameter manager allocates processors in a processor set to n model adjustment parameter groups according to the number of requests corresponding to n request types and the average processing time corresponding to n model adjustment parameter groups; n is a positive integer; one model adjustment parameter group corresponds to one request type; one parameter group identifier is used to indicate one model adjustment parameter group; one processor information is used to indicate one processor.

[0308] For details, please refer to Fig.10, Fig.10 Schematic diagram of an interactive data processing method provided in an embodiment of the present application. Fig.10 As shown, the interaction process may include steps S51 to S5a.

[0309] Steps S51-S52 are executed by the parameter manager. Step S51: allocate processors in the processor set to the n model adjustment parameter groups according to the number of requests corresponding to the n request types and the average processing time corresponding to the n model adjustment parameter groups. Step S52: construct a request routing table according to the processors assigned to the n model adjustment parameter groups and the n parameter group identifiers. The parameter manager determines the processor information of the processor assigned to each model adjustment parameter group, determines the parameter group identifier corresponding to each model adjustment parameter group, constructs an index relationship between a parameter group identifier and the processor information corresponding to the model adjustment parameter group indicated by the parameter group identifier, and constructs the index relationship corresponding to the n parameter group identifiers into a request routing table. Step S53: The parameter manager sends the request routing table to the request router.

[0310] Steps S54 to S57 are executed by the request router. Step S54: obtain a service request carrying a target parameter group identifier.

[0311] The service request of the embodiment of the present application is equivalent to the second service request in the embodiment corresponding to the figure, so the processing process of the request router for the service request can be referred to above. Figure 7 The corresponding embodiment includes steps S308 to S311.

[0312] In addition, please refer to the above for the process of parameter manager assigning processors to n model adjustment parameter groups. Figure 3 as well as Figure 6 The descriptions in the corresponding embodiments are not repeated here.

[0313] Step S402: query the target parameter group identifier in the request routing table to obtain candidate processor information having an index relationship with the target parameter group identifier in the request routing table.

[0314] For details, please refer to Fig.10 , step S402 may include Fig.10 Step S55-step S56 in the above process. Step S55: query the target parameter group identifier in the request routing table. Step S56: obtain candidate processor information having an index relationship with the target parameter group identifier in the request routing table.

[0315] Step S403: Determine a candidate processor indicated by the candidate processor information in the processor set; the candidate processor is loaded with the target model adjustment parameter group indicated by the target parameter group identifier.

[0316] Step S404, determine the target processor from the candidate processors, and distribute the business request to the target processor through the request router; the target processor is used to add the loaded target model adjustment parameter group to the base model according to the target parameter group identifier carried by the business request; the base model with the added target model adjustment parameter group is used to process the business request.

[0317] Specifically, the memory utilization of the candidate processors is obtained, and the memory availability of the candidate processors is determined according to the memory utilization of the candidate processors; and the target processor is determined among the candidate processors according to the memory availability, the request queue length of the candidate processors, and the network delay of the candidate processors.

[0318] Among them, according to the memory availability, the request queue length of the candidate processors and the network delay of the candidate processors, the specific process of determining the target processor among the candidate processors may include: determining the first inverse of the request queue length of the candidate processor, and determining the second inverse of the network delay of the candidate processor; performing weighted fusion processing on the memory availability, the first inverse and the second inverse to obtain the score of the candidate processor; obtaining the maximum score among the scores of the candidate processors, and determining the processor with the maximum score among the candidate processors as the target processor.

[0319] For details, please refer to Fig.10 , step S404 may include Fig.10 Step S57-Step S58 in the process. Step S57: Determine the target processor from the candidate processors indicated by the candidate processor information. Step S58: Request the router to send the service request to the target processor.

[0320] Step S59-step S5a is executed by the target processor. Step S59, obtain the target parameter group identifier carried in the service request. Step S5a, add the target model adjustment parameter group indicated by the target parameter group identifier to the base model, and process the service request through the base model with the added target model adjustment parameter group.

[0321] Fig.10 The three components described, namely the parameter manager, the request router and the target processor, can be deployed on the same device or on different devices.

[0322] Among them, this step may also include: in the request router, obtaining the request queue corresponding to the target processor; the request queue includes c business requests; c is a positive integer; the business requests with the same request type in the c business requests are divided into the same business request set to obtain d business request sets; d is a positive integer, and d is less than or equal to c; counting the number of business requests included in the d business request sets, and obtaining the number of requests corresponding to the d business request sets; determining the request types corresponding to the d business request sets, and determining the processing priorities corresponding to the d request types according to the d request numbers; wherein there is a positive relationship between the processing priority corresponding to a request type and the number of requests corresponding to the request type; sending the processing priorities corresponding to the d request types to the target processor through the request router; the target processor is used to process the c business requests according to the processing priorities corresponding to the d request types.

[0323] Among them, this step may also include: in the request router, determining the waiting processing time threshold of the service request; sending the waiting processing time threshold to the target processor through the request router; when the target processor determines that the waiting processing time of the service request exceeds the waiting processing time threshold, the service request is returned to the request router and the service request in the request queue is deleted; through the request router, the service request returned by the target processor is obtained, and the service request is redistributed and processed.

[0324] The processors in the processor cluster are the computing units that actually perform inference tasks. Each processor contains the following: 1. Basic language model, i.e., base model; 2. Dynamically loaded model adjustment parameter group; 3. Local request queue; 4. Resource detection component.

[0325] The processor dynamically loads or unloads the model adjustment parameter group according to the load instruction or unload instruction sent by the parameter manager. The processor adopts an efficient memory management strategy, such as paging to store the model adjustment parameter group.

[0326] To improve GPU utilization, the GPU (as a processor) implements an intelligent batching strategy, which can include the following:

[0327] a) Dynamic batch size: The batch size is adjusted dynamically based on the current load and request characteristics (e.g. average processing time, priority of request type).

[0328] b) Request aggregation: Business requests of the same request type are aggregated together to reduce the switching overhead of the model adjustment parameter group.

[0329] c) Time window batch processing: collect business requests within a fixed time window to balance latency and throughput.

[0330] Combination Figure 3 , Figure 6 , Figure 7 as well as Fig. 9 It can be seen that the use of this application includes but is not limited to the following beneficial effects:

[0331] 1. Significantly improve system performance

[0332] a) Reduce request latency

[0333] Effect: Average request response time reduced by 40-60%.

[0334] reason:

[0335] The dynamic model tuning parameter group distribution algorithm ensures that frequently used model tuning parameter groups are available on multiple GPUs.

[0336] Intelligent request routing sends business requests to the most suitable GPU, reducing waiting time.

[0337] The model tuning parameter group preloading strategy reduces the cold start time.

[0338] b) Improve system throughput

[0339] Effect: The overall system throughput is increased by 50-100%.

[0340] reason:

[0341] Batch optimization allows the GPU to process multiple business requests more efficiently.

[0342] Dynamic load balancing ensures that all GPU resources are fully utilized.

[0343] The frequency of switching the model adjustment parameter group is reduced, and the related overhead is reduced.

[0344] c) Optimize resource utilization

[0345] Effect: 30-50% increase in GPU utilization.

[0346] reason:

[0347] The dynamic model adjusts the parameter group distribution to allocate resources according to actual needs.

[0348] Adaptive batching adjusts the batch size based on the current load.

[0349] The multi-level cache strategy reduces the model adjustment parameter group module loading time.

[0350] 2. Enhance system flexibility and adaptability

[0351] a) Dynamically adapt to load changes

[0352] Effect: The system can maintain stability when the load suddenly increases, and the peak processing capacity is increased by 40%.

[0353] reason:

[0354] The adaptive model adjusts the parameter group distribution algorithm to adjust resource allocation in real time.

[0355] Elastic resource management supports dynamic addition or removal of GPU nodes.

[0356] Request prediction and resource reservation mechanisms respond to load changes in advance.

[0357] b) Support diverse model adjustment parameter groups

[0358] Effect: A single system can simultaneously serve more than 100 different model adjustment parameter groups to meet diverse needs.

[0359] reason:

[0360] Dynamic model adjustment parameter group loading / unloading mechanism.

[0361] Fine-grained resource management and request routing.

[0362] An extensible plugin system enables fast integration of new model tuning parameter sets.

[0363] c) Quickly adapt to new demands

[0364] Results: 80% reduction in deployment and integration time for new model tuning parameter sets.

[0365] reason:

[0366] Standardized registration and management mechanism for model tuning parameter groups.

[0367] Dynamic configuration updates do not require a system restart.

[0368] Automated performance optimization and resource allocation.

[0369] 3. Improve system reliability and stability

[0370] a) Enhanced fault tolerance

[0371] Effect: When 20% of the GPU nodes fail, the system can still maintain more than 90% of its service capacity.

[0372] reason:

[0373] Fault detection and automatic recovery mechanisms.

[0374] Automatic redistribution of model tuning parameter groups and requests.

[0375] Distributed architecture reduces the impact of single points of failure.

[0376] b) Improve system stability

[0377] Effect: The system's average trouble-free operation time increased by 200%.

[0378] reason:

[0379] Comprehensive detection and early warning systems detect potential problems in a timely manner.

[0380] Adaptive resource management prevents resource exhaustion.

[0381] Optimized memory management reduces memory-related crashes.

[0382] For further information, see Fig.11 , Fig.11 This is a schematic diagram of the structure of a data processing device provided in an embodiment of the present application. Figure 1 The above-mentioned data processing device 1 can be used to execute the corresponding steps of the method provided in the embodiment of the present application. Fig.11 As shown, the data processing device 1 may include: a determination module 11 , an acquisition module 12 and an allocation module 13 .

[0383] The determination module 11 is used to determine the number of requests corresponding to the n request types, and determine the model adjustment parameter groups corresponding to the n request types; n is a positive integer;

[0384] An acquisition module 12 is used to acquire average processing times corresponding to n model adjustment parameter groups respectively; the average processing time corresponding to a model adjustment parameter group refers to the average time for processing a business request belonging to a request type corresponding to the model adjustment parameter group by using a base model to which the model adjustment parameter group is added;

[0385] The allocation module 13 is used to allocate processors in the processor set to n model adjustment parameter groups respectively according to the number of requests corresponding to the n request types and the n average processing times; the processors in the processor set are used to load the allocated model adjustment parameter groups, and to process business requests belonging to the request types corresponding to the loaded model adjustment parameter groups through the loaded model adjustment parameter groups and the base model.

[0386] The specific description of the data processing device 1 can be found in the above-mentioned data processing method, which will not be repeated here.

[0387] For further information, see Fig.12 , Fig.12 This is a schematic diagram of the structure of a data processing device provided in an embodiment of the present application. Figure 2The above-mentioned data processing device 2 can be used to execute the corresponding steps in the method provided in the embodiment of the present application. Fig.12 As shown, the data processing device 2 may include: an acquisition module 21 , a determination module 22 and a distribution module 23 .

[0388] The acquisition module 21 is used to acquire the service request carrying the target parameter group identifier through the request router, and acquire the request routing table according to the service request; the request routing table is generated by the parameter manager according to n parameter group identifiers and n processor information after the processors in the processor set are respectively allocated to the n model adjustment parameter groups according to the number of requests corresponding to the n request types and the average processing time corresponding to the n model adjustment parameter groups; n is a positive integer; one model adjustment parameter group corresponds to one request type; one parameter group identifier is used to indicate one model adjustment parameter group; one processor information is used to indicate one processor;

[0389] The acquisition module 21 is further used to query the target parameter group identifier in the request routing table, and obtain the candidate processor information having an index relationship with the target parameter group identifier in the request routing table;

[0390] The determination module 22 is used to determine the candidate processor indicated by the candidate processor information in the processor set; the candidate processor is loaded with the target model adjustment parameter group indicated by the target parameter group identifier;

[0391] The distribution module 23 is used to determine the target processor from the candidate processors and distribute the business request to the target processor through the request router; the target processor is used to add the loaded target model adjustment parameter group to the base model according to the target parameter group identifier carried by the business request; the base model with the target model adjustment parameter group added is used to process the business request.

[0392] The specific description of the data processing device 2 can be found in the aforementioned data processing method, which will not be repeated here.

[0393] In this application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0394] For further information, see Fig.13 , Fig.13 is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. The computer device may be Figure 1 At least one of the terminal device or service server shown. Fig.13 As shown, the computer device 1000 may include: at least one processor 1001, such as a CPU, at least one network interface 1004, a user interface 1003, a memory 1005, and at least one communication bus 1002. The communication bus 1002 is used to realize connection and communication between these components.

[0395] In some embodiments, the user interface 1003 may include a display screen (Display), a keyboard (Keyboard), and the network interface 1004 may optionally include a standard wired interface, a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or a non-volatile memory (non-volatile memory), such as at least one disk memory. The memory 1005 may also be at least one storage device located away from the aforementioned processor 1001.

[0396] like Fig.13 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a device control application.

[0397] exist Fig.13 In the computer device 1000 shown, the network interface 1004 can provide network communication functions; the user interface 1003 is mainly used to provide an input interface for the user; and the processor 1001 can be used to call the device control application stored in the memory 1005 to implement the method steps in each embodiment of the present application.

[0398] It should be understood that the computer device 1000 described in the embodiment of the present application can execute the description of the data processing method or device in the above embodiments, which will not be repeated here. In addition, the description of the beneficial effects of adopting the same method will not be repeated.

[0399] The present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the description of the data processing method or device in the above embodiments is implemented, which will not be repeated here. In addition, the description of the beneficial effects of the same method will not be repeated.

[0400] The computer-readable storage medium may be the data processing device provided in any of the aforementioned embodiments or the internal storage unit of the computer device, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc., provided on the computer device.

[0401] Furthermore, the computer-readable storage medium may also include both an internal storage unit of the computer device and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0402] The present application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. The processor of the computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device can execute the description of the data processing method or device in the above embodiments, which will not be repeated here. In addition, the description of the beneficial effects of the same method will not be repeated.

[0403] The terms "first", "second", etc. in the description, claims, and drawings of the embodiments of the present application are used to distinguish different objects, rather than to describe a specific order. In addition, the term "comprising" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, device, product, or equipment that includes a series of steps or units is not limited to the listed steps or modules, but optionally includes steps or modules that are not listed, or optionally includes other step units inherent to these processes, methods, devices, products, or equipment.

[0404] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0405] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A data processing method, characterized in that: The method is performed by a computer device, and the method comprises: Determine the number of requests corresponding to n request types, and determine the model adjustment parameter groups corresponding to the n request types, wherein n is a positive integer; Obtain the average processing times corresponding to n model adjustment parameter groups respectively; the average processing time corresponding to a model adjustment parameter group refers to the average time for processing business requests belonging to the request type corresponding to the model adjustment parameter group by using the base model to which the model adjustment parameter group is added; According to the number of requests corresponding to the n request types and the n average processing times, processors in the processor set are respectively allocated to the n model adjustment parameter groups; the processors in the processor set are used to load the allocated model adjustment parameter groups, and to process business requests belonging to the request type corresponding to the loaded model adjustment parameter groups through the loaded model adjustment parameter groups and the base model.

2. The method according to claim 1, characterized in that The allocating processors in the processor set to the n model adjustment parameter groups respectively according to the request quantities respectively corresponding to the n request types and the n average processing times, comprises: Determine, according to the number of requests corresponding to the n request types and the n average processing times, the number of distributions corresponding to the n model adjustment parameter groups respectively; the number of distributions corresponding to the n model adjustment parameter groups respectively are all less than the total number of processors in the processor set; According to the distribution quantities corresponding to the n model adjustment parameter groups, processors in the processor set are respectively allocated to the n model adjustment parameter groups; the distribution quantity corresponding to a model adjustment parameter group is used to allocate processors in the processor set to the model adjustment parameter group; the number of processors allocated to a model adjustment parameter group is the same as the distribution quantity corresponding to the model adjustment parameter group.

3. The method according to claim 2, characterized in that The determining, according to the request quantities respectively corresponding to the n request types and the n average processing times, the distribution quantities respectively corresponding to the n model adjustment parameter groups includes: Generating constraint conditions corresponding to the n model adjustment parameter groups; the constraint conditions are used to indicate conditions that need to be satisfied when the n model adjustment parameter groups are loaded into the processor set; Obtaining initial distribution quantities corresponding to the n model adjustment parameter groups, respectively, and determining an initial total request processing time according to the request quantities corresponding to the n request types, respectively, the n average processing times, and the n initial distribution quantities; According to the initial total request processing time and the constraint condition, the n initial distribution quantities are adjusted to obtain the distribution quantities corresponding to the n model adjustment parameter groups respectively.

4. The method according to claim 3, characterized in that The n model adjustment parameter groups include a first model adjustment parameter group; the first model adjustment parameter group is any one of the n model adjustment parameter groups; the processor set includes a first processor; The first processor is any processor in the processor set; The generating of the constraint conditions corresponding to the n model adjustment parameter groups includes: The first model adjustment parameter group is loaded into at least one processor in the processor set and determined as a first constraint condition; Obtaining the memory of the first processor, and determining a memory utilization upper limit value corresponding to the memory of the first processor according to a memory utilization threshold; The total occupied space of the model adjustment parameter group loaded on the first processor is less than or equal to the upper limit of memory utilization, and is determined as a second constraint condition; The first constraint condition and the second constraint condition are determined as the constraint conditions corresponding to the n model adjustment parameter groups.

5. The method according to claim 3, characterized in that: The n model adjustment parameter groups include a first model adjustment parameter group; the first model adjustment parameter group is any one of the n model adjustment parameter groups; The determining the initial total request processing time according to the number of requests respectively corresponding to the n request types, the n average processing times and the n initial distribution numbers includes: Obtain a time optimizer, and input the number of requests, the n average processing times, and the n initial distribution numbers respectively corresponding to the n request types into the time optimizer; the time optimizer includes an objective function; By using the objective function, the number of requests corresponding to the first model adjustment parameter group and the average processing time corresponding to the first model adjustment parameter group are multiplied to obtain the request processing time corresponding to the first model adjustment parameter group; Performing ratio processing on the request processing time corresponding to the first model adjustment parameter group and the initial distribution quantity corresponding to the first model adjustment parameter group to obtain the initial request processing time corresponding to the first model adjustment parameter group; The initial request processing times corresponding to the n model adjustment parameter groups are summed to obtain the initial total request processing time.

6. The method according to claim 5, characterized in that The adjusting process of the n initial distribution quantities according to the initial total request processing time and the constraint condition to obtain the distribution quantities corresponding to the n model adjustment parameter groups respectively includes: inputting the constraints into the time optimizer; In the time optimizer, the n initial distribution quantities are adjusted according to the initial total request processing time and the constraint condition to obtain n intermediate distribution quantities corresponding to the n model adjustment parameter groups respectively; the n intermediate distribution quantities match the constraint condition, and the total request processing time corresponding to the n intermediate distribution quantities is less than the initial total request processing time; When the total request processing time corresponding to the n intermediate distribution quantities is the optimal solution of the objective function, the n intermediate distribution quantities are determined as the distribution quantities corresponding to the n model adjustment parameter groups respectively.

7. The method according to claim 2, characterized in that The n model adjustment parameter groups include a first model adjustment parameter group; the first model adjustment parameter group is any one of the n model adjustment parameter groups; The allocating processors in the processor set to the n model adjustment parameter groups respectively according to the distribution quantities respectively corresponding to the n model adjustment parameter groups, comprises: Obtaining a historical distribution quantity corresponding to the first model adjustment parameter group, and comparing the historical distribution quantity with a distribution quantity corresponding to the first model adjustment parameter group to obtain a quantity comparison result; According to the quantity comparison result and the second processor corresponding to the first model adjustment parameter group, a processor in the processor set is allocated to the first model adjustment parameter group; the second processor is the processor allocated to the first model adjustment parameter group according to the historical distribution quantity, and the number of the second processors is the same as the historical distribution quantity.

8. The method according to claim 7, characterized in that The allocating a processor in the processor set to the first model adjustment parameter group according to the quantity comparison result and the second processor corresponding to the first model adjustment parameter group includes: If the quantity comparison result is that the historical distribution quantity is the same as the distribution quantity corresponding to the first model adjustment parameter group, the second processor is allocated as the processor corresponding to the first model adjustment parameter group; the second processor belongs to the processor set; If the historical distribution quantity is less than the distribution quantity corresponding to the first model adjustment parameter group, determine a third processor different from the second processor in the processor set, and allocate the third processor and the second processor as processors corresponding to the first model adjustment parameter group; the number of the third processors is the same as the first number difference; the first number difference refers to the difference between the historical distribution quantity and the distribution quantity corresponding to the first model adjustment parameter group; If the number of historical distributions is greater than the number of distributions corresponding to the first model adjustment parameter group, a fourth processor is determined in the second processor and the fourth processor is assigned as the processor corresponding to the first model adjustment parameter group; the number of the fourth processors is the same as the number of distributions corresponding to the first model adjustment parameter group.

9. The method according to claim 8, characterized in that The method further comprises: If the number of historical distributions is less than the number of distributions corresponding to the first model adjustment parameter group, generating a loading instruction for the first model adjustment parameter group in the parameter manager; The loading instruction is sent to the third processor through the parameter manager; the third processor is used to load the first model adjustment parameter group according to the loading instruction; If the number of historical distributions is greater than the number of distributions corresponding to the first model adjustment parameter group, then generating, in the parameter manager, an uninstallation instruction for the first model adjustment parameter group; The parameter manager sends the unloading instruction to the fifth processor; the fifth processor is used to unload the loaded first model adjustment parameter group according to the unloading instruction; the fifth processor includes the processors in the second processor except the fourth processor.

10. The method according to claim 2, characterized in that The n model adjustment parameter groups include a second model adjustment parameter group; the second model adjustment parameter group is any one of the n model adjustment parameter groups; The determining, according to the request quantities respectively corresponding to the n request types and the n average processing times, the distribution quantities respectively corresponding to the n model adjustment parameter groups includes: Multiplying the number of requests corresponding to the second model adjustment parameter group and the average processing time corresponding to the second model adjustment parameter group to obtain the request processing time corresponding to the second model adjustment parameter group; A ratio processing is performed on the request processing time corresponding to the second model adjustment parameter group and the occupied space of the second model adjustment parameter group to obtain the distribution quantity corresponding to the second model adjustment parameter group.

11. The method according to claim 10, characterized in that The allocating processors in the processor set to the n model adjustment parameter groups respectively according to the distribution quantities respectively corresponding to the n model adjustment parameter groups, comprises: Sorting the distribution quantities corresponding to the n model adjustment parameter groups from large to small to obtain n sorted distribution quantities; According to the sorted n distribution numbers, the processors in the processor set are allocated to the n model adjustment parameter groups in sequence; wherein the number of processors in the processor set allocated to the second model adjustment parameter group is the same as the distribution number corresponding to the second model adjustment parameter group; the order in which the second model adjustment parameter group is allocated to the processors is the same as the order of the distribution numbers corresponding to the second model adjustment parameter group in the sorted n distribution numbers; the total occupied space of the model adjustment parameter group loaded by the processor to which the second model adjustment parameter group is allocated is less than or equal to the available memory of the processor to which the second model adjustment parameter group is allocated.

12. The method according to claim 1, characterized in that The method further comprises: In the parameter manager, determining the processor information corresponding to the n model adjustment parameter groups respectively; wherein one processor information is used to indicate the processor to which one model adjustment parameter group is allocated; Obtain parameter group identifiers corresponding to the n model adjustment parameter groups respectively; wherein one parameter group identifier is used to indicate one model adjustment parameter group; According to the n parameter group identifiers and the n processor information, n index relationships are constructed; wherein one index relationship includes one parameter group identifier and one processor information; Determine the n index relationships as a request routing table; The request routing table is sent to the request router through the parameter manager; the request router is used to forward the second business request to the target processor in the processor set according to the request routing table when determining the target model adjustment parameter group corresponding to the second business request; the target processor refers to the processor loaded with the target model adjustment parameter group; the target model adjustment parameter group refers to the model adjustment parameter group corresponding to the request type to which the second business request belongs.

13. The method according to claim 12, characterized in that The computer device includes the request router; The method further comprises: In the request router, determining the parameter group identifier carried by the second service request as the target parameter group identifier; Querying the target parameter group identifier in the request routing table, obtaining processor information in the request routing table that has an index relationship with the target parameter group identifier, and determining the obtained processor information as candidate processor information; Determine, in the processor set, a candidate processor indicated by the candidate processor information; the candidate processor is loaded with the target model adjustment parameter group indicated by the target parameter group identifier; A target processor is determined among the candidate processors, and the second business request is distributed to the target processor through the request router; the target processor is used to add the loaded target model adjustment parameter group to the base model according to the parameter group identifier carried by the second business request; the base model to which the target model adjustment parameter group is added is used to process the second business request.

14. The method according to claim 13, characterized in that The step of determining a target processor from among the candidate processors comprises: Acquire the memory utilization of the candidate processor, and determine the memory availability of the candidate processor according to the memory utilization of the candidate processor; Determine a first reciprocal of a request queue length of the candidate processor, and determine a second reciprocal of a network delay of the candidate processor; Performing weighted fusion processing on the memory availability, the first inverse, and the second inverse to obtain a score of the candidate processor; The maximum score among the scores of the candidate processors is obtained, and the processor with the maximum score among the candidate processors is determined as the target processor.

15. A data processing method, characterized in that: include: Obtain a service request carrying a target parameter group identifier, and obtain a request routing table according to the service request; The request routing table is generated by the parameter manager according to n parameter group identifiers and n processor information after the parameter manager allocates processors in the processor set to the n model adjustment parameter groups according to the number of requests corresponding to the n request types and the average processing time corresponding to the n model adjustment parameter groups; n is a positive integer; one model adjustment parameter group corresponds to one request type; one parameter group identifier is used to indicate one model adjustment parameter group; one processor information is used to indicate one processor; Query the target parameter group identifier in the request routing table to obtain candidate processor information in the request routing table that has an index relationship with the target parameter group identifier; In the processor set, determining a candidate processor indicated by the candidate processor information; the candidate processor is loaded with the target model adjustment parameter group indicated by the target parameter group identifier; Determine a target processor among the candidate processors, and distribute the service request to the target processor; The target processor is used to add the loaded target model adjustment parameter group to the base model according to the target parameter group identifier carried by the business request; the base model added with the target model adjustment parameter group is used to process the business request.

16. A data processing device, characterized in that: The device is run on a computer device, and the device includes: A determination module, used to determine the number of requests corresponding to n request types, and determine the model adjustment parameter groups corresponding to the n request types; n is a positive integer; An acquisition module, used to acquire average processing times corresponding to n model adjustment parameter groups respectively; the average processing time corresponding to a model adjustment parameter group refers to the average time for processing business requests belonging to the request type corresponding to the model adjustment parameter group by using a base model to which the model adjustment parameter group is added; An allocation module is used to allocate processors in a processor set to the n model adjustment parameter groups respectively according to the number of requests corresponding to the n request types and n average processing times; the processors in the processor set are used to load the allocated model adjustment parameter groups, and to process business requests belonging to the request types corresponding to the loaded model adjustment parameter groups through the loaded model adjustment parameter groups and the base model.

17. A data processing device, characterized in that: include: An acquisition module, used to acquire a service request carrying a target parameter group identifier, and acquire a request routing table according to the service request; The request routing table is generated by the parameter manager according to n parameter group identifiers and n processor information after the parameter manager allocates processors in the processor set to the n model adjustment parameter groups according to the number of requests corresponding to the n request types and the average processing time corresponding to the n model adjustment parameter groups; n is a positive integer; one model adjustment parameter group corresponds to one request type; one parameter group identifier is used to indicate one model adjustment parameter group; one processor information is used to indicate one processor; The acquisition module is further used to query the target parameter group identifier in the request routing table, and obtain candidate processor information in the request routing table that has an index relationship with the target parameter group identifier; A determination module, configured to determine, in the processor set, a candidate processor indicated by the candidate processor information; The candidate processor is loaded with a target model adjustment parameter group indicated by the target parameter group identifier; A distribution module, used to determine a target processor from the candidate processors and distribute the service request to the target processor; The target processor is used to add the loaded target model adjustment parameter group to the base model according to the target parameter group identifier carried by the business request; the base model added with the target model adjustment parameter group is used to process the business request.

18. A computer device, characterized in that: include: Processor, memory, and network interface; The processor is connected to the memory and the network interface, wherein the network interface is used to provide a data communication function, the memory is used to store a computer program, and the processor is used to call the computer program so that the computer device executes the method described in any one of claims 1 to 15.

19. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded and executed by a processor, so that a computer device having the processor executes the method according to any one of claims 1 to 15.

20. A computer program product, characterized in that The computer program product comprises a computer program, which is stored in a computer-readable storage medium. The computer program is suitable for being read and executed by a processor, so that a computer device having the processor executes the method according to any one of claims 1 to 15.

Images