A large language model scheduling and training method, system, device and medium
By selecting lightweight sub-vertical models from the comprehensive vertical model to answer vertical questions, and utilizing text self-classification and iterative pre-training, the problem of balancing training computational power and representational ability of question-answering models is solved, achieving efficient answering of vertical questions.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHANGHAI GANGLIAN E COMMERCE
- Filing Date
- 2023-10-11
- Publication Date
- 2026-05-05
AI Technical Summary
In the process of vertical knowledge question answering, it is difficult to balance the training computing power requirements and representation capabilities of question answering models. Lightweight models have insufficient representation capabilities, while heavyweight models have high computing resource requirements.
By selecting multiple lightweight sub-vertical models from the comprehensive vertical model, we can answer the vertical questions input by users. We generate sub-vertical models through text self-classification and cyclic pre-training, and use the semantic similarity between the center point and the vertical question vector to generate high-quality answer results.
It balances the computational requirements for training question-answering models with their representational capabilities, reducing the burden on computational resources and improving the quality and relevance of responses.
Smart Images
Figure CN117332087B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence, and in particular to a scheduling and training method, system, device and medium for a large language model. Background Technology
[0002] As the number of parameters in a large language model increases, the demands on computational resources and training samples become higher. To address the problems associated with large-parameter models, current approaches primarily employ model compression, pruning, or quantization to compress them into lightweight models, thereby reducing their computational resource consumption.
[0003] However, lightweight models have a significant gap in representation capabilities compared to models with a large number of parameters. Therefore, in practical applications, it is necessary to select an appropriate model size based on specific needs and scenarios, and to strike a balance between the resource requirements of large parameter models and the representation capabilities of small parameter models.
[0004] Currently, in the process of vertical knowledge question answering, the user-inputted vertical question is typically fed into a single question answering model, and the answer is returned to the user based on the model's output. If a question answering model uses a large number of parameters, although this improves its representational power, it places higher demands on computational resources during training. Conversely, if a lightweight model is used, while reducing the computational burden during training, it also reduces the model's representational power. Currently, it is difficult to achieve a good balance between the computational requirements for training and the representational power of question answering models. Summary of the Invention
[0005] In order to better balance the computational power requirements and representation capabilities of the training of the response model, this application provides a scheduling and training method, system, device and medium for a large language model.
[0006] Firstly, this application provides a scheduling and training method for a large language model, the method comprising the following steps:
[0007] Obtain vertical category questions;
[0008] Based on the vertical problem, at least one sub-vertical model is selected from the comprehensive vertical model. The comprehensive vertical model contains multiple sub-vertical models, and each sub-vertical model is a lightweight model.
[0009] The vertical question is input into the selected sub-vertical model to obtain multiple first answer results output by the sub-vertical model;
[0010] The first answer results are organized and combined to obtain the second answer result corresponding to the vertical question, thus completing the solution to the vertical question.
[0011] By adopting the above technical solution, for the vertical question input by the user, at least one sub-vertical model is selected from the comprehensive vertical model to answer the vertical question. The representational capability of the model is ensured by calling multiple sub-vertical models. Simultaneously, these sub-vertical models are all lightweight models, and the computational resource requirements for training multiple lightweight models are far less than those for training a large-parameter model, reducing the computational burden during training. This effectively balances the computational requirements for question-answering model training with its representational capability.
[0012] Optionally, before selecting at least one sub-vertical model from the comprehensive vertical model based on the vertical problem, the method further includes a training process for the comprehensive vertical model, which specifically includes the following steps:
[0013] Retrieve vertical text data;
[0014] The vertical text data is self-classified to obtain multiple vertical text clusters;
[0015] The comprehensive vertical model is pre-trained iteratively using each of the vertical text clusters to generate the subdivided vertical model corresponding to each of the vertical text clusters.
[0016] By employing the above technical solution, self-classification of vertical texts is achieved. During the text classification process, no classification target is set; instead, texts are automatically classified based on their own semantics, thereby determining vertical text clusters in different vertical domains. The comprehensive vertical model is trained based on these automatically segmented vertical text clusters, resulting in subdivided vertical models for each vertical domain, ensuring the representational ability of each subdivided vertical model for its corresponding vertical domain.
[0017] Optionally, the process of performing text self-classification on the vertical text data to obtain multiple vertical text clusters specifically includes:
[0018] The vertical text data is sliced to obtain multiple semantic slices;
[0019] Several sets of similar slices are identified from all semantic slices, and each set of similar slices contains multiple semantic slices.
[0020] Calculate the set score for each of the aforementioned similar slice sets;
[0021] Select the sets of similar slices with the highest scores as the vertical text clusters, and determine the center point of each vertical text cluster.
[0022] By adopting the above technical solution, the vertical text clusters are determined by selecting the highest-scoring sets of similar slices. The high semantic similarity among the semantic slices in the similar slice sets ensures a high degree of aggregation of the vertical text clusters, thus enabling them to better represent specific vertical domains. Simultaneously, the core points of the vertical text clusters are determined, allowing for the description of the main features of the vertical text clusters.
[0023] Optionally, several sets of similar slices are identified from all semantic slices, specifically including:
[0024] Select any one of the semantic slices from all the semantic slices;
[0025] Calculate the semantic similarity between the selected semantic slice and the other semantic slices respectively;
[0026] Without replacement, extract a certain number of other semantic slices whose semantic similarity is greater than the first similarity threshold to complete the construction of the similar slice set.
[0027] By adopting the above technical solution, when determining the set of similar slices, they are extracted from all semantic slices without replacement, ensuring that there are no identical semantic slices between different sets of similar slices. At the same time, the number of semantic slices contained in the set of similar slices is limited, thereby avoiding to some extent the erosion of sparse classes by dense classes caused by large classes eroding small classes or imbalance in corpus content.
[0028] Optionally, the calculation of the set score for each of the aforementioned similar slice sets specifically includes:
[0029] Calculate the semantic similarity between any two semantic slices in the set of similar slices;
[0030] Calculate the first similarity average and the first similarity standard deviation of the similar slice set based on the semantic similarity between any two semantic slices in the similar slice set;
[0031] The first similarity average and the first similarity standard deviation are normalized to obtain the second similarity average and the second similarity standard deviation.
[0032] The set score is calculated by processing the second similarity average and the second similarity standard deviation using a preset set score calculation formula.
[0033] By employing the above technical solution and calculating set scores, the similarity distribution of similar slice sets can be comprehensively considered, taking into account both the average similarity level within each set and the degree of dispersion of similarity. This allows for the ranking of similar slice sets, prioritizing those with higher scores for subsequent processing, thereby improving the effectiveness and accuracy of clustering.
[0034] Optionally, in selecting at least one subdivided vertical model from the comprehensive vertical model based on the vertical problem, the specific steps include:
[0035] The vertical problem is vectorized to obtain the vertical problem vector;
[0036] Calculate the semantic similarity between the vertical question vector and the core point of the similar slice set corresponding to each of the subdivided vertical models;
[0037] Select all the subdivided vertical category models whose semantic similarity is greater than the second similarity threshold.
[0038] By adopting the above technical solution, the core point is used to describe the main features of the similar slice set. By scheduling the semantic similarity between the core point and the vertical question vector and the subdivided vertical model corresponding to the vertical question, the correspondence between the subdivided vertical model and the vertical question is guaranteed.
[0039] Optionally, the process of organizing and combining the first answer results to obtain the second answer result corresponding to the vertical category question specifically includes:
[0040] Obtain vertical question prompts based on the vertical question vector;
[0041] The first answer results are concatenated, and the vertical question prompt words are added before all the concatenated text to obtain the second answer results.
[0042] By adopting the above technical solution, the corresponding vertical question prompts are determined based on the user's original query vertical question vector. The vertical question prompts can provide effective contextual information for the generation of the second answer result, so that the second answer result obtained by splicing the vertical question prompts can better adapt to the query intent and context, and be more targeted to the vertical question when generating the answer, thereby improving the quality of the answer and making it more relevant and readable.
[0043] A second aspect of this application provides a scheduling and training system for a large language model, the system comprising the following modules:
[0044] The user vertical category question acquisition module is used to acquire vertical category questions;
[0045] The subdivided vertical category model selection module is used to select at least one subdivided vertical category model from the comprehensive vertical category model according to the vertical category problem. The comprehensive vertical category model contains multiple subdivided vertical category models, and each subdivided vertical category model is a lightweight model.
[0046] The first answer result determination module is used to input the vertical question into the selected sub-vertical model to obtain multiple first answer results output by the sub-vertical model;
[0047] The second answer result determination module is used to organize and splice each of the first answer results to obtain the second answer result corresponding to the vertical question, thereby completing the answer to the vertical question.
[0048] In a third aspect of this application, an electronic device is provided;
[0049] The electronic device includes a processor, a memory, a user interface, and a network interface. The memory is used to store instructions, the user interface and the network interface are used to communicate with other devices, and the processor is used to execute the instructions stored in the memory to enable the electronic device to perform a scheduling and training method for a large language model.
[0050] A fourth aspect of this application provides a computer-readable storage medium;
[0051] The computer-readable storage medium stores instructions that, when executed, perform a scheduling and training method for a large language model.
[0052] In summary, one or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages:
[0053] 1. For user-input vertical questions, at least one sub-vertical model is selected from the comprehensive vertical model to answer the question. Calling multiple sub-vertical models ensures the model's representational capability. Simultaneously, these sub-vertical models are lightweight; the training computational resource requirements of multiple lightweight models are far less than those of a large-parameter model, reducing the computational burden during training. This approach effectively balances the computational requirements of the question-answering model with its representational capability.
[0054] 2. The system performs self-classification of vertical texts. During text classification, no classification target is set; instead, the system automatically classifies texts based on their semantics, thus identifying vertical text clusters for different vertical domains. The comprehensive vertical model is trained based on these automatically defined vertical text clusters, resulting in subdivided vertical models for each vertical domain, ensuring the representational ability of each subdivided model for its corresponding vertical domain.
[0055] 3. By scheduling the semantic similarity between the core point and the vertical question vector and the corresponding sub-vertical model, the correspondence between the sub-vertical model and the vertical question is guaranteed. Attached Figure Description
[0056] Figure 1 This is a flowchart illustrating a scheduling and training method for a large language model provided in an embodiment of this application.
[0057] Figure 2 This is a schematic diagram of the process for training a comprehensive vertical category model provided in an embodiment of this application.
[0058] Figure 3 This is a schematic diagram of the structure of a scheduling and training system for a large language model disclosed in an embodiment of this application.
[0059] Figure 4 This is a schematic diagram of the structure of an electronic device disclosed in an embodiment of this application.
[0060] Explanation of reference numerals in the attached diagram: 301, User vertical question acquisition module; 302, Subdivided vertical model selection module; 303, First answer result determination module; 304, Second answer result determination module; 400, Electronic device; 401, Processor; 402, Communication bus; 403, User interface; 404, Network interface; 405, Memory. Detailed Implementation
[0061] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.
[0062] In the description of the embodiments of this application, the words "for example" or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design that is described as "for example" or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design options. Rather, the use of the words "for example" or "for instance" is intended to present the relevant concepts in a specific manner.
[0063] In the description of the embodiments of this application, the term "multiple" means two or more. For example, multiple systems means two or more systems, and multiple screen terminals means two or more screen terminals. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the indicated technical features. Thus, a feature defined with "first" or "second" may explicitly or implicitly include one or more of that feature. The terms "comprising," "including," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.
[0064] Reference Figure 1 This application provides a scheduling and training method for a large language model, which specifically includes the following steps:
[0065] S11: Obtaining vertical category information.
[0066] Specifically, vertical text and data refer to text and data related to a particular field or industry, which typically possess a high degree of specialization and specific domain background. Correspondingly, vertical questions are questions raised by users regarding a specific domain.
[0067] The system retrieves user-inputted vertical questions from the front-end user-facing application, and stores these questions in query format.
[0068] In one feasible embodiment of this application, after obtaining the vertical category questions, sensitive word cleaning is performed on the vertical category questions to ensure the text quality of the obtained vertical category questions.
[0069] S12: Select at least one subdivided vertical model from the comprehensive vertical model based on the vertical problem.
[0070] Specifically, the vertical problem is vectorized to obtain a vertical problem vector. In one feasible embodiment of this application, the vectorization is performed using BERT vectorization. Based on the vertical problem vector, at least one sub-vertical model corresponding to the vertical problem is scheduled in the comprehensive vertical model.
[0071] It should be noted that the comprehensive vertical model includes multiple sub-vertical models, each corresponding to a specific vertical field, and each sub-vertical model is a lightweight model.
[0072] Reference Figure 2 Before scheduling the subdivided vertical models, there is also a training process for the comprehensive vertical model. The specific training process is as follows:
[0073] S21: Obtain vertical text data.
[0074] The collected vertical text data is obtained from a pre-set vertical text database. The vertical text data contains professional data from multiple vertical fields, including but not limited to vertical field literature data, vertical field expert Q&A data, vertical field patent data, and vertical field search data.
[0075] S22: Perform text self-classification on the vertical text data to obtain multiple vertical text clusters.
[0076] Text self-classification of vertical text data divides all vertical text data into multiple vertical text clusters. It can be understood that vertical text data from different vertical fields are mixed together in the vertical text data, while the vertical text clusters obtained after text self-classification only contain vertical text data from specific vertical fields.
[0077] Optionally, the vertical text data can be self-classified using a semantic slice self-clustering algorithm. It should be noted that all semantic similarity calculations involved in the semantic slice self-clustering algorithm are based on the cosine distance of vectors. The semantic slice self-clustering algorithm is as follows:
[0078] Step 1: Slice and vectorize all vertical text data to obtain multiple semantic slices, and put all the obtained semantic slices into the semantic slice library.
[0079] Specifically, since the amount of vertical text data is large, the traditional method of counting by the number of texts is inconvenient. In one feasible embodiment of this application, all methods of evaluating the number of texts are expressed in terms of text size. During the calculation process, the estimation is based on the text size corresponding to the unit token. Each text string is a whole, and the value is cut for the whole text. There is no process of cutting a part of the text.
[0080] Optionally, BERT vectorization can be used for vectorization of vertical text data.
[0081] Step 2: Select any semantic slice x1 from the semantic slice library, and at the same time, select several similar semantic slices that meet the preset selection conditions.
[0082] The preset selection conditions specifically include:
[0083] Condition 1: The similarity between the similar semantic slice and semantic slice x1 is greater than the first similarity threshold E'. Optionally, the first similarity threshold is set to 0.95.
[0084] Condition 2: The number of similar semantic slices selected is less than 10,000.
[0085] Condition 3: The number of similar semantic slices selected is greater than 100.
[0086] After selecting similar semantic slices, a similar slice set is constructed by combining randomly selected semantic slices with similar semantic slices. It should be noted that the semantic slice extraction process is performed without replacement in the semantic slice database, and the number of semantic slices in the similar slice set is denoted as n. x1 .
[0087] Step 3: Calculate the average first similarity E of the currently selected set of similar slices. x1 The first similarity standard deviation D x1 And the first similarity average E x1 The first similarity standard deviation D x1 After normalization, the average second similarity E is obtained. x1 ′ and the second similarity standard deviation D x1 ′.
[0088] Among them, the average first similarity E x1 The first similarity standard deviation D is the average semantic similarity between all pairwise semantic slices in the similar slice set. x1 The standard deviation of semantic similarity between all pairwise semantic slices in the similar slice set.
[0089] The normalization method is as follows:
[0090]
[0091] Step 4: Calculate the percentage P of the number of semantic slices in the current set of similar slices in the semantic slice database.
[0092] Step 5: Randomly select another semantic slice x2 from the semantic slice database, and repeat Step 2-Step 3. After each completion of the construction of the similar slice set, calculate P.
[0093] Step 6: When P > 0.4, stop the loop from Step 2 to Step 3.
[0094] It is understandable that after the above steps, the set of similar semantic slices x1, x2, x3, ..., xn has been obtained.
[0095] Step 7: Update the second similarity standard deviation D for each set of similar semantic slices. xi ′.
[0096] D xi The specific calculation formula for ′ is:
[0097]
[0098] Among them, D xi Let D be the first similarity standard deviation of the i-th set of similar semantic slices, min(D xi ) is the minimum first similarity standard deviation among all similar semantic slice sets, max(D xi ) represents the largest first similarity standard deviation among all similar semantic slice sets.
[0099] Step 8: Calculate the set score A for each set of similar slices.
[0100] The set score A is calculated using the set score calculation formula, specifically: A=αE′ xi +βD xi ′;
[0101] Among them, E′ xi Let D′ be the average of the second similarity of the i-th set of similar slices. xi Let α be the second similarity standard deviation of the i-th similar slice set, and β be adjustment coefficients. In a preferred embodiment of this application, α = 0.7 and β = 0.3.
[0102] Step 9: Select the top N similar slice sets according to the set score A, and take the top N similar slice sets with the highest set score A as the vertical text cluster.
[0103] Preferably, N is 10. If there are fewer than 10 similar slice sets, all slices are taken out.
[0104] Optionally, to further cluster the semantic vectors contained in the vertical text clusters and determine the centroids of the vertical text clusters, after determining the vertical text clusters, it is also necessary to perform a semantic vector self-clustering algorithm in the vertical text clusters. The semantic vector self-clustering algorithm is as follows:
[0105] In a vertical text cluster, a semantic vector is randomly selected as a seed. Then, the 1000 semantic vectors with the highest similarity to the seed are found among the remaining semantic vectors, forming a set Y. After every 100 seed selections, the ratio Pr of the number of duplicate semantic vectors in Y to the total number of semantic vectors is calculated. When Pr is greater than or equal to 0.95, seed selection is stopped. The number of each semantic vector in Y is counted, and the semantic vector with the largest number is taken as the centroid of the vertical text cluster. In the remaining semantic vectors, the 1000 semantic vectors with a similarity higher than 0.95 to the centroid are found, forming a new set C. The above steps are repeated until no semantic vector with a similarity higher than 1.95 can be found. The semantic vectors in C are used as the result of semantic vector self-clustering, and the centroids corresponding to the vertical text clusters are obtained.
[0106] Step 10: For semantic slices in the set of similar slices that were not selected, put them back into the semantic slice database.
[0107] Step 11: Repeat Step 2-Step 10 until no similar semantic slices that meet the selection criteria can be found in Step 2.
[0108] Optionally, if the execution time of Step 2 is greater than 10 minutes, the loop will also be exited.
[0109] Step12: Take center1, center2,..., center n To achieve the goal, the remaining semantic slices y1, y2, y3, ..., ym in the current semantic slice library are placed in center1, center2, ..., center2. n In the search, for each y m take out like Then y m Merge into the corresponding cluster n .
[0110] in, For y m Relative to each center n The highest semantic similarity, min(E) n )for corresponding center n The semantic similarity between the least similar internal point and the core point.
[0111] Step 13: Output the updated vertical text clusters cluster1, cluster2, ..., cluster n And the center points center1, center2, ..., center corresponding to the vertical text clusters. n .
[0112] S23: Perform cyclic pre-training on the comprehensive vertical model through each vertical text cluster to generate a subdivided vertical model corresponding to each vertical text cluster.
[0113] The comprehensive vertical model is pre-trained iteratively using all the obtained vertical text clusters to generate subdivided vertical models corresponding to each vertical text cluster.
[0114] Optionally, the cyclic pre-training of the comprehensive vertical model can be performed using LLM cyclic pre-training.
[0115] S13: Input the vertical question into the selected sub-vertical model to obtain multiple first answer results output by the sub-vertical model;
[0116] Specifically, through the training process of the above-mentioned comprehensive vertical model, multiple sub-vertical models that have been trained are obtained. Each sub-vertical model has the ability to represent the vertical domain corresponding to the vertical text cluster used for training.
[0117] Based on the vertical problem, a sub-vertical model corresponding to the vertical problem is selected from the comprehensive vertical model to solve the vertical problem. The selection of the sub-vertical model specifically includes:
[0118] Vectorize the vertical question to obtain the vertical question vector; calculate the semantic similarity between the vertical question vector and the core points of the similar slice set corresponding to each sub-vertical model; select all sub-vertical models whose semantic similarity is greater than the second similarity threshold.
[0119] By inputting the vertical question vector into the selected sub-vertical model, multiple first answer results are obtained from the output of multiple sub-vertical models.
[0120] S14: Organize and combine the results of each first answer to obtain the second answer corresponding to the vertical question, thus completing the solution to the vertical question.
[0121] Specifically, multiple first-response results are concatenated to determine the second-response result corresponding to the vertical category question.
[0122] Optionally, a prompt word can be added before the entire concatenated text to improve the descriptiveness of the second answer.
[0123] Reference Figure 3 This application also provides a scheduling and training system for large language models, which specifically includes the following modules:
[0124] User vertical category question acquisition module 301 is used to acquire vertical category questions;
[0125] The subdivided vertical model selection module 302 is used to select at least one subdivided vertical model from the comprehensive vertical model based on the vertical problem. The comprehensive vertical model contains multiple subdivided vertical models, and each subdivided vertical model is a lightweight model.
[0126] The first answer result determination module 303 is used to input the vertical question into the selected sub-vertical model to obtain multiple first answer results output by the sub-vertical model;
[0127] The second answer result determination module 304 is used to organize and splice the first answer results to obtain the second answer result corresponding to the vertical question, thus completing the answer to the vertical question.
[0128] It should be noted that the above embodiments of the apparatus are only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.
[0129] This application also discloses an electronic device 400. (See reference...) Figure 4 , Figure 4 This is a schematic diagram of the structure of an electronic device 400 disclosed in an embodiment of this application. The electronic device 400 may include: at least one processor 401, at least one network interface 404, a user interface 403, a memory 405, and at least one communication bus 402.
[0130] The communication bus 402 is used to enable communication between these components.
[0131] The user interface 403 may include a display screen and a camera. Optionally, the user interface 403 may also include a standard wired interface and a wireless interface.
[0132] The network interface 404 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface).
[0133] The processor 401 may include one or more processing cores. The processor 401 connects to various parts of the server using various interfaces and lines, and performs various server functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in memory 405, and by calling data stored in memory 405. Optionally, the processor 401 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 401 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content to be displayed on the screen; and the modem handles wireless communication. It is understood that the modem may also not be integrated into the processor 401 and may be implemented as a separate chip.
[0134] The memory 405 may include random access memory (RAM) or read-only memory. Optionally, the memory 405 may include a non-transitory computer-readable storage medium. The memory 405 may be used to store instructions, programs, code, code sets, or instruction sets. The memory 405 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-described method embodiments, etc.; the data storage area may store data involved in the above-described method embodiments, etc. Optionally, the memory 405 may also be at least one storage device located remotely from the aforementioned processor 401. (Refer to...) Figure 4 The memory 405, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and an application program for scheduling and training a large language model.
[0135] exist Figure 4In the illustrated electronic device 400, the user interface 403 is mainly used to provide an input interface for the user and acquire user input data; while the processor 401 can be used to call an application program stored in the memory 405 that stores a scheduling and training method for a large language model. When executed by one or more processors 401, the electronic device 400 performs one or more of the methods described in the above embodiments. It should be noted that, for the foregoing method embodiments, for the sake of simplicity, they are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.
[0136] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.
[0137] In the various embodiments provided in this application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some service interface; the indirect coupling or communication connection between apparatuses or units may be electrical or other forms.
[0138] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0139] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0140] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage device 405. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage device 405 and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage device 405 includes various media capable of storing program code, such as a USB flash drive, external hard drive, magnetic disk, or optical disk.
[0141] The above description is merely an exemplary embodiment of this disclosure and should not be construed as limiting the scope of this disclosure. Any equivalent changes and modifications made in accordance with the teachings of this disclosure shall still fall within the scope of this disclosure. Other embodiments of this disclosure will be readily apparent to those skilled in the art upon consideration of the specification and the disclosure of practical truths.
[0142] This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not described in this disclosure. The specification and embodiments are to be considered exemplary only, and the scope and spirit of this disclosure are defined by the claims.
Claims
1. A scheduling and training method for a large language model, characterized in that, The method includes the following steps: Obtain vertical category questions; Based on the vertical problem, at least one sub-vertical model is selected from the comprehensive vertical model. The comprehensive vertical model contains multiple sub-vertical models, and each sub-vertical model is a lightweight model. The vertical question is input into the selected sub-vertical model to obtain multiple first answer results output by the sub-vertical model; The first answer results are organized and combined to obtain the second answer result corresponding to the vertical question, thus completing the solution to the vertical question. Before selecting at least one sub-vertical model from the comprehensive vertical model based on the vertical problem, the process also includes training the comprehensive vertical model, which specifically includes the following steps: Retrieve vertical text data; The vertical text data is self-classified to obtain multiple vertical text clusters; The comprehensive vertical model is pre-trained iteratively using each of the vertical text clusters to generate the subdivided vertical model corresponding to each of the vertical text clusters. The text self-classification of the aforementioned vertical text data to obtain multiple vertical text clusters specifically includes: The vertical text data is sliced to obtain multiple semantic slices; Several sets of similar slices are identified from all semantic slices, and each set of similar slices contains multiple semantic slices. Calculate the set score for each of the aforementioned similar slice sets; Select the sets of similar slices with the highest scores as the vertical text clusters, and determine the core point of each vertical text cluster. In selecting at least one subdivided vertical model from the comprehensive vertical model based on the aforementioned vertical problem, the specific steps include: The vertical problem is vectorized to obtain the vertical problem vector; Calculate the semantic similarity between the vertical question vector and the core point of the similar slice set corresponding to each of the subdivided vertical models; Select all the subdivided vertical category models whose semantic similarity is greater than the second similarity threshold.
2. The scheduling and training method for a large language model according to claim 1, characterized in that, Several sets of similar slices were identified from all semantic slices, specifically including: Select any one of the semantic slices from all the semantic slices; Calculate the semantic similarity between the selected semantic slice and the other semantic slices respectively; Without replacement, extract a certain number of other semantic slices whose semantic similarity is greater than the first similarity threshold to complete the construction of the similar slice set.
3. The scheduling and training method for a large language model according to claim 1, characterized in that, The calculation of the set score for each of the aforementioned similar slice sets specifically includes: Calculate the semantic similarity between any two semantic slices in the set of similar slices; Calculate the first similarity average and the first similarity standard deviation of the similar slice set based on the semantic similarity between any two semantic slices in the similar slice set; The first similarity average and the first similarity standard deviation are normalized to obtain the second similarity average and the second similarity standard deviation. The set score is calculated by processing the second similarity average and the second similarity standard deviation using a preset set score calculation formula.
4. The scheduling and training method for a large language model according to claim 1, characterized in that, The process of organizing and combining the first answer results to obtain the second answer result corresponding to the vertical category question specifically includes: Obtain vertical question prompts based on the vertical question vector; The first answer results are concatenated, and the vertical question prompt words are added before all the concatenated text to obtain the second answer results.
5. A scheduling and training system for a large language model, characterized in that, The system includes: The user vertical category question acquisition module (301) is used to acquire vertical category questions; The subdivided vertical category model selection module (302) is used to select at least one subdivided vertical category model from the comprehensive vertical category model based on the vertical category problem. The comprehensive vertical category model contains multiple subdivided vertical category models, and each subdivided vertical category model is a lightweight model. Before selecting at least one sub-vertical model from the comprehensive vertical model based on the vertical problem, the process also includes training the comprehensive vertical model, which specifically includes the following steps: Retrieve vertical text data; The vertical text data is self-classified to obtain multiple vertical text clusters; The comprehensive vertical model is pre-trained iteratively using each of the vertical text clusters to generate the subdivided vertical model corresponding to each of the vertical text clusters. The text self-classification of the aforementioned vertical text data to obtain multiple vertical text clusters specifically includes: The vertical text data is sliced to obtain multiple semantic slices; Several sets of similar slices are identified from all semantic slices, and each set of similar slices contains multiple semantic slices. Calculate the set score for each of the aforementioned similar slice sets; Select the sets of similar slices with the highest scores as the vertical text clusters, and determine the core point of each vertical text cluster. In selecting at least one subdivided vertical model from the comprehensive vertical model based on the aforementioned vertical problem, the specific steps include: The vertical problem is vectorized to obtain the vertical problem vector; Calculate the semantic similarity between the vertical question vector and the core point of the similar slice set corresponding to each of the subdivided vertical models; Select all the subdivided vertical category models whose semantic similarity is greater than the second similarity threshold; The first answer result determination module (303) is used to input the vertical question into the selected subdivided vertical model to obtain multiple first answer results output by the subdivided vertical model; The second answer result determination module (304) is used to organize and splice each of the first answer results to obtain the second answer result corresponding to the vertical question, thereby completing the answer to the vertical question.
6. An electronic device, characterized in that, The device includes a processor (401), a memory (405), a user interface (403), and a network interface (404). The memory (405) is used to store instructions. The user interface (403) and the network interface (404) are used to communicate with other devices. The processor (401) is used to execute the instructions stored in the memory (405) to cause the electronic device (400) to perform the method as described in any one of claims 1-4.
7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed, perform the steps of the method as described in any one of claims 1-4.
Citation Information
Patent Citations
Visual question and answer implementation method and method based on visual question and answer test model
CN115221369A