Language model determination method and device, electronic equipment and program product

Abstract

The invention discloses a language model determination method and device, electronic equipment and a program product, and relates to the technical field of artificial intelligence, the determination method comprises the following steps: receiving a query request sent by a target terminal, and carrying out feature analysis on query information carried by the query request to obtain a target feature vector; determining a candidate model set according to the target feature vector and a model feature vector of each model in a preset model feature database; allocating different preset weights to the performance data corresponding to each model in the candidate model set; and determining a model score of each model according to the performance data and a preset weight, and determining a target language model from the candidate model set based on the model scores. The technical problem that the resource utilization rate is low due to the fact that the language model cannot be accurately determined in the prior art is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of artificial intelligence, and in particular, to a method and apparatus for determining a language model, an electronic device, and a program product. Background Art

[0002] Currently, the application of LLM (Large Language Model) faces problems of model selection complexity and low resource utilization rate: (1) Users need to manually call the APIs (Application Programming Interfaces) of different models, resulting in cumbersome operations; (2) Different LLM models have significant differences in different task types, and it is necessary to understand the performance differences of each model in tasks such as text generation and code parsing, resulting in a high operation threshold for users; (3) It is necessary to manage multiple API interfaces and authentication processes simultaneously, increasing the system complexity and maintenance cost.

[0003] In response to the above problems, no effective solution has been proposed yet. Summary of the Invention

[0004] Embodiments of the present invention provide a method and apparatus for determining a language model, an electronic device, and a program product, so as to at least solve the technical problem in the related art that the language model cannot be accurately determined, resulting in low resource utilization rate.

[0005] According to one aspect of the embodiments of the present application, a method for determining a language model is provided, including: receiving a query request sent by a target terminal, and performing feature analysis on the query information carried in the query request to obtain a target feature vector, where the target feature vector is obtained by splicing multiple feature vectors of the query information, and each feature vector is used to represent different feature information; determining a candidate model set according to the target feature vector and the model feature vectors of each model in a preset model characteristic database, where the preset model characteristic database at least includes: multiple models, each model corresponds to multiple performance data, and each performance data includes a model feature vector; assigning different preset weights to the performance data corresponding to each model in the candidate model set; determining the model score of each model according to the performance data and the preset weights, and based on the model score, determining a target language model from the candidate model set, where the target language model is used to process the query request of the target terminal.

[0006] Further, the steps of performing feature analysis on the query information carried in the query request to obtain the target feature vector include: preprocessing the query information to obtain the processed query information, and performing semantic extraction on the processed query information to obtain the semantic feature vector; classifying the semantic feature vector to obtain the task type of the semantic feature vector, and determining the professional type to which the processed query information belongs according to the task type; determining the complexity level of the processed query information, and generating feature vectors for the task type, professional type, and complexity level respectively; and concatenating all the feature vectors to obtain the target feature vector.

[0007] Further, before determining the candidate model set based on the target feature vector and the model feature vectors of each model in the preset model feature database, it further includes: assigning a model identifier to each model, and determining the model name and model version of the model, where there is an associated relationship between the model identifier and the model name and model version; determining the constraint conditions of each model, where the constraint conditions are used to determine the functions of each model; and constructing the preset model feature database based on the model identifier and the constraint conditions.

[0008] Further, the steps of constructing the preset model feature database based on the model identifier and the constraint conditions include: determining the performance data of each model, where the performance data at least includes: performance task data, metric data, the performance task data at least includes: task type, task performance score for each task type, and the metric data at least includes: performance metric, performance metric value corresponding to each performance metric; performing feature analysis on the task type, task performance score, performance metric, and performance metric value to obtain the model feature vector of each performance data, and performing feature analysis on the constraint conditions to obtain the model feature vector of the constraint conditions; and constructing the preset model feature database based on the model feature vector of the performance data associated with each model identifier and the model feature vector of the constraint conditions.

[0009] Further, after determining the target language model from the candidate model set based on the model score, it further includes: converting the query information into the target query information in the preset input format, where the preset input format is the input format corresponding to the target language model; searching for the target interface of the target language model in the preset interface library and calling the target interface; and when it is detected that the target language model processes the target query information and outputs the query result, converting the query result into the preset output format to obtain the target result.

[0010] Further, during the process of invoking the target interface, it further includes: monitoring the invocation of the target interface, and in the case where it is detected that the target interface fails to be successfully invoked, re-invoking the target interface according to a preset retry mechanism; in the case where it is detected that the target interface fails to be re-invoked successfully, re-determining the target language model based on the model score.

[0011] Further, after determining the target language model from the candidate model set, it further includes: receiving feedback data of the target terminal and obtaining the current performance data of the target language model during the process of processing the query request, where the feedback data at least includes: explicit feedback data and implicit feedback data, the explicit feedback data is the score given by the target terminal to the target result, and the implicit feedback data at least includes: the duration of the query information, and the duration is the time between the moment when feature analysis of the query information starts and the moment when feature analysis of other query information starts, and the other query information is the information other than the query information; updating the preset model characteristic database according to the feedback data and the current performance data to obtain an updated preset model characteristic database, and adjusting the preset weight of each performance data corresponding to each model to obtain an adjusted preset weight; re-determining the target language model based on the updated preset model characteristic database and the adjusted preset weight.

[0012] According to another aspect of the embodiments of the present application, there is also provided a device for determining a language model, including: a first analysis unit, configured to receive a query request sent by a target terminal and perform feature analysis on the query information carried in the query request to obtain a target feature vector, where the target feature vector is obtained by splicing multiple feature vectors of the query information, and each feature vector is used to represent different feature information; a first determination unit, configured to determine a candidate model set according to the target feature vector and the model feature vectors of each model in the preset model characteristic database, where the preset model characteristic database at least includes: multiple models, each model corresponds to multiple performance data, and each performance data includes a model feature vector; a first allocation unit, configured to allocate different preset weights to the performance data corresponding to each model in the candidate model set; a second determination unit, configured to determine the model score of each model according to the performance data and the preset weight, and determine the target language model from the candidate model set based on the model score, where the target language model is used to process the query request of the target terminal.

[0013] Further, the first analysis unit includes: a first processing module, configured to preprocess the query information to obtain processed query information, and perform semantic extraction on the processed query information to obtain a semantic feature vector; a first classification module, configured to classify the semantic feature vector to obtain the task type of the semantic feature vector, and determine the professional type to which the processed query information belongs according to the task type; a first determination module, configured to determine the complexity level of the processed query information, and generate feature vectors for the task type, professional type, and complexity level respectively; a first splicing module, configured to splice all the feature vectors to obtain a target feature vector.

[0014] Further, the apparatus for determining a language model includes: a first allocation module, configured to assign a model identifier to each model, and determine the model name and model version of the model before determining a candidate model set based on the target feature vector and the model feature vector of each model in the preset model feature database, where there is an associated relationship between the model identifier and the model name and model version; a second determination module, configured to determine the constraint condition of each model, where the constraint condition is used to determine the function of each model; a first construction module, configured to construct the preset model feature database based on the model identifier and the constraint condition.

[0015] Further, the first construction module includes: a first determination sub-module, configured to determine the performance data of each model, where the performance data at least includes: performance task data, metric data, the performance task data at least includes: task type, task performance score for each task type, and the metric data at least includes: performance metric, performance metric value corresponding to each performance metric; a first analysis sub-module, configured to perform feature analysis on the task type, task performance score, performance metric, and performance metric value to obtain the model feature vector of each performance data, and perform feature analysis on the constraint condition to obtain the model feature vector of the constraint condition; a first construction sub-module, configured to construct the preset model feature database based on the model feature vector of the performance data associated with each model identifier and the model feature vector of the constraint condition.

[0016] Further, the apparatus for determining a language model includes: a first conversion module, configured to convert the query information into target query information in a preset input format after determining a target language model from the candidate model set based on the model score, where the preset input format is the input format corresponding to the target language model; a first invocation module, configured to find the target interface of the target language model in the preset interface library and invoke the target interface; a second conversion module, configured to convert the query result into a preset output format to obtain a target result when it is detected that the target language model processes the target query information and outputs a query result.

[0017] Further, the first invocation module includes: a first monitoring sub-module, configured to monitor the invocation of the target interface, and re-invoke the target interface according to a preset retry mechanism when it is detected that the target interface fails to be invoked successfully; a second determination sub-module, configured to re-determine the target language model according to the model score when it is detected that the target interface fails to be re-invoked successfully.

[0018] Further, the apparatus for determining a language model includes: a first acquisition module, configured to, after determining the target language model from the candidate model set, receive feedback data of the target terminal and acquire current performance data of the target language model during the process of processing a query request, where the feedback data at least includes explicit feedback data and implicit feedback data, the explicit feedback data is a score of the target terminal for the target result, and the implicit feedback data at least includes: the duration of the query information, which is the duration between the moment when feature analysis of the query information starts and the moment when feature analysis of other query information starts, and the other query information is information other than the query information; a first update module, configured to update a preset model characteristic database according to the feedback data and the current performance data to obtain an updated preset model characteristic database, and adjust preset weights of each performance data corresponding to each model to obtain adjusted preset weights; a third determination module, configured to re-determine the target language model based on the updated preset model characteristic database and the adjusted preset weights.

[0019] According to another aspect of the embodiments of the present application, there is also provided a computer program product, including a non-volatile computer-readable storage medium storing a computer program, where the computer program, when executed by a processor, implements the method for determining a language model according to any one of the above.

[0020] According to another aspect of the embodiments of the present application, there is also provided an electronic device, including one or more processors and a memory, where the memory is configured to store one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the method for determining a language model according to any one of the above.

[0021] In the present invention, a query request sent by a target terminal is received, feature analysis is performed on query information carried in the query request to obtain a target feature vector, a candidate model set is determined according to the target feature vector and model feature vectors of each model in a preset model characteristic database, different preset weights are assigned to performance data corresponding to each model in the candidate model set, a model score of each model is determined according to the performance data and the preset weights, and a target language model is determined from the candidate model set based on the model score, thereby solving the technical problem in the related art that the language model cannot be accurately determined, resulting in low resource utilization.

[0022] In the present invention, by receiving a query request from a target terminal and performing feature analysis on the query information in the request, a target feature vector can be generated. This vector is obtained by concatenating multiple feature vectors, and each feature vector represents a different feature of the query information, such as semantic content, task type, complexity, and domain, etc. Then, based on the target feature vector and a preset model characteristic database including performance data of multiple large language models (including model feature vectors), a candidate model set that matches the target feature vector is selected, and different preset weights are assigned to the performance data corresponding to each model in the candidate model set. After that, based on the performance data and preset weights of each model in the candidate model set, the model score of each model is determined, and according to the model scores, a target language model for processing the query request of the target terminal is determined from the candidate model set. By analyzing data such as the semantic features, task type, and complexity of the user query, an appropriate LLM model is automatically selected and invoked, achieving the technical effect of accurately determining the language model and thus improving resource utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The drawings described herein are used to provide a further understanding of the present invention and form a part of the present invention. The schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation to the present invention. In the drawings:

[0024] Figure 1 A hardware structure block diagram of a computer terminal (or mobile device) for implementing a method for determining a language model is shown;

[0025] Figure 2 is a flowchart of a method for determining a language model according to Embodiment 1 of the present application;

[0026] Figure 3 is a schematic diagram of an optional intelligent routing decision process according to Embodiment 1 of the present application;

[0027] Figure 4 is a schematic diagram of an optional input analysis engine process according to Embodiment 1 of the present application;

[0028] Figure 5 is a schematic diagram of an optional process for dynamically invoking a language model API according to Embodiment 1 of the present application;

[0029] Figure 6 is a schematic diagram of an optional continuous learning system process according to Embodiment 1 of the present application;

[0030] Figure 7 is a schematic diagram of an optional intelligent selection process of a large language model based on input analysis according to Embodiment 1 of the present application;

[0031] Figure 8 It is a schematic diagram of an optional process for determining a language model according to Embodiment 1 of the present application;

[0032] Figure 9 It is a schematic diagram of an optional device for determining a language model according to an embodiment of the present application;

[0033] Figure 10 It is a structural block diagram of an electronic device according to an embodiment of the present application. Detailed implementation manners

[0034] In order to enable those skilled in the art to better understand the solution of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without making creative efforts shall fall within the protection scope of the present invention.

[0035] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily need to be used to describe a specific order or sequence. It should be understood that such data used can be interchanged under appropriate circumstances so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0036] It should be noted that the relevant information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) collected and involved in the present invention are all information and data authorized by the user or fully authorized by all parties. And the collection, storage, use, processing, transmission, provision, disclosure and application of relevant data and other processing all comply with the relevant laws, regulations and standards of the relevant regions, take necessary confidentiality measures, do not violate public order and good customs, and provide corresponding operation entrances for users to choose to authorize or refuse. For example, an interface is set between the present system and relevant users or institutions. Before obtaining relevant information, a request for obtaining information needs to be sent to the aforementioned users or institutions through the interface, and after receiving the consent information feedback from the aforementioned users or institutions, the relevant information is obtained.

[0037] Since the current method calls a high-performance language model based on model performance, and there may be situations where a high-performance and high-cost model does not need to be called, and continuing to call a high-performance model will cause resource waste, there is currently a lack of an automated mechanism to match models according to task characteristics to improve resource utilization. To solve the above problems, the present invention can, through intelligent analysis and decision-making, automatically identify and call an adapted LLM model according to the characteristics of the input query (including semantic features, task type, complexity, etc.), realize the matching of user queries and large language models, improve the problem-solving efficiency, and reduce the user operation complexity and resource consumption by optimizing resource allocation.

[0038] The present invention will be described in detail below in conjunction with various embodiments.

[0039] Embodiment 1

[0040] According to an embodiment of the present application, an embodiment of a method for determining a language model is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0041] The method embodiment provided by the first embodiment of the present application can be executed on a mobile terminal, a computer terminal, or a similar computing device. Figure 1 A hardware structure block diagram of a computer terminal (or mobile device) for implementing the method for determining a language model is shown. As Figure 1 shown, the computer terminal 10 (or mobile device) may include one or more ( Figure 1 shown as 102a, 10 _ 2b,..., 102n in the figure) processors 102 (the processor 102 may include, but is not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may further include: a display, a keyboard, a cursor control device, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply, and / or a camera, where the network interface may be connected to a wired and / or wireless network. Those of ordinary skill in the art can understand that Figure 1 the structure shown is only schematic and does not limit the structure of the above-mentioned electronic device. For example, the computer terminal 10 may further include more or fewer components than Figure 1 shown in the figure, or have a different configuration from Figure 1 shown in the figure.

[0042] It should be noted that one or more of the above-mentioned processors 102 and / or other data processing circuits can generally be referred to as "data processing circuits" herein. The data processing circuit can be embodied in software, hardware, firmware, or any combination thereof, in whole or in part. In addition, the data processing circuit can be a single independent processing module, or be incorporated, in whole or in part, into any one of other elements in the computer terminal 10 (or mobile device). As involved in the embodiments of the present application, the data processing circuit is a processor control (for example, the selection of a variable resistance terminal path connected to an interface).

[0043] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage devices corresponding to the language model determination method in the embodiments of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, implements the above-mentioned language model determination method. The memory 104 can include high-speed random access memory, and can also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memories. In some instances, the memory 104 can further include a memory remotely set relative to the processor 102, and these remote memories can be connected to the computer terminal 10 through a network. Examples of the above-mentioned network include, but are not limited to, the Internet, intranet, local area network, mobile communication network, and combinations thereof.

[0044] The transmission device 106 is used to receive or send data via a network. Specific examples of the above-mentioned network can include the wireless network provided by the communication provider of the computer terminal 10. In one instance, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station and thus communicate with the Internet. In one instance, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0045] The display can be, for example, a touch-screen liquid crystal display (LCD), which enables the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0046] Under the above operating environment, the present application provides a method for determining a language model as shown in Figure 2 below. Figure 2 is a flowchart of the method for determining a language model according to Embodiment 1 of the present application. As shown in Figure 2 below, the method includes the following steps:

[0047] Step S201: Receive the query request sent by the target terminal, and perform feature analysis on the query information carried in the query request to obtain a target feature vector. The target feature vector is obtained by concatenating multiple feature vectors of the query information, and each feature vector is used to represent different feature information.

[0048] Optionally, receive the query request sent from the target terminal (such as the terminal device of the user or application). The query can be text input, speech-to-text conversion, or any form of inquiry based on natural language.

[0049] In the embodiment of the present invention, in-depth analysis can be performed on the query information carried in the query request to determine feature information such as the semantic characteristics of the query, the model type suitable for processing the query, the complexity of the query information, and the query domain, and feature vectors can be generated respectively for the above feature information. Each characteristic vector can be concatenated to obtain the target feature vector.

[0050] Step S202: Determine a candidate model set according to the target feature vector and the model feature vectors of each model in the preset model characteristic database. The preset model characteristic database at least includes: multiple models, each model corresponds to multiple performance data, and each performance data includes a model feature vector.

[0051] Optionally, the preset model characteristic database is a database storing information of multiple large language models, and each model has its corresponding performance data and feature vectors.

[0052] In the embodiment of the present invention, the target feature vector can be compared with each model feature vector in the preset model characteristic database (such as by calculating the similarity or matching degree) to identify and determine the candidate model set that matches the features, so as to obtain which models can process the current query.

[0053] Step S203: Assign different preset weights to the performance data corresponding to each model in the candidate model set.

[0054] In the embodiment of the present invention, different preset weights (set according to the requirements of different scenarios) are assigned to the performance data corresponding to each model in the candidate model set (including but not limited to the historical performance data of the model, the latency time, the cost-benefit ratio, etc.). For example, in a scenario where cost needs to be controlled, the weight of the cost-benefit ratio may be higher, and in a scenario with high real-time requirements, the weight of the response time may be greater.

[0055] Step S204: Determine the model score of each model according to the performance data and the preset weights, and based on the model scores, determine the target language model from the candidate model set. The target language model is used to process the query request of the target terminal.

[0056] In an embodiment of the present invention, according to model performance data and preset weights, model scores are calculated for each candidate model respectively to quantify the adaptability and efficiency of the model in processing a specific query. Among them, the preset weights can be automatically adjusted by machine learning or manually adjusted.

[0057] Exemplarily, the model score (Score) can be calculated through the following multiple performance data and preset weights: Score = w1 * TaskMatch + w2 * DomainMatch + w3 * HistPerformance + w4 * CostEfficiency + w5 * ResponseTime, where TaskMatch is the task type adaptability (the matching degree between the candidate model and the current query task type), DomainMatch is the domain professionalism matching degree (the expertise degree of the model in the professional field or sub-field to which the query belongs), HistPerformance is the historical performance (based on the statistical data of calling the model in the past, reflecting the actual performance of the model on similar queries), CostEfficiency is the cost-benefit ratio (the ratio between the cost of model invocation and the expected gain), ResponseTime is the response time (the time required for the model to process the query and return a response), w1 is the task type adaptability weight, w2 is the domain professionalism matching degree weight, w3 is the historical performance weight, w4 is the cost-benefit ratio weight, and w5 is the response time estimation weight.

[0058] In an embodiment of the present invention, the model with the highest model score can be selected from the candidate model set as the target language model. Compared with other candidate models, the target language model can provide better response quality and service efficiency.

[0059] Optionally, when making a final decision, the following real-time factors can also be considered, including the current load status of the system, the availability of each model API, the priorities set by the user (such as time priority or quality priority), cost budget constraints, etc.

[0060] In some alternative embodiments, the task type, model characteristics, and historical performance can be constructed into a graph structure, and a graph neural network can be used to learn the optimal matching relationship to obtain the target language model; the model selection can also be modeled as a reinforcement learning problem, with the model as the agent and the user satisfaction as the reward signal, and the selection strategy can be continuously optimized by interacting with the environment; a historical query-model-effect case library can also be constructed to find the most similar historical cases for new queries and reuse the successful model selection.

[0061] Figure 3It is a schematic diagram of an optional intelligent routing decision-making process according to Embodiment 1 of the present application. By combining the feature vector (i.e., the target feature vector) and the model feature database (i.e., the preset model feature database), a candidate model set can be obtained. Then, based on the user preference data (for example, the task type of the user's current query request is the same as that of the previous query request, and the user has a relatively high score for the response result of the target language model determined last time), a model matching score is calculated for each model in the candidate model set to select a single model. Additionally, the final routing decision (i.e., obtaining the target language model) can be determined according to factors such as the system load status and cost control strategy.

[0062] Exemplarily, the user's query information is to write a poem about spring. It is detected that the user has a high satisfaction with the creative writing response of model F in the past, and the query information is analyzed for features, obtaining a task type of creative writing, a medium complexity, and a feature vector of [0.0, 0.0, 0.0, 0.9, 0.0, 0.1, 0.0]. While considering the task type, the user preference is used as an important weight. For example, the optimal model for creative writing is model E with a score of 0.9, while the user's historical preference model is model F with a score of 0.85. Model F can be selected because the user satisfaction weight improves the comprehensive score of F.

[0063] In summary, by receiving the user's query and analyzing the features of the query information carried in the query request, the target feature vector can be obtained. By comparing the target feature vector of the query with the feature vectors of each model in the preset model feature database, a candidate model set that highly matches the query is intelligently screened out. Then, by performing weighted calculations on the performance data of each model in the candidate model set, a suitable target language model is dynamically selected, avoiding resource waste and low efficiency, and thus solving the technical problem in the related art that the language model cannot be accurately determined, resulting in low resource utilization.

[0064] In order to accurately obtain the target feature vector, in the method for determining the language model provided in Embodiment 1 of the present application, the query information is preprocessed to obtain the processed query information, and semantic extraction is performed on the processed query information to obtain the semantic feature vector; the semantic feature vector is classified to obtain the task type of the semantic feature vector, and based on the task type, the professional type to which the processed query information belongs is determined; the complexity level of the processed query information is determined, and feature vectors are generated for the task type, professional type, and complexity level respectively; all the feature vectors are concatenated to obtain the target feature vector.

[0065] In the embodiment of the present invention, preprocessing the query information carried in the query request to obtain the processed query information includes, but is not limited to, standardization processes such as text cleaning, word segmentation, and stop word removal to eliminate text noise and ensure the accuracy of subsequent analysis.

[0066] In the embodiments of the present invention, a deep learning model (such as BER (Bidirectional Encoder Representations from Transformers, that is, the encoder representation based on the bidirectional Transformer (a deep learning model architecture), etc.) can be used to extract the semantics of the processed query information to obtain a semantic feature vector. Methods such as keyword matching, syntactic analysis, and predefined rules can be used to further analyze the semantic feature vector and classify it into predefined task types, such as creative writing, code generation, data analysis, etc., to clarify the specific task requirements of the query (for example, identify that the query is asking to create a poem), and the professional field or sub-field to which the query belongs can be identified according to the task type, such as literature, computer science, financial analysis, etc., aiming to capture the professional nature of the query (for example, determine that the query is related to the literature field).

[0067] In some alternative embodiments, the model can be directly used for task classification without the need to specifically train a classifier.

[0068] In the embodiments of the present invention, by evaluating the complexity of the processed query information, such as whether the query involves complex logical reasoning or specific technical details, the complexity level of the query information is obtained, and independent feature vectors are generated for the task type, professional type, and complexity level respectively. Then, all the feature vectors are concatenated to obtain a target feature vector. By understanding the complexity of the query information, its task type and professional field are located, so as to generate a target feature vector that highly summarizes the characteristics of the target query, which can improve the accuracy and efficiency of subsequent model selection.

[0069] Exemplarily, the user's query information is to explain the physical principle of quantum entanglement and represent it with mathematical formulas. After analysis, the task type is knowledge Q&A and mathematical explanation, the complexity is high, the professional field is quantum physics, and the target feature vector is [0.0, 0.2, 0.0, 0.0, 0.9, 0.8, 0.0]. All models can be evaluated, such as Model A: a high-performance general model with high cost and a score of 0.92; Model D: a model with expertise in the scientific field with medium cost and a score of 0.89; Model E: a lightweight knowledge model with low cost and a score of 0.65. Since the query involves complex scientific concepts and mathematical formulas, Model A can be selected.

[0070] Figure 4 It is a schematic diagram of an optional input analysis engine process according to Embodiment 1 of the present application, such as Figure 3As shown, first, the user query on the target terminal is input into the preprocessing module to obtain the processed query information. Then, the processed query information is input into the semantic understanding module to obtain the semantic feature vector. The semantic feature vector is input into the task classifier for classification to obtain the task type of the semantic feature vector. Then, the query information is evaluated by the complexity evaluator to obtain the complexity level of the processed query information. And the processed query information is analyzed by the context analyzer, combined with the task type, to determine the professional type to which the processed query information belongs. After that, feature vectors are generated for the task type, professional type, and complexity level respectively, and all the feature vectors are concatenated to output the analysis result (i.e., the target feature vector).

[0071] In the method for determining the language model provided in Embodiment 1 of the present application, in order to accurately construct the preset model feature database, a model identifier is assigned to each model, and the model name and model version of the model are determined. Among them, there is an associated relationship between the model identifier and the model name and model version. The constraint conditions for each model are determined, where the constraint conditions are used to determine the functions of each model. Based on the model identifier and the constraint conditions, the preset model feature database is constructed.

[0072] In the embodiment of the present invention, a unique model identifier (Model ID) is assigned to each LLM model, which is used to uniquely identify and index the model in the preset model feature database to ensure the accurate invocation of the model in subsequent processing. For example, model A will be assigned a specific ID (such as Model001).

[0073] In the embodiment of the present invention, for each model, basic information such as its model name and model version is recorded, and this information forms a one-to-one correspondence with the model identifier, which is convenient for managing the update and iteration of different models. For example, the name of model B may be CodeGen and the version is 2.0.

[0074] In the embodiment of the present invention, the constraint conditions for each model are analyzed and determined, including but not limited to whether the model supports code generation, mathematical operation ability, expertise in a specific field, maximum input length limit, whether it supports multi-language processing, etc. The definition of the constraint conditions helps to understand the applicable scope and special technical requirements of the model. For example, model F may only support creative writing tasks and is not applicable to code generation. Based on the model identifier and the constraint conditions, the preset model feature database can be constructed.

[0075] To improve the accuracy of constructing a preset model feature database, in the method for determining a language model provided in Embodiment 1 of this application, performance data of each model is determined, where the performance data at least includes: performance task data, metric data, the performance task data at least includes: task type, task performance score for each task type, and the metric data at least includes: performance metric, performance metric value corresponding to each performance metric; perform feature analysis on the task type, task performance score, performance metric, and performance metric value to obtain the model feature vector of each performance data, and perform feature analysis on the constraint conditions to obtain the model feature vector of the constraint conditions; based on the model feature vector of the performance data associated with each model identifier and the model feature vector of the constraint conditions, construct a preset model feature database.

[0076] Optionally, the performance data of the model at least includes performance task data, metric data, the performance task data includes the task types processed by the model, for example, task types such as creative writing, code generation, text summarization, etc., and the task performance score of the model under each task type; the metric data at least includes performance metrics and their corresponding performance metric values, including but not limited to performance metrics such as latency, throughput, cost per token (lexical unit), maximum input length, etc.

[0077] In an embodiment of the present invention, perform feature analysis on the performance data and the constraint conditions to generate the model feature vector of each performance data and the constraint conditions, and by vectorizing the performance of the model in different dimensions, it is convenient for subsequent decision algorithms to process. For example, the model feature vector of the performance metric can be expressed as [latency, throughput, cost per token].

[0078] In an embodiment of the present invention, according to the model feature vector of the performance data associated with each model identifier and the model feature vector of the constraint conditions, construct a preset model feature database. The structure of the preset model feature database can be set to include model identifier, model feature vector, task performance score vector, performance metric value vector, and constraint condition vector. For example, the following entries may be included in the preset model feature database: Model 123: [latency, throughput, cost], [creative writing score, code generation score], [multilingual support, does not support batch processing].

[0079] It should be noted that the above examples are only for illustrative purposes and do not impose any limitations on the present invention.

[0080] To accurately obtain the target result, in the method for determining a language model provided in Embodiment 1 of this application, the query information is converted into target query information in a preset input format, where the preset input format is the input format corresponding to the target language model; the target interface of the target language model is searched for in the preset interface library, and the target interface is called; when it is detected that the target language model processes the target query information and outputs a query result, the query result is converted into a preset output format to obtain the target result.

[0081] In an embodiment of the present invention, the query information is converted into target query information in a preset input format (i.e., the input format corresponding to the target language model) to match the input format requirements of the target language model. For example, the target model is GPT-3 (Generative Pre-trained Transformer 3, a large language model). By converting the format of the query information, it conforms to the input specification of GPT-3, including but not limited to correct JSON (JavaScript Object Notation, a lightweight data exchange format) encapsulation, specific parameter settings, etc.

[0082] Optionally, the preset interface library stores the API details of each model, including the authentication process, request format, response processing logic, etc.

[0083] In an embodiment of the present invention, by querying the preset interface library, the target interface of the target language model can be located. According to the target language model identifier, the corresponding target interface information is obtained from the preset interface library, and the target interface is called. When it is detected that the target language model completes the processing of the target query information and outputs a query result, the query result can be converted into a unified preset output format (a preset format), and the target result is obtained. Even when facing various response formats of different models, users can receive and understand information in the same format.

[0084] Exemplarily, the query result returned by GPT-3 is text wrapped in HTML (Hypertext Markup Language) format. The HTML tags can be stripped and converted into a plain text or JSON format response to meet the expectations of the user side.

[0085] In some optional embodiments, the API call request can be placed in a message queue and processed by a dedicated consumer service to improve system throughput and fault tolerance; service mesh technology can also be used to manage the call of model APIs, providing advanced features such as circuit breaking, load balancing, and traffic control.

[0086] In some other alternative embodiments, for application scenarios with high requirements for response time, the lightweight model can be deployed on edge nodes close to users (such as network edge servers, Internet of Things devices, smartphones, etc.), and the model can be directly invoked. Only complex queries are sent to the main model in the cloud, reducing the latency of data transmission to the central cloud and then back, and enabling faster response to query requests.

[0087] To improve the accuracy of determining the target language model, in the method for determining a language model provided in Embodiment 1 of this application, the invocation of the target interface is monitored. In the case where the target interface is not successfully invoked, according to a preset retry mechanism, the target interface is re-invoked; in the case where the target interface is not successfully re-invoked, a new target language model is re-determined according to the model score.

[0088] In an embodiment of the present invention, when the target interface is invoked, a real-time monitoring mechanism can be started to track the entire process of the invocation, including response time, invocation status code, whether there is a return result, etc. In the case where the target interface is not successfully invoked, according to a preset retry mechanism (including parameters such as the upper limit of the number of retries and the retry interval time, for example, after the first failure, wait for 5 seconds and then perform the first retry, with a maximum of 3 retries), the target interface is re-invoked. For example, it is monitored that the target interface has a response timeout or returns an error status code, such as HTTP 500 Internal Server Error.

[0089] In an embodiment of the present invention, according to the preset retry mechanism, an attempt can be made to re-invoke the target interface. In the case where the target interface is not successfully re-invoked (for example, the target interface cannot be successfully invoked after multiple retries), a new eligible alternative model can be re-selected as the new target language model according to the model score in the preset model feature database.

[0090] Figure 5 is a schematic diagram of an optional dynamic invocation language model API process according to Embodiment 1 of this application; as Figure 5 shown, by combining the routing decision (i.e., the target language model) with the API configuration library (i.e., the preset interface library), an API adapter is selected, and a dedicated adapter is implemented for each LLM to handle specific authentication, request format, and response parsing logic (i.e., converting the query information into the format required by each model API and converting the response of the target language model into the preset output format), so as to invoke a single API, implement the invocation of the target language model and the response processing of the target language model. At the same time, the entire process of the invocation is monitored to implement error handling and retry mechanisms, and then the response of the target language model is output to the target terminal.

[0091] In order to further improve the accuracy of determining the target language model, in the language model determination method provided in Example 1 of the present application, feedback data from the target terminal is received, and current performance data of the target language model in the process of processing the query request is obtained, wherein the feedback data includes at least: explicit feedback data and implicit feedback data, the explicit feedback data is the score of the target result by the target terminal, and the implicit feedback data includes at least: the duration of the query information, the duration is the time between the moment when the feature analysis of the query information starts and the moment when the feature analysis of other query information starts, and the other query information is information other than the query information; based on the feedback data and the current performance data, the preset model feature database is updated to obtain an updated preset model feature database, and the preset weight of each performance data corresponding to each model is adjusted to obtain the adjusted preset weight; based on the updated preset model feature database and the adjusted preset weight, the target language model is re-determined.

[0092] Optionally, by collecting and analyzing user feedback and the real-time performance of the model, the preset model feature database can be continuously optimized and the weight of performance data can be adjusted to make model selection more accurate and improve the overall response quality and efficiency of the system.

[0093] In an embodiment of the present invention, feedback data from a target terminal is received, and the feedback data includes explicit feedback data and implicit feedback data. The explicit feedback data refers to the user's direct rating or comment on the response result through the target terminal, and the implicit feedback data involves user behavior data, such as the duration of the query information, that is, the time interval from the start of feature analysis of the query information to the feature analysis of other query information.

[0094] For example, the user may give a 5-star rating to the poem generated by model E. At the same time, it is detected that the user immediately initiates another request (a request that is inconsistent with the query information carried by the current query request) after receiving the response, which indicates that the user is satisfied with the target result of the model response.

[0095] In some optional embodiments, it is possible not to rely on a pre-built model feature database, but to adjust the model selection probability in real time based on user feedback through online learning; it is also possible to set up an automated model testing process to regularly evaluate the performance of each model on various tasks and update performance data.

[0096] Figure 6 This is a schematic diagram of an optional continuous learning system process according to Example 1 of the present application, such as Figure 6As shown, user feedback and system operation data are collected, and feedback analysis and scoring are performed to update model performance and evaluate routing decisions. At the same time, according to the system operation data, the performance metric values corresponding to the performance metrics are calculated to update model performance and evaluate routing decisions (i.e., evaluate the preset weights and performance data), analyze the accuracy of the routing decisions, identify the patterns and causes of decision errors, thereby updating the model feature database (i.e., updating the preset model feature database) and optimizing the routing algorithm (i.e., updating the algorithm for calculating the model score), and routing strategy testing can be performed to evaluate the improvement effect. In this way, a new routing strategy can be executed to select a suitable target language model.

[0097] In an embodiment of the present invention, real-time performance data (i.e., current performance data, such as average response time, cost consumption, etc.) of the target language model during the process of processing a query request is recorded. According to the received feedback data and the current performance data, the preset model feature database is updated to obtain an updated preset model feature database. Such an update may include recording new performance data, adjusting the task performance score, and updating the performance evaluation of the model on specific task types. At the same time, the preset weights corresponding to each performance data of each model can be adjusted to obtain adjusted preset weights. According to the updated preset model feature database and the adjusted preset weights, the target language model is re-determined.

[0098] Figure 7 is a schematic diagram of an optional intelligent selection process of a large language model based on input analysis according to Embodiment 1 of the present application, as Figure 7 shown, first, the user query is input into the input analysis engine to obtain a target feature vector, and then combined with the intelligent routing decision system and the dynamic API call framework to obtain a candidate model set (such as LLM Model 1, LLM Model 2... LLM Model n). Then, according to the performance data of each model in the candidate model set and the corresponding preset weights, the model score of each model is calculated. According to the model score, one of the models is selected to obtain the target language model, and this model is called to obtain the user response output. After that, through the feedback collection system and the continuous learning system, the model feature database (i.e., the preset model feature database) is updated, and at the same time, the routing decision in the intelligent routing decision system is updated.

[0099] Figure 8 is a schematic diagram of an optional process for determining a language model according to Embodiment 1 of the present application, as Figure 8As shown, the user submits a query request through a unified interface. The query request carries query information. The input analysis engine analyzes the characteristics of the query information and generates a feature vector (i.e., the target feature vector). The feature vector is passed to the intelligent routing decision system. The intelligent routing decision system queries the model feature database (i.e., the preset model feature database) to obtain the performance data of each model. The intelligent routing decision system executes the model selection algorithm to determine a single optimal model and passes the single-model routing decision to the dynamic API call framework. The dynamic API call framework calls the API of the selected single LLM model (the best LLM model, i.e., the target language model). The best LLM model processes the query and returns a model response. The dynamic API call framework receives the response, processes it, and returns the final answer (i.e., the target result) to the user. The user provides feedback, including explicit feedback or implicit feedback. The continuous learning system collects this feedback data, updates the model performance data, and optimizes the routing algorithm.

[0100] The method for determining a language model provided by the embodiments of the present application can analyze the semantic features, task types, and complexity of a query through deep learning techniques to generate a feature vector. Then, combined with the performance data in the preset model feature database, it automatically selects and calls a single most suitable LLM model. At the same time, it can monitor the call and has a call retry mechanism. After that, by collecting user feedback and real-time performance data, it dynamically adjusts the weights of the model performance data and updates the preset model feature database to achieve dynamic adjustment of model features and continuous improvement of strategies. It can not only improve the response quality according to user feedback but also quickly respond and re-select the best LLM model for task processing when the model performance fluctuates or user preferences change, thereby improving resource utilization efficiency and reducing operation complexity and resource consumption.

[0101] It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. And although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0102] Embodiment 2

[0103] The embodiments of the present application also provide a device for determining a language model. It should be noted that the device for determining a language model in the embodiments of the present application can be used to execute the method for determining a language model provided by the embodiments of the present application. The following introduces the device for determining a language model provided by the embodiments of the present application.

[0104] According to the embodiments of the present application, there is also provided a device for implementing the above method for determining a language model. Figure 9 is a schematic diagram of an optional device for determining a language model according to the embodiments of the present application, as Figure 9As shown in the figure, the determination device of the language model may include: a first analysis unit 90, a first determination unit 91, a first allocation unit 92, and a second determination unit 93.

[0105] Among them, the first analysis unit 90 is configured to receive a query request sent by a target terminal, and perform feature analysis on the query information carried in the query request to obtain a target feature vector, where the target feature vector is obtained by splicing a plurality of feature vectors of the query information, and each feature vector is used to represent different feature information;

[0106] The first determination unit 91 is configured to determine a candidate model set according to the target feature vector and the model feature vector of each model in the preset model characteristic database, where the preset model characteristic database at least includes: a plurality of models, each model corresponds to a plurality of performance data, and each performance data includes a model feature vector;

[0107] The first allocation unit 92 is configured to allocate different preset weights to the performance data corresponding to each model in the candidate model set;

[0108] The second determination unit 93 is configured to determine the model score of each model according to the performance data and the preset weights, and determine the target language model from the candidate model set based on the model score, where the target language model is used to process the query request of the target terminal.

[0109] The determination device of the language model provided by the embodiments of the present application can receive a query request sent by a target terminal through the first analysis unit 90, perform feature analysis on the query information carried in the query request to obtain a target feature vector, and can determine a candidate model set according to the target feature vector and the model feature vector of each model in the preset model characteristic database through the first determination unit 91. The first allocation unit 92 can allocate different preset weights to the performance data corresponding to each model in the candidate model set, and the second determination unit 93 can determine the model score of each model according to the performance data and the preset weights, and determine the target language model from the candidate model set based on the model score.

[0110] Optionally, the first analysis unit includes: a first processing module, configured to preprocess the query information to obtain the preprocessed query information, and perform semantic extraction on the preprocessed query information to obtain a semantic feature vector; a first classification module, configured to classify the semantic feature vector to obtain the task type of the semantic feature vector, and determine the professional type to which the preprocessed query information belongs according to the task type; a first determination module, configured to determine the complexity level of the preprocessed query information, and generate feature vectors for the task type, the professional type, and the complexity level respectively; a first splicing module, configured to splice all the feature vectors to obtain a target feature vector.

[0111] Optionally, the determining device of the language model includes: a first allocation module, configured to allocate a model identifier to each model, and determine the model name and model version of the model before determining the candidate model set according to the target feature vector and the model feature vectors of each model in the preset model characteristic database, where there is an associated relationship between the model identifier and the model name and model version; a second determination module, configured to determine the constraint conditions of each model, where the constraint conditions are used to determine the functions of each model; a first construction module, configured to construct the preset model characteristic database based on the model identifier and the constraint conditions.

[0112] Optionally, the first construction module includes: a first determination sub-module, configured to determine the performance data of each model, where the performance data at least includes: performance task data, metric data, the performance task data at least includes: task type, task performance score of each task type, and the metric data at least includes: performance metric, performance metric value corresponding to each performance metric; a first analysis sub-module, configured to perform feature analysis on the task type, task performance score, performance metric, and performance metric value to obtain the model feature vector of each performance data, and perform feature analysis on the constraint conditions to obtain the model feature vector of the constraint conditions; a first construction sub-module, configured to construct the preset model characteristic database based on the model feature vector of the performance data associated with each model identifier and the model feature vector of the constraint conditions.

[0113] Optionally, the determining device of the language model includes: a first conversion module, configured to convert the query information into target query information in a preset input format after determining the target language model from the candidate model set based on the model score, where the preset input format is the input format corresponding to the target language model; a first invocation module, configured to find the target interface of the target language model in the preset interface library and invoke the target interface; a second conversion module, configured to convert the query result into a preset output format to obtain the target result when it is detected that the target language model processes the target query information and outputs the query result.

[0114] Optionally, the first invocation module includes: a first monitoring sub-module, configured to monitor the invocation of the target interface, and re-invoke the target interface according to the preset retry mechanism when it is detected that the target interface is not successfully invoked; a second determination sub-module, configured to re-determine the target language model according to the model score when it is detected that the target interface is not successfully re-invoked.

[0115] Optionally, the determining device of the language model includes: a first acquisition module, configured to receive feedback data of the target terminal and acquire current performance data of the target language model during the process of processing a query request after determining the target language model from a candidate model set, where the feedback data at least includes: explicit feedback data and implicit feedback data, the explicit feedback data is a score of the target result by the target terminal, and the implicit feedback data at least includes: the duration of the query information, where the duration is the duration between the moment when feature analysis of the query information starts and the moment when feature analysis of other query information starts, and the other query information is information other than the query information; a first update module, configured to update a preset model feature database according to the feedback data and the current performance data to obtain an updated preset model feature database, and adjust preset weights of each performance data corresponding to each model to obtain adjusted preset weights; a third determination module, configured to re-determine the target language model based on the updated preset model feature database and the adjusted preset weights.

[0116] The above-mentioned determining device of the language model may further include a processor and a memory. The above-mentioned first analysis unit 90, first determination unit 91, first allocation unit 92, second determination unit 93, etc. are all stored in the memory as program units, and the corresponding functions are implemented by the processor executing the above program units stored in the memory.

[0117] The above-mentioned processor includes a kernel, and the kernel retrieves the corresponding program units from the memory. One or more kernels may be provided, and by adjusting the kernel parameters, the model score of each model is determined according to the performance data and the preset weights, and based on the model scores, the target language model is determined from the candidate model set.

[0118] The above-mentioned memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of, for example, read-only memory (ROM) or flash memory (flash RAM), and the memory includes at least one storage chip.

[0119] It should be noted here that the above-mentioned first analysis unit 90, first determination unit 91, first allocation unit 92, second determination unit 93 correspond to steps S201 to S204 in Embodiment 1. The examples and application scenarios implemented by the above units and the corresponding steps are the same, but are not limited to the content disclosed in the above Embodiment 1. It should be noted that the above units may be hardware components or software components stored in a memory (for example, memory 104) and processed by one or more processors (for example, processors 102a, 102b,..., 102n), and the above units may also be part of a device and can run in the computer terminal 10 provided in Embodiment 1.

[0120] Example 3

[0121] An embodiment of the present application may provide a computer terminal, and the computer terminal may be any computer terminal device in a computer terminal group. Optionally, in this embodiment, the above computer terminal may also be replaced with a mobile terminal or other terminal devices such as an electronic device.

[0122] Optionally, in this embodiment, the above computer terminal may be located in at least one of multiple network devices in a computer network.

[0123] In this embodiment, the above computer terminal may execute the program code of the following steps in the method for determining a language model: receiving a query request sent by a target terminal, and performing feature analysis on the query information carried in the query request to obtain a target feature vector, where the target feature vector is obtained by splicing multiple feature vectors of the query information, and each feature vector is used to represent different feature information; determining a candidate model set according to the target feature vector and the model feature vectors of each model in a preset model characteristic database, where the preset model characteristic database at least includes: multiple models, each model corresponds to multiple performance data, and each performance data includes a model feature vector; assigning different preset weights to the performance data corresponding to each model in the candidate model set; determining the model score of each model according to the performance data and the preset weights, and based on the model score, determining a target language model from the candidate model set, where the target language model is used to process the query request of the target terminal.

[0124] Optionally, the above computer terminal may execute the program code of the following steps in the method for determining a language model: preprocessing the query information to obtain processed query information, and performing semantic extraction on the processed query information to obtain a semantic feature vector; classifying the semantic feature vector to obtain the task type of the semantic feature vector, and determining the professional type to which the processed query information belongs according to the task type; determining the complexity level of the processed query information, and generating feature vectors for the task type, professional type, and complexity level respectively; splicing all the feature vectors to obtain a target feature vector.

[0125] Optionally, the above computer terminal may execute the program code of the following steps in the method for determining a language model: assigning a model identifier to each model, and determining the model name and model version of the model, where there is an associated relationship between the model identifier and the model name and model version; determining the constraint conditions of each model, where the constraint conditions are used to determine the function of each model; constructing a preset model characteristic database based on the model identifier and the constraint conditions.

[0126] Optionally, the above computer terminal may execute the program code of the following steps in the method for determining a language model: determining the performance data of each model, where the performance data at least includes: performance task data and metric data, the performance task data at least includes: task type and the task performance score of each task type, and the metric data at least includes: performance metrics and the performance metric values corresponding to each performance metric; performing feature analysis on the task type, task performance score, performance metrics, and performance metric values to obtain the model feature vector of each performance data, and performing feature analysis on the constraint conditions to obtain the model feature vector of the constraint conditions; constructing a preset model characteristic database based on the model feature vector of the performance data associated with each model identifier and the model feature vector of the constraint conditions.

[0127] Optionally, the above computer terminal may execute the program code of the following steps in the method for determining a language model: converting the query information into target query information in a preset input format, where the preset input format is the input format corresponding to the target language model; searching for the target interface of the target language model in the preset interface library and calling the target interface; when it is detected that the target language model processes the target query information and outputs a query result, converting the query result into a preset output format to obtain the target result.

[0128] Optionally, the above computer terminal may execute the program code of the following steps in the method for determining a language model: monitoring the call of the target interface, and when it is detected that the target interface fails to be called successfully, re-calling the target interface according to the preset retry mechanism; when it is detected that the target interface fails to be re-called successfully, re-determining the target language model according to the model score.

[0129] Optionally, the above computer terminal may execute the program code of the following steps in the method for determining a language model: receiving the feedback data of the target terminal and obtaining the current performance data of the target language model during the process of processing the query request, where the feedback data at least includes: explicit feedback data and implicit feedback data, the explicit feedback data is the score of the target result by the target terminal, and the implicit feedback data at least includes: the duration of the query information, and the duration is the duration between the moment when feature analysis of the query information starts and the moment when feature analysis of other query information starts, and the other query information is information other than the query information; updating the preset model characteristic database according to the feedback data and the current performance data to obtain an updated preset model characteristic database, and adjusting the preset weights of each performance data corresponding to each model to obtain adjusted preset weights; re-determining the target language model based on the updated preset model characteristic database and the adjusted preset weights.

[0130] Optionally, Figure 10is a block diagram of an electronic device according to an embodiment of the present application. As Figure 10 shown, the electronic device may include: one or more ( Figure 10 only one is shown in the figure) processors 1002, a memory 1004, a storage controller, and a peripheral interface, wherein the peripheral interface is connected to a radio frequency module, an audio module, and a display.

[0131] Among them, the memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the language model determination method and device in the embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, implements the above-mentioned language model determination method. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some instances, the memory may further include a memory remotely provided with respect to the processor, and these remote memories may be connected to the terminal through a network. Examples of the above network include but are not limited to the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof.

[0132] The processor may call the information and application programs stored in the memory through a transmission device to execute the above steps in the above-mentioned language model determination method.

[0133] Adopting the embodiment of the present application, a solution for determining a language model is provided. Through a query analysis method of deep learning, user inputs are accurately classified into multiple task types, query characteristics are represented by multi-dimensional feature vectors, and intelligent matching is performed with a preset model characteristic database to achieve optimal single model selection. By comprehensively considering multiple factors such as task type, domain professionalism, historical performance, cost-effectiveness, and response time, a comprehensive score is given to candidate models to obtain a target language model, and the interface is uniformly called and the query results are standardized to obtain a target result. By collecting user feedback and system operation data, the routing decision weights are automatically adjusted, the routing decision algorithm is continuously optimized, and the model characteristic data is updated to achieve personalized model selection. On the premise of ensuring response quality, factors such as API call cost and response time can be considered to achieve optimal resource allocation, thereby solving the technical problem in the related art that the language model cannot be accurately determined, resulting in low resource utilization.

[0134] Those of ordinary skill in the art can understand that Figure 10 the structure shown is only schematic, and the electronic device may also be a terminal device such as a smart phone, a tablet computer, a personal digital assistant, and a mobile Internet device (MID). Figure 10 It does not limit the structure of the above-mentioned electronic device. For example, the electronic device may further include moreFigure 10 more or fewer components (such as network interfaces, display devices, etc.) shown, or having a configuration different from that Figure 10 shown.

[0135] Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium. The storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, an optical disc, etc.

[0136] Embodiment 4

[0137] An embodiment of the present application also provides a storage medium. Optionally, in this embodiment, the above storage medium can be used to store the program code executed by the method for determining the language model provided in the first embodiment above.

[0138] Optionally, in this embodiment, the above storage medium can be located in any one of the computer terminals in a computer terminal group in a computer network, or in any one of the mobile terminals in a mobile terminal group.

[0139] The present application also provides a computer program product, which is suitable for executing a program for the steps of the method for determining the language model when executed on a data processing device.

[0140] The serial numbers of the above embodiments of the present application are only for description and do not represent the advantages or disadvantages of the embodiments.

[0141] In the above embodiments of the present application, the descriptions of the various embodiments have their own emphases. For the parts not detailed in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0142] In the several embodiments provided by the present application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only illustrative. For example, the division of the units is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point, the displayed or discussed coupling or direct coupling or communication connection to each other can be through some interfaces, and the indirect coupling or communication connection of the units or modules can be in an electrical or other form.

[0143] The unit described as a separation component may or may not be physically separated. The component shown as a unit may or may not be a physical unit, that is, it may be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0144] In addition, each functional unit in various embodiments of the present application may be integrated in a processing unit, may exist separately as individual physical units, or two or more units may be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0145] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present application. The aforementioned storage medium includes: USB flash drives, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), mobile hard disks, magnetic disks, or optical disks and other various media that can store program codes.

[0146] The above are only the preferred embodiments of the present application. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present application, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present application.

Claims

1. A method for determining a language model, characterized in that, Including: Receiving a query request sent by a target terminal, and performing feature analysis on the query information carried in the query request to obtain a target feature vector, where the target feature vector is obtained by splicing multiple feature vectors of the query information, and each feature vector is used to represent different feature information; Determining a candidate model set according to the target feature vector and the model feature vectors of each model in a preset model characteristic database, where the preset model characteristic database at least includes: multiple models, each model corresponding to multiple performance data, and each performance data includes the model feature vector; Assigning different preset weights to the performance data corresponding to each model in the candidate model set; Determining the model score of each model according to the performance data and the preset weights, and based on the model score, determining a target language model from the candidate model set, where the target language model is used to process the query request of the target terminal.

2. The method for determining the language model according to claim 1, characterized in that, The step of performing feature analysis on the query information carried in the query request to obtain a target feature vector includes: Performing preprocessing on the query information to obtain processed query information, and performing semantic extraction on the processed query information to obtain a semantic feature vector; Classifying the semantic feature vector to obtain the task type of the semantic feature vector, and determining the professional type to which the processed query information belongs according to the task type; Determining the complexity level of the processed query information, and generating the feature vectors for the task type, the professional type, and the complexity level respectively; Splicing all the feature vectors to obtain the target feature vector.

3. The method for determining a language model according to claim 1, wherein Before determining the candidate model set according to the target feature vector and the model feature vectors of each model in the preset model characteristic database, it further includes: Assigning a model identifier to each model, and determining the model name and model version of the model, where there is an associated relationship between the model identifier and the model name and the model version; Determining the constraint conditions of each model, where the constraint conditions are used to determine the functions of each model; Constructing the preset model characteristic database based on the model identifier and the constraint conditions.

4. The method for determining the language model according to claim 3, characterized in that, The step of constructing the preset model characteristic database based on the model identifier and the constraint conditions includes: Determining the performance data of each model, where the performance data at least includes: performance task data, index data, the performance task data at least includes: task type, task performance score of each task type, and the index data at least includes: performance index, performance index value corresponding to each performance index; Performing feature analysis on the task type, the task performance score, the performance index, and the performance index value to obtain the model feature vector of each performance data, and performing feature analysis on the constraint conditions to obtain the model feature vector of the constraint conditions; Construct the preset model characteristic database based on the model feature vectors associated with the performance data of each of the said models and the model feature vectors of the said constraint conditions.

5. The method for determining a language model according to claim 1, wherein After determining the target language model from the candidate model set based on the model scores, it further includes: Convert the query information into target query information in a preset input format, where the preset input format is the input format corresponding to the target language model; Search for the target interface of the target language model in the preset interface library and call the target interface; When it is detected that the target language model processes the target query information and outputs a query result, convert the query result into a preset output format to obtain a target result.

6. The method for determining a language model according to claim 5, wherein During the process of calling the target interface, it further includes: Monitor the call of the target interface. When it is monitored that the target interface is not successfully called, re-call the target interface according to a preset retry mechanism; When it is monitored that the target interface is not successfully re-called, re-determine the target language model according to the model scores.

7. The method for determining a language model according to claim 1, wherein After determining the target language model from the candidate model set, it further includes: Receive the feedback data of the target terminal and obtain the current performance data of the target language model during the process of processing the query request, where the feedback data at least includes: explicit feedback data and implicit feedback data, the explicit feedback data is the score of the target result by the target terminal, and the implicit feedback data at least includes: the duration of the query information, and the duration is the duration between the moment when the feature analysis of the query information starts and the moment when the feature analysis of other query information starts, and the other query information is information other than the query information; Update the preset model characteristic database according to the feedback data and the current performance data to obtain an updated preset model characteristic database, and adjust the preset weights of each performance data corresponding to each model to obtain the adjusted preset weights; Re-determine the target language model based on the updated preset model characteristic database and the adjusted preset weights.

8. An apparatus for determining a language model, characterized in that It includes: A first analysis unit, configured to receive a query request sent by a target terminal and perform feature analysis on the query information carried in the query request to obtain a target feature vector, where the target feature vector is obtained by splicing multiple feature vectors of the query information, and each feature vector is used to represent different feature information; A first determination unit, configured to determine a candidate model set according to the target feature vector and the model feature vectors of each model in the preset model characteristic database, where the preset model characteristic database at least includes: multiple said models, each model corresponds to multiple performance data, and each performance data includes the model feature vector; A first allocation unit, configured to allocate different preset weights to the performance data corresponding to each model in the candidate model set; A second determination unit, configured to determine a model score for each of the models according to the performance data and the preset weights, and determine a target language model from the candidate model set based on the model scores, where the target language model is used to process the query request of the target terminal.

9. A computer program product, characterized in that, It includes a non-volatile computer-readable storage medium that stores a computer program, and when the computer program is executed by a processor, it implements the method for determining a language model according to any one of claims 1 to 7.

10. An electronic device, characterized in that, It includes one or more processors and a memory, where the memory is used to store one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method for determining a language model according to any one of claims 1 to 7.

Images