Data processing method, terminal equipment and computer readable storage medium

Abstract

The invention is suitable for the technical field of artificial intelligence, and provides a data processing method, terminal equipment and a computer readable storage medium. In the embodiment of the invention, sampling data is automatically acquired, and the sampling data comprises first input information of an initial routing model and first output information used for indicating a first generation model; and adjusting the first output information in the sampling data according to the model efficiency of the candidate model connected with the initial routing model so as to generate training data. Through the method, the training data of the routing model can be automatically generated, manual labeling is not needed, manpower is saved, and efficiency is improved; and the training data generated according to the model efficiency is beneficial to training the routing model for accurate routing, so that the generation model routed by the trained routing model has relatively high model efficiency.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of artificial intelligence, and particularly relates to a data processing method, a terminal device and a computer readable storage medium. BACKGROUND

[0002] At present, with the vigorous development of artificial intelligence and large model technology, performance evaluation and iterative optimization of models have become key links. As an important part of the application scenario of large models, the core role of the routing model is to accurately route the input request or task to the appropriate large model according to certain rules or conditions, so as to realize efficient and accurate response.

[0003] At present, in the training process of the routing model, manual collection and labeling of training data are usually required. This way has high labor cost and low efficiency of training set construction, which affects the training efficiency of the routing model. SUMMARY

[0004] The embodiments of the application provide a data processing method, a terminal device and a computer readable storage medium, which can improve the generation efficiency of training data.

[0005] In a first aspect, the embodiments of the application provide a data processing method, comprising: obtaining sampling data; wherein each group of sampling data comprises first input information and first output information of an initial routing model, the initial routing model is used to select a model from candidate models to respond to the input information according to the input information; the candidate model comprises at least one generation model connected with the initial routing model; and the first output information is used to indicate a first generation model in the candidate model; selecting a second generation model with the highest performance from the candidate model according to the first input information; if the second generation model is the first generation model, regarding the sampling data as training data; if the second generation model is not the first generation model, generating a group of training data according to the second generation model; wherein the training data is used to train the initial routing model; the training data comprises first input information and second output information of the initial routing model, and the second output information is used to indicate the second generation model.

[0006] In the embodiments of the present application, the sampling data is automatically acquired, and the data includes first input information of an initial routing model and first output information used for indicating a first generation model; the first output information in the sampling data is adjusted according to model performance of a candidate model connected with the initial routing model, so as to generate training data. Through the above method, the training data of the routing model can be automatically generated without manual annotation, manpower is saved, and efficiency is improved; and the training data generated according to the model performance is beneficial to training the routing model with accurate routing, so that the generation model routed by the trained routing model has high model performance.

[0007] In a possible implementation of the first aspect, the selecting, from the candidate models, a second generation model with the highest model performance according to the first input information comprises: evaluating the model performance according to input and output information of the first generation model to obtain a first score; wherein the input and output information of the first generation model is obtained according to the first input information; the model performance includes a first performance indicator and a second performance indicator; if the first score is greater than a preset value, selecting the second generation model in the order of a first sequence; wherein the first sequence is a sequence obtained by arranging the models in the candidate models from inferior to superior according to the first performance indicator; if the first score is less than the preset value, selecting the second generation model in the order of a second sequence; wherein the second sequence is a sequence obtained by arranging the models in the candidate models from superior to inferior according to the second performance indicator; the priority of the first performance indicator is lower than the priority of the second performance indicator.

[0008] Optionally, the first performance indicator is model cost, and the second performance indicator is model effect.

[0009] In the above manner, the model performance is evaluated by comprehensively considering the first performance indicator and the second performance indicator, which is beneficial to selecting the second generation model with the optimal comprehensive performance, and the training data generated accordingly provides reliable data basis for the training of the subsequent routing model, which is beneficial to training the routing model with accurate routing. In addition, in the case that the first score is greater than the preset value, the selection is performed according to the first performance indicator with the lower priority; in the case that the first score is less than the preset value, the selection is performed according to the second performance indicator with the higher priority. This is equivalent to considering the performance indicator with the lower priority after ensuring the performance indicator with the higher priority, which is beneficial to selecting the second generation model with the optimal comprehensive performance.

[0010] In a possible implementation of the first aspect, the selecting, from the candidate models, a second generation model with the highest model performance according to the first input information comprises: Select the second generation model according to the model efficiency of the remaining model, starting from the first generation model; The score of the model efficiency of the second generation model is greater than the preset value, and the score of the model efficiency of the next model of the second generation model in the first sequence is less than the preset value.

[0011] In this way, when the model efficiency of the first generation model meets the requirement, the model selection is performed by giving priority to the performance index with a lower priority, so that the selected model can meet the first performance index while meeting the model efficiency requirement, thereby facilitating selection of the second generation model with optimal comprehensive performance.

[0012] In a possible implementation of the first aspect, the second generation model is selected in the order of the second sequence, including: Select the second generation model according to the model efficiency, starting from the first model of the second sequence; The score of the model efficiency of the second generation model is greater than the preset value, and the score of the model efficiency of the next model of the second generation model in the second sequence is less than the preset value.

[0013] In this way, when the model efficiency of the first generation model does not meet the requirement, the model selection is performed by giving priority to the performance index with a higher priority, so that the selected model can meet the second performance index preferentially, thereby facilitating selection of the second generation model with optimal comprehensive performance.

[0014] In a possible implementation of the first aspect, the method further includes: Generating a training set according to a plurality of groups of training data; Training the initial routing model according to the training data with the first data proportion in the training set to obtain a trained model; Evaluating the trained model according to test data in the training set to obtain an evaluation result, wherein the test data is training data other than the training data with the first data proportion in the training set.

[0015] In this way, the training set and the test set are automatically constructed without manual construction, thereby saving labor costs and improving the construction efficiency of the training set and the test set, and facilitating improvement of the model training efficiency.

[0016] In a possible implementation of the first aspect, after evaluating the trained model according to the test data in the training set to obtain an evaluation result, the method further includes: If the evaluation result indicates that the evaluation passes, the initial routing model is updated according to the training model to obtain a target routing model; If the evaluation result indicates that the evaluation fails, the initial routing model is recorded as the target routing model.

[0017] In this way, the routing model is updated only when the model evaluation passes, which can ensure the routing effect of the routing model in the application process, so that the routing model can route a more appropriate generation model to achieve efficient and accurate response.

[0018] In a possible implementation manner of the first aspect, the evaluation of the training model according to the test data in the training set to obtain an evaluation result comprises: For each set of test data, input first input information in the test data to a generation model indicated by second output information in the test data to obtain third output information; Calculate the semantic similarity between the reference output information corresponding to the test data and the third output information to obtain a similarity score; Determine the evaluation result according to the similarity score corresponding to each set of test data.

[0019] In this way, the semantic similarity is used to evaluate the training model, which can accurately find the deficiencies of the model in semantic understanding and generation, thereby facilitating the enhancement of the semantic generalization ability of the model.

[0020] In a second aspect, an embodiment of the present application provides a data processing apparatus, comprising: An acquisition unit is configured to acquire sampling data; each set of the sampling data comprises first input information and first output information of an initial routing model, the initial routing model is configured to select a model from candidate models to respond to input information according to the input information; the candidate models comprise at least one generation model connected to the initial routing model; the first output information is configured to indicate a first generation model in the candidate models; A selection unit is configured to select a second generation model with the highest performance from the candidate models according to the first input information; A generation unit is configured to, if the second generation model is the first generation model, use the sampling data as training data; The generation unit is further configured to, if the second generation model is not the first generation model, generate a set of training data according to the second generation model; the training data is configured to train the initial routing model; the training data comprises first input information and second output information of the initial routing model, and the second output information is configured to indicate the second generation model.

[0021] In a third aspect, an embodiment of the present application provides a terminal device, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, and the processor implements the data processing method according to any one of the first aspect when executing the computer program.

[0022] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the data processing method according to any one of the first aspect.

[0023] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when running on a terminal device, causes the terminal device to execute the data processing method according to any one of the first aspect.

[0024] It can be understood that the beneficial effects of the second aspect to the fifth aspect can be referred to the related description of the first aspect, which will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0026] Figure 1 is a schematic diagram of the application architecture of the routing model provided by the embodiment of the present application; Figure 2 is a schematic diagram of the generation model provided by the embodiment of the present application; Figure 3 is a schematic diagram of the transform model architecture provided by the embodiment of the present application; Figure 4 is a flowchart of the data processing method provided by the embodiment of the present application; Figure 5 is a schematic diagram of the overall flow provided by the embodiment of the present application; Figure 6 is a structural block diagram of the data processing apparatus provided by the embodiment of the present application; Figure 7 is a structural schematic diagram of the terminal device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0027] In the following description, for purposes of explanation and not limitation, specific details are set forth such as particular architectures, technologies, techniques, etc. in order to provide a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application can be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.

[0028] It is to be understood that the terminology "includes", "has", "holds", "contains" and / or "comprising", when used in this specification and in the following claims, indicates the presence of the described features, integers, steps, operations, elements, and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0029] It is also to be understood that the terminology "and / or" when used in this specification and in the following claims, refers to at least one of the items, or any combination of the items, and includes all possible combinations when used in the description of items.

[0030] As used in this specification and in the claims, the terms "if" and "when" can be interpreted to mean "upon" or "in response to a determination" or "in response to a detection" depending on the context. Similarly, the phrase "if it is determined" or "if [a described condition or event] is detected" can be interpreted to mean "upon determining" or "in response to determining" or "upon detecting [the described condition or event]" or "in response to detecting [the described condition or event]" depending on the context.

[0031] In addition, the terms "first", "second", "third", etc. in the description of the present application and in the following claims are only used to distinguish descriptions, and cannot be understood as indicating or implying relative importance.

[0032] Reference in the specification to "one embodiment" or "some embodiments" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the application. The appearances of the phrase "in one embodiment" or "in some embodiments" in various places in the specification are not necessarily all referring to the same embodiment, although it can. The terms "comprises", "comprising", "includes", "including", "has", "having", "contains", "containing", or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises, includes, has, contains a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus.

[0033] In the current booming development of artificial intelligence and large model technology, performance evaluation and iterative optimization of models have become key links. As an important part of the application scenario of large models, the core function of the routing model is to accurately route the input request or task to the appropriate large model according to certain rules or conditions, so as to achieve efficient and accurate response.

[0034] Currently, in the training process of the routing model, it is usually necessary to manually collect and label the training data, which is high in labor cost and low in efficiency of constructing the training set, thereby affecting the training efficiency of the routing model.

[0035] Based on this, the embodiment of the present application provides a data processing method. In the embodiment of the present application, the sampling data is automatically acquired, and the data includes the first input information of the initial routing model and the first output information for indicating the first generation model; the first output information in the sampling data is adjusted according to the model performance of the candidate model connected with the initial routing model, thereby generating the training data. Through the above method, the training data of the routing model can be automatically generated, without manual labeling, saving manpower and improving efficiency; and the training data generated according to the model performance is beneficial to training the routing model with accurate routing, so that the generation model routed by the trained routing model has high model performance.

[0036] First, the application scenario of the routing model is introduced.

[0037] Referring to Figure 1 is a schematic diagram of an application architecture of the routing model provided by the embodiment of the present application. As an example but not limitation, as shown in Figure 1 , the input end of the routing model is used to receive the input information of the user, and the output end of the routing model is used to connect at least one generation model.

[0038] Figure 1 N generation models are shown in , and N is not specifically limited in actual application, that is, the routing model can be connected with only one generation model, or can be connected with two or more generation models. In the embodiment of the present application, the generation model connected with the output end of the routing model can be recorded as a candidate model.

[0039] Among them, the routing model is used to route to the appropriate generation model according to the input information, that is, to select one generation model from the candidate model to respond to the input information. The generation model is a model with the ability to respond to input information, such as a neural network model, a large language model (LLM), etc., or the generation model can also refer to a model architecture with the ability to respond to input information, which not only includes a certain model, but also includes subsequent retrieval and enhancement processes of the model.

[0040] As an example but not limitation, referring to Figure 2 is a schematic diagram of a generation model provided by the embodiment of the present application. As an example but not limitation, as shown in Figure 2 , the generation model refers to a model architecture with the ability to respond to input information, which includes an embedded model, a database and an enhancement model.

[0041] In the response process, the embedded model vectorizes the input information to obtain a representation vector, and sends the representation vector to the vector database; the database searches for related content according to the representation vector; the enhanced model performs retrieval enhancement processing according to the input information and the related content searched by the database to generate an enhanced prompt word; the enhanced prompt word is input into the embedded model to obtain output information, and the output information is displayed to the user.

[0042] Optionally, the embedded model can adopt a transform model architecture. For example, refer to Figure 3 , which is a schematic diagram of the transform model architecture provided by the embodiment of the present application. As an example but not limitation, as shown in Figure 3 , the transform model architecture includes an input part, an encoder, a decoder and an output part.

[0043] The input part includes an input embedding layer (input embeddings) and a position encoding layer (PositionalEncoding). The input embedding layer is used to convert the input information into word indexes through a vocabulary lookup, and then convert the word indexes into high-dimensional vectors, which can be called word embeddings (or word vectors). The position encoding layer is used to represent the sequential information of each word in the input information.

[0044] The encoder includes a multi-head attention layer (Multi-Head Self-Attention), a feedforward network (Feedforward Network). The multi-head attention layer "allows" the model to view the information of the entire input sequence at once without considering the order of the words, and each "head" independently calculates the self-attention score to capture different patterns of input information from different angles. The feedforward network is used to further process the vector representation after the multi-head attention layer. In addition, the normalization and residual layer (Add&Norm) is connected behind the multi-head attention layer and the feedforward network, which is used for normalization and residual processing of feature data, so that information can be more smoothly transmitted in the network.

[0045] The decoder includes a multi-head attention layer, a feedforward network, and a normalization and residual layer. Among them, the multi-head attention layer of the decoder includes both a self-attention layer for its own historical output (to prevent future information leakage), and an attention layer for the encoder output (to ensure that relevant information of the input sequence can be paid attention to in the decoding process).

[0046] The output part includes a linear mapping layer (Linear) and a softmax layer. The linear mapping layer is used to map the information to the size of the target vocabulary space, and the softmax layer is used to calculate the prediction probability of each possible result.

[0047] Based on the application architecture shown in Figure 1 The application process of the routing model can include: after the user inputs information, the routing model selects a target model from the candidate models according to the input information of the user, and inputs the input information of the user into the target model; the target model responds according to the input information and obtains an output result. As shown in Figure 1 The target model selected by the routing model according to the input information is the generation model 2, and the input information is input into the generation model 2. The generation model 2 responds and obtains the output information.

[0048] For example, in a search application scenario, the user inputs a keyword for searching (input information); the routing model selects a target model from the candidate models according to the keyword, and inputs the keyword into the target model; the target model searches for content matching the keyword and outputs the content as output information to the user.

[0049] For another example, in an intelligent question and answer application scenario, the user inputs a question; the routing model selects a target model from the candidate models according to the input question, and inputs the input question into the target model; the target model outputs reply content matching the input question as output information to the user.

[0050] In some implementations, the routing model also has an information rewriting capability. Specifically, after the routing model determines the target model, the input information is rewritten into information in an input format conforming to the target model, and the rewritten input information is input into the target model.

[0051] Based on the application architecture of the routing model shown in Figure 1 The following describes a data processing method of an embodiment of the present application. Referring to Figure 4 FIG. 1 is a flowchart of a data processing method provided by an embodiment of the present application, which is an example and is not limited, and the method can include the following steps: S101, obtaining sampling data.

[0052] Each set of sampling data includes first input information and first output information of an initial routing model, and the initial routing model is used to select a model from candidate models according to input information to respond to the input information; the candidate models include at least one generation model connected to the initial routing model; and the first output information is used to indicate a first generation model in the candidate models.

[0053] Optionally, the sampling data can be obtained according to historical data. For example, in an intelligent question and answer application scenario, log data of a set of historical conversations is obtained, the input content of the user in the set of historical conversations is obtained as the first input information according to the log data, and the output information of the routing model in the set of historical conversations is obtained as the first output information.

[0054] S102, selecting a second generation model with the highest model performance from the candidate models according to the first input information.

[0055] The model performance can be evaluated according to a single performance index. However, in this way, the performance of the generation model cannot be comprehensively evaluated from multiple aspects.

[0056] In view of this, in an embodiment, the model performance can be evaluated according to multiple performance indexes. Specifically, S102 can include: evaluating the model performance according to the input and output information of the first generation model to obtain a first score; wherein the input and output information of the first generation model is obtained according to the first input information; the model performance includes a first performance index and a second performance index; if the first score is greater than a preset value, selecting the second generation model in the order of a first sequence; wherein the first sequence is a sequence obtained by arranging the models in the candidate models in the order from bad to good according to the first performance index; if the first score is less than the preset value, selecting the second generation model in the order of a second sequence; wherein the second sequence is a sequence obtained by arranging the models in the candidate models in the order from good to bad according to the second performance index.

[0057] The priority of the first performance index is lower than the priority of the second performance index.

[0058] Optionally, the first performance index is model cost, and the second performance index is model effect.

[0059] It can be understood that the purpose of evaluating the model performance is to select a generation model that takes into account both cost and effect from all candidate models with a model performance score greater than a preset value. When the first score is greater than the preset value, it indicates that the first generation model has met the model performance at this time, and the cost needs to be considered first, and then the selection is performed in the order from high to low (the order from bad to good according to the first performance index). When the first score is less than the preset value, it indicates that the first generation model does not meet the model performance at this time, and the effect needs to be considered first, and then the selection is performed in the order from good to bad (the order from good to bad according to the second performance index).

[0060] Optionally, the first sequence and the second sequence can be generated in advance. For example, the index values of the first performance index and the second performance index of each generation model are obtained; each generation model is sorted according to the index value of the first performance index to obtain the first sequence; and each generation model is sorted according to the index value of the second performance index to obtain the second sequence.

[0061] By the above manner, the model performance is evaluated by comprehensively considering the first performance index and the second performance index, which is beneficial to select the second generation model with the optimal comprehensive performance, and the training data generated according to the second generation model provides a reliable data basis for the training of the subsequent routing model, which is beneficial to train the routing model with high routing accuracy. In addition, in the case that the first score is greater than the preset value, the first generation model is selected according to the first performance index with lower priority; in the case that the first score is less than the preset value, the second generation model is selected according to the second performance index with higher priority. That is, the performance index with higher priority is guaranteed, and then the performance index with lower priority is considered, which is beneficial to select the second generation model with the optimal comprehensive performance.

[0062] In one implementation manner, the second generation model is selected in the order of the first sequence, including: The second generation model is selected according to the model performance of the remaining model based on the first generation model as the starting point.

[0063] The remaining model is a model located after the first generation model in the first sequence; the score of the model performance of the second generation model is greater than the preset value, and the score of the model performance of the next model of the second generation model in the first sequence is less than the preset value.

[0064] Specifically, the next model of the first generation model in the first sequence is obtained to obtain a third generation model; the model performance is evaluated according to the input and output information of the third generation model to obtain a second score; the input and output information of the third generation model is obtained according to the first input information; if the second score is less than the preset value, the first generation model is recorded as the second generation model; if the second score is greater than the preset value, the next model of the third generation model in the first sequence is continuously obtained until the second generation model is selected.

[0065] The first input information is input into the third generation model to obtain the output information of the third generation model. The first input information and the output information of the third generation model obtained based on the first input information are recorded as the input and output information of the third generation model obtained according to the first input information.

[0066] For example, the first sequence includes model A, model B, model C and model D in turn, indicating that the model cost of model A is the highest and the model cost of model D is the lowest. If the first generation model is model B and the first score is greater than the preset value, the first input information is input into model C to obtain output information of model C, and the model efficiency is evaluated according to the input and output information of model C to obtain score C1; if score C1 is less than the preset value, model B is recorded as the second generation model; if score C1 is greater than the preset value, the first input information is input into model D to obtain output information of model D, and the model efficiency is evaluated according to the input and output information of model D to obtain score D1; if score D1 is less than the preset value, model C is recorded as the second generation model; if score D1 is greater than the preset value, model D1 is recorded as the second generation model.

[0067] In the above manner, when the model efficiency of the first generation model meets the requirement, the performance index with lower priority is given priority, and the model is selected accordingly, so that the selected model can meet the first performance index while meeting the model efficiency requirement, thereby facilitating selection of the second generation model with the optimal comprehensive performance.

[0068] In one implementation, the second generation model is selected in the order of the second sequence, including: Starting from the first model of the second sequence, the second generation model is selected according to the model efficiency.

[0069] The score of the model efficiency of the second generation model is greater than the preset value, and the score of the model efficiency of the next model of the second generation model in the second sequence is less than the preset value.

[0070] Specifically, the first generation model is obtained from the second sequence to obtain a fourth generation model; the model efficiency is evaluated according to the input and output information of the fourth generation model to obtain a third score; the input and output information of the fourth generation model is obtained according to the first input information; if the third score is less than the preset value, the fourth generation model is recorded as the second generation model; if the third score is greater than the preset value, the next model of the fourth generation model is obtained from the second sequence until the second generation model is selected.

[0071] The first input information is input into the fourth generation model to obtain output information of the fourth generation model. The first input information and the output information of the fourth generation model obtained based on the first input information are recorded as the input and output information of the fourth generation model obtained according to the first input information.

[0072] It can be understood that the order of the models in the first sequence and the second sequence can be different. For example, model A has the best effect, but its cost can also be higher.

[0073] For example, the second sequence includes model A, model C, model D and model B in turn, indicating that the effect of model A is the best and the effect of model B is the worst. If the first generation model is model D and the first score is less than the preset value, the first input information is input into model A to obtain the output information of model A, and the model performance is evaluated according to the input and output information of model A to obtain score A2. If score A2 is still less than the preset value, model A is recorded as the second generation model. If score A2 is greater than the preset value, the first input information is input into model C to obtain the output information of model C, and the model performance is evaluated according to the input and output information of model C to obtain score C2. If score C2 is less than the preset value, model A is recorded as the second generation model. If score C2 is greater than the preset value, since the first score of model D is less than the preset value, model C is recorded as the second generation model.

[0074] In the above manner, when the model performance of the first generation model does not meet the requirements, the performance index with higher priority is given priority, and the model is selected accordingly, so that the selected model can preferentially meet the second performance index, thereby facilitating the selection of the second generation model with the best comprehensive performance.

[0075] It should be noted that only the first performance index and the second performance index are used as examples in the above embodiments. In actual application, more performance indexes can be used to evaluate the model performance, and the implementation principle is the same as that of using the first performance index and the second performance index for evaluation, which will not be described here.

[0076] S103, if the second generation model is the first generation model, the sampling data is used as the training data.

[0077] S104, if the second generation model is not the first generation model, a set of training data is generated according to the second generation model.

[0078] The training data is used to train the initial routing model; the training data includes the first input information of the initial routing model and the second output information, and the second output information is used to indicate the second generation model.

[0079] Figure 4 In the embodiment, the sampling data is automatically obtained, and the data includes the first input information of the initial routing model and the first output information used to indicate the first generation model. The first output information in the sampling data is adjusted according to the model performance of the candidate model connected to the initial routing model, so as to generate the training data. Through the above method, the training data of the routing model can be automatically generated without manual annotation, which saves manpower and improves efficiency. Moreover, the training data generated according to the model performance is conducive to training the routing model with accurate routing, so that the generated model routed by the trained routing model has high model performance.

[0080] It can be understood that through the method of S101-S104, multiple sets of training data can be obtained, the training set is generated from the multiple sets of training data, and the initial routing model is trained according to the training set.

[0081] In one embodiment, the process of training the initial routing can include the following steps: generating a training set according to the multiple sets of training data; training the initial routing model according to the training data with the first data proportion in the training set to obtain a training model; evaluating the training model according to the test data in the training set to obtain an evaluation result; wherein the test data is the training data in the training set except the training data with the first data proportion.

[0082] Optionally, the training process can be performed offline. For example, as shown in the flowchart, Figure 5 the routing model in the current application process can be trained offline, that is, the routing model in the current application process can be used as the initial routing model in the training process.

[0083] The first data proportion represents the ratio of the number of training data participating in the training to the total number of training sets. For example, the first data proportion is 9 / 10, the total number of training data included in the training set is 100, then 100x(9 / 10)=90 groups of training data in the training set are used to train the initial routing model, and the remaining 10 groups of training data are used to evaluate the training model.

[0084] In the above manner, the training set and the test set are automatically constructed, without the need for manual construction, saving human cost, improving the construction efficiency of the training set and the test set, and facilitating the improvement of the efficiency of model training.

[0085] In one implementation of evaluating the training model, it can include: for each set of test data, input the first input information in the test data to the generated model indicated by the second output information in the test data to obtain third output information; calculate the semantic similarity between the reference output information corresponding to the test data and the third output information to obtain a similarity score; determine the evaluation result according to the similarity score corresponding to each set of test data.

[0086] It can be understood that the reference output information corresponding to the test data is the output information obtained by inputting the first input information in the test data to the generated model indicated by the second output information in the test data.

[0087] For example, the calculation method of semantic similarity is shown in the following code: def recursive_score(sol, exp, weight, semantic_key_list, semantic_threshold, need_semantic=False) -> float: if not isinstance(exp, dict): # Leaf node, score if value and type are the same, otherwise 0 points if need_semantic: semantic_score = semantic_judgment_score(sol, exp) semantic_score = semantic_score * weight if semantic_score > semantic_threshold else 0 return semantic_score return weight if sol == exp and type(sol) == type(exp) else 0.0 keys = exp.keys() n = len(keys) if n == 0: return weight if sol == exp else 0.0 score = 0.0 for k in keys: if k in sol: score += recursive_score(sol[k], exp[k], weight / n, semantic_key_list, need_semantic = True if k in semantic_key_list else False) return score Optionally, if the similarity scores corresponding to each set of test data are all greater than the preset score, the evaluation passes; if there is any set of test data corresponding to a similarity score less than or equal to the preset score, the evaluation fails. Alternatively, if the number of sets of test data with similarity scores greater than the preset score reaches a preset number, the evaluation passes; if the number of sets of test data with similarity scores greater than the preset score is less than the preset number, the evaluation fails. Alternatively, if the number of sets of test data with similarity scores greater than the preset score is greater than the number of sets of test data with similarity scores less than the preset score, the evaluation passes; if the number of sets of test data with similarity scores greater than the preset score is less than the number of sets of test data with similarity scores less than the preset score, the evaluation fails.

[0088] In the above manner, the semantic similarity is used to evaluate the training model, which can accurately find the deficiencies of the model in semantic understanding and generation, thereby facilitating the enhancement of the semantic generalization ability of the model.

[0089] In one embodiment, after evaluating the training model according to the remaining data in the training set to obtain an evaluation result, the method further comprises: If the evaluation result indicates that the evaluation passes, updating the initial routing model according to the training model to obtain a target routing model; If the evaluation result indicates that the evaluation fails, recording the initial routing model as the target routing model.

[0090] As shown in the flow, Figure 5 If the evaluation passes, replacing the routing model currently used in the application process with the target routing model; if the evaluation fails, continuing to use the current initial routing model in the application process.

[0091] In the above manner, the routing model is updated only after the model evaluation passes, which can ensure the routing effect of the routing model in the application process, so that the routing model can route a more appropriate generation model to achieve efficient and accurate response.

[0092] As shown in the flow, Figure 5 The overall flow can include an application process and a training process.

[0093] In the application process, after the user inputs information, the routing model selects a target model from the candidate models according to the user's input information, and inputs the user's input information into the target model; the target model responds according to the input information to obtain an output result.

[0094] The input information, the output information of the target model, and the output information of the routing model involved in each interaction with the user can be logged and stored.

[0095] In the training process, the routing model currently being applied is denoted as an initial routing model. A plurality of sets of sampling data are obtained from logs, and a plurality of sets of training data are generated according to the above-mentioned Figure 4 The method described in the embodiments generates a set of training data according to each set of sampling data; generates a training set according to the plurality of sets of training data; trains the initial routing model according to the training data with a first data proportion in the training set to obtain a training model; performs model evaluation on the training model according to the remaining data in the training set to obtain an evaluation result; if the evaluation result indicates that the evaluation passes, updates the initial routing model according to the training model to obtain a target routing model, and deploys the target routing model to the application process; if the evaluation result indicates that the evaluation fails, continues to use the current initial routing model in the application process.

[0096] It should be understood that the size of the serial number of each step in the above embodiments does not mean the order of execution, and the execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0097] Corresponding to the data processing method described in the above embodiments, Figure 6 is a structural block diagram of a data processing apparatus provided by the embodiments of the present application, and only parts related to the embodiments of the present application are shown for ease of description.

[0098] Referring to Figure 6 , the apparatus comprises: An obtaining unit 61 is configured to obtain sampling data; wherein each set of the sampling data comprises first input information and first output information of an initial routing model, the initial routing model is configured to select a model from candidate models to respond to input information according to the input information; the candidate models comprise at least one generation model connected to the initial routing model; and the first output information is used to indicate a first generation model in the candidate models.

[0099] A selecting unit 62 is configured to select a second generation model with the highest performance from the candidate models according to the first input information.

[0100] A generating unit 63 is configured to, if the second generation model is the first generation model, use the sampling data as training data.

[0101] The generating unit 63 is further configured to, if the second generation model is not the first generation model, generate a set of training data according to the second generation model; wherein the training data is used to train the initial routing model; and the training data comprises first input information and second output information of the initial routing model, and the second output information is used to indicate the second generation model.

[0102] Optionally, the selecting unit 62 is further configured to: The model performance is evaluated according to input and output information of the first generation model, and a first score is obtained; wherein the input and output information of the first generation model is obtained according to the first input information; the model performance includes a first performance index and a second performance index; If the first score is greater than a preset value, the second generation model is selected according to a first sequence; wherein the first sequence is a sequence obtained by arranging the models in the candidate models from poor to good according to the first performance index; If the first score is less than a preset value, the second generation model is selected according to a second sequence; wherein the second sequence is a sequence obtained by arranging the models in the candidate models from good to poor according to the second performance index; the priority of the first performance index is lower than the priority of the second performance index.

[0103] Optionally, the selection unit 62 is further configured to: select the second generation model according to the model performance of the remaining models, starting from the first generation model; wherein the remaining models are the models in the first sequence after the first generation model; the score of the model performance of the second generation model is greater than the preset value, and the score of the model performance of the next model of the second generation model in the first sequence is less than the preset value.

[0104] Optionally, the selection unit 62 is further configured to: select the second generation model according to the model performance, starting from the first model of the second sequence; wherein the score of the model performance of the second generation model is greater than the preset value, and the score of the model performance of the next model of the second generation model in the second sequence is less than the preset value.

[0105] Optionally, the device 6 further includes a training unit 64 configured to: generate a training set according to a plurality of groups of the training data; train the initial routing model according to the training data with a first data proportion in the training set, to obtain a trained model; evaluate the trained model according to test data in the training set, to obtain an evaluation result; wherein the test data is the training data in the training set except the training data with the first data proportion.

[0106] Optionally, the training unit 64 is further configured to: if the evaluation result indicates that the evaluation is passed, update the initial routing model according to the trained model, to obtain a target routing model; If the evaluation result indicates that the evaluation fails, the initial routing model is recorded as the target routing model.

[0107] Optionally, the training unit 64 is further configured to: For each set of test data, input first input information in the test data to a generated model indicated by second output information in the test data, to obtain third output information; Calculate semantic similarity between reference output information corresponding to the test data and the third output information, to obtain a similarity score; Determine the evaluation result according to the similarity score corresponding to each set of test data.

[0108] It should be noted that the information interaction, execution process and the like between the above apparatuses / units are based on the same concept as the method embodiments of the present application, and specific functions and technical effects brought by the same can be referred to the method embodiments part, which will not be repeated here.

[0109] In addition, Figure 6 The data processing apparatus shown can be a software unit, a hardware unit, or a software and hardware combined unit built into an existing terminal device, can be integrated into the terminal device as an independent plug-in, or can exist as an independent terminal device.

[0110] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above functional units and modules is exemplified, and in actual application, the above functions can be completed by different functional units and modules according to needs, that is, the internal structure of the apparatus is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit, and the integrated unit can be realized in the form of hardware or software. In addition, the specific names of each functional unit and module are only for easy distinction, and do not limit the protection scope of the present application. The specific working process of the units and modules in the system can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0111] Figure 7 is a structural schematic diagram of a terminal device provided by the present embodiment. As Figure 7 shown, the terminal device 7 of this embodiment includes at least one processor 70 ( Figure 7The terminal device 7 can include, but not limited to, a processor, a memory, etc. Those skilled in the art can understand that the terminal device 7 can include more or less components, or combine some components, or include different components, for example, the terminal device 7 can also include an input / output device, a network access device, etc.

[0112] The terminal device can be a desktop computer, a notebook computer, a palm computer, a cloud server, and the like. The terminal device can include, but is not limited to, a processor, a memory. Those skilled in the art can understand that the terminal device can include more or less components, or combine some components, or include different components, for example, the terminal device can also include an input / output device, a network access device, etc. Figure 7 The terminal device 7 is only an example and does not constitute a limitation on the terminal device 7, and can include more or less components than those shown, or combine some components, or different components, for example, can also include an input / output device, a network access device, etc.

[0113] The processor 70 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic components, discrete hardware components, etc. The general-purpose processor can be a microprocessor or can also be any conventional processor.

[0114] The memory 71 can be an internal storage unit of the terminal device 7 in some embodiments, for example, a hard disk or a memory of the terminal device 7. The memory 71 can also be an external storage device of the terminal device 7 in other embodiments, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Further, the memory 71 can include both the internal storage unit and the external storage device of the terminal device 7. The memory 71 is used to store an operating system, an application program, a boot loader, data, and other programs, for example, program codes of the computer program, etc. The memory 71 can also be used to temporarily store data that has been output or will be output.

[0115] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the steps in each of the above method embodiments.

[0116] The embodiment of the present application provides a computer program product. When the computer program product is run on a terminal device, the terminal device is caused to execute the steps in each of the above method embodiments.

[0117] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the present application can implement all or part of the processes in the above embodiments by a computer program to instruct related hardware to complete, and the computer program can be stored in a computer readable storage medium. The computer program is executed by a processor to implement the steps in each of the above method embodiments. The computer program includes computer program code, which can be in the form of source code, object code, executable files or some intermediate forms. The computer readable medium at least includes any entity or device capable of carrying the computer program code to the device / terminal equipment, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunications signal and a software distribution medium. For example, a U disk, a mobile hard disk, a magnetic disk or an optical disk, etc. In some jurisdictions, according to legislation and patent practice, the computer readable medium cannot be an electrical carrier signal and a telecommunications signal.

[0118] In the above embodiments, the description of each embodiment has its own focus. The parts not described or recorded in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0119] Those skilled in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0120] In the embodiments provided by the present application, it should be understood that the disclosed apparatus / terminal device and method can be implemented in other manners. For example, the embodiments of the apparatus / terminal device described above are merely schematic, and the division of the modules or units is merely logical function division, and there can be another division manner in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between the units can be indirect couplings or communication connections through some interfaces, devices or units, and can be electrical, mechanical or in other forms.

[0121] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiments.

[0122] The above-described embodiments are merely used to illustrate the technical solutions of the present application, but not limit the present application; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: the technical solutions recorded in the foregoing embodiments can still be modified, or some technical features can be replaced by equivalent replacements; and these modifications or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A data processing method, characterized by, include: Acquire sampling data; wherein each set of sampling data includes first input information and first output information of an initial routing model, the initial routing model being used to select a model from candidate models to respond to the input information based on the input information; the candidate models include at least one generator model connected to the initial routing model; the first output information is used to indicate the first generator model among the candidate models; Based on the first input information, select the second generative model with the highest model performance from the candidate models; If the second generative model is the first generative model, then the sampled data is used as training data; If the second generation model is not the first generation model, then a set of training data is generated based on the second generation model; wherein, the training data is used to train the initial routing model; the training data includes the first input information and the second output information of the initial routing model, and the second output information is used to instruct the second generation model.

2. The data processing method of claim 1, wherein, The step of selecting the second generative model with the highest model performance from the candidate models based on the first input information includes: The model performance is evaluated based on the input and output information of the first generative model to obtain a first score; wherein the input and output information of the first generative model is obtained based on the first input information; the model performance includes a first performance index and a second performance index. If the first score is greater than a preset value, the second generative model is selected according to the order of the first sequence; wherein, the first sequence is the sequence obtained by arranging the candidate models in order from worst to best according to the first performance index; If the first score is less than a preset value, the second generating model is selected according to the order of the second sequence; wherein, the second sequence is a sequence obtained by arranging the candidate models in order of the second performance index from best to worst; the priority of the first performance index is lower than the priority of the second performance index.

3. The data processing method of claim 2, wherein, The step of selecting the second generative model according to the order of the first sequence includes: Starting with the first generative model, the second generative model is selected based on the model performance of the remaining models; Wherein, the remaining model is the model in the first sequence that follows the first generated model; the model performance score of the second generated model is greater than the preset value, and the model performance score of the next model in the first sequence is less than the preset value.

4. The data processing method of claim 2, wherein, The step of selecting the second generative model according to the order of the second sequence includes: Starting with the first model of the second sequence, the second generative model is selected based on model performance; Wherein, the model performance score of the second generated model is greater than the preset value, and the model performance score of the next model of the second generated model in the second sequence is less than the preset value.

5. The data processing method of claim 1, wherein, The method further includes: A training set is generated based on the multiple sets of training data described above; The initial routing model is trained based on the training data representing the first data proportion in the training set to obtain the training model; The training model is evaluated according to test data in the training set, and an evaluation result is obtained; wherein the test data is training data other than the training data with the first data proportion in the training set.

6. The data processing method of claim 5, wherein, After the training model is evaluated according to the test data in the training set, and the evaluation result is obtained, the method further comprises: If the evaluation result indicates that the evaluation passes, the initial routing model is updated according to the training model to obtain a target routing model; If the evaluation result indicates that the evaluation fails, the initial routing model is recorded as the target routing model.

7. The data processing method of claim 5, wherein, The evaluation of the training model according to the test data in the training set to obtain the evaluation result comprises: For each set of test data, input first input information in the test data to a generated model indicated by second output information in the test data to obtain third output information; Calculate the semantic similarity between the reference output information corresponding to the test data and the third output information to obtain a similarity score; Determine the evaluation result according to the similarity score corresponding to each set of test data.

8. The data processing method of claim 2, wherein, The first performance indicator is model cost, and the second performance indicator is model effect.

9. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the method of any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 9. The computer program is executed by the processor to implement the method of any one of claims 1 to 8.

Images