Method and device for determining model performance, equipment, medium and product

Abstract

The invention discloses a method and device for determining model performance, equipment, a medium and a product. The method comprises the steps of obtaining a plurality of first questions output by a first language model by inputting a plurality of first answers corresponding to at least part of user data question and answer history data into the first language model, and obtaining a plurality of question and answer pairs including a plurality of second questions and the corresponding plurality of second answers. The method further includes obtaining a plurality of answers output by a second language model by inputting a plurality of first questions and a plurality of second questions to the second language model, the user being able to use the second language model. The method further includes determining a degree of matching between a plurality of answers output by the second language model and a plurality of predetermined answers, the plurality of predetermined answers including a plurality of first answers and a plurality of second answers. The method further includes determining a performance of the second language model based on the determined degree of matching. Therefore, the evaluated model can be evaluated by using the output of the use case model, and the model performance can be determined in multiple dimensions.

Description

Technical Field

[0001] Embodiments of the present disclosure relate generally to the field of computers, and in particular to methods and apparatuses, devices, media, and products for determining model performance. Background Art

[0002] Language models, especially the large language model (LLM) that has emerged in recent years, have developed into an influential and widely cited technology. One of its core functions is that it greatly simplifies the previously complex and cumbersome process of knowledge acquisition, opening up new paths for rapid retrieval, understanding and application of information.

[0003] With the technical empowerment of language models, users can overcome the previously high knowledge barriers and achieve functions equivalent to those difficult-to-use professional tools, thereby reducing the entry threshold and learning costs. The application of language models not only expands the scope of data circulation but also improves processing efficiency, thereby increasing the possibility of enterprises landing digital transformation and digital drive. Summary of the invention

[0004] Embodiments of the present disclosure provide a solution for determining model performance.

[0005] In a first aspect of the present disclosure, a method for determining model performance is provided, the method comprising obtaining a plurality of first questions output by a first language model by inputting a plurality of first answers corresponding to at least part of user data question and answer history data into the first language model. The method also comprises obtaining a plurality of question and answer pairs, the plurality of question and answer pairs comprising a plurality of second questions and a plurality of corresponding second answers. The method also comprises obtaining a plurality of answers output by a second language model by inputting a plurality of first questions and a plurality of second questions into a second language model, the second language model being usable by a user. The method also comprises determining a degree of match between a plurality of answers output by the second language model and a plurality of predetermined answers, the plurality of predetermined answers comprising a plurality of first answers input into the first language model and a plurality of second answers obtained. The method also comprises determining the performance of the second language model based on the determined degree of match.

[0006] In a second aspect of the present disclosure, there is provided a device for determining model performance, the device comprising a first question acquisition module, configured to obtain a plurality of first questions output by a first language model by inputting a plurality of first answers corresponding to at least part of the user data question and answer history data into the first language model. The device also comprises a second question acquisition module, configured to obtain a plurality of question and answer pairs, the plurality of question and answer pairs comprising a plurality of second questions and a plurality of corresponding second answers. The device also comprises a question input module, configured to obtain a plurality of answers output by a second language model by inputting a plurality of first questions and a plurality of second questions into a second language model, the second language model being usable by a user. The device also comprises a matching degree determination module, configured to determine a matching degree between a plurality of answers output by the second language model and a plurality of predetermined answers, the plurality of predetermined answers comprising a plurality of first answers input into the first language model and a plurality of second answers obtained. The device also comprises a performance determination module, configured to determine the performance of the second language model based on the determined matching degree.

[0007] According to a third aspect of the present disclosure, an electronic device is provided. The computing device includes a processor and a memory, wherein instructions are stored in the memory, and when the instructions are executed by the processor, the processor executes a method or process according to an embodiment of the present disclosure.

[0008] According to a fourth aspect of the present disclosure, a machine-readable storage medium is provided, wherein machine-executable instructions are stored on the machine-readable storage medium, and when the machine-executable instructions are executed by a processor, the processor is enabled to execute a method or process according to an embodiment of the present disclosure.

[0009] In a fifth aspect of the present disclosure, a computer program product is provided. The computer program product is tangibly stored on a non-transitory computer-readable storage medium and includes a computer program, which, when executed by a processor of a computer, causes the processor to perform a method or process according to an embodiment of the present disclosure.

[0010] Please note that the summary is provided to introduce a series of concepts in a simplified form, which will be further described in the detailed description below. The summary is not intended to identify key features or essential features of the disclosure, nor is it intended to limit the scope of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other purposes, features and advantages of the present disclosure will become more clearly understood by describing the embodiments of the present disclosure in more detail in conjunction with the accompanying drawings, in which: Figure 1 is a diagram schematically illustrating an example environment in which methods and / or processes according to embodiments of the present disclosure may be implemented; Figure 2 is a flow chart schematically illustrating a method for determining model performance according to an embodiment of the present disclosure; Figure 3 A diagram schematically illustrates an example input / output flow 300 for a use case model and an assessed model according to an embodiment of the present disclosure; Figure 4 is a diagram schematically illustrating an example process 400 of matching degree determination according to an embodiment of the present disclosure; Figure 5 is a diagram schematically illustrating an example process 500 of model adjustment based on matching degree according to an embodiment of the present disclosure; Figure 6 is a diagram schematically illustrating an apparatus for determining model performance according to an embodiment of the present disclosure; Figure 7 is a schematic block diagram of an example device that may be used to implement embodiments according to the present disclosure.

[0012] Throughout the drawings, same or similar reference numbers generally refer to same or similar elements. DETAILED DESCRIPTION

[0013] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0014] In the description of the embodiments of the present disclosure, the term "including" and its variations should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc. may refer to different or the same objects, unless explicitly indicated to be different.

[0015] As mentioned above, language models greatly simplify the process of knowledge acquisition, not only speeding up the retrieval and retrieval of information, but also promoting the deep understanding and efficient application of information. However, in order to ensure that accurate knowledge is reasonably fed back to users, current language model technology still faces many challenges and needs to be improved.

[0016] When users use language models to acquire knowledge, it is difficult to form an objective evaluation of model performance. Without an accurate grasp of the performance of the language model, it is difficult to implement effective adaptive adjustments or update strategies to adapt to changing scenario requirements or personalized customization for specific users. This dilemma limits the full release of the potential of language models in improving user experience and meeting specific task requirements.

[0017] In related solutions, recording, playback and manual testing are more commonly used. Questions are asked manually using historical questions of users, and the rationality of the questions is analyzed manually and the fit of the answers is calculated. However, problems with manual intervention are often subjective, so it is impossible to ensure the objectivity of the evaluation of the model. Instead, the evaluation results reflect the subjective thinking and judgment of the evaluator more. The problem construction in related solutions is more of a positive deduction, without a result orientation. In addition, the efficiency and coverage of related solutions are relatively low. When constructing problems manually, the problems are not only relatively simple, but also the number is not considerable.

[0018] At least to solve at least some of the above and other potential problems, an embodiment of the present disclosure proposes a scheme for determining model performance. The scheme includes obtaining multiple first questions output by the first language model by inputting multiple first answers corresponding to at least part of the user data question and answer history data into the first language model. The scheme also includes obtaining multiple question and answer pairs, and the multiple question and answer pairs include multiple second questions and corresponding multiple second answers. The scheme also includes obtaining multiple answers output by the second language model by inputting multiple first questions and multiple second questions into the second language model, and the second language model can be used by the user. The scheme also includes determining the degree of match between the multiple answers output by the second language model and multiple predetermined answers, and the multiple predetermined answers include multiple first answers input to the first language model and multiple second answers obtained. The scheme also includes determining the performance of the second language model based on the determined degree of match.

[0019] In this way, the use case model can be efficiently used to quickly obtain a large number of effective use cases for evaluating the model being evaluated, where the problems are user-related in multiple dimensions, so that the user's specific usage scenarios and personalized needs can be more comprehensively integrated in the evaluation of the model being evaluated. In addition, the entire model evaluation process is non-human intervention, and the objectivity of the evaluation process is ensured by measuring the match between the model output and the expected output.

[0020] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0021] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0022] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0023] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0024] Reference below Figures 1 to 7 It should be understood that these exemplary embodiments are provided only to enable those skilled in the art to better understand and implement the embodiments of the present disclosure, and are not intended to limit the scope of the present disclosure in any way.

[0025] Figure 1 1 is a schematic diagram schematically illustrating an example environment 100 in which methods and / or processes according to embodiments of the present disclosure may be implemented. Figure 1 As shown in FIG. 1 , environment 100 includes user 110, local computing nodes 121 and 122, and computing node 123 arranged in the cloud. Figure 1 The implementation environment of the embodiment of the present disclosure is described by taking a mixed arrangement computing system as an example. It should be understood that this is only an exemplary and non-restrictive description, and other different types of computing systems are also feasible, such as distributed or centralized computing systems. The appropriate computing environment configuration can be selected according to actual use needs.

[0026] User 110 may be an individual or an organization (e.g., a business). User 110 may ask questions or inquiries through language description, such as inputting the language description of the question or query via a corresponding interface component in a graphical user interface (GUI). Other input methods may be possible, such as via an audio input device such as a microphone.

[0027] Multiple computing nodes with computing capabilities can process inputs from user 110, and the processed results will be output to user 110 by multiple computing nodes. In an exemplary but non-restrictive manner, computing nodes 121 and 122 can be arranged locally in user 110, while computing node 123 can be arranged in the cloud. The cloud can indicate a service model built based on cloud or distributed technology. In this model, computing resources, storage resources, etc. are coupled together through a network to form a schedulable and scalable resource collection. At least a portion of these resources can be dynamically accessed or allocated to complete tasks without the need to locally own or manage these physical resources.

[0028] A computing node may be a device with computing capabilities. For example, a computing node may be provided with a processor and a memory, etc., or may be equipped with a dedicated accelerator (such as a graphics processing unit (GPU)). In addition, a computing node may store and maintain data. The data and models involved in the method for determining model performance according to an embodiment of the present disclosure (for example, data to be processed and the language model used, etc.) may be centrally arranged on one node, or may be distributedly deployed on some of the multiple nodes.

[0029] like Figure 1 As shown in the example, the local computing nodes 121 and 122 and the computing node 123 at the cloud can be interconnected to communicate with each other through the network. Figure 1 Taking the architecture in as an example, computing nodes 121, 122, and 123 can communicate via a network to achieve, for example, data synchronization or sharing between nodes. Multiple computing nodes in environment 100 can process computing tasks in parallel. In addition, multiple computing nodes in environment 100 can have redundancy and fault tolerance mechanisms to ensure reliable execution of computing tasks. When a computing node fails, the system can automatically migrate the job to other nodes that are working normally to ensure the continuity and availability of the task.

[0030] Examples of computing nodes may include supercomputers, personal computers, laptop computers, vehicle-mounted computing devices, mobile devices (such as smart phones, tablet computers, etc.), wearable electronic devices, multimedia devices, personal digital assistants (PDAs), smart home devices, consumer electronic products, or a combination of any one or more of the above devices, etc. It should be understood that the computing nodes described herein are merely exemplary and non-restrictive, and for example, other different types of computing nodes may also be used.

[0031] It should be understood that for ease of explanation and illustration purposes, Figure 1Only limited components and exemplary connection relationships are shown. It should be understood that this is not intended to limit the scope of the present disclosure, and there may be other environments with different numbers, configurations, etc. Customized arrangements may be selected according to specific usage scenarios and needs, and the same is true in the following.

[0032] Combined with the above Figure 1 Describes an example environment in which the methods and / or processes according to embodiments of the present disclosure may be implemented. Figure 2 The method 200 for determining model performance according to an embodiment of the present disclosure is described. Through the method 200, the output of the use case model can be used to evaluate the model being evaluated, and the model performance can be determined in multiple dimensions.

[0033] Figure 2 2 is a flowchart schematically illustrating a method 200 for determining model performance according to an embodiment of the present disclosure. At 210, a plurality of first answers corresponding to at least part of the user data question and answer history data are input to the first language model to obtain a plurality of first questions output by the first language model. According to an embodiment of the present disclosure, based on the user data question and answer history data associated with the user, the first language model as a use case model obtains questions using answers in a result-oriented manner. Through this process, a large number of effective use cases can be quickly obtained, and the large number of questions obtained have multiple dimensions related to the user.

[0034] At 220, a plurality of question-answer pairs are obtained, the plurality of question-answer pairs including a plurality of second questions and a plurality of corresponding second answers. According to an embodiment of the present disclosure, after obtaining a plurality of result-oriented first questions, a question-answer pair different from that at 210 is introduced. Such a question-answer pair may be a pre-stored user-related knowledge resource, which can provide a new dimension, thereby further ensuring the recall accuracy and user-specificity of the language model.

[0035] At 230, a plurality of first questions and a plurality of second questions are input to the second language model to obtain a plurality of answers output by the second language model, and the second language model can be used by the user. According to an embodiment of the present disclosure, the plurality of first questions at 210 and the plurality of second questions at 220 are input to the second language model as the evaluated model to obtain an output including a plurality of answers.

[0036] At 240, the degree of match between the multiple answers output by the second language model and the multiple predetermined answers is determined, and the multiple predetermined answers include the multiple first answers input to the first language model and the multiple second answers obtained. Here, the predetermined answer indicates an expected answer or an accurate answer. According to an embodiment of the present disclosure, the performance of the evaluated model is determined based on the difference between the multiple first answers input to the use case model and the multiple second answers in the multiple question-answer pairs obtained and the multiple answers output by the evaluated model. Compared with manual subjective evaluation, the use of a matching algorithm will ensure the objectivity of the determination of model performance.

[0037] At 250, based on the determined matching degree, the performance of the second language model is determined. According to an embodiment of the present disclosure, by determining the matching degree between the output of the evaluated model and the input of the use case model, the performance of the evaluated model can be objectively determined. Further, based on the evaluation results, effective adaptive adjustment or update strategies can be implemented in a targeted manner.

[0038] According to the method 200 for determining model performance of the embodiment of the present disclosure, a large number of effective use cases can be quickly obtained by efficiently using the use case model for evaluating the model being evaluated, wherein the problems are of multiple dimensions related to the user, so that the specific usage scenarios and personalized needs of the user can be more comprehensively integrated in the evaluation of the model being evaluated. In addition, the entire model evaluation process is non-human intervention, and the objectivity of the evaluation process is ensured by measuring the matching degree between the model output and the expected output.

[0039] Figure 3 Schematically illustrates an example input / output flow 300 for a use case model and an evaluated model according to an embodiment of the present disclosure. According to an embodiment of the present disclosure, the user data question and answer history data 303 associated with the user may indicate the user's data question and answer history. Figure 3 As shown in , user data question and answer history data 303 may be input into a first language model 310 as a use case model to obtain an output including a plurality of first questions 308 .

[0040] As described above, multiple question-answer pairs are obtained to further promote the recall accuracy and user-specificity of the language model. In some embodiments, such multiple question-answer pairs can be obtained from user help data 306, which includes multiple second questions 312 for which the user seeks help from the human and corresponding multiple second answers obtained from the human. It should be understood that the user help data 306 is exemplary and non-restrictive, and academic research data and the like are also feasible.

[0041] According to an embodiment of the present disclosure, user data question and answer history data 303 and user help document 306 can be converted into a structured document, which can include meta information for user data question and answer history data 303 and user help document 306 to characterize user data question and answer history data 303 and user help document 306. In an exemplary but non-restrictive manner, based on the first meta information for user data question and answer history data 303, question and answer knowledge can be generated, which is associated with the user's business line and is used to create a knowledge base. Business question and answer pairs are generated by extracting business information from the meta information including the first meta information using at least one of the first language model and the second language model. In this way, in the process of constructing a knowledge question and answer framework, the user's business nature will be considered so that the language model can answer in a more business-friendly manner. The second language model 320 as the evaluated model can output multiple answers 314 based on multiple first questions 308 and multiple second questions 312.

[0042] Figure 4 4 is a diagram schematically illustrating an example process 400 of determining a degree of match according to an embodiment of the present disclosure. After obtaining a plurality of answers 314 output by the evaluated model, a degree of match 404 indicating the difference between the plurality of answers 314 and a plurality of predetermined answers 402 is determined. The embodiment of the present disclosure will determine the degree of match between the plurality of answers 314 and the plurality of predetermined answers 402 from multiple levels.

[0043] According to an embodiment of the present disclosure, the first degree of match between the multiple answers 314 and the multiple predetermined answers 402 can be determined by calculating the ratio between the number of identical words that appear in the multiple answers 314 output by the second language model (the model being evaluated) and the multiple predetermined answers 402 in a consistent context order and the expected number (i.e., the number of identical words / the number of expected result words).

[0044] According to an embodiment of the present disclosure, the second degree of matching between the multiple answers 314 and the multiple predetermined answers 402 can be determined by calculating the ratio between the number of identical English words appearing in the multiple answers 314 output by the second language model (the model being evaluated) and the multiple predetermined answers 402 and the expected number, and the upper and lower case letters are ignored when determining the identical English words in the multiple answers 314 and the multiple predetermined answers 402. That is, the number of post-consistent words / the number of expected result words is ignored.

[0045] According to an embodiment of the present disclosure, the third degree of match between the multiple answers 314 and the multiple predetermined answers 402 can be determined by calculating the ratio between the number of synonyms appearing in the multiple answers 314 output by the second language model (the model being evaluated) and the multiple predetermined answers 402 and the expected number (i.e., the number of synonyms / the number of expected result words).

[0046] According to an embodiment of the present disclosure, the fourth degree of matching between the multiple answers 314 and the multiple predetermined answers 402 can be determined by calculating the ratio between the number of identical words appearing in the multiple answers 314 and the multiple predetermined answers 402 output by the second language model (the model being evaluated) and the expected number, and the context order is ignored when determining the identical words in the multiple answers 314 and the multiple predetermined answers 402. That is, the number of post-consistent words / the number of expected result words is ignored.

[0047] According to an embodiment of the present disclosure, based on a first weight corresponding to the first degree of match, a second weight corresponding to the second degree of match, a third weight corresponding to the third degree of match, and a fourth weight corresponding to the fourth degree of match, a matching degree 404 can be obtained by taking a weighted average of the first degree of match, the second degree of match, the third degree of match, and the fourth degree of match, wherein the first weight is greater than the second weight, the second weight is greater than the third weight, and the third weight is greater than the fourth weight.

[0048] By way of example and non-limiting example, the first weight may be 100%, the second weight may be 90%, the third weight may be 80%, and the fourth weight may be 30%. This is for example only and is not intended to limit the scope of the embodiments of the present disclosure. Through the above-mentioned weighted average matching degree calculation scheme, the results calculated by different matching algorithms are weighted and calculated to obtain the final matching degree.

[0049] As in the above example, in the matching weight algorithm, the word order has the lowest weight. In some embodiments, for high-frequency word order problems, and confirming that the word meanings are consistent, the model results can be judged by fixed values ​​to indicate the model. For example, when ABC appears, each data returns CAB to reduce invalid noise.

[0050] Figure 5 Schematically illustrates an example process 500 of model adjustment based on matching degree according to an embodiment of the present disclosure. After obtaining the matching degree 404 between the multiple answers 314 output by the second language model (the evaluated model) and the multiple predetermined answers 402, a corresponding adjustment or update strategy can be applied to the second language model to improve the model performance.

[0051] According to an embodiment of the present disclosure, in response to determining that the matching degree 404 is lower than the first threshold, the second language model may be trained using the plurality of first questions output by the second language model and the corresponding plurality of first answers. Figure 5As shown in , when the matching degree 404 falls into the first threshold interval 502 below the first threshold, a large number of use cases obtained from the use case model are fed into the evaluated model for training (at 508). In this way, the ineffective training time can be reduced, and the model can be made to recognize the correct input and output with greater strength.

[0052] According to an embodiment of the present disclosure, in response to determining that the matching degree 404 is higher than the first threshold and lower than the second threshold, synonym training and case-ignoring training may be performed on the second language model. Figure 5 As shown in , when the matching degree 404 falls into the second threshold interval 504 which is higher than the first threshold and lower than the second threshold, the evaluated model is connected to the synonym word library, trained with certain synonym cases, and case-ignoring training is performed (510). After each training, the question is thrown into the evaluated model to observe the growth of the matching degree.

[0053] According to an embodiment of the present disclosure, in response to determining that the matching degree 404 is higher than the second threshold, during the training of the second language model, the number of question-answer pairs from the user data question-answer history data may be increased. Figure 5 As shown in , when the matching degree 404 falls into the third threshold interval 506 which is higher than the second threshold, the number of questions from the user data question and answer history data is increased so that the model understands the original intention of the user data question and answer, or describes the user data question and answer more clearly, thereby improving the model's understanding of the user data question and answer itself, thereby improving the model recall (at 512).

[0054] According to the embodiments of the present disclosure, the use case input itself is constantly iterating to ensure that the output evaluation questions continue to maintain accuracy, scientificity, and business nature. In some embodiments, in response to the first language model or the second language model failing to determine the meaning of the predetermined word, the predetermined word can be described in the prompt word. For example, ABC is a special function of xxx, which represents the meaning of xxx, thereby improving the business understanding ability of the model.

[0055] Figure 6 Schematically illustrates an apparatus 600 for determining model performance according to an embodiment of the present disclosure, and the apparatus 600 may include multiple units or modules for performing the steps or actions in the method or process discussed above. Figure 6As shown in , the device 600 includes: a first question acquisition module, configured to obtain a plurality of first questions output by the first language model by inputting a plurality of first answers corresponding to at least part of the user data question and answer history data into the first language model; a second question acquisition module, configured to obtain a plurality of question and answer pairs, the plurality of question and answer pairs including a plurality of second questions and a corresponding plurality of second answers; a question input module, configured to obtain a plurality of answers output by the second language model by inputting the plurality of first questions and the plurality of second questions into the second language model, the second language model being usable by the user; a matching degree determination module, configured to determine the matching degree between the plurality of answers output by the second language model and a plurality of predetermined answers, the plurality of predetermined answers including the plurality of first answers input to the first language model and the plurality of second answers obtained; and a performance determination module, configured to determine the performance of the second language model based on the determined matching degree.

[0056] In some embodiments, the plurality of question-answer pairs are obtained from user help data, and the user help data includes the plurality of second questions for which the user seeks help from a human and the plurality of second answers obtained from the human.

[0057] In some embodiments, the device 600 further includes a conversion module configured to convert the user data question and answer history data and the user help data into a structured document, wherein the structured document includes meta-information for the user data question and answer history data and the user help data.

[0058] In some embodiments, wherein the user data question and answer history data is associated with the user's business line, the device 600 further includes: a question and answer knowledge module, configured to generate question and answer knowledge based on first meta-information for the user data question and answer history data, wherein the question and answer knowledge is associated with the user's business line and is used to create a knowledge base; and a business knowledge module, configured to generate a business question and answer pair by extracting business information from the meta-information including the first meta-information using at least one of the first language model and the second language model.

[0059] In some embodiments, the device 600 further includes a calculation module configured to: determine a first degree of match between the multiple answers and the multiple predetermined answers by calculating a ratio between the number of identical words in the multiple answers output by the second language model and the multiple predetermined answers and an expected number of identical words in the context order; determine a second degree of match between the multiple answers and the multiple predetermined answers by calculating a ratio between the number of identical English words in the multiple answers output by the second language model and the multiple predetermined answers and an expected number of identical words, ignoring the case of letters when determining the identical English words in the multiple answers and the multiple predetermined answers; determine a third degree of match between the multiple answers and the multiple predetermined answers by calculating a ratio between the number of synonyms in the multiple answers output by the second language model and the multiple predetermined answers and an expected number of identical words; determine a fourth degree of match between the multiple answers and the multiple predetermined answers by calculating a ratio between the number of identical words in the multiple answers output by the second language model and the multiple predetermined answers and an expected number of identical words, ignoring the context order when determining the identical words in the multiple answers and the multiple predetermined answers.

[0060] In some embodiments, the matching degree determination module 640 is further configured to obtain the matching degree by taking a weighted average of the first matching degree, the second matching degree, the third matching degree and the fourth matching degree based on a first weight corresponding to the first matching degree, a second weight corresponding to the second matching degree, a third weight corresponding to the third matching degree, and a fourth weight corresponding to the fourth matching degree, wherein the first weight is greater than the second weight, the second weight is greater than the third weight, and the third weight is greater than the fourth weight.

[0061] In some embodiments, the device 600 also includes a training module, which is configured to: in response to determining that the match degree is lower than a first threshold, train the second language model using the multiple first questions and the corresponding multiple first answers output by the first language model; in response to determining that the match degree is higher than the first threshold and lower than a second threshold, perform synonym training and case-ignoring training on the second language model; and in response to determining that the match degree is higher than the second threshold, increase the number of question and answer pairs from the user data question and answer history data during training for the second language model.

[0062] In some embodiments, the apparatus 600 further comprises a description module configured to: in response to the first language model or the second language model failing to determine the meaning of the predetermined word, describe the predetermined word in the prompt word.

[0063] Figure 7 FIG. 1 is a block diagram of an electronic device 700 according to some embodiments of the present disclosure. The device 700 may be a device or apparatus described in an embodiment of the present disclosure. Figure 7 As shown, the device 700 includes a central processing unit (CPU) and / or a graphics processing unit (GPU) 701, which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 702 or loaded from a storage unit 708 to a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The CPU / GPU 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704. Although not shown in FIG. Figure 7 As shown in FIG. 7 , device 700 may further include a coprocessor.

[0064] A number of components in the device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0065] The various methods or processes described above may be performed by the CPU / GPU 701. For example, in some embodiments, the methods may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the CPU / GPU 701, one or more steps or actions in the methods or processes described above may be performed.

[0066] In some embodiments, the methods and processes described above may be implemented as a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present disclosure.

[0067] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. The computer readable storage medium used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (for example, a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0068] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0069] The computer program instructions for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, and conventional procedural programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as a separate software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby implementing various aspects of the present disclosure.

[0070] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0071] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0072] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the equipment, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each frame in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the frame can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous frames can actually be executed substantially in parallel, and they can also be executed in the opposite order sometimes, depending on the functions involved. It should also be noted that each frame in the block diagram and / or flow chart, and the combination of frames in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0073] Various embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the various embodiments, practical applications, or technical improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the various embodiments disclosed herein.

Claims

1. A method for determining model performance, comprising: Acquire a plurality of first questions output by the first language model by inputting a plurality of first answers corresponding to at least a portion of user data question and answer history data into the first language model, wherein the user data question and answer history data indicates a data question and answer interaction history of the user; Acquire a plurality of question-answer pairs, wherein the plurality of question-answer pairs include a plurality of second questions and a plurality of corresponding second answers; obtaining a plurality of answers output by the second language model by inputting the plurality of first questions and the plurality of second questions into a second language model, wherein the second language model can be used by the user; Determining a degree of match between the plurality of answers output by the second language model and a plurality of predetermined answers, the plurality of predetermined answers including the plurality of first answers input to the first language model and the plurality of second answers obtained; as well as Based on the determined degree of match, performance of the second language model is determined.

2. The method according to claim 1, wherein: The multiple question-answer pairs are obtained from user help data, and the user help data includes the multiple second questions for which the user seeks help from a human and the multiple second answers obtained from the human.

3. The method according to claim 2, further comprising: Convert the user data question and answer history data and the user help data into a structured document, The structured document includes meta information for the user data question and answer history data and the user help data.

4. The method according to claim 3, wherein the user data question and answer history data is associated with the user's business line, the method further comprising: generating question-and-answer knowledge based on first meta information of question-and-answer history data of the user data, wherein the question-and-answer knowledge is associated with the business line of the user and is used to create a knowledge base; as well as A business question-answer pair is generated by extracting business information from the meta information including the first meta information using at least one of the first language model and the second language model.

5. The method according to claim 1, further comprising: Determine a first degree of match between the plurality of answers and the plurality of predetermined answers by calculating a ratio between the number of identical words in the plurality of answers output by the second language model and the plurality of predetermined answers and an expected number of identical words in the same context order; Determine a second degree of match between the plurality of answers and the plurality of predetermined answers by calculating a ratio between the number of identical English words that appear in the plurality of answers output by the second language model and the plurality of predetermined answers and an expected number, ignoring the case of letters when determining the identical English words in the plurality of answers and the plurality of predetermined answers; determining a third degree of matching between the plurality of answers and the plurality of predetermined answers by calculating a ratio between the number of synonyms appearing in the plurality of answers output by the second language model and the plurality of predetermined answers and an expected number; A fourth degree of match between the multiple answers and the multiple predetermined answers is determined by calculating a ratio between the number of identical words appearing in the multiple answers output by the second language model and the multiple predetermined answers and an expected number, and the context order is ignored when determining the identical words in the multiple answers and the multiple predetermined answers.

6. The method according to claim 5, wherein determining the degree of matching further comprises: The matching degree is obtained by weighted averaging the first matching degree, the second matching degree, the third matching degree, and the fourth matching degree based on a first weight corresponding to the first matching degree, a second weight corresponding to the second matching degree, a third weight corresponding to the third matching degree, and a fourth weight corresponding to the fourth matching degree, The first weight is greater than the second weight, the second weight is greater than the third weight, and the third weight is greater than the fourth weight.

7. The method according to claim 1, further comprising: In response to determining that the degree of match is lower than a first threshold, training the second language model using the plurality of first questions output by the first language model and the corresponding plurality of first answers; In response to determining that the degree of match is higher than the first threshold and lower than a second threshold, performing synonym training and case-ignoring training on the second language model; as well as In response to determining that the degree of match is higher than the second threshold, during training for the second language model, the number of question-answer pairs from the user data question-answer history data is increased.

8. The method according to claim 1, further comprising: In response to the first language model or the second language model failing to determine the meaning of a predetermined word, the predetermined word is described in a prompt word.

9. An apparatus for determining model performance, comprising: A first question acquisition module is configured to acquire a plurality of first questions output by the first language model by inputting a plurality of first answers corresponding to at least part of the user data question and answer history data into the first language model, wherein the user data question and answer history data indicates the user's data question and answer interaction history; A second question acquisition module is configured to acquire a plurality of question-answer pairs, wherein the plurality of question-answer pairs include a plurality of second questions and a plurality of corresponding second answers; a question input module configured to obtain a plurality of answers output by a second language model by inputting the plurality of first questions and the plurality of second questions into the second language model, wherein the second language model can be used by the user; a matching degree determination module configured to determine a matching degree between the plurality of answers output by the second language model and a plurality of predetermined answers, the plurality of predetermined answers including the plurality of first answers input to the first language model and the plurality of second answers obtained; as well as The performance determination module is configured to determine the performance of the second language model based on the determined matching degree.

10. An electronic device comprising: processor; as well as A memory coupled to the processor, wherein instructions are stored in the memory, and when the instructions are executed by the processor, the processor is caused to perform the method according to any one of claims 1 to 8.

11. A machine-readable storage medium having machine-executable instructions stored thereon, wherein the machine-executable instructions, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 8.

12. A computer program product comprising a computer program which, when executed by a processor of a computer, causes the processor to perform the method according to any one of claims 1 to 8.

Images