Data processing method and related apparatus

Abstract

The application provides a data processing method and related device; the method comprises: determining a first index value based on a first text and a second text; the first text is a conversation text of a first object and a second object generated by a first model, and the second text is a conversation text of a third object and the second object; the first index value is an index value for representing the conversation quality of the first object and the second object; determining a second index value based on the conversation response duration of the first object and the second object; the second index value is an index value for representing the conversation response speed of the first object; determining an evaluation score for the first model based on the first index value and the second index value; determining a text generation model adapted to the first object based on the evaluation score. Through the application, the text generation effect of the text generation model corresponding to the first object can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and particularly relates to a data processing method and related device. BACKGROUND

[0002] For a classification task or a regression task in evaluating the text generation effect of a large language model, a loss function is generally used in the related art to measure the difference between the predicted value of the large language model and the true value, and the difference is taken as a key indicator for optimizing the model parameters. The smaller the value of the loss function, the closer the predicted value of the large language model is to the true value, and the better the performance of the large language model, that is, the better the text generation effect. In the loss function in the related art, mean squared error (MSE), cross-entropy loss, mean absolute error (MAE), or log-likelihood loss, etc. are all based on the true value and the predicted value as the benchmark, and by means of different formulas, the difference between the two is reflected to some extent, and then the performance optimization of the large language model is guided. SUMMARY

[0003] The embodiments of the present application provide a data processing method and related device, which can improve the text generation effect of a data processing model corresponding to a first object.

[0004] The technical solution of the embodiments of the present application is as follows:

[0005] The embodiments of the present application provide a data processing method, which comprises: determining a first indicator value based on a first text and a second text; the first text is a conversation text of a first object and a second object generated by a first model, and the second text is a conversation text of a third object and the second object; the first indicator value is an indicator value for representing the conversation quality of the first object and the second object; determining a second indicator value based on the conversation response duration of the first object and the second object; the second indicator value is an indicator value for representing the conversation response speed of the first object; determining an evaluation score for the first model based on the first indicator value and the second indicator value; and determining a text generation model adapted to the first object based on the evaluation score.

[0006] The embodiment of the present application provides a data processing apparatus, comprising: a first determining module, configured to determine a first index value based on a first text and a second text; the first text is a conversation text of a first object and a second object generated by a first model, and the second text is a conversation text of a third object and the second object; the first index value is an index value used for representing conversation quality of the first object and the second object; a second determining module, configured to determine a second index value based on a conversation response duration of the first object and the second object; the second index value is an index value used for representing conversation response speed of the first object; a third determining module, configured to determine an evaluation score of the first model based on the first index value and the second index value; and a fourth determining module, configured to determine a text generation model adapted to the first object based on the evaluation score.

[0007] The embodiment of the present application provides an electronic device, comprising: a memory, configured to store computer executable instructions; and a processor, configured to execute the computer executable instructions stored in the memory, so as to implement the data processing method provided by the embodiment of the present application.

[0008] The embodiment of the present application provides a computer readable storage medium, which stores computer executable instructions, and is configured to implement the data processing method provided by the embodiment of the present application when executed by a processor.

[0009] The embodiment of the present application provides a computer program product, which comprises executable instructions stored in a computer readable storage medium; wherein a processor of an electronic device reads the executable instructions from the computer readable storage medium and executes the executable instructions, so as to implement the data processing method provided by the embodiment of the present application.

[0010] The embodiment of the present application has the following beneficial effects:

[0011] In the data processing process, first, based on the conversation text of the first object and the second object generated by the first model and the conversation text of the third object and the second object, a first index value for representing the conversation quality of the first object and the second object is determined; then, based on the conversation response time length of the first object and the second object, a second index value for representing the conversation response speed of the first object is determined; then, the first index value and the second index value are integrated to determine an evaluation score for the first model; finally, based on the evaluation score, a text generation model adapted to the first object is determined. In this way, through the first index value, the conversation quality of the first object and the second object can be evaluated, that is, from the text content level, the conversation quality when the first object and the second object have a conversation generated by the first model can be evaluated; at the same time, through the second index value, the conversation response speed of the first object can be evaluated, that is, from the conversation response efficiency level, the instant feedback ability when the first object and the second object have a conversation can be evaluated; by integrating the two index values to obtain the evaluation score, the comprehensive performance of the first object in the conversation quality and the conversation response efficiency can be comprehensively and accurately reflected, so that the text generation model adapted to the first object is determined based on the evaluation score, which can make the determined text generation model meet the conversation content demand of the first object and match the expectation of the response time efficiency of the first object, thereby effectively improving the text generation effect of the text generation model and ensuring that the text generation model can better meet the dual demands of conversation quality and conversation response efficiency in actual scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0012] Figure 1 is a structural schematic diagram of a data processing system provided by an embodiment of the present application;

[0013] Figure 2 is an optional flowchart of a data processing method provided by an embodiment of the present application;

[0014] Figure 3 is another optional flowchart of a data processing method provided by an embodiment of the present application;

[0015] Figure 4 is an optional implementation schematic diagram of determining a third index value provided by an embodiment of the present application;

[0016] Figure 5 is another optional implementation schematic diagram of determining a third index value provided by an embodiment of the present application;

[0017] Figure 6 is an optional implementation schematic diagram of determining a fourth index value provided by an embodiment of the present application;

[0018] Figure 7 is another optional implementation schematic diagram of determining a fourth index value provided by an embodiment of the present application;

[0019] Figure 8 FIG. 1 is an implementation schematic diagram of determining a fifth index value provided by an embodiment of the present application;

[0020] Figure 9 FIG. 2 is a flow schematic diagram of a method for evaluating text generation effect of a large language model provided by an embodiment of the present application;

[0021] Figure 10 FIG. 3 is a structural schematic diagram of a data processing apparatus provided by an embodiment of the present application;

[0022] Figure 11 FIG. 4 is a component structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings, and the described embodiments should not be regarded as limiting the present application. All other embodiments obtained by a person of ordinary skill in the art without making any creative labor fall within the scope of protection of the present application.

[0024] In the following description, “some embodiments” are described, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0025] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as commonly understood by a person skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0026] The relevant data collection and processing in the embodiments of the present application should strictly comply with the requirements of relevant national laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of authorization of laws and regulations and the personal information subject.

[0027] The core task of a large language model is text generation, but it is unreasonable to accurately measure the pros and cons of the text generation result of the large language model from a single dimension. Therefore, the single loss function in the related art cannot be directly used for the large language model and obtain a good text generation effect.

[0028] Based on at least one problem existing in the related art, embodiments of the present application respectively evaluate the text generation effect of the large language model through four dimensions, i.e., evaluating the service quality of the intelligent agent, to obtain four evaluation scores. Finally, the four evaluation scores are integrated to obtain a total evaluation value (i.e., an evaluation score) for the intelligent agent, and the total evaluation value is used to determine whether the text generation effect of the large language model or the service quality of the intelligent agent needs to be optimized. By optimizing the performance of the large language model, the text generation effect of the large language model is improved.

[0029] Referring to Figure 1 , Figure 1 is a structural schematic diagram of a data processing system 100 architecture provided by embodiments of the present application, for implementing a data processing application, a terminal 400 connects a server 200 through a network 300, the network 300 can be a wide area network or a local area network, or a combination of the two.

[0030] The terminal 400 is configured to send a data processing request to the server 200, and the server 200 is configured to determine a first index value based on the first text and the second text in response to the data processing request; then determine a second index value based on the conversation response duration of the first object and the second object; then determine an evaluation score for the first model based on the first index value and the second index value; and finally determine a text generation model adapted to the first object based on the evaluation score. After obtaining the text generation model adapted to the first object, the server 200 can also send the model parameters of the text generation model adapted to the first object to the terminal 400, so as to output the model parameters of the text generation model adapted to the first object at the terminal 400, or perform subsequent text generation operations based on the model parameters of the text generation model adapted to the first object.

[0031] In some embodiments, the server 200 can be a standalone physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and basic cloud computing services such as big data and artificial intelligence platforms. The terminal 400 can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, or a vehicle-mounted terminal, but is not limited thereto. The terminal 400 and the server 200 can be directly or indirectly connected through wired or wireless communication, which is not limited in embodiments of the present application.

[0032] The data processing method provided by each embodiment of the present application can be executed by an electronic device, which can be a server or a terminal, that is, the data processing method of each embodiment of the present application can be executed by a server, a terminal, or interaction between the server and the terminal.

[0033] Referring to Figure 2 , Figure 2 is an optional flowchart of the data processing method provided by the embodiments of the present application, which will be described in combination with the steps shown in the figure, taken as an example of the execution subject of the data processing method as a server, the method comprises the following steps S101 to S104: Figure 2

[0034] Step S101, determining a first index value based on a first text and a second text.

[0035] In the embodiments of the present application, the first object refers to one of the subjects participating in the conversation, and the conversation performance (such as quality and response speed, etc.) of the first object is the core object of evaluation and optimization. The first object can be an intelligent system, an AI model or an automated tool that needs to realize interactive functions through a data processing model. For example, the first object can be an intelligent customer service model (such as an automatic reply customer service of an e-commerce platform), an intelligent question and answer robot (such as a mobile phone voice assistant) or a tutoring AI in online education, etc. The second object refers to a common subject that has conversation interaction with the first object and the third object, and is the initiator or demander of the conversation. The demand of the second object is the core goal of the conversation. For example, the second object can be a user who consults commodity problems to an intelligent customer service, a mobile terminal user who asks questions to a voice assistant or a student who asks for help to a tutoring AI, etc. The third object refers to a subject that has conversation interaction with the second object, and its conversation performance can be used as a reference or benchmark for evaluating the conversation quality of the first object. The third object can be a professional or standard process artificial or system. For example, the third object can be an artificial customer service of an e-commerce platform, a teacher in the education field, a doctor in the medical field, etc.

[0036] ​The conversation text refers to the text record generated by the first object and the second object, or the third object and the second object in the conversation process. The conversation text includes the questions, replies, instructions and feedback of both parties, and is the core basis for evaluating the conversation quality. The first text is the conversation text of the first object and the second object generated by the first model, that is, the first text is the text record generated by the first object and the second object in the interaction process, which is directly used as the material for evaluating the conversation quality of the first object. The second text is the conversation text of the third object and the second object, that is, the second text is the text record generated by the third object and the second object in the interaction process, which is used as the reference material for evaluating the conversation quality of the first object, such as comparing the differences in the replies of the first object and the third object to the same question. For example, the user transfers to a human when the intelligent customer service cannot solve the problem, the patient further consults the doctor after communicating with the intelligent inquiry assistant, or the student supplements the question to the teacher after the tutoring AI answers. The first model refers to the algorithm model supporting the first object to realize the conversation interaction function, which is constructed based on natural language processing (NLP) or machine learning technology. The first model is used to understand the intention of the second object, generate reply content and control the response rhythm. The conversation text of the first object and the second object can be generated through the first model. For example, the first model can be a dialogue generation model behind the intelligent customer service or a question and answer matching model of the tutoring AI. The model structure of the first model is not limited here.

[0037] The first index value is an index value used to represent the conversation quality of the first object and the second object. The first index value can be determined by analyzing the content matching degree (for example, the consistency of the answers to the same question) of the first text and the second text, and the content validity (for example, the semantic integrity and text fluency) of the first text, and the like. The conversation quality of the first object and the second object is determined by the first index value.

[0038] In some embodiments, after obtaining the conversation text of the first object and the second object and the conversation text of the third object and the second object, the conversation quality of the first object and the second object can be evaluated through multiple dimensions to obtain evaluation scores in multiple dimensions respectively. Finally, the evaluation scores in multiple dimensions are integrated to obtain the first index value used to represent the conversation quality of the first object and the second object.

[0039] In step S102, the second index value is determined based on the conversation response duration of the first object and the second object.

[0040] In the embodiments of the present application, the session response duration refers to the time interval for the first object to generate and return a reply after receiving the message (such as a question or an instruction) of the second object, which is the core data for measuring the response speed of the first object. For example, the user sends a message to the intelligent customer service to inquire about the logistics of the order, and the intelligent customer service replies with the logistics information after 1.2 seconds, which is the session response duration; if the response durations of multiple interactions are 1.2 seconds, 0.8 seconds and 1.5 seconds respectively, then the average response duration is 1.17 seconds. The second index value is an index value for representing the session response speed of the first object, which can be determined based on the statistical result of the session response duration (for example, the average response duration, the longest response duration or the response duration standard rate, etc.). After determining the session response duration of the first object and the second object, the second index value for representing the session response speed of the first object can be determined according to the session response duration.

[0041] In step S103, the evaluation score for the first model is determined based on the first index value and the second index value.

[0042] In the embodiments of the present application, the evaluation score refers to the score obtained by comprehensively considering the first index value (session quality) and the second index value (session response speed), which is used to comprehensively evaluate the overall performance of the first model in generating the session text of the first object and the second object, that is, to evaluate the effect in the session quality level and the efficiency in the response speed level. The larger the evaluation score is, the better the comprehensive performance of the first model in the above multiple dimensions is. The evaluation score for the first model can be determined based on the first index value and the second index value by weighted calculation or other fusion algorithm, etc. For example, the first index value of the intelligent customer service is 80 points, the second index value is 90 points, and if the weight of each is 50%, then the evaluation score is (80x50%+90x50%) = 85 points.

[0043] In step S104, the text generation model adapted to the first object is determined based on the evaluation score.

[0044] In the embodiments of the present application, the text generation model adapted to the first object is a model that can meet the requirements of the first object for the session quality and the session speed in the actual session scenario. The text generation model can exhibit more suitable performance in the dialogue generation task related to the first object.

[0045] In some embodiments, according to whether the first model corresponding to the first object is an open-source large language model, two cases can be divided: in the first case, when the first model corresponding to the first object is an open-source large language model, after obtaining the evaluation score, if the evaluation score is less than the preset evaluation threshold, it indicates that the text generation effect of the first model corresponding to the first object is poor, and the first model can be fine-tuned according to the evaluation score, for example, adjusting the model parameters or model structure of the first model, and then continuing to determine the evaluation score through the fine-tuned first model, until the re-determined evaluation score is greater than the preset evaluation threshold, that is, the fine-tuned first model corresponding to the re-determined evaluation score can be determined as the text generation model adapted to the first object. For example, after fine-tuning the intelligent customer service model, the first index value is improved from 80% to 92%, the second index value is shortened from 1.5 seconds to 0.8 seconds, the evaluation score is improved from 85 points to 96 points, and the user complaint rate is reduced by 30%.

[0046] In the second case, when the first model corresponding to the first object is a non-open-source large language model, the text generation effects of multiple first models can be evaluated based on the first text and the second text, and the conversation response time of the first object and the second object, to obtain an evaluation score corresponding to each first model. Then, the evaluation scores are sorted, the first model corresponding to the maximum evaluation score is selected, and the first model corresponding to the maximum evaluation score is determined as the text generation model adapted to the first object. For example, the first object is a medical AI inquiry assistant, mainly used for communicating with patients about symptoms and answering basic health questions, and multiple candidate first models are multiple medical special-purpose dialogue models. The evaluation score of one of the candidate first models is only 75 points, and there are problems of inaccurate medical terminology and delayed response in the dialogue. After comparing multiple candidate first models, it is determined that a certain medical special-purpose dialogue model has an evaluation score of 92 points in this scenario, not only the dialogue quality is high and conforms to the medical specification, but also the response speed is fast, so the medical special-purpose dialogue model is the text generation model adapted to the medical AI inquiry assistant.

[0047] The data processing method provided in the embodiments of the present application comprises the following steps: in the data processing process, first, a first index value for representing the conversation quality of the first object and the second object is determined based on the conversation text of the first object and the second object generated by the first model and the conversation text of the third object and the second object; then, a second index value for representing the conversation response speed of the first object is determined based on the conversation response duration of the first object and the second object; then, an evaluation score for the first model is determined by comprehensively determining the first index value and the second index value; finally, a text generation model adapted to the first object is determined based on the evaluation score. In this way, the conversation quality of the first object and the second object can be evaluated through the first index value, that is, the conversation quality when the first object and the second object have a conversation generated by the first model can be evaluated from the text content level; at the same time, the conversation response speed of the first object can be evaluated through the second index value, that is, the instant feedback capability when the first object and the second object have a conversation can be evaluated from the conversation response efficiency level; the comprehensive performance of the first object in the conversation quality and the conversation response efficiency can be comprehensively and accurately reflected by comprehensively determining the evaluation score of the two index values, so that the text generation model adapted to the first object can be determined based on the evaluation score, which can make the determined text generation model meet the conversation content requirements of the first object and match the expectations of the first object on the response time efficiency, thereby effectively improving the text generation effect of the text generation model and ensuring that the text generation model can better meet the dual requirements of the conversation quality and the conversation response efficiency in the actual scene.

[0048] The data processing method in the embodiments of the present application will be described below in combination with the interaction process between the server and the terminal in the data processing system. It should be noted that the data processing method is a data processing method realized by the interaction between the server and the terminal, which is substantially the same as the data processing method executed by the server in the above embodiments, and the difference is that the actions performed by the terminal in the execution process of the data processing method are described in the embodiments of the present application, and some steps can be executed by the terminal or the server. Therefore, for the steps in the embodiments of the present application which are the same as those in the above embodiments but have different execution subjects, the embodiments of the present application are only illustratively described, and in the implementation process, any one execution subject can be executed, which is not limited in the embodiments of the present application.

[0049] Referring to Figure 3 , Figure 3 is another optional flowchart of the data processing method provided in the embodiments of the present application, as shown in Figure 3 , the method comprises the following steps S201 to S213:

[0050] In step S201, the terminal receives the data processing operation input by the user.

[0051] In the embodiments of the present application, the data processing operation can be an operation input by a user through a client of the data processing system. The user can input the data processing operation on the client of the data processing system. In the data processing system, a data processing function page can be provided, and the user can input the data processing operation on the data processing function page. In some embodiments, the first text and the second text can be input at the same time when the data processing operation is input. The first text is the conversation text of the first object and the second object, and the second text is the conversation text of the third object and the second object.

[0052] In step S202, the terminal generates a data processing request in response to the data processing operation.

[0053] In the embodiments of the present application, the data input by the user can be encapsulated into the data processing request. For example, the first text and the second text input by the user can be encapsulated into the data processing request.

[0054] In step S203, the terminal sends the data processing request to the server.

[0055] In step S204, the server determines the third index value based on the first text and the second text in response to the data processing request.

[0056] In the embodiments of the present application, when the data processing request encapsulates the first text and the second text, the data processing request is parsed to directly obtain the first text and the second text. After obtaining the first text and the second text, the third index value is determined based on the first text and the second text. The third index value is an index value for representing the identity of the first object.

[0057] In some embodiments, referring to Figure 4 , Figure 4 It is shown that in step S204, the server determines the third index value based on the first text and the second text, which can also be implemented through the following steps S2041 to S2042:

[0058] In step S2041, a first text vector of the first conversation text in the first text and a second text vector of the second text are determined.

[0059] In the embodiments of the present application, the first conversation text is the conversation text of the first object in the first text.

[0060] In some embodiments, the server determines a first text vector of the first conversation text in the first text, and a second text vector of the second text, which can be achieved by: first, performing word segmentation processing on the first conversation text to obtain a first word sequence of each sentence in the first conversation text; then, replacing a third word in the first word sequence with a preset placeholder to obtain a second word sequence; and finally, performing vectorization processing on the words in the second word sequence to obtain the first text vector; and performing vectorization processing on the second text to obtain the second text vector.

[0061] In the embodiments of the present application, the word segmentation processing refers to the process of splitting continuous text into independent words according to certain rules. The first word sequence refers to an ordered set of words formed by each sentence in the first conversation text after word segmentation processing, which retains the word order of the original sentence. The preset placeholder is a symbol or character that is pre-set and used to replace a specific word. The placeholder itself has no actual semantic meaning and is only used as a placeholder marker. The third word refers to a word in the first word sequence that represents the identity of the first object and the second object, which is also the target word that needs to be replaced. The second word sequence refers to a new word sequence formed after replacing the third word in the first word sequence with the preset placeholder, which eliminates the interference of the identity word on the semantic analysis of the text. The vectorization processing refers to the process of converting words or text into numerical vectors, which can quantify the semantic information of words into calculable vectors through a pre-set model (such as Word2Vec or BERT, etc.). The first text vector refers to a vector representing the core semantics of the first conversation text obtained by integrating all word vectors after vectorization processing of the second word sequence. The second text vector refers to a vector obtained by vectorization processing of the second text, which is used for comparative analysis with the first text vector.

[0062] In some embodiments, the first conversation text is split by sentence, and word segmentation tools such as Jieba can be used to perform word segmentation processing sentence by sentence. The third word in the first word sequence that represents the identity of the first object and the second object is identified, and the third word is replaced one by one with a preset placeholder (such as [MASK]) to obtain a second word sequence. A pre-trained word vector model is used to perform vectorization processing on each word in the second word sequence to obtain a vector for each word. All word vectors are integrated to obtain a first text vector of the first conversation text. The same word vector model and integration method as the first conversation text can also be used to perform vectorization processing on the word sequence in the second text to obtain a second text vector of the second text.

[0063] Here, by means of word segmentation processing, replacing the third word representing the identity with a space symbol, obtaining a second word sequence, and then performing vectorization processing, the interference of the identity word on the text vector can be excluded, and the core semantic content of the conversation text can be focused on. The first text vector and the second text vector obtained in this way can better reflect the substantive semantics of the text, reduce the influence of irrelevant factors on the evaluation, and make the third index value determined based on the vector more in line with the real semantic association between the first object conversation text and the second text, thereby improving the accuracy of the evaluation.

[0064] In step S2042, the third index value is determined based on the first text vector and the second text vector.

[0065] In some embodiments, referring to Figure 5 , Figure 5 It is shown that in step S2042, the server determines the third index value based on the first text vector and the second text vector, which can also be implemented by the following steps S501 to S503:

[0066] In step S501, a first sentence vector of each sentence in the first conversation text is determined from the first text vector.

[0067] In the embodiments of the present application, the first sentence vector is extracted from the first text vector, which is a vector for representing the core semantics of each sentence in the first conversation text, and can accurately reflect the semantic information of a single sentence in the first conversation text. The first sentence vector of each sentence can be obtained by dividing the first text vector according to the sentences of the first conversation text and integrating the word vectors corresponding to each sentence.

[0068] In step S502, a second sentence vector of each sentence in the second conversation text and a third sentence vector of each sentence in the third conversation text are determined from the second text vector.

[0069] In the embodiments of the present application, the second sentence vector is extracted from the second text vector, which is a vector for representing the core semantics of each sentence in the second conversation text, and can accurately reflect the semantic information of a single sentence in the second conversation text. The third sentence vector is extracted from the third text vector, which is a vector for representing the core semantics of each sentence in the third conversation text, and can accurately reflect the semantic information of a single sentence in the third conversation text. The second conversation text is the conversation text of the third object in the second text, and the third conversation text is the conversation text of the second object in the second text. The second sentence vector and the third sentence vector can be obtained by splitting the text vectors corresponding to the second conversation text and the third conversation text from the second text vector, dividing the text vectors corresponding to the second conversation text and the third conversation text according to the sentences, and integrating the word vectors corresponding to each sentence.

[0070] In step S503, the third index value is determined based on the first sentence vector, the second sentence vector and the third sentence vector.

[0071] In some embodiments, the server determines the third index value based on the first sentence vector, the second sentence vector and the third sentence vector, which can be achieved by the following manner: first, determining a first similarity based on the first sentence vector and the second sentence vector, and determining a second similarity based on the first sentence vector and the third sentence vector; then, determining a first quantity of the third words belonging to the first type and a second quantity of the third words belonging to the second type in the first conversation text; finally, determining the third index value based on the first similarity, the second similarity, the first quantity and the second quantity.

[0072] In the embodiments of the present application, the first similarity refers to the semantic similarity between the first sentence vector and the second sentence vector, which is used to measure the semantic matching degree of the first object and the third object in the corresponding sentence, and is usually represented by a value between 0 and 1, and the closer the value is to 1, the higher the similarity. The second similarity refers to the semantic similarity between the first sentence vector and the third sentence vector, which is used to measure the semantic matching degree of the first object and the second object in the corresponding sentence, and is also represented by a value between 0 and 1. The third words belonging to the first type are words used to represent the identity of the first object, and the third words belonging to the second type are words used to represent the identity of the second object. The first type refers to the category of the third words used to represent the identity of the first object, which includes all words that can identify the identity of the first object. The second type refers to the category of the third words used to represent the identity of the second object, which includes all words that can identify the identity of the second object. The first quantity refers to the total number of the third words belonging to the first type in the first conversation text, which is used to reflect the frequency of the identity representation of the first object. The second quantity refers to the total number of the third words belonging to the second type in the first conversation text, which is used to reflect the frequency of the identity representation of the second object. The third index value is obtained by comprehensively considering the first similarity, the second similarity, the first quantity and the second quantity, and is used to represent the comprehensive index value of the identity representation accuracy and the semantic matching degree of the first object.

[0073] In some embodiments, in the first conversation text, the number of words matching the agent identity word library, i.e. the first quantity, and the number of words matching the user identity word library, i.e. the second quantity, are determined. The similarity between the first sentence vector and the second sentence vector is calculated and summed up to obtain the first similarity, and the similarity between the first sentence vector and the third sentence vector is also calculated and summed up to obtain the second similarity. The first similarity is divided by the second similarity to obtain a proportion value. The proportion of the identity word quantity is calculated by dividing the first quantity by the sum of 1 and the second quantity to obtain another proportion value. The product of the above two proportion values is the third index value.

[0074] Here, by combining the similarity of the first sentence vector and the second sentence vector, the third sentence vector, and the number of third words of the first type and the second type, the matching degree of the conversation content of the first object and its own identity can be comprehensively quantified. Whether the conversation is consistent with the identity, whether it is irrelevant, and the proximity to the artificial seat are evaluated through semantic similarity, and the rationality of the identity representation is reflected by the number of identity words, so that the third index value can objectively and accurately reflect the identity rationality of the intelligent seat, providing a reliable basis for evaluating and optimizing the conversation performance of the first object.

[0075] Here, by extracting sentence vectors from text vectors, distinguishing the sentence vectors corresponding to the conversation texts of different objects in the second text, and analyzing based on the sentence vectors, the text semantics are finely disassembled. This sentence-level vector comparison can more finely capture the semantic association of the first object and the third object and the second object in specific sentences, avoiding the generality that may exist in overall text vector analysis, making the determination of the third index value more targeted and accurate, and thus more accurately evaluating the identity rationality of the first object.

[0076] Here, by determining the first text vector of the first conversation text and the second text vector of the second text to determine the third index value, the text is converted into a vector form for analysis, which can accurately capture the semantic information of the text by using the mathematical properties of the vector, avoiding the subjectivity and ambiguity of direct evaluation based on the text content. By measuring the association between the conversation text of the first object and the second text through vector-level operations, the semantic matching degree of the two can be more objectively reflected, so that the third index value can effectively represent the identity rationality of the first object, providing a scientific quantitative basis for subsequent evaluation and optimization.

[0077] In step S205, the server determines a fourth index value based on the first text and the second text.

[0078] In the embodiments of the present application, the fourth index value is an index value for representing the semantic completeness of the conversation text belonging to the first object in the first text. The semantic completeness refers to the completeness and comprehensiveness of the conversation content of the first object in the first text in expressing the intention and conveying the information, which is specifically reflected in the integrity of single sentence semantics (for example, whether the core meaning is clearly expressed or information is missing) and the coherence of context (for example, whether the logic between sentences is smooth or whether a complete semantic chain can be formed).

[0079] In some embodiments, referring to Figure 6 , Figure 6 It is shown that in step S205, the server determines a fourth index value based on the first text and the second text, which can also be implemented by the following steps S2051 to S2054:

[0080] Step S2051, the first conversation text in the first text is masked to obtain the first masked text.

[0081] Step S2052, the sentence group in the first text is masked to obtain the second masked text.

[0082] Step S2053, the second conversation text in the second text is masked to obtain the third masked text.

[0083] In the embodiments of the present application, the first conversation text is the conversation text belonging to the first object in the first text, and the second conversation text is the conversation text belonging to the third object in the second text. Masking refers to replacing specific words, phrases or sentences in the text with a preset symbol [MASK] to highlight the core semantic structure or exclude irrelevant information interference, which is used for focusing on the analysis of key content of the text. The first masked text refers to the text obtained by masking the first conversation text belonging to the first object in the first text, which is used to retain the core semantics of the first object conversation and exclude variable interference information. The sentence group refers to a set composed of adjacent sentences in the first text, which includes the first sentence and the second sentence adjacent to the first sentence, and is used to analyze the context coherence. The second masked text refers to the text obtained by masking the sentence group in the first text, which is used to focus on the context semantic association of the sentence group and exclude variable interference information. The third masked text refers to the text obtained by masking the second conversation text belonging to the third object in the second text, and the semantic meaning of the second conversation text is complete. The second conversation text can be used as a reference for the conversation text of the first object, and is used for comparative analysis of semantic completeness.

[0084] In some embodiments, for the first conversation text, the sentence group and the second conversation text, the corresponding masking objects, i.e. variable interference information in the text, such as user name or mailbox information, are determined respectively. A preset masking symbol, such as [MASK], is selected to ensure that the masking symbol has no actual semantics and is uniform, which is convenient for subsequent text analysis. The first conversation text, the sentence group and the second conversation text are traversed sentence by sentence respectively, and the masking objects are identified and replaced with the set masking symbol. After replacing all masking objects, the processed sentences are integrated to form a complete masked text, obtaining the first masked text, the second masked text and the third masked text, which ensures that the text structure of the masked text is consistent with the original text, and only the masking objects are replaced.

[0085] Step S2054, based on the first masked text, the second masked text and the third masked text, a fourth index value is determined.

[0086] In this embodiment of the application, the fourth index value is determined by comparing the semantic matching degree of the first mask text, the second mask text and the third mask text, reflecting the completeness of the expression of the first object.

[0087] In some embodiments, see Figure 7 , Figure 7 The step S2054, in which the server determines the fourth index value based on the first mask text, the second mask text, and the third mask text, can also be achieved through the following steps S701 to S704:

[0088] Step S701: Encode the first mask text to obtain a first encoding vector; encode the second mask text to obtain a second encoding vector; encode the third mask text to obtain a third encoding vector.

[0089] In this embodiment, encoding processing refers to the process of converting masked text into a fixed-dimensional numerical vector. A pre-trained model (e.g., BERT or GPT) captures the semantic features of the masked text, enabling mathematical operations and comparative analysis. The first encoded vector is obtained by encoding the first masked text and is used to quantify the core semantic information of the first masked text. The second encoded vector is obtained by encoding the second masked text and is used to quantify the contextual semantic features of the second masked text. The third encoded vector is obtained by encoding the third masked text and is used to quantify the core semantic information of the third masked text, serving as a reference for the conversational text of the first object.

[0090] In some embodiments, a pre-trained text encoding model can be selected to ensure that the model output vector dimension is uniform. The masked text (first masked text, second masked text, and third masked text) is formatted according to the model requirements, for example, by adding start characters [CLS] and end characters [SEP]. The formatted text is input into the model, and the encoding vectors output by the last layer of the model are extracted and used as the first encoding vector, the second encoding vector, and the third encoding vector, respectively.

[0091] Step S702: Determine the first sentence class vector based on the sentence type of each sentence in the first masked text; determine the second sentence class vector based on the sentence type of each sentence in the second masked text; determine the third sentence class vector based on the sentence type of each sentence in the second masked text.

[0092] In the embodiments of the present application, the sentence type refers to the classification of the function or use of the sentence in the conversation, and the sentence type includes a question sentence, a statement sentence, an imperative sentence, and a response sentence, etc., which are used to distinguish the expression intention of the sentence. The first sentence type vector is a vector generated based on the sentence type of each sentence in the first masked text, which is used to quantitatively represent the distribution characteristics of the sentence type in the first masked text. The second sentence type vector is a vector generated based on the sentence type of each sentence in the second masked text, which is used to quantitatively represent the sequence characteristics of the sentence type in the sentence group. The third sentence type vector is a vector generated based on the sentence type of each sentence in the third masked text, which is used to quantitatively represent the distribution characteristics of the sentence type in the third masked text.

[0093] In some embodiments, the mapping relationship between the preset sentence type and the numerical value is, for example, the question sentence is mapped to 0, the statement sentence is mapped to 1, the imperative sentence is mapped to 2, and the response sentence is mapped to 3. The first masked text, the second masked text, and the third masked text are traversed respectively, the sentence type of each sentence in the masked text is judged sentence by sentence, and the corresponding numerical value is converted to form a type sequence. Finally, the type sequence is converted into a fixed-dimensional vector through embedding layer mapping or extension padding, etc., to obtain the first sentence type vector, the second sentence type vector, and the third sentence type vector.

[0094] In step S703, the third text vector is determined based on the first encoding vector and the first sentence type vector; the fourth text vector is determined based on the second encoding vector and the second sentence type vector; and the fifth text vector is determined based on the third encoding vector and the third sentence type vector.

[0095] In the embodiments of the present application, the third text vector is a vector obtained by fusing the first encoding vector and the first sentence type vector, which comprehensively combines the semantic information and the sentence type characteristics of the first masked text. The fourth text vector is a vector obtained by fusing the second encoding vector and the second sentence type vector, which comprehensively combines the context semantic association and the sentence type sequence characteristics of the second masked text. The fifth text vector is a vector obtained by fusing the third encoding vector and the third sentence type vector, which comprehensively combines the semantic information and the sentence type characteristics of the third masked text.

[0096] In step S704, the fourth index value is determined based on the third text vector, the fourth text vector, and the fifth text vector.

[0097] In some embodiments, the server determines the fourth index value based on the third text vector, the fourth text vector, and the fifth text vector, which can be achieved by the following manner: first, determining the third similarity based on the third text vector and the fifth text vector, and determining the fourth similarity based on the fourth text vector and the fifth text vector; and then, determining the fourth index value based on the third similarity and the fourth similarity.

[0098] In the embodiments of the present application, the third similarity refers to the semantic similarity between the third text vector and the fifth text vector, and is used to measure the matching degree of the core semantics of the conversation text of the first object and the core semantics of the conversation text of the third object, and is usually represented by a value between 0 and 1. The closer the value is to 1, the higher the similarity. In addition, since the semantics of the conversation text of the third object is complete, the higher the third similarity, the more complete the semantics of the conversation text of the first object. The fourth similarity refers to the semantic similarity between the fourth text vector and the fifth text vector, and is used to measure the matching degree of the context semantic association of the conversation text of the first object and the core semantics of the conversation text of the third object, and is also represented by a value between 0 and 1. The higher the fourth similarity, the better the context coherence of the conversation text of the first object. The fourth index value is obtained by comprehensively considering the third similarity and the fourth similarity, and is used to quantify the index value representing the semantic completeness of the conversation text of the first object, and reflects the comprehensive performance of the expression of the first object in terms of single sentence semantic completeness and context coherence.

[0099] In some embodiments, the cosine similarity algorithm can be used to calculate the third similarity between the third text vector and the fifth text vector, and to calculate the fourth similarity between the fourth text vector and the fifth text vector. The weights of the third similarity and the fourth similarity can be set according to the evaluation requirements. If the single sentence semantic matching is emphasized, the weight of the third similarity is higher. If the context coherence is emphasized, the weight of the fourth similarity is higher. The third similarity and the fourth similarity are weighted and summed according to the weights to obtain the fourth index value.

[0100] Here, the third similarity and the fourth similarity are determined based on the third text vector, the fourth text vector and the fifth text vector, and then the fourth index value is determined. By quantifying the semantic association between different text vectors, the matching degree of the conversation text of the first object and the conversation text of the third object and the user-related content is directly reflected. This evaluation method based on similarity converts the semantic completeness into a quantifiable index, avoids the deviation of subjective judgment, and makes the fourth index value objectively reflect the completeness of the expression of the first object, thereby providing a clear basis for optimizing the conversation performance of the first object.

[0101] Here, the mask text is processed by encoding to obtain an encoded vector, a sentence type vector is determined in combination with the sentence type, and then the text vector is fused to determine the fourth index value, thereby realizing multi-dimensional feature extraction from the surface of the text to the deep semantic. The encoded vector captures the semantic information of the text, and the sentence type vector reflects the functional attributes of the sentence. The combination of the two makes the text vector more comprehensively represent the text features, improves the depth and richness of the evaluation of the fourth index value, and makes the evaluation of the semantic completeness of the first object more accurate and reliable.

[0102] Here, by performing mask processing on the first conversation text, the sentence group and the second conversation text to obtain corresponding mask texts, and then determining the fourth index value based on the mask texts, the core semantic structure of the text can be focused on, and irrelevant information interference can be excluded; by comparing the dialogue texts of the first object and the third object, the semantic completeness is evaluated from the single sentence and the context coherence dimension, so that the evaluation is more in line with the actual expression scene, which provides an intuitive and targeted analysis basis for the determination of the fourth index value, and helps to accurately measure the semantic completeness of the conversation text of the first object.

[0103] In step S206, the server determines a fifth index value based on the first word in the first text and the second word adjacent to the first word.

[0104] In the embodiments of the present application, the fifth index value is an index value for representing the text fluency of the conversation text belonging to the first object in the first text. The text fluency refers to the smoothness, semantic coherence and logical naturalness of the conversation text belonging to the first object in the first text in language expression, which is specifically reflected in the semantic association closeness between adjacent words and the smoothness of the overall expression.

[0105] In some embodiments, referring to Figure 8 , Figure 8 It is shown that in step S206, the server determines a fifth index value based on the first word in the first text and the second word adjacent to the first word, which can also be implemented by the following steps S2061 to S2063:

[0106] In step S2061, a first mapping point corresponding to the first word in the first coordinate system is determined, and a second mapping point corresponding to the second word in the first coordinate system is determined.

[0107] In the embodiments of the present application, the first word refers to any one word in the conversation text belonging to the first object in the first text. The first coordinate system is a high-dimensional vector coordinate system for mapping the word vector of the word to a spatial point. The first mapping point is the spatial point corresponding to the word vector of the first word in the first coordinate system, and the coordinate value of the first mapping point can be the word vector value of the word, which quantitatively represents the semantic information of the first word. The second word refers to the word adjacent to the first word, including the word adjacent before and after. The second mapping point is the spatial point corresponding to the word vector of the second word in the first coordinate system, and the coordinate value of the second mapping point is the word vector value of the word, which is used to calculate the gradient change with the first mapping point.

[0108] In some embodiments, the conversation text belonging to the first object in the first text can be segmented word by word to obtain a word sequence containing the first word and its adjacent second word. The segmented word sequence is input into a large language model to obtain a word vector of each word, wherein the word vector dimensions of the first word and the second word are the same as the dimensions of the first coordinate system. The numerical value of the word vector of the first word is taken as the coordinate value to find the corresponding spatial point in the first coordinate system, which is the first mapping point; similarly, the numerical value of the word vector of the second word is taken as the coordinate value to find the corresponding spatial point in the first coordinate system, which is the second mapping point.

[0109] In step S2062, the gradient change sum between the first mapping point and the second mapping point is determined based on the coordinate value of the first mapping point and the coordinate value of the second mapping point.

[0110] In the embodiments of the present application, the gradient change sum between the first mapping point and the second mapping point refers to the sum (or sum of squares, etc.) of the absolute values of the coordinate difference values of the first mapping point and the second mapping point in each dimension in the first coordinate system, which is used to measure the semantic association closeness and change amplitude of adjacent words, and further reflects the text fluency.

[0111] In some embodiments, the coordinate values of the first mapping point and the second mapping point in each dimension of the first coordinate system can be extracted respectively. For the case of three adjacent words, first, the first mapping point and the second mapping point corresponding to the first two words are taken to calculate the gradient change between them; then, the mapping points corresponding to the last two words are taken to calculate the gradient change between them; finally, the two gradient changes are added to obtain the gradient change sum between the mapping points corresponding to the three adjacent words.

[0112] In step S2063, the fifth index value is determined based on the gradient change sum.

[0113] In the embodiments of the present application, the smaller the gradient change sum, the closer the semantic association of adjacent words, and the higher the text fluency; on the contrary, the lower the text fluency. If there is a combination of three adjacent words, the gradient change sum of the mapping points of the three adjacent words is calculated, and the reciprocal of the gradient change sum is determined as the fifth index value; if there are multiple combinations of three adjacent words, the average value of the reciprocals of the gradient change sums of all combinations is calculated, and the average value is determined as the fifth index value.

[0114] Here, by converting the words into mapping points in a high-dimensional coordinate system and calculating the gradient change sum between adjacent mapping points to evaluate the text fluency, the fluency of the first object conversation text can be objectively and quantitatively reflected from the semantic level, avoiding the subjective randomness of manual evaluation; with the characteristics of the high-dimensional vector space, the semantic association between words can be accurately captured, making the fifth index value more truly reflect the text fluency; and this way has clear steps and strong operability, and is suitable for text fluency evaluation in various scenarios, providing a reliable basis for subsequent determination of the first index value and optimization of the data processing model, which is conducive to improving the expression effect of the first object conversation text and the user's interactive experience.

[0115] In step S207, the server determines the average of the third index value, the fourth index value and the fifth index value as the first index value.

[0116] In the embodiments of the present application, after obtaining the third index value, the fourth index value and the fifth index value, the average of the third index value, the fourth index value and the fifth index value is calculated, and the average is determined as the first index value.

[0117] Here, by multi-dimensionally disassembling the conversation quality evaluation, the first object interaction performance is accurately quantified. The third index value is used to evaluate the identity rationality of the interaction subject, i.e., to judge whether the conversation content of the first object is consistent with its identity, whether it is irrelevant to the question, and whether the conversation performance of the first object is close to the third object; the fourth index value focuses on the content quality to promote the model to provide more comprehensive information; the fifth index value optimizes the language expression to improve the naturalness of human-computer dialogue; and the comprehensive average balances the multi-dimensional indexes to avoid one-sidedness of a single dimension, making the evaluation result more objective and comprehensive. This hierarchical evaluation mechanism not only can locate the model defects, but also can guide targeted optimization in the subsequent, and finally improve the overall conversation quality and user experience.

[0118] In step S208, the server determines the first text length of the first conversation text in the first text and the second text length of the fourth conversation text in the first text.

[0119] In the embodiments of the present application, the first conversation text is the conversation text of the first object in the first text, and the first text length refers to the total number of characters or words of the conversation text of the first object in the first text, which can be used to measure the amount of information replied by the first object; the fourth conversation text is the conversation text of the second object in the first text, and the second text length refers to the total number of characters or words of the conversation text of the second object in the first text, which can be used to measure the complexity of the user's question.

[0120] In some embodiments, the first text can be split into alternating utterances of the first object and the second object by dialogue turn, and the attribution of each round of utterance is marked. The total length of the utterance marked as the first object in each round of utterance is counted as the first text length, and the total length of the utterance marked as the second object in each round of utterance is counted as the second text length.

[0121] In step S209, the server determines the second index value based on the conversation response duration of the first object and the second object, the first text length and the second text length.

[0122] In the embodiments of the present application, the second index value is an index value for quantifying the response efficiency of the first object in the conversation, and the second index value comprehensively considers the conversation response duration of the first object and the second object, the information amount of the first object reply and the complexity of the second object question.

[0123] In some embodiments, after determining the first text length and the second text length of each round of utterance, the sum of the first text length and the second text length of each round of utterance can be determined as the total text length of each round of utterance. For each turn, the single-turn response time consumption rate of the turn is obtained by dividing the conversation response duration of the turn by the total text length of the turn. The average value of the single-turn response time consumption rates of all turns is determined as the second index value.

[0124] Here, by separately calculating the sum of the first text length and the second text length by turn, and combining the single-turn response time consumption rate obtained by the conversation response duration of the corresponding turn, and then taking the average value to determine the second index value, the response efficiency of the first object in each round of conversation can be more accurately reflected, and the overall evaluation deviation caused by the large difference between the conversation content length and the response duration in different turns can be avoided. At the same time, the total length of the conversation text of both parties in each round and the response duration are comprehensively considered, which can more comprehensively reflect the instant response ability of the first object in processing conversations of different lengths and different complexities, so that the second index value is more objective and targeted. It provides a more reliable basis for subsequent determination of evaluation score based on the index value and optimization of data processing model, and further improves the adaptability and response effect of the model in the actual dialogue scene.

[0125] In step S210, the server determines the evaluation score of the first object based on the first index value and the second index value.

[0126] In the embodiments of the present application, for determining the evaluation score of the first object based on the first index value and the second index value, the specific meaning explanation and specific implementation involved in this step can be referred to the description of step S103 in the above, which will not be repeated here.

[0127] In step S211, the server determines the text generation model adapted to the first object based on the evaluation score.

[0128] In the embodiments of the present application, for determining the text generation model adapted to the first object based on the evaluation score, the specific meaning explanation and specific implementation involved in this step can be referred to the description of step S104, which will not be repeated here.

[0129] In step S212, the server sends the model parameters of the text generation model to the terminal.

[0130] In step S213, the terminal outputs the model parameters of the text generation model.

[0131] In the embodiments of the present application, the identity rationality evaluation, the semantic integrity evaluation, the text fluency evaluation and the response rapidity evaluation are comprehensively evaluated by constructing a multi-dimensional quantitative index system, which realizes comprehensive and objective evaluation of the conversation performance of the first object. For the identity rationality evaluation, the semantic matching degree and the number of identity words are used to accurately judge the degree of fit between the conversation content of the first object and its own identity and the closeness of the conversation content of the third object. For the semantic integrity evaluation, the mask text and vector analysis are used to measure the completeness of the conversation content expression of the first object from the dimensions of single sentence semantics and context coherence. For the text fluency evaluation, the high-dimensional vector gradient change is used to quantify the semantic association and sentence smoothness of adjacent words in the conversation content of the first object. For the response rapidity evaluation, the ratio of response time to text length is used to reflect the processing efficiency of the first object. These evaluations complement each other, avoiding the subjectivity of manual judgment, and capturing the subtle features in the conversation of the first object, providing a scientific basis for optimizing the conversation model of the first object and improving the interactive experience. Thus, the first object continuously approaches the artificial level in terms of semantic accuracy, expression completeness, communication fluency and response timeliness, and the text generation effect of the data processing model corresponding to the first object is greatly improved.

[0132] The data processing method provided by the embodiments of the present application can be applied in various scenarios. For example, application scenario one: intelligent customer service scenario, the first object corresponds to an intelligent customer service model, the second object is a user, and the third object is a human customer service. By analyzing the conversation text between the intelligent customer service and the user and the conversation text between the human customer service and the user, the identity rationality, semantic completeness, text fluency and response speed of the intelligent customer service are evaluated, and after the total evaluation score is obtained by comprehensive evaluation, the intelligent customer service model is optimized, and the service quality and response speed thereof are improved. Application scenario two: intelligent question and answer assistant scenario, the first object corresponds to a question and answer assistant model, the second object is a user, and the third object is a domain expert. By using the dialogue text between the question and answer assistant and the user and the supplementary dialogue text between the expert and the user, the identity rationality, semantic completeness, text fluency and response speed of the question and answer assistant are evaluated, and after the total evaluation score is obtained by comprehensive evaluation, the question and answer assistant model is optimized according to the total evaluation score, so that the question and answer assistant can more accurately and quickly answer the user's questions. Application scenario three: online education tutoring scenario, the first object corresponds to a tutoring robot model, the second object is a student, and the third object is a teacher. By analyzing the learning conversation text between the tutoring robot and the student and the after-school communication text between the teacher and the student, the identity rationality, semantic completeness, text fluency and response speed of the tutoring robot are evaluated, and after the total evaluation score is obtained by comprehensive evaluation, the tutoring robot model is optimized according to the total evaluation score, so that the tutoring robot can better meet the learning needs of the student and improve the tutoring efficiency and effect.

[0133] In the following, an exemplary application of the embodiments of the present application in an actual application scenario will be described.

[0134] The embodiments of the present application provide an evaluation method for text generation effect of a large language model. The method evaluates the text generation effect of the large language model through four dimensions, i.e., evaluating the service quality of the intelligent agent, and correspondingly obtaining four evaluation scores. Finally, the four evaluation scores are comprehensively evaluated to obtain a total evaluation value for the intelligent agent. Whether the text generation effect of the large language model or the service quality of the intelligent agent needs to be optimized is determined by the total evaluation value. The flowchart of the method is shown in Figure 9 .

[0135] In step S901, the historical conversation text between the intelligent agent and the user in the customer service scenario (i.e., the first text described above) and the historical conversation text between the human agent and the user (i.e., the second text described above) are obtained.

[0136] In step S902, identity rationality evaluation is performed.

[0137] First, identity word detection is performed, the intelligent seat text to be detected (i.e. the first session text described above) is extracted from the historical session text, the intelligent seat text to be detected is processed by word segmentation sentence by sentence, and the words (i.e. the third words described above) related to the address words, positions and relationships, such as company names, work numbers and colleagues, etc. indicating identity are recalled using the full word matching method; it should be noted that the seat identity word library and the user identity word library need to be constructed in advance for the recall process. Then, the identity phrase or short sentence detection is performed, the phrases or short sentences in the intelligent seat text after deleting the identity words form a data set sets_auto_seats (assuming m pieces), and the sentence vectors (i.e. the first sentence vectors described above) thereof are obtained piece by piece. Special, the position of deletion is replaced by the same empty symbol vector. The user text (i.e. the second session text described above) is extracted from the historical session text, the complete user text forms a data set sets_customer (assuming n pieces), and the sentence vectors (i.e. the second sentence vectors described above) thereof are obtained piece by piece. The artificial seat text (i.e. the third session text described above) is extracted from the historical session text, the complete artificial seat text forms a data set sets_seats (assuming o pieces), and the sentence vectors (i.e. the third sentence vectors described above) thereof are obtained piece by piece. Each sentence vector in sets_auto_seats (assuming m pieces) and sets_seats (assuming o pieces) is paired two by two, and the vector loss sum (i.e. the first similarity described above) is calculated by the following formula (1). Each sentence vector in sets_auto_seats (assuming m pieces) and sets_customer (assuming n pieces) is paired two by two, and the vector loss sum (i.e. the second similarity described above) is calculated by the following formula (2):

[0138] (1)

[0139] (2)

[0140] wherein, represents the i-th sentence vector corresponding to sets_auto_seats, represents the p-th sentence vector corresponding to sets_seats, represents the j-th sentence vector corresponding to sets_customer. ∑ represents the accumulation function.

[0141] Finally, the identity rationality evaluation is performed according to the following formula (3), and the identity rationality evaluation score (i.e. the third index value described above) is obtained.

[0142] (3)

[0143] wherein, represents the identity rationality evaluation score of the intelligent seat, represents the number of words in the intelligent agent text that can be matched to the words in the agent identity library (i.e., the first number described above), represents the number of words in the intelligent agent text that can be matched to the words in the user identity library (i.e., the second number described above). As can be seen from the formula, the more words in the intelligent agent text that can be matched to the words in the agent identity library than to the words in the user identity library, the higher the identity rationality evaluation score; the higher the similarity between the text in the intelligent agent text excluding the identity words and the artificial agent text compared to the similarity between the text in the intelligent agent text excluding the identity words and the user text, the higher the identity rationality evaluation score of the intelligent agent.

[0144] Step 903, table meaning integrity evaluation.

[0145] First, replace the specific variable information in the table meaning complete text in the artificial agent text with Mask, for example, the original artificial agent text is: Are you Ms. Bai? The standardized template is: Are you [Mask] Ms.? Then, extract the set of standardized templates (i.e., the third mask text described above), denoted as template_sets, and perform template encoding on template_sets. For the template encoding method, at the semantic level, directly input the elements in template_sets into any large language model, call the sentence vector function, obtain the sentence vector of template_sets, where [Mask] is treated as a normal blank, and the obtained encoding result is denoted as ; at the sentence class level, using the full matching method with the word library and the punctuation library, match the words or punctuation marks in the artificial agent text that can indicate the sentence class, present them in the form of a list, and determine the sentence class of the template template_sets according to the principle of highest frequency, and obtain the corresponding vector representation (i.e., the third sentence class vector described above) in the sentence class vector dictionary using the sentence class name, denoted as .

[0146] Obtain all sentence class names, which can include declarative sentences, interrogative sentences (yes / no questions, specific questions, choice questions, and positive / negative questions), imperative sentences, and exclamatory sentences; input the above sentence class names into any large language model to obtain their word vector representations. Since there is a hierarchical relationship between the interrogative sentence, the yes / no question, the specific question, the choice question, and the positive / negative question, the word vectors of the interrogative sentence, the yes / no question, the specific question, the choice question, and the positive / negative question are multiplied by the word vector of the interrogative sentence to obtain the final sentence class vector representation of the above four types of interrogative sentences. The mapping relationship between all the above sentence classes and the sentence class vector representations can be stored in the sentence class vector dictionary. Then, template encoding is merged to obtain the encoding result (i.e., the third encoding vector described above).

[0147] Similarly, the specific variable information in the intelligent agent text is replaced with Mask, and the set of standardized templates (i.e., the first masked text described above) is extracted, denoted as intelligence_frame_sets, and template encoding is performed, and the encoding result is denoted as (i.e., the first encoding vector described above), as single-sentence comparison data; the intelligent agent text is combined with its context 1 sentence (either user text or intelligent agent text), forming a sentence group, the specific variable information in the sentence group is replaced with Mask, and the set of standardized templates (i.e., the second masked text described above) is extracted, denoted as intelligence_frame_sets, and template encoding is performed, and the encoding result is denoted as (i.e., the second encoding vector described above), as joint sentence comparison data.

[0148] Finally, the semantic completeness evaluation is performed according to the following formula (4) to obtain the semantic completeness evaluation score (i.e., the fourth index value described above).

[0149] (4)

[0150] wherein, represents the semantic completeness evaluation score of the intelligent agent, and are two different preset weight values, represents the i-th vector in represents the p-th vector in represents the j-th vector in .

[0151] Step S904, text fluency evaluation. First, the intelligent agent text is sent into a large language model after being divided into sentences, and the word vector sequence of each sentence in the intelligent agent text is obtained; then, the word vector of each word in the word vector sequence is mapped in a high-dimensional vector coordinate system, represented by a point. Next, the sum of the gradients between the mapped points of the word vectors of adjacent three words in the high-dimensional vector coordinate system (i.e., the total gradient change described above) is calculated according to the following formula (5).

[0152] (5)

[0153] wherein, , and represent the coordinate values of the x-axis, y-axis and z-axis directions of the mapped points in the high-dimensional vector coordinate system, represents a scalar function, represents the Laplacian of the scalar function , which can also be written as ,​​ , and These are three spatial coordinates in a spherical coordinate system. scalar function The second-order partial derivatives with respect to x, y, and z in a rectangular coordinate system reflect the rate of change of the quantity along the x, y, and z axes. express radial coordinates in spherical coordinate system The first-order partial derivative reflects Rate of change along the radial direction; Indicates to " Multiply The radial first-order partial derivative is then used to calculate the radial partial derivative, which is then used to correct for radial variations in spherical coordinates. express Polar angle in spherical coordinates The first-order partial derivative reflects The rate of change along the polar angle; Indicates to " Multiply The first-order partial derivative of the polar angle is used to calculate the partial derivative of the polar angle, which is then used to correct for changes in the polar angle direction in spherical coordinates. express Aspect of azimuth in spherical coordinates The second-order partial derivative reflects The rate of change of the rate of change along the azimuth direction; This represents the correction factor for the azimuth direction in spherical coordinates, due to the polar angle. Changes will affect the measurement scale of the azimuth direction. In short, the left side of formula (5) is the expression for the Laplacian operator in Cartesian coordinates, and the right side is its equivalent transformation form in spherical coordinates. This coordinate transformation achieves the calculation conversion of the same physical quantity in different coordinate systems, facilitating calculations in different scenarios. The spatial variation characteristics are analyzed. Finally, the text fluency is evaluated using the following formula (6), and the text fluency evaluation score is obtained. (i.e., the fifth indicator value mentioned above).

[0154] (6)

[0155] Step S905, Response speed assessment. The response speed assessment score is a direct indicator that most effectively impacts user experience, regardless of the specific content of the conversation. Response speed can be calculated from the end of the user's speech. From the beginning, until the smart agent started speaking. To eliminate the impact of text length on the response time assessment score, the user text length is adjusted according to the time difference. (i.e., the second text length mentioned above) and the text length of the intelligent agent The sum of (i.e., the length of the first text mentioned above) is used as the denominator, as shown in the following formula (7):

[0156] (7)

[0157] in, This represents the evaluation score for the responsiveness of the smart agent (i.e., the second indicator value mentioned above).

[0158] Step S906: Based on the above evaluation results, calculate the performance score of the large language model. (Large Language Model Performance Score) (That is, the above assessment score) can be calculated using the following formula (8):

[0159] (8)

[0160] This application provides a new approach to improving the text generation performance of large language models by introducing the evaluation of large language model text generation performance into the intelligent customer service scenario and conducting evaluations of text generation performance across multiple dimensions.

[0161] The following description continues to illustrate the exemplary structure of the data processing apparatus 455 provided in the embodiments of this application as a software module. In some embodiments, such as Figure 10 As shown, the software modules in the data processing device 455 may include: a first determining module 4551, used to determine a first indicator value based on a first text and a second text; the first text is conversation text between a first object and a second object generated by a first model, and the second text is conversation text between a third object and the second object; the first indicator value is an indicator value used to characterize the conversation quality between the first object and the second object; a second determining module 4552, used to determine a second indicator value based on the conversation response duration between the first object and the second object; the second indicator value is an indicator value used to characterize the conversation response speed of the first object; a third determining module 4553, used to determine an evaluation score for the first model based on the first indicator value and the second indicator value; and a fourth determining module 4554, used to determine a text generation model adapted to the first object based on the evaluation score.

[0162] In some embodiments, the first determining module 4551 is further configured to: determine a third index value based on the first text and the second text; the third index value is an index value for representing the identity of the first object; determine a fourth index value based on the first text and the second text; the fourth index value is an index value for representing the semantic completeness of first conversation text in the first text; the first conversation text is conversation text of the first object in the first text; determine a fifth index value based on a first word in the first text and a second word adjacent to the first word; the fifth index value is an index value for representing the text fluency of the first conversation text; and determine a mean value of the third index value, the fourth index value and the fifth index value as the first index value.

[0163] In some embodiments, the first determining module 4551 is further configured to: determine a first text vector of the first conversation text, and a second text vector of the second text; and determine the third index value based on the first text vector and the second text vector.

[0164] In some embodiments, the first determining module 4551 is further configured to: perform word segmentation processing on the first conversation text to obtain a first word sequence of each sentence in the first conversation text; replace a third word in the first word sequence with a preset placeholder to obtain a second word sequence; the third word is a word for representing the identity of the first object and the second object; perform vectorization processing on the words in the second word sequence to obtain the first text vector; and perform vectorization processing on the second text to obtain the second text vector.

[0165] In some embodiments, the first determining module 4551 is further configured to: determine a first sentence vector of each sentence in the first conversation text from the first text vector; determine a second sentence vector of each sentence in the second conversation text and a third sentence vector of each sentence in the third conversation text from the second text vector; the second conversation text is conversation text of the third object in the second text, and the third conversation text is conversation text of the second object in the second text; and determine the third index value based on the first sentence vector, the second sentence vector and the third sentence vector.

[0166] In some embodiments, the first determining module 4551 is further configured to: determine a first similarity based on the first sentence vector and the second sentence vector, and determine a second similarity based on the first sentence vector and the third sentence vector; determine a first quantity of third words belonging to a first type and a second quantity of third words belonging to a second type in the first conversation text; the third words belonging to the first type are words used to represent the identity of the first object, and the third words belonging to the second type are words used to represent the identity of the second object; determine the third indicator value based on the first similarity, the second similarity, the first quantity, and the second quantity.

[0167] In some embodiments, the first determining module 4551 is further configured to: perform mask processing on the first conversation text in the first text to obtain a first mask text; perform mask processing on a sentence group in the first text to obtain a second mask text; the sentence group includes a first sentence in the first text and a second sentence adjacent to the first sentence; perform mask processing on the second conversation text in the second text to obtain a third mask text; and determine the fourth indicator value based on the first mask text, the second mask text, and the third mask text.

[0168] In some embodiments, the first determining module 4551 is further configured to: perform encoding processing on the first mask text to obtain a first encoding vector; perform encoding processing on the second mask text to obtain a second encoding vector; perform encoding processing on the third mask text to obtain a third encoding vector; determine a first sentence class vector based on the sentence type of each sentence in the first mask text; determine a second sentence class vector based on the sentence type of each sentence in the second mask text; determine a third sentence class vector based on the sentence type of each sentence in the third mask text; determine a third text vector based on the first encoding vector and the first sentence class vector; determine a fourth text vector based on the second encoding vector and the second sentence class vector; determine a fifth text vector based on the third encoding vector and the third sentence class vector; and determine the fourth indicator value based on the third text vector, the fourth text vector, and the fifth text vector.

[0169] In some embodiments, the first determining module 4551 is further configured to: determine a third similarity based on the third text vector and the fifth text vector, and determine a fourth similarity based on the fourth text vector and the fifth text vector; and determine the fourth indicator value based on the third similarity and the fourth similarity.

[0170] In some embodiments, the first determining module 4551 is further configured to: determine a first mapping point corresponding to the first word in a first coordinate system, and a second mapping point corresponding to the second word in the first coordinate system; determine a gradient change sum between the first mapping point and the second mapping point based on coordinate values of the first mapping point and the second mapping point; and determine the fifth index value based on the gradient change sum.

[0171] In some embodiments, the second determining module 4552 is further configured to: determine a first text length of the first session text and a second text length of fourth session text in the first text; the fourth session text is session text belonging to the second object in the first text; and determine the second index value based on a session response duration of the first object and the second object, the first text length, and the second text length.

[0172] It should be noted that the description of the device embodiments of the present application is similar to the description of the above method embodiments, has similar beneficial effects as the method embodiments, and therefore will not be described here. For technical details not disclosed in the device embodiments of the present application, please refer to the description of the method embodiments of the present application for understanding.

[0173] Correspondingly, the present application provides an electronic device, Figure 11 The electronic device provided in the embodiments of the present application has a component structure diagram as shown in Figure 11 The electronic device 1200 at least includes a processor 1201, a communication interface 1202, and a storage medium 1203 configured to store executable instructions, wherein the processor 1201 generally controls the overall operation of the electronic device 1200.

[0174] The communication interface 1202 can enable the electronic device to communicate with other terminals or servers through a network.

[0175] The storage medium 1203 is configured to store instructions and applications executable by the processor 1201, and can also cache data to be processed by the processor 1201 and modules in the electronic device 1200, and can be implemented by a FLASH or a Random Access Memory (RAM).

[0176] The embodiments of the present application provide a computer readable storage medium, wherein computer executable instructions are stored, when the computer executable instructions are executed by a processor, the processor will execute the data processing method provided by the embodiments of the present application, for example, the data processing method shown in Figure 2 The data processing method.

[0177] The embodiment of the present application provides a computer program product, which comprises computer executable instructions stored in a computer readable storage medium. The processor of the electronic device reads the computer executable instructions from the computer readable storage medium, and the processor executes the computer executable instructions, so that the electronic device executes the data processing method provided by the embodiment of the present application.

[0178] In some embodiments, the computer readable storage medium can be RAM, ROM, flash memory, magnetic surface memory, optical disc, or CD-ROM memory, etc.; and can also be various devices comprising one or any combination of the above storage medium.

[0179] In some embodiments, the computer executable instructions can be in the form of programs, software, software modules, scripts or codes, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and can be deployed in any form, including being deployed as independent programs or being deployed as modules, components, subroutines or other units suitable for use in a computing environment.

[0180] As an example, the computer executable instructions can but not necessarily correspond to files in a file system, can be stored in a part of a file storing other programs or data, for example, stored in one or more scripts in a Hyper Text Markup Language (HTML) document, stored in a single file dedicated to the program in question, or stored in multiple cooperative files (for example, files storing one or more modules, subroutines or code parts).

[0181] As an example, the computer executable instructions can be deployed to be executed on one electronic device, or executed on multiple electronic devices located in one place, or executed on multiple electronic devices distributed in multiple places and interconnected through a communication network.

[0182] The above only describes the embodiments of the present application, and is not used to limit the protection scope of the present application. Any modification, equivalent replacement and improvement made within the spirit and scope of the present application shall be included in the protection scope of the present application.

Claims

1. A data processing method, characterized by, The method comprises: determining a first index value based on first text and second text; the first text is conversation text of a first object and a second object generated by a first model, and the second text is conversation text of a third object and the second object; the first index value is an index value for representing conversation quality of the first object and the second object; wherein the first object is an intelligent system, an artificial intelligence model or an automated tool, the second object is an initiator or a demander of a conversation, and the third object is a human or a system with professional ability or standard process; the first index value is a mean value of a third index value, a fourth index value and a fifth index value; the third index value is an index value for representing identity of the first object, and the third index value is determined based on the first text and the second text; the fourth index value is an index value for representing semantic completeness of first conversation text in the first text, and the first conversation text is conversation text of the first object in the first text; and the fifth index value is an index value for representing text fluency of the first conversation text; determining a second index value based on conversation response duration of the first object and the second object; the second index value is an index value for representing conversation response speed of the first object; determining an evaluation score for the first model based on the first index value and the second index value; determining a text generation model adapted to the first object based on the evaluation score.

2. The method of claim 1, wherein, The method comprises: determining a third index value based on the first text and the second text; determining a fourth index value based on the first text and the second text; determining a fifth index value based on a first word in the first text and a second word adjacent to the first word; determining a mean value of the third index value, the fourth index value and the fifth index value as the first index value.

3. The method of claim 2, wherein, The method comprises: determining a first text vector of the first conversation text and a second text vector of the second text; determining the third index value based on the first text vector and the second text vector.

4. The method of claim 3, wherein, The method comprises: performing word segmentation processing on the first conversation text to obtain a first word sequence of each sentence in the first conversation text; replacing a third word in the first word sequence with a preset placeholder to obtain a second word sequence; the third word is a word for representing identity of the first object and the second object; performing vectorization processing on the second word sequence to obtain the first text vector; performing vectorization processing on the second text to obtain the second text vector.

5. The method of claim 4, wherein, The method comprises: determining the third index value based on the first text vector and the second text vector. determine a first sentence vector of each sentence in the first conversation text from the first text vector; determine a second sentence vector of each sentence in the second conversation text and a third sentence vector of each sentence in the third conversation text from the second text vector; the second conversation text is conversation text of the second text belonging to the third object, and the third conversation text is conversation text of the second text belonging to the second object; determine the third index value based on the first sentence vector, the second sentence vector and the third sentence vector.

6. The method of claim 5, wherein, The determination of the third index value based on the first sentence vector, the second sentence vector and the third sentence vector comprises: determine a first similarity based on the first sentence vector and the second sentence vector, and determine a second similarity based on the first sentence vector and the third sentence vector; determine a first number of third words belonging to a first type and a second number of third words belonging to a second type in the first conversation text; the third words belonging to the first type are words used to represent the identity of the first object, and the third words belonging to the second type are words used to represent the identity of the second object; determine the third index value based on the first similarity, the second similarity, the first number and the second number.

7. The method of claim 5, wherein, The determination of the fourth index value based on the first text and the second text comprises: mask processing the first conversation text in the first text to obtain a first masked text; mask processing a sentence group in the first text to obtain a second masked text; the sentence group comprises a first sentence in the first text and a second sentence adjacent to the first sentence; mask processing the second conversation text in the second text to obtain a third masked text; determine the fourth index value based on the first masked text, the second masked text and the third masked text.

8. The method of claim 7, wherein, The determination of the fourth index value based on the first masked text, the second masked text and the third masked text comprises: perform encoding processing on the first masked text to obtain a first encoding vector, perform encoding processing on the second masked text to obtain a second encoding vector, and perform encoding processing on the third masked text to obtain a third encoding vector; determine a first sentence type vector based on the sentence type of each sentence in the first masked text, determine a second sentence type vector based on the sentence type of each sentence in the second masked text, and determine a third sentence type vector based on the sentence type of each sentence in the third masked text; determine a third text vector based on the first encoding vector and the first sentence type vector, determine a fourth text vector based on the second encoding vector and the second sentence type vector, and determine a fifth text vector based on the third encoding vector and the third sentence type vector; determine the fourth index value based on the third text vector, the fourth text vector and the fifth text vector.

9. The method of claim 8, wherein, The fourth index value is determined based on the third text vector, the fourth text vector, and the fifth text vector, including: a third similarity is determined based on the third text vector and the fifth text vector, and a fourth similarity is determined based on the fourth text vector and the fifth text vector; the fourth index value is determined based on the third similarity and the fourth similarity.

10. The method of claim 2, wherein, The fifth index value is determined based on a first word in the first text and a second word adjacent to the first word, including: a first mapping point corresponding to the first word in a first coordinate system is determined, and a second mapping point corresponding to the second word in the first coordinate system is determined; a gradient change sum between the first mapping point and the second mapping point is determined based on coordinate values of the first mapping point and the second mapping point; the fifth index value is determined based on the gradient change sum.

11. The method according to any one of claims 2 to 10, characterized in that, The second index value is determined based on a conversation response duration of the first object and the second object, including: a first text length of the first conversation text and a second text length of a fourth conversation text in the first text are determined; the fourth conversation text is conversation text of the second object in the first text; the second index value is determined based on the conversation response duration of the first object and the second object, the first text length, and the second text length.

12. An electronic device, comprising: including: a memory for storing computer executable instructions; a processor for executing the computer executable instructions stored in the memory, to implement the data processing method in any one of claims 1 to 11.

13. A computer-readable storage medium, characterized in that, computer executable instructions stored in the memory, the computer executable instructions are executed by the processor to implement the data processing method in any one of claims 1 to 11.

Images