Data processing method and related device

Abstract

The invention provides a data processing method and a related device. The method comprises the steps of determining a first index value based on a first text and a second text; the first text is a session text of a first object and a second object generated by a first model, and the second text is a session text of a third object and the second object; the first index value is used for representing the session quality of the first object and the second object; determining a second index value based on the session response duration of the first object and the second object; the second index value is an index value used for representing the session response speed of the first object; determining an evaluation score for the first model based on the first index value and the second index value; and determining a text generation model matched with the first object based on the evaluation score. Through the method, the text generation effect of the text generation model corresponding to the first object can be improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data processing method and related apparatus. Background Technology

[0002] In classification or regression tasks where the text generation performance of large language models is evaluated, a loss function is generally used to measure the difference between the model's predictions and the actual values. This difference is then used as a key indicator for optimizing model parameters. A smaller loss function value indicates that the model's predictions are closer to the actual values, resulting in better performance and, consequently, better text generation. Among the loss functions used in these techniques, Mean Squared Error (MSE), Cross-Entropy Loss, Mean Absolute Error (MAE), and Log-Likelihood Loss all use the actual and predicted values ​​as a benchmark, reflecting the difference between them to some extent through different formulas, thereby guiding the performance optimization of large language models. Summary of the Invention

[0003] This application provides a data processing method and related apparatus that can improve the text generation effect of the data processing model corresponding to the first object.

[0004] The technical solution of this application embodiment is implemented as follows: This application provides a data processing method, the method comprising: determining a first indicator value based on a first text and a second text; the first text being conversation text between a first object and a second object generated by a first model, and the second text being conversation text between a third object and the second object; the first indicator value being an indicator value used to characterize the conversation quality between the first object and the second object; determining a second indicator value based on the conversation response duration between the first object and the second object; the second indicator value being an indicator value used to characterize the conversation response speed of the first object; determining an evaluation score for the first model based on the first indicator value and the second indicator value; and determining a text generation model adapted to the first object based on the evaluation score.

[0005] This application provides a data processing apparatus, comprising: a first determining module, configured to determine a first indicator value based on a first text and a second text; the first text being conversation text between a first object and a second object generated by a first model, and the second text being conversation text between a third object and the second object; the first indicator value being an indicator value characterizing the conversation quality between the first object and the second object; a second determining module, configured to determine a second indicator value based on the conversation response duration between the first object and the second object; the second indicator value being an indicator value characterizing the conversation response speed of the first object; a third determining module, configured to determine an evaluation score for the first model based on the first indicator value and the second indicator value; and a fourth determining module, configured to determine a text generation model adapted to the first object based on the evaluation score.

[0006] This application provides an electronic device, including: a memory for storing computer-executable instructions; and a processor for executing the computer-executable instructions stored in the memory to implement the data processing method provided in this application.

[0007] This application provides a computer-readable storage medium storing computer-executable instructions for implementing the data processing method provided in this application when executed by a processor.

[0008] This application provides a computer program product including executable instructions stored in a computer-readable storage medium. When the processor of an electronic device reads and executes the executable instructions from the computer-readable storage medium, it implements the data processing method provided in this application.

[0009] The embodiments of this application have the following beneficial effects: In the data processing, firstly, based on the conversation text between the first object and the second object generated by the first model, and the conversation text between the third object and the second object, a first indicator value is determined to characterize the conversation quality between the first object and the second object; then, based on the conversation response duration between the first object and the second object, a second indicator value is determined to characterize the conversation response speed of the first object; next, by combining the first indicator value and the second indicator value, an evaluation score for the first model is determined; finally, based on the evaluation score, a text generation model adapted to the first object is determined. Thus, the first indicator value can be used to evaluate the conversation quality between the first and second objects, that is, to evaluate the conversation quality of the first and second objects generated by the first model from the perspective of text content. At the same time, the second indicator value can be used to evaluate the conversation response speed of the first object, that is, to evaluate the instantaneous feedback capability of the first and second objects in the conversation from the perspective of conversation response efficiency. By combining these two indicator values ​​to obtain an evaluation score, the overall performance of the first object in terms of conversation quality and conversation response efficiency can be comprehensively and accurately reflected. Based on this evaluation score, a text generation model suitable for the first object can be determined. This ensures that the determined text generation model not only meets the conversation content requirements of the first object, but also matches the first object's expectations for response timeliness, thereby effectively improving the text generation effect of the text generation model and ensuring that the text generation model can better meet the dual requirements of conversation quality and conversation response efficiency in real-world scenarios. Attached Figure Description

[0010] Figure 1 This is a schematic diagram of the structure of the data processing system provided in the embodiments of this application; Figure 2 This is an optional flowchart illustrating the data processing method provided in an embodiment of this application; Figure 3 This is another optional flowchart illustrating the data processing method provided in the embodiments of this application; Figure 4 This is an optional implementation diagram of determining the value of the third indicator provided in an embodiment of this application; Figure 5 This is another optional implementation diagram of determining the value of the third indicator provided in the embodiments of this application; Figure 6 This is an optional implementation diagram of determining the value of the fourth indicator provided in an embodiment of this application; Figure 7 This is another optional implementation diagram of determining the value of the fourth indicator provided in the embodiments of this application; Figure 8 This is a schematic diagram illustrating the implementation of determining the fifth index value provided in an embodiment of this application; Figure 9This is a flowchart illustrating the evaluation method for the text generation effect of a large language model provided in the embodiments of this application; Figure 10 This is a schematic diagram of the structure of the data processing apparatus provided in the embodiments of this application; Figure 11 This is a schematic diagram of the composition structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0011] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0012] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0013] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.

[0014] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant national laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0015] The core task of large language models is text generation, but it is unreasonable to accurately measure the quality of text generation results from a single dimension. Therefore, a single loss function in related techniques cannot be directly applied to large language models to achieve good text generation results.

[0016] To address at least one problem with related technologies, this application evaluates the text generation performance of a large language model across four dimensions, i.e., assessing the service quality of the intelligent agent, resulting in four evaluation scores. Finally, the four evaluation scores are combined to obtain a total evaluation value (i.e., evaluation score) for the intelligent agent. This total evaluation value is used to determine whether the text generation performance of the large language model or the service quality of the intelligent agent needs optimization. By optimizing the performance of the large language model, the text generation performance of the large language model is improved.

[0017] See Figure 1 , Figure 1 This is a schematic diagram of the architecture of the data processing system 100 provided in this application embodiment. In order to support a data processing application, the terminal 400 connects to the server 200 through the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0018] Terminal 400 sends a data processing request to server 200. Server 200 responds to the data processing request by determining a first metric value based on a first text and a second text; then, determining a second metric value based on the session response duration between the first and second objects; next, determining an evaluation score for the first model based on the first and second metric values; and finally, determining a text generation model adapted to the first object based on the evaluation score. After obtaining the text generation model adapted to the first object, server 200 can also send the model parameters of the text generation model adapted to the first object to terminal 400, so that terminal 400 can output the model parameters of the text generation model adapted to the first object, or perform subsequent text generation operations based on the model parameters of the text generation model adapted to the first object.

[0019] In some embodiments, server 200 may be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Terminal 400 may be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or in-vehicle terminal, but is not limited to these. Terminal 400 and server 200 can be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment.

[0020] The data processing methods provided in the embodiments of this application can be executed by an electronic device, which can be a server or a terminal. That is, the data processing methods in the embodiments of this application can be executed by a server, by a terminal, or by interaction between a server and a terminal.

[0021] See Figure 2 , Figure 2 This is an optional flowchart illustrating the data processing method provided in the embodiments of this application, which will be combined with... Figure 2 The steps shown are explained below, taking the server as the execution subject of the data processing method as an example. The method includes the following steps S101 to S104: Step S101: Determine the first indicator value based on the first text and the second text.

[0022] In this embodiment, the first object refers to one of the subjects participating in the conversation. The conversation performance of the first object (e.g., quality and response speed) is the core object for evaluation and optimization. The first object can be an intelligent system, AI model, or automated tool that needs to implement interactive functions through a data processing model. For example, the first object can be an intelligent customer service model (such as an automatic reply customer service on an e-commerce platform), an intelligent question-and-answer robot (such as a mobile voice assistant), or a tutoring AI in online education. The second object refers to a common subject that interacts with both the first and third objects in the conversation. It is the initiator or requester of the conversation, and the request of the second object is the core objective of the conversation. For example, the second object can be a user consulting intelligent customer service about product issues, a mobile terminal user asking a question to a voice assistant, or a student asking a tutoring AI for help with a problem. The third object refers to a subject that has conversational interaction with the second object, and whose conversation performance can serve as a reference or benchmark for evaluating the conversation quality of the first object. The third object can be a human or system with professional capabilities or standard procedures. For example, the third object can be a human customer service representative on an e-commerce platform, a teacher in the education field, or a doctor in the medical field.

[0023] Conversational text refers to the written records generated during conversations between the first and second parties, and between the third and second parties. It includes questions, replies, instructions, and feedback from both sides, and is the core basis for evaluating conversation quality. The first text is the conversational text between the first and second parties generated by the first model; it is the written record generated during their interaction and serves as direct material for evaluating the conversational quality of the first party. The second text is the conversational text between the third and second parties; it is the written record generated during their interaction and serves as reference material for evaluating the conversational quality of the first party, such as comparing the responses of the first and third parties to the same question. Examples include users being transferred to a human agent when intelligent customer service cannot resolve their issues, patients consulting doctors after communicating with intelligent medical assistants, or students asking teachers follow-up questions after receiving answers from tutoring AI. The first model refers to the algorithmic model that supports the first party's conversational interaction function. It is built based on technologies such as Natural Language Processing (NLP) or machine learning. The first model is used to understand the second party's intent, generate response content, and control the response pace. The first model can generate the conversational text between the first and second parties. For example, the first model could be a dialogue generation model behind an intelligent customer service system or a question-and-answer matching model that guides AI. No specific limitations are placed on the model structure of the first model here.

[0024] The first metric is used to characterize the conversation quality between the first and second objects. It can be determined by analyzing multiple dimensions, such as the content matching degree between the first and second texts (e.g., consistency in answers to the same question) and the content validity of the first text (e.g., semantic completeness and text fluency). The first metric is used to determine the conversation quality between the first and second objects.

[0025] In some embodiments, after obtaining the conversation text between the first object and the second object, and the conversation text between the third object and the second object, the conversation quality between the first object and the second object can be evaluated through multiple dimensions, obtaining evaluation scores under each dimension. Finally, the evaluation scores under multiple dimensions are combined to obtain a first index value used to characterize the conversation quality between the first object and the second object.

[0026] Step S102: Determine the second indicator value based on the session response duration between the first object and the second object.

[0027] In this embodiment, the session response duration refers to the time interval between the first object receiving a message (such as a question or instruction) from the second object and generating and returning a response. It is the core data for measuring the response speed of the first object. For example, if a user sends a message to the intelligent customer service to inquire about order logistics, and the intelligent customer service replies with logistics information after 1.2 seconds, this 1.2 seconds is the session response duration. If the response durations of multiple interactions are 1.2 seconds, 0.8 seconds, and 1.5 seconds respectively, then the average response duration is 1.17 seconds. The second indicator value is used to characterize the session response speed of the first object. The second indicator value can be determined based on the statistical results of the session response duration (e.g., average response duration, longest response duration, or response duration scaling rate). After determining the session response duration between the first object and the second object, the second indicator value used to characterize the session response speed of the first object can be determined based on this session response duration.

[0028] Step S103: Determine the evaluation score for the first model based on the first indicator value and the second indicator value.

[0029] In this embodiment, the evaluation score refers to the score obtained by combining the first indicator value (conversation quality) and the second indicator value (conversation response speed). This score is used to comprehensively evaluate the overall performance of the first model in generating conversational text between the first and second objects, assessing both the effectiveness in terms of conversation quality and the efficiency in terms of response speed. The higher the evaluation score, the better the overall performance of the first model across the aforementioned multiple dimensions. The evaluation score for the first model can be determined based on the first and second indicator values ​​through weighted calculation or other fusion algorithms. For example, if the first indicator value for intelligent customer service is 80 points and the second indicator value is 90 points, and the weights are each 50%, then the evaluation score is (80×50%+90×50%)=85 points.

[0030] Step S104: Based on the evaluation score, determine the text generation model that is suitable for the first object.

[0031] In this embodiment, the text generation model adapted to the first object is one that can meet the first object's requirements for conversation quality and speed in actual conversation scenarios. The text generation model exhibits more suitable performance in dialogue generation tasks related to the first object.

[0032] In some embodiments, depending on whether the first model corresponding to the first object is an open-source large language model, there are two cases: First, when the first model corresponding to the first object is an open-source large language model, after obtaining the evaluation score, if the evaluation score is less than a preset evaluation threshold, it indicates that the text generation effect of the first model corresponding to the first object is poor. Based on the evaluation score, the first model can be fine-tuned, for example, by adjusting the model parameters or model structure. Then, the evaluation score is re-determined using the fine-tuned first model until the re-determined evaluation score is greater than the preset evaluation threshold. In this case, the fine-tuned first model corresponding to the re-determined evaluation score can be identified as the text generation model adapted to the first object. For example, after fine-tuning the intelligent customer service model, the first indicator value increased from 80% to 92%, the second indicator value decreased from 1.5 seconds to 0.8 seconds, the evaluation score increased from 85 points to 96 points, and the user complaint rate decreased by 30%.

[0033] In the second scenario, when the first model corresponding to the first object is a non-open-source large language model, the text generation performance of multiple first models can be evaluated based on the first and second texts, as well as the conversation response time between the first and second objects, to obtain an evaluation score for each first model. Then, the evaluation scores are sorted, and the first model with the highest evaluation score is selected as the text generation model suitable for the first object. For example, if the first object is a medical AI consultation assistant, mainly used to communicate symptoms and answer basic health questions with patients, and multiple candidate first models are medical-specific dialogue models, one of which has an evaluation score of only 75 points, exhibiting inaccurate medical terminology and response delays, a medical-specific dialogue model with an evaluation score of 92 points in this scenario is determined to be the text generation model suitable for the medical AI consultation assistant. This model not only demonstrates high dialogue quality conforming to medical standards but also exhibits fast response speed.

[0034] The data processing method provided in this application embodiment firstly determines a first indicator value to characterize the conversation quality between the first object and the second object based on the conversation text between the first object and the second object generated by the first model, and the conversation text between the third object and the second object; then, it determines a second indicator value to characterize the conversation response speed of the first object based on the conversation response duration between the first object and the second object; next, it combines the first indicator value and the second indicator value to determine an evaluation score for the first model; finally, it determines a text generation model adapted to the first object based on the evaluation score. Thus, the first indicator value can be used to evaluate the conversation quality between the first and second objects, that is, to evaluate the conversation quality of the first and second objects generated by the first model from the perspective of text content. At the same time, the second indicator value can be used to evaluate the conversation response speed of the first object, that is, to evaluate the instantaneous feedback capability of the first and second objects in the conversation from the perspective of conversation response efficiency. By combining these two indicator values ​​to obtain an evaluation score, the overall performance of the first object in terms of conversation quality and conversation response efficiency can be comprehensively and accurately reflected. Based on this evaluation score, a text generation model suitable for the first object can be determined. This ensures that the determined text generation model not only meets the conversation content requirements of the first object, but also matches the first object's expectations for response timeliness, thereby effectively improving the text generation effect of the text generation model and ensuring that the text generation model can better meet the dual requirements of conversation quality and conversation response efficiency in real-world scenarios.

[0035] The data processing method in this application embodiment will be described below in conjunction with the interaction process between the server and the terminal in the data processing system. It should be noted that the data processing method here is implemented through interaction between the server and the terminal, and is essentially the same as the data processing method executed by the server in the above embodiments. The only difference is that this application embodiment also describes the actions performed by the terminal during the execution of the data processing method. Furthermore, some steps can be executed by either the terminal or the server. Therefore, for steps in this embodiment that are the same as those in the above embodiments but have different execution subjects, this embodiment is merely illustrative. In the implementation process, any execution subject can perform the steps, and this application embodiment does not limit this.

[0036] See Figure 3 , Figure 3 This is another optional flowchart illustrating the data processing method provided in the embodiments of this application, such as... Figure 3 As shown, the method includes the following steps S201 to S213: Step S201: The terminal receives data processing operations input by the user.

[0037] In this embodiment, the data processing operation can be an operation input by a user through a client of the data processing system. The user can input the data processing operation on the client side of the data processing system. The data processing system can provide a data processing function page where the user can input the data processing operation. In some embodiments, when inputting the data processing operation, a first text and a second text can also be input simultaneously. The first text is the conversation text between a first object and a second object, and the second text is the conversation text between a third object and a second object.

[0038] In step S202, the terminal generates a data processing request in response to the data processing operation.

[0039] In this embodiment of the application, user-input data can be encapsulated into a data processing request. For example, the first text and the second text input by the user can be encapsulated into a data processing request.

[0040] In step S203, the terminal sends a data processing request to the server.

[0041] In step S204, the server responds to the data processing request and determines a third indicator value based on the first text and the second text.

[0042] In this embodiment of the application, when a data processing request encapsulates a first text and a second text, the data processing request is parsed to directly obtain the first text and the second text. After obtaining the first text and the second text, a third indicator value is determined based on the first text and the second text. The third indicator value is an indicator value used to characterize the identity of the first object.

[0043] In some embodiments, see Figure 4 , Figure 4 The step S204, in which the server determines the third indicator value based on the first text and the second text, can also be achieved through the following steps S2041 to S2042: Step S2041: Determine the first text vector of the first conversation text in the first text, and the second text vector of the second text.

[0044] In this embodiment of the application, the first conversation text is the conversation text belonging to the first object in the first text.

[0045] In some embodiments, the server determines the first text vector of the first session text in the first text, and the second text vector of the second text, which can be achieved in the following way: First, the first session text is segmented to obtain the first word sequence of each sentence in the first session text; then, the third word in the first word sequence is replaced by a preset space character to obtain the second word sequence; finally, the words in the second word sequence are vectorized to obtain the first text vector; the second text is vectorized to obtain the second text vector.

[0046] In this embodiment, word segmentation refers to the process of splitting continuous text into independent words according to certain rules. The first word sequence refers to the ordered set of words formed by each sentence of the first conversation text after word segmentation, preserving the original word order. Preset empty spaces are pre-defined symbols or characters used to replace specific words; the empty spaces themselves have no actual semantic meaning and only serve as placeholders. The third word refers to the word in the first word sequence used to represent the identities of the first and second objects, and is also the target word to be replaced. The second word sequence refers to the new word sequence formed after replacing the third word in the first word sequence with the preset empty spaces, eliminating the interference of identity words on the semantic analysis of the text. Vectorization refers to the process of converting words or text into numerical vectors. The semantic information of words can be quantified into computable vectors using a pre-defined model (e.g., Word2Vec or BERT). The first text vector refers to the vector representing the core semantics of the first conversation text obtained by integrating all word vectors after vectorization of the second word sequence. The second text vector refers to the vector obtained after vectorization of the second text, used for comparative analysis with the first text vector.

[0047] In some embodiments, the first conversation text is segmented sentence by sentence using word segmentation tools such as Jieba. Third words representing the identities of the first and second objects in the first word sequence are identified, and these third words are replaced one by one with preset space characters (e.g., [MASK]) to obtain the second word sequence. Each word in the second word sequence is vectorized using a pre-trained word vector model to obtain a vector for each word; all word vectors are then integrated to obtain the first text vector of the first conversation text. Alternatively, the same word vector model and integration method as the first conversation text can be used to vectorize the word sequence in the second text to obtain the second text vector.

[0048] Here, by segmenting words, replacing the third word representing identity with empty characters to obtain the second word sequence, and then vectorizing it, we can eliminate the interference of identity words on the text vector and focus on the core semantic content of the conversation text. The first and second text vectors obtained in this way can better reflect the substantive semantics of the text, reduce the influence of irrelevant factors on the evaluation, and make the third index value determined based on the vector more in line with the true semantic relationship between the first object's conversation text and the second text, thus improving the accuracy of the evaluation.

[0049] Step S2042: Determine the third index value based on the first text vector and the second text vector.

[0050] In some embodiments, see Figure 5 , Figure 5 The step S2042, in which the server determines the third index value based on the first text vector and the second text vector, can also be achieved through the following steps S501 to S503: Step S501: Determine the first sentence vector of each sentence in the first conversation text from the first text vector.

[0051] In this embodiment, the first sentence vector is extracted from the first text vector and is used to represent the core semantics of each sentence in the first conversational text, accurately reflecting the semantic information of a single sentence in the first conversational text. The first text vector can be divided according to the sentences of the first conversational text, and the word vectors corresponding to each sentence can be integrated to obtain the first sentence vector for each sentence.

[0052] Step S502: Determine the second sentence vector of each sentence in the second conversation text and the third sentence vector of each sentence in the third conversation text from the second text vector.

[0053] In this embodiment, the second sentence vector is extracted from the second text vector and is used to represent the core semantics of each sentence in the second conversational text, accurately reflecting the semantic information of a single sentence in the second conversational text. The third sentence vector is extracted from the third text vector and is used to represent the core semantics of each sentence in the third conversational text, accurately reflecting the semantic information of a single sentence in the third conversational text. The second conversational text is the conversational text belonging to the third object within the second text, and the third conversational text is the conversational text belonging to the second object within the second text. Text vectors corresponding to the second and third conversational texts can be extracted from the second text vector. These text vectors are then divided by sentence, and the word vectors corresponding to each sentence are integrated to obtain the second sentence vector and the third sentence vector.

[0054] Step S503: Determine the third index value based on the first sentence vector, the second sentence vector, and the third sentence vector.

[0055] In some embodiments, the server determines a third index value based on a first sentence vector, a second sentence vector, and a third sentence vector, which can be achieved in the following way: First, a first similarity is determined based on the first sentence vector and the second sentence vector, and a second similarity is determined based on the first sentence vector and the third sentence vector; then, a first number of third words belonging to a first type and a second number of third words belonging to a second type in the first conversation text are determined; finally, the third index value is determined based on the first similarity, the second similarity, the first number, and the second number.

[0056] In this embodiment, the first similarity refers to the semantic similarity between the first sentence vector and the second sentence vector, used to measure the semantic matching degree between the first object and the third object in the corresponding sentences, usually represented by a value between 0 and 1, with the closer the value is to 1, the higher the similarity. The second similarity refers to the semantic similarity between the first sentence vector and the third sentence vector, used to measure the semantic matching degree between the first object and the second object in the corresponding sentences, also represented by a value between 0 and 1. Third words belonging to the first type are words used to characterize the identity of the first object, and third words belonging to the second type are words used to characterize the identity of the second object. The first type refers to the category of third words used to characterize the identity of the first object, including all words that can identify the identity of the first object. The second type refers to the category of third words used to characterize the identity of the second object, including all words that can identify the identity of the second object. The first quantity refers to the total number of third words belonging to the first type in the first conversation text, used to reflect the frequency of the characterization of the first object's identity. The second quantity refers to the total number of third words belonging to the second type in the first conversation text, used to reflect the frequency of the characterization of the second object's identity. The third indicator value is a comprehensive indicator value obtained by combining the first similarity, the second similarity, the first quantity, and the second quantity, used to characterize the accuracy of the first object's identity representation and semantic matching degree.

[0057] In some embodiments, in the first conversation text, the number of words matching the agent identity lexicon (i.e., the first number) and the number of words matching the user identity lexicon (i.e., the second number) are determined. The similarity between the first sentence vector and the second sentence vector is calculated and summed to obtain the first similarity; similarly, the similarity between the first sentence vector and the third sentence vector is calculated and summed to obtain the second similarity. The first similarity is divided by the second similarity to obtain a ratio. The ratio of the number of identity words is then calculated, i.e., the first number divided by the sum of 1 and the second number, to obtain another ratio. Multiplying these two ratios yields the third index value.

[0058] Here, by combining the similarity between the first sentence vector and the second and third sentence vectors, as well as the number of third words in the first and second types, we can comprehensively quantify the matching degree between the conversation content of the first object and its own identity. We can assess whether the conversation is consistent with the identity, whether the answer is irrelevant, and how close it is to the human agent through semantic similarity, and reflect the rationality of the identity representation by the number of identity words. This allows the third indicator value to objectively and accurately reflect the rationality of the intelligent agent's identity, providing a reliable basis for evaluating and optimizing the conversation performance of the first object.

[0059] Here, by extracting sentence vectors from text vectors, we can distinguish the sentence vectors corresponding to the conversational texts of different objects in the second text, and perform analysis based on the sentence vectors, thus achieving a refined decomposition of text semantics. This sentence-level vector comparison can capture the semantic relationship between the first object and the third object, as well as the second object, in more detail on specific sentences, avoiding the generality that may exist in the overall text vector analysis, making the determination of the third indicator value more targeted and accurate, and thus more accurately assessing the rationality of the first object's identity.

[0060] Here, the third indicator value is determined by identifying the first text vector of the first conversation text and the second text vector of the second text. Converting the text into vector form for analysis allows for the precise capture of semantic information through the mathematical properties of vectors, avoiding the subjectivity and ambiguity of directly evaluating based on text content. By measuring the association between the first object's conversation text and the second text through vector-level operations, the semantic matching degree between the two can be reflected more objectively. This enables the third indicator value to effectively characterize the rationality of the first object's identity, providing a scientific quantitative basis for subsequent evaluation and optimization.

[0061] In step S205, the server determines the fourth indicator value based on the first text and the second text.

[0062] In this embodiment of the application, the fourth index value is an index value used to characterize the semantic completeness of the conversational text belonging to the first object in the first text. Semantic completeness refers to the completeness and comprehensiveness of the conversational content of the first object in the first text in expressing intent and conveying information. Specifically, it is reflected in the completeness of the semantics of a single sentence (e.g., whether the core meaning is clearly expressed or whether information is omitted) and the coherence of the context (e.g., whether the logic between sentences is smooth or whether a complete semantic chain can be formed).

[0063] In some embodiments, see Figure 6 , Figure 6 The step S205, in which the server determines the fourth indicator value based on the first text and the second text, can also be achieved through the following steps S2051 to S2054: Step S2051: Mask the first conversation text in the first text to obtain the first masked text.

[0064] Step S2052: Mask the sentence groups in the first text to obtain the second masked text.

[0065] Step S2053: Mask the second conversation text in the second text to obtain the third masked text.

[0066] In this embodiment, the first conversation text is the conversation text belonging to the first object within the first text, and the second conversation text is the conversation text belonging to the third object within the second text. Masking refers to replacing specific words, phrases, or sentences in the text with a preset symbol [MASK] to highlight the core semantic structure or eliminate irrelevant information interference, used to focus on the analysis of key text content. The first masked text is the text obtained after masking the first conversation text belonging to the first object within the first text, used to preserve the core semantics of the first object's conversation and eliminate variable interference information. A sentence group refers to a set of adjacent sentences in the first text, including the first sentence and the second sentence adjacent to it, used to analyze contextual coherence. The second masked text is the text obtained after masking the sentence group in the first text, used to focus on the contextual semantic relationship of the sentence group and eliminate variable interference information. The third masked text is the text obtained after masking the second conversation text belonging to the third object within the second text. The second conversation text has complete semantic meaning and can be used as a reference for the conversation text of the first object for comparative analysis of semantic completeness.

[0067] In some embodiments, for the first conversation text, sentence groups, and second conversation text, corresponding mask objects are determined, i.e., variable interference information in the text, such as usernames or email addresses. Preset mask symbols are selected, such as [MASK], to ensure that the mask symbols have no actual semantic meaning and are consistent, facilitating subsequent text analysis. The first conversation text, sentence groups, and second conversation text are traversed sentence by sentence to identify the mask objects to be masked, and these mask objects are replaced with the set mask symbols. After replacing all mask objects, the processed sentences are integrated to form complete masked text, resulting in the first masked text, the second masked text, and the third masked text. The text structure of the masked text is ensured to be consistent with the original text, only replacing the mask objects.

[0068] Step S2054: Determine the fourth index value based on the first mask text, the second mask text, and the third mask text.

[0069] In this embodiment of the application, the fourth index value is determined by comparing the semantic matching degree of the first mask text, the second mask text and the third mask text, reflecting the completeness of the expression of the first object.

[0070] In some embodiments, see Figure 7 , Figure 7 The step S2054, in which the server determines the fourth index value based on the first mask text, the second mask text, and the third mask text, can also be achieved through the following steps S701 to S704: Step S701: Encode the first mask text to obtain a first encoding vector; encode the second mask text to obtain a second encoding vector; encode the third mask text to obtain a third encoding vector.

[0071] In this embodiment, encoding processing refers to the process of converting masked text into a fixed-dimensional numerical vector. A pre-trained model (e.g., BERT or GPT) captures the semantic features of the masked text, enabling mathematical operations and comparative analysis. The first encoded vector is obtained by encoding the first masked text and is used to quantify the core semantic information of the first masked text. The second encoded vector is obtained by encoding the second masked text and is used to quantify the contextual semantic features of the second masked text. The third encoded vector is obtained by encoding the third masked text and is used to quantify the core semantic information of the third masked text, serving as a reference for the conversational text of the first object.

[0072] In some embodiments, a pre-trained text encoding model can be selected to ensure that the model output vector dimension is uniform. The masked text (first masked text, second masked text, and third masked text) is formatted according to the model requirements, for example, by adding start characters [CLS] and end characters [SEP]. The formatted text is input into the model, and the encoding vectors output by the last layer of the model are extracted and used as the first encoding vector, the second encoding vector, and the third encoding vector, respectively.

[0073] Step S702: Determine the first sentence class vector based on the sentence type of each sentence in the first masked text; determine the second sentence class vector based on the sentence type of each sentence in the second masked text; determine the third sentence class vector based on the sentence type of each sentence in the second masked text.

[0074] In this embodiment, sentence type refers to the classification of a sentence's function or purpose in a conversation. Sentence types include interrogative sentences, declarative sentences, imperative sentences, and response sentences, used to distinguish the expressive intent of sentences. The first sentence class vector is a vector generated based on the sentence type of each sentence in the first masked text, used to quantitatively represent the distribution characteristics of sentence types in the first masked text. The second sentence class vector is a vector generated based on the sentence type of each sentence in the second masked text, used to quantitatively represent the sequence characteristics of sentence types in a group of sentences. The third sentence class vector is a vector generated based on the sentence type of each sentence in the third masked text, used to quantitatively represent the distribution characteristics of sentence types in the third masked text.

[0075] In some embodiments, a predefined mapping relationship between sentence types and numerical values ​​is established. For example, a question is mapped to 0, a statement to 1, an imperative to 2, and a response to 3. The first, second, and third masked texts are traversed, and the sentence type of each sentence in the masked text is determined and converted into a corresponding numerical value, forming a type sequence. Finally, through embedding layer mapping or extended padding, the type sequence is converted into a fixed-dimensional vector, resulting in the first sentence class vector, the second sentence class vector, and the third sentence class vector.

[0076] Step S703: Determine the third text vector based on the first encoding vector and the first sentence class vector; determine the fourth text vector based on the second encoding vector and the second sentence class vector; determine the fifth text vector based on the third encoding vector and the third sentence class vector.

[0077] In this embodiment, the third text vector is obtained by fusing the first encoding vector and the first sentence class vector, integrating the semantic information and sentence type features of the first masked text. The fourth text vector is obtained by fusing the second encoding vector and the second sentence class vector, integrating the contextual semantic association and sentence type sequence features of the second masked text. The fifth text vector is obtained by fusing the third encoding vector and the third sentence class vector, integrating the semantic information and sentence type features of the third masked text.

[0078] Step S704: Determine the fourth index value based on the third text vector, the fourth text vector, and the fifth text vector.

[0079] In some embodiments, the server determines the fourth index value based on the third text vector, the fourth text vector, and the fifth text vector, which can be achieved by: first, determining the third similarity based on the third text vector and the fifth text vector, and then determining the fourth similarity based on the fourth text vector and the fifth text vector; and then determining the fourth index value based on the third similarity and the fourth similarity.

[0080] In this embodiment, the third similarity refers to the semantic similarity between the third and fifth text vectors. It measures the matching degree between the core semantics of the first object's conversational text and the core semantics of the third object's conversational text, and is usually represented by a value between 0 and 1, with the closer the value is to 1, the higher the similarity. Furthermore, since the third object's conversational text is semantically complete, the higher the third similarity, the more complete the meaning of the first object's conversational text. The fourth similarity refers to the semantic similarity between the fourth and fifth text vectors. It measures the matching degree between the contextual semantic association of the first object's conversational text and the core semantics of the third object's conversational text, and is also represented by a value between 0 and 1. The higher the fourth similarity, the better the contextual coherence of the first object's conversational text. The fourth index value is obtained by combining the third and fourth similarities and is used to quantify the semantic completeness of the first object's conversational text, reflecting the comprehensive performance of the first object's expression in terms of single-sentence semantic completeness and contextual coherence.

[0081] In some embodiments, a cosine similarity algorithm can be used to calculate the third similarity between the third and fifth text vectors, and the fourth similarity between the fourth and fifth text vectors. The weights of the third and fourth similarities are set according to the evaluation requirements; if the emphasis is on single-sentence semantic matching, the third similarity has a higher weight; if the emphasis is on contextual coherence, the fourth similarity has a higher weight. The third and fourth similarities are then weighted and summed to obtain the fourth index value.

[0082] Here, the third and fourth similarities are determined based on the third, fourth, and fifth text vectors, and then the fourth index value is determined. By quantifying the semantic association between different text vectors, the degree of matching between the first object's verbal text and the third object's verbal text and user-related content is intuitively reflected. This similarity-based evaluation method transforms semantic completeness into a calculable quantitative index, avoids the bias of subjective judgment, and enables the fourth index value to objectively reflect the completeness of the first object's expression, providing a clear basis for optimizing the first object's conversational performance.

[0083] Here, the masked text is encoded to obtain an encoding vector, which is then combined with the sentence type to determine the sentence class vector. These vectors are then fused to obtain a text vector used to determine the fourth indicator value. This achieves multi-dimensional feature extraction from the surface to the deep semantics of the text. The encoding vector captures the semantic information of the text, while the sentence class vector reflects the functional attributes of the sentence. The combination of the two makes the text vector more comprehensively represent the text features, improves the depth and richness of the evaluation of the fourth indicator value, and makes the evaluation of the semantic integrity of the first object more accurate and reliable.

[0084] Here, by masking the first conversation text, sentence groups, and second conversation text respectively to obtain the corresponding masked text, and then determining the fourth indicator value based on these masked texts, we can focus on the core semantic structure of the text and eliminate interference from irrelevant information. By comparing the spoken texts of the first and third objects, we evaluate the semantic integrity from the dimensions of single sentence and contextual coherence, making the evaluation more in line with the actual expression scenario. This provides an intuitive and targeted analytical basis for determining the fourth indicator value and helps to accurately measure the semantic integrity of the first object's conversation text.

[0085] In step S206, the server determines the fifth index value based on the first word in the first text and the second word adjacent to the first word.

[0086] In this embodiment, the fifth index value is used to characterize the text fluency of the conversational text belonging to the first object in the first text. Text fluency refers to the smoothness of language expression, semantic coherence, and logical naturalness of the conversational text belonging to the first object in the first text, specifically reflected in the tightness of semantic connections between adjacent words and the overall smoothness of expression.

[0087] In some embodiments, see Figure 8 , Figure 8 The step S206, in which the server determines the fifth index value based on the first word in the first text and the second word adjacent to the first word, can also be achieved through the following steps S2061 to S2063: Step S2061: Determine the first mapping point corresponding to the first word in the first coordinate system, and the second mapping point corresponding to the second word in the first coordinate system.

[0088] In this embodiment, the first word refers to any word in the conversational text belonging to the first object in the first text. The first coordinate system is a high-dimensional vector coordinate system used to map the word vectors of words to spatial points. The first mapping point is the spatial point corresponding to the word vector of the first word in the first coordinate system, and the coordinate value of the first mapping point can be the word vector value of the word, which quantifies the semantic information of the first word. The second word refers to the word adjacent to the first word, including words adjacent before and after it. The second mapping point is the spatial point corresponding to the word vector of the second word in the first coordinate system, and the coordinate value of the second mapping point is the word vector value of the word, used to calculate the gradient change with the first mapping point.

[0089] In some embodiments, the conversational text belonging to the first object in the first text can be segmented sentence by sentence to obtain a word sequence containing the first word and its adjacent second words. The segmented word sequence is input into a large language model to obtain the word vector of each word, wherein the word vector dimension of the first word and the second word is the same as the dimension of the first coordinate system. The word vector value of the first word is used as the coordinate value to find the corresponding spatial point in the first coordinate system, which is the first mapping point; similarly, the word vector value of the second word is used as the coordinate value to find the corresponding spatial point in the first coordinate system, which is the second mapping point.

[0090] Step S2062: Based on the coordinate values ​​of the first mapping point and the second mapping point, determine the sum of gradient changes between the first mapping point and the second mapping point.

[0091] In this embodiment, the sum of gradient changes between the first mapping point and the second mapping point refers to the sum of the absolute values ​​(or sum of squares, etc.) of the coordinate differences between the first mapping point and the second mapping point in each dimension in the first coordinate system. This is used to measure the semantic closeness and variation of adjacent words, thereby reflecting the fluency of the text.

[0092] In some embodiments, the coordinate values ​​of the first mapping point and the second mapping point in each dimension of the first coordinate system can be extracted respectively. For the case of three adjacent words, first take the first mapping point and the second mapping point corresponding to the first two words and calculate the gradient change between them; then take the mapping point corresponding to the last two words and calculate the gradient change between them in the same way; finally, add the two gradient changes together to get the sum of the gradient changes between the mapping points corresponding to the three adjacent words.

[0093] Step S2063: Determine the value of the fifth index based on the sum of gradient changes.

[0094] In this embodiment, the smaller the sum of gradient changes, the closer the semantic relationship between adjacent words, and the higher the text fluency; conversely, the lower the text fluency. If there is a combination of three adjacent words, the sum of gradient changes of the mapping points of the three adjacent words is calculated, and the reciprocal of the sum of gradient changes is determined as the fifth index value; if there are multiple combinations of three adjacent words, the average of the reciprocals of the sum of gradient changes of all combinations is calculated, and this average value is determined as the fifth index value.

[0095] Here, text fluency is evaluated by transforming words into mapping points in a high-dimensional coordinate system and calculating the sum of gradient changes between adjacent mapping points. This objectively and quantitatively reflects the fluency of the first-object conversation text from a semantic perspective, avoiding the subjective arbitrariness of manual evaluation. By leveraging the characteristics of high-dimensional vector space, the semantic relationships between words can be accurately captured, making the fifth indicator value more realistically reflect the degree of text fluency. Moreover, this method has clear steps, strong operability, and is applicable to text fluency evaluation in various scenarios. It provides a reliable basis for the subsequent determination of the first indicator value and the optimization of the data processing model, which is conducive to improving the expressive effect of the first-object conversation text and the user's interactive experience.

[0096] In step S207, the server determines the average of the third, fourth, and fifth indicator values ​​as the first indicator value.

[0097] In this embodiment of the application, after obtaining the third, fourth, and fifth index values, the average value of the third, fourth, and fifth index values ​​is calculated, and the average value is determined as the first index value.

[0098] Here, by breaking down the conversation quality assessment into multiple dimensions, precise quantification of the first subject's interactive performance is achieved. A third indicator assesses the legitimacy of the interacting subject's identity, determining whether the first subject's conversation content aligns with their identity, whether their answers are irrelevant, and whether their conversational performance closely resembles that of the third subject. A fourth indicator focuses on content quality, prompting the model to provide more comprehensive information. A fifth indicator optimizes language expression, enhancing the naturalness of human-computer dialogue. A comprehensive average balances multiple indicators, avoiding the bias of a single dimension and making the assessment results more objective and comprehensive. This hierarchical assessment mechanism not only identifies model defects but also guides targeted optimization in the future, ultimately improving overall conversation quality and user experience.

[0099] In step S208, the server determines the first text length of the first session text in the first text and the second text length of the fourth session text in the first text.

[0100] In this embodiment of the application, the first conversation text is the conversation text belonging to the first object in the first text, and the first text length refers to the total number of characters or words in the conversation text belonging to the first object in the first text. The first text length can be used to measure the amount of information in the reply from the first object; the fourth conversation text is the conversation text belonging to the second object in the first text, and the second text length refers to the total number of characters or words in the conversation text belonging to the second object in the first text. The second text length can be used to measure the complexity of the user's question.

[0101] In some embodiments, the first text can be split into alternating speeches by a first object and a second object according to the dialogue rounds, and the party to which each round of speech belongs can be marked. The total length of the speeches marked as belonging to the first object in each round of speech is counted to obtain the length of the first text, and the total length of the speeches marked as belonging to the second object in each round of speech is counted to obtain the length of the second text.

[0102] Step S209: The server determines the second indicator value based on the session response duration between the first object and the second object, the length of the first text, and the length of the second text.

[0103] In this embodiment of the application, the second index value is an index value used to quantify the session response efficiency of the first object. The second index value comprehensively considers the session response duration between the first object and the second object, the amount of information in the reply from the first object, and the complexity of the problem from the second object.

[0104] In some embodiments, after determining the first and second text lengths of each round of speech, the sum of the first and second text lengths of each round of speech can be determined as the total text length of each round of speech. For each round, the single-round response time rate of that round is obtained by dividing the session response duration of that round by the total text length of that round. The average of the single-round response time rates of all rounds is determined as the second metric value.

[0105] Here, by calculating the sum of the lengths of the first and second texts for each round, and combining this with the conversation response time for the corresponding round to obtain the single-round response time rate, and then taking the average to determine the second indicator value, we can more accurately reflect the response efficiency of the first party in each round of dialogue, avoiding overall evaluation bias caused by large differences in the length of dialogue content and response time in different rounds. At the same time, by comprehensively considering the total length of the dialogue text and the response time of both parties in each round, we can more comprehensively reflect the first party's real-time response capability when handling dialogues of different lengths and complexities, making the second indicator value more objective and targeted. This provides a more reliable basis for determining the evaluation score based on this indicator value and optimizing the data processing model, thereby improving the model's adaptability and response effect in real dialogue scenarios.

[0106] In step S210, the server determines the evaluation score of the first object based on the first indicator value and the second indicator value.

[0107] In this embodiment of the application, the specific meaning and implementation of the step of determining the evaluation score of the first object based on the first index value and the second index value can be found in the description of step S103 above, and will not be repeated here.

[0108] In step S211, the server determines the text generation model that is suitable for the first object based on the evaluation score.

[0109] In this embodiment of the application, the specific meaning and implementation of the step of determining the text generation model that is suitable for the first object based on the evaluation score can be found in the description of step S104 above, and will not be repeated here.

[0110] In step S212, the server sends the model parameters of the text generation model to the terminal.

[0111] Step S213: The terminal outputs the model parameters of the text generation model.

[0112] In this embodiment, a comprehensive and objective evaluation of the first object's conversational performance is achieved by integrating identity rationality assessment, semantic integrity assessment, text fluency assessment, and response speed assessment through a multi-dimensional quantitative indicator system. For identity rationality assessment, the semantic matching degree and the number of identity words are used to accurately determine the degree of fit between the first object's conversational content and its own identity, as well as its similarity to the conversational content of the third object. For semantic integrity assessment, masked text and vector analysis are used to measure the completeness of the first object's conversational content expression from the dimensions of single-sentence semantics and contextual coherence. For text fluency assessment, high-dimensional vector gradient changes are used to quantify the semantic association between adjacent words and the fluency of sentences in the first object's conversational content. For response speed assessment, the ratio of response time to text length reflects the first object's processing efficiency. These assessments complement each other, avoiding the subjectivity of human judgment while capturing subtle features in the first object's conversation. This provides a scientific basis for optimizing the first object's conversation model and improving the interactive experience, thereby enabling the first object to continuously approach human-level performance in terms of semantic accuracy, expression completeness, communication fluency, and response timeliness. The text generation effect of the data processing model corresponding to the first object is greatly improved.

[0113] The data processing method provided in this application can be applied to various scenarios. For example, in scenario one: intelligent customer service, the first object corresponds to the intelligent customer service model, the second object is the user, and the third object is the human customer service representative. By analyzing the conversation text between the intelligent customer service representative and the user, as well as the conversation text between the human customer service representative and the user, the rationality of the intelligent customer service representative's identity, semantic completeness, text fluency, and response speed are evaluated. After obtaining a comprehensive evaluation score, the intelligent customer service model is optimized to improve its service quality and response speed. In scenario two: intelligent question-and-answer assistant, the first object corresponds to the question-and-answer assistant model, the second object is the user, and the third object is the domain expert. Using the conversation text between the question-and-answer assistant and the user, as well as the supplementary conversation text between the expert and the user, the rationality of the question-and-answer assistant's identity, semantic completeness, text fluency, and response speed are evaluated. After obtaining a comprehensive evaluation score, the question-and-answer assistant model is optimized based on the comprehensive evaluation score to make it more accurate and faster in answering user questions. In scenario three: online education and tutoring, the first object corresponds to the tutoring robot model, the second object is the student, and the third object is the teacher. By analyzing the learning conversation texts between the tutoring robot and students, as well as the after-class communication texts between teachers and students, the tutoring robot's identity rationality, semantic completeness, text fluency, and responsiveness are evaluated. After obtaining a total evaluation score, the tutoring robot model is optimized based on the total evaluation score to make it more in line with students' learning needs and improve tutoring efficiency and effectiveness.

[0114] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0115] This application provides a method for evaluating the text generation performance of a large language model. This method evaluates the text generation performance of the large language model from four dimensions, i.e., evaluating the service quality of the intelligent agent, resulting in four evaluation scores. Finally, the four evaluation scores are combined to obtain a total evaluation value for the intelligent agent. This total evaluation value is used to determine whether it is necessary to optimize the text generation performance of the large language model or the service quality of the intelligent agent. A flowchart of this method is shown below. Figure 9 As shown.

[0116] Step S901: Obtain the historical conversation text between the intelligent agent and the user in the customer service scenario (i.e., the first text mentioned above), and the historical conversation text between the human agent and the user (i.e., the second text mentioned above).

[0117] Step S902, Identity Reasonability Assessment.

[0118] First, identity word detection is performed. The text of the intelligent agent to be detected (i.e., the first conversation text mentioned above) is extracted from historical conversation text. The text is then segmented sentence by sentence, and whole-word matching is used to recall words indicating identity, such as titles, positions, and relationships (i.e., the third type of words mentioned above), for example, company name, employee ID, and colleagues. It is important to note that an agent identity lexicon and a user identity lexicon need to be built in advance for the recall process. Next, identity phrase or short sentence detection is performed. Phrases or short sentences after removing identity words from the intelligent agent text are compiled into a dataset `sets_auto_seats` (assuming there are m entries), and their sentence vectors are obtained for each entry (i.e., the first sentence vector mentioned above). Specifically, deleted positions are replaced with the same empty space vector. Finally, user text (i.e., the second conversation text mentioned above) is extracted from historical conversation text. The complete user text is compiled into a dataset `sets_customer` (assuming there are n entries), and its sentence vectors are obtained for each entry (i.e., the second sentence vector mentioned above). Extract the human agent text from the historical conversation text (i.e., the third conversation text mentioned above), and form a dataset sets_seats (assuming there are o texts) from the complete human agent texts. Then, obtain the sentence vector of each text (i.e., the third sentence vector mentioned above). Pair up each sentence vector in sets_auto_seats (assuming there are m texts) and sets_seats (assuming there are o texts), and calculate the vector loss sum (i.e., the first similarity mentioned above) using the following formula (1). Pair up each sentence vector in sets_auto_seats (assuming there are m texts) and sets_customer (assuming there are n texts), and calculate the vector loss sum (i.e., the second similarity mentioned above) using the following formula (2): (1) (2) in, This represents the vector of the i-th sentence corresponding to sets_auto_seats. This represents the vector of the p-th sentence corresponding to sets_seats. This represents the vector of the j-th sentence corresponding to sets_customer. ∑ represents the accumulation function.

[0119] Finally, the identity rationality assessment is carried out according to the following formula (3) to obtain the identity rationality assessment score (i.e. the third indicator value mentioned above).

[0120] (3) in, This indicates the score for assessing the legitimacy of the smart agent's identity. This represents the number of words in the intelligent agent's text that can be matched with words in the agent's identity dictionary (i.e., the first number mentioned above). This represents the number of words in the intelligent agent's text that can match words from the user identity dictionary (i.e., the second quantity mentioned above). As the formula shows, the more words in the intelligent agent's text that can match words from the agent's identity dictionary than words from the user's identity dictionary, the higher the identity legitimacy assessment score. Furthermore, the higher the similarity ratio between the intelligent agent's text (excluding identity terms) and the human agent's text, compared to the similarity ratio between the intelligent agent's text (excluding identity terms) and the user's text, the higher the intelligent agent's identity legitimacy assessment score.

[0121] Step 903, semantic integrity assessment.

[0122] First, replace the specific variable information in the semantically complete text of the human agent's text with a mask. For example, the original human agent's text is: "Are you Ms. Bai**?" The standardized template is: "Are you Ms. [Mask]?" Then, extract the set of standardized templates (i.e., the third masked text mentioned above), denoted as `template_sets`, and perform template encoding on `template_sets`. For the template encoding method, at the semantic level, the elements in `template_sets` are directly input into any large language model, and the sentence vector function is called to obtain the sentence vector of `template_sets`, where `[Mask]` is treated as an ordinary empty space. The obtained encoding result is denoted as... At the sentence class level, using a full match against the lexicon and punctuation library, words or punctuation marks that can indicate sentence classes in the human agent's text are matched and presented in list form. Based on the principle of highest frequency, the sentence class of the template_sets is determined, and the corresponding vector representation (i.e., the third sentence class vector mentioned above) is obtained from the sentence class vector dictionary using the sentence class name, denoted as... .

[0123] All sentence class names are obtained, which can include declarative sentences, interrogative sentences (yes / no / specific / alternative / positive / negative), imperative sentences, and exclamatory sentences. These sentence class names are then input into any large language model to obtain their word vector representations. Since there are hierarchical relationships between interrogative sentences, yes / no questions, specific questions, alternative questions, and positive / negative questions, the word vectors of interrogative sentences / yes / no / specific / alternative questions / positive / negative questions are multiplied by the word vectors of the interrogative sentences to obtain the final sentence class vector representations for these four types of interrogative sentences. The mapping relationship between all sentence classes and sentence class vector representations can be stored in a sentence class vector dictionary. Next, template encoding merging is performed to obtain the encoding result. (i.e., the third encoding vector mentioned above).

[0124] Similarly, the specific variable information in the intelligent agent's text is replaced with a mask, and a set of standardized templates (i.e., the first masked text mentioned above) is extracted and denoted as intelligence_frame_sets. Template encoding is then performed, and the encoding result is denoted as... (i.e., the first encoded vector mentioned above), serving as single-sentence comparison data; combining the intelligent agent's text with its context sentence (which could be either user text or intelligent agent text) to form sentence groups; replacing the specific variable information in the sentence groups with a mask; extracting the set of standardized templates (i.e., the second masked text mentioned above), denoted as intelligence_frame_sets, and performing template encoding, the encoding result is denoted as... (i.e., the second encoding vector mentioned above), as comparison data for joint sentences.

[0125] Finally, the semantic integrity assessment is performed according to the following formula (4) to obtain the semantic integrity assessment score (i.e. the fourth indicator value mentioned above).

[0126] (4) in, This indicates the semantic completeness assessment score of the smart agent. and These are two different preset weight values. express The i-th vector in express The p-th vector in express The j-th vector in the array.

[0127] Step S904, text fluency evaluation. First, the intelligent agent text is segmented sentence by sentence and fed into the large language model to obtain the word vector sequence of each sentence in the intelligent agent text; then, the word vector of each word in the word vector sequence is mapped to a high-dimensional vector coordinate system and represented by a point. Next, the sum of gradients between the mapping points of the word vectors of three adjacent words in the high-dimensional vector coordinate system (i.e., the sum of the gradient changes mentioned above) is calculated according to the following formula (5).

[0128] (5) in, , and These represent the coordinate values ​​of the mapped point in the x-axis, y-axis, and z-axis directions, respectively. Represent a scalar function. Represents scalar functions The Laplace operator can also be abbreviated as , , and These are three spatial coordinates in a spherical coordinate system. scalar function The second-order partial derivatives with respect to x, y, and z in a rectangular coordinate system reflect the rate of change of the quantity along the x, y, and z axes. express radial coordinates in spherical coordinate system The first-order partial derivative reflects Rate of change along the radial direction; Indicates to " Multiply The radial first-order partial derivative is then used to calculate the radial partial derivative, which is then used to correct for radial variations in spherical coordinates. express Polar angle in spherical coordinates The first-order partial derivative reflects The rate of change along the polar angle; Indicates to " Multiply The first-order partial derivative of the polar angle is used to calculate the partial derivative of the polar angle, which is then used to correct for changes in the polar angle direction in spherical coordinates. express Aspect of azimuth in spherical coordinates The second-order partial derivative reflects The rate of change of the rate of change along the azimuth direction; This represents the correction factor for the azimuth direction in spherical coordinates, due to the polar angle. Changes will affect the measurement scale of the azimuth direction. In short, the left side of formula (5) is the expression for the Laplacian operator in Cartesian coordinates, and the right side is its equivalent transformation form in spherical coordinates. This coordinate transformation achieves the calculation conversion of the same physical quantity in different coordinate systems, facilitating calculations in different scenarios. The spatial variation characteristics are analyzed. Finally, the text fluency is evaluated using the following formula (6), and the text fluency evaluation score is obtained. (i.e., the fifth indicator value mentioned above).

[0129] (6) Step S905, Response speed assessment. The response speed assessment score is a direct indicator that most effectively impacts user experience, regardless of the specific content of the conversation. Response speed can be calculated from the end of the user's speech. From the beginning, until the smart agent started speaking. To eliminate the impact of text length on the response time assessment score, the user text length is adjusted according to the time difference. (i.e., the second text length mentioned above) and the text length of the intelligent agent The sum of (i.e., the length of the first text mentioned above) is used as the denominator, as shown in the following formula (7): (7) in, This represents the evaluation score for the responsiveness of the smart agent (i.e., the second indicator value mentioned above).

[0130] Step S906: Based on the above evaluation results, calculate the performance score of the large language model. (Large Language Model Performance Score) (That is, the above assessment score) can be calculated using the following formula (8): (8) This application provides a new approach to improving the text generation performance of large language models by introducing the evaluation of large language model text generation performance into the intelligent customer service scenario and conducting evaluations of text generation performance across multiple dimensions.

[0131] The following description continues to illustrate the exemplary structure of the data processing apparatus 455 provided in the embodiments of this application as a software module. In some embodiments, such as Figure 10 As shown, the software modules in the data processing device 455 may include: a first determining module 4551, used to determine a first indicator value based on a first text and a second text; the first text is conversation text between a first object and a second object generated by a first model, and the second text is conversation text between a third object and the second object; the first indicator value is an indicator value used to characterize the conversation quality between the first object and the second object; a second determining module 4552, used to determine a second indicator value based on the conversation response duration between the first object and the second object; the second indicator value is an indicator value used to characterize the conversation response speed of the first object; a third determining module 4553, used to determine an evaluation score for the first model based on the first indicator value and the second indicator value; and a fourth determining module 4554, used to determine a text generation model adapted to the first object based on the evaluation score.

[0132] In some embodiments, the first determining module 4551 is further configured to: determine a third indicator value based on the first text and the second text; the third indicator value is an indicator value used to characterize the identity of the first object; determine a fourth indicator value based on the first text and the second text; the fourth indicator value is an indicator value used to characterize the semantic integrity of the first conversational text in the first text; the first conversational text is the conversational text in the first text belonging to the first object; determine a fifth indicator value based on a first word in the first text and a second word adjacent to the first word; the fifth indicator value is an indicator value used to characterize the text fluency of the first conversational text; and determine the average of the third indicator value, the fourth indicator value, and the fifth indicator value as the first indicator value.

[0133] In some embodiments, the first determining module 4551 is further configured to: determine a first text vector of the first conversation text and a second text vector of the second text; and determine the third index value based on the first text vector and the second text vector.

[0134] In some embodiments, the first determining module 4551 is further configured to: perform word segmentation on the first conversation text to obtain a first word sequence for each sentence in the first conversation text; replace the third word in the first word sequence with a preset empty character to obtain a second word sequence; the third word is a word used to characterize the identity of the first object and the second object; perform vectorization on the words in the second word sequence to obtain the first text vector; and perform vectorization on the second text to obtain the second text vector.

[0135] In some embodiments, the first determining module 4551 is further configured to: determine a first sentence vector for each sentence in the first conversation text from the first text vector; determine a second sentence vector for each sentence in the second conversation text and a third sentence vector for each sentence in the third conversation text from the second text vector; the second conversation text is a conversation text in the second text that belongs to the third object, and the third conversation text is a conversation text in the second text that belongs to the second object; and determine the third index value based on the first sentence vector, the second sentence vector, and the third sentence vector.

[0136] In some embodiments, the first determining module 4551 is further configured to: determine a first similarity based on the first sentence vector and the second sentence vector, and determine a second similarity based on the first sentence vector and the third sentence vector; determine a first number of third words belonging to a first type and a second number of third words belonging to a second type in the first conversation text; wherein the third words belonging to the first type are words used to characterize the identity of the first object, and the third words belonging to the second type are words used to characterize the identity of the second object; and determine the third index value based on the first similarity, the second similarity, the first number, and the second number.

[0137] In some embodiments, the first determining module 4551 is further configured to: perform masking processing on the first conversation text in the first text to obtain a first masked text; perform masking processing on the sentence group in the first text to obtain a second masked text; the sentence group includes a first sentence in the first text and a second sentence adjacent to the first sentence; perform masking processing on the second conversation text in the second text to obtain a third masked text; and determine the fourth index value based on the first masked text, the second masked text, and the third masked text.

[0138] In some embodiments, the first determining module 4551 is further configured to: encode the first masked text to obtain a first encoded vector; encode the second masked text to obtain a second encoded vector; encode the third masked text to obtain a third encoded vector; determine a first sentence class vector based on the sentence type of each sentence in the first masked text; determine a second sentence class vector based on the sentence type of each sentence in the second masked text; determine a third sentence class vector based on the sentence type of each sentence in the third masked text; determine a third text vector based on the first encoded vector and the first sentence class vector; determine a fourth text vector based on the second encoded vector and the second sentence class vector; determine a fifth text vector based on the third encoded vector and the third sentence class vector; and determine a fourth index value based on the third text vector, the fourth text vector, and the fifth text vector.

[0139] In some embodiments, the first determining module 4551 is further configured to: determine a third similarity based on the third text vector and the fifth text vector, and determine a fourth similarity based on the fourth text vector and the fifth text vector; and determine the fourth index value based on the third similarity and the fourth similarity.

[0140] In some embodiments, the first determining module 4551 is further configured to: determine a first mapping point corresponding to the first word in a first coordinate system, and a second mapping point corresponding to the second word in the first coordinate system; determine the sum of gradient changes between the first mapping point and the second mapping point based on the coordinate values ​​of the first mapping point and the second mapping point; and determine the fifth index value based on the sum of gradient changes.

[0141] In some embodiments, the second determining module 4552 is further configured to: determine the first text length of the first conversation text and the second text length of the fourth conversation text in the first text; the fourth conversation text is the conversation text in the first text belonging to the second object; and determine the second index value based on the conversation response duration between the first object and the second object, the first text length, and the second text length.

[0142] It should be noted that the description of the apparatus in this application embodiment is similar to the description of the method embodiment described above, and has similar beneficial effects as the method embodiment; therefore, it will not be repeated. For technical details not disclosed in this apparatus embodiment, please refer to the description of the method embodiment of this application for understanding.

[0143] Correspondingly, this application provides an electronic device. Figure 11 This is a schematic diagram of the composition structure of the electronic device provided in the embodiments of this application, such as... Figure 11 As shown, the electronic device 1200 includes at least: a processor 1201, a communication interface 1202, and a storage medium 1203 configured to store executable instructions, wherein: the processor 1201 generally controls the overall operation of the electronic device 1200.

[0144] The communication interface 1202 enables electronic devices to communicate with other terminals or servers via a network.

[0145] Storage medium 1203 is configured to store instructions and applications executable by processor 1201, and can also cache data to be processed or processed by various modules in processor 1201 and electronic device 1200, which can be implemented by flash memory or random access memory (RAM).

[0146] This application provides a computer-readable storage medium storing computer-executable instructions. When these instructions are executed by a processor, they cause the processor to perform the data processing method provided in this application. For example, ... Figure 2 The data processing method is shown.

[0147] This application provides a computer program product including computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the data processing method described in this application.

[0148] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0149] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0150] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0151] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0152] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A data processing method, characterized in that, The method includes: Based on the first text and the second text, a first indicator value is determined; the first text is the conversation text between the first object and the second object generated by the first model, and the second text is the conversation text between the third object and the second object; the first indicator value is an indicator value used to characterize the conversation quality between the first object and the second object. A second metric value is determined based on the session response duration between the first object and the second object; the second metric value is a metric value used to characterize the session response speed of the first object. Based on the first indicator value and the second indicator value, an evaluation score is determined for the first model; Based on the evaluation score, a text generation model suitable for the first object is determined.

2. The method according to claim 1, characterized in that, The process of determining the first indicator value based on the first text and the second text includes: Based on the first text and the second text, a third indicator value is determined; the third indicator value is an indicator value used to characterize the identity of the first object. Based on the first text and the second text, a fourth indicator value is determined; the fourth indicator value is an indicator value used to characterize the semantic integrity of the first conversational text in the first text; the first conversational text is the conversational text in the first text that belongs to the first object; A fifth index value is determined based on the first word in the first text and the second word adjacent to the first word; the fifth index value is an index value used to characterize the text fluency of the first conversational text. The average of the third, fourth, and fifth indicator values ​​is determined as the first indicator value.

3. The method according to claim 2, characterized in that, The determination of the third indicator value based on the first text and the second text includes: Determine the first text vector of the first conversation text, and the second text vector of the second text; The third index value is determined based on the first text vector and the second text vector.

4. The method according to claim 3, characterized in that, Determining the first text vector of the first conversation text and the second text vector of the second text includes: The first conversation text is segmented to obtain the first word sequence of each sentence in the first conversation text; A second word sequence is obtained by replacing the third word in the first word sequence with a preset empty character; the third word is a word used to characterize the identities of the first object and the second object. The words in the second word sequence are vectorized to obtain the first text vector; The second text is vectorized to obtain the second text vector.

5. The method according to claim 4, characterized in that, Determining the third indicator value based on the first text vector and the second text vector includes: Determine the first sentence vector of each sentence in the first conversation text from the first text vector; The second sentence vector of each sentence in the second conversation text and the third sentence vector of each sentence in the third conversation text are determined from the second text vector; the second conversation text is the conversation text in the second text that belongs to the third object, and the third conversation text is the conversation text in the second text that belongs to the second object; The third index value is determined based on the first sentence vector, the second sentence vector, and the third sentence vector.

6. The method according to claim 5, characterized in that, The step of determining the third index value based on the first sentence vector, the second sentence vector, and the third sentence vector includes: A first similarity is determined based on the first sentence vector and the second sentence vector, and a second similarity is determined based on the first sentence vector and the third sentence vector; Determine the first number of third words belonging to the first type and the second number of third words belonging to the second type in the first conversation text; the third words belonging to the first type are words used to characterize the identity of the first object, and the third words belonging to the second type are words used to characterize the identity of the second object; The third index value is determined based on the first similarity, the second similarity, the first quantity, and the second quantity.

7. The method according to claim 5, characterized in that, The step of determining the fourth indicator value based on the first text and the second text includes: The first conversation text in the first text is masked to obtain the first masked text; The sentence group in the first text is masked to obtain the second masked text; the sentence group includes the first sentence in the first text and the second sentence adjacent to the first sentence. The second conversation text in the second text is masked to obtain the third masked text; The fourth index value is determined based on the first mask text, the second mask text, and the third mask text.

8. The method according to claim 7, characterized in that, The step of determining the fourth index value based on the first mask text, the second mask text, and the third mask text includes: The first masked text is encoded to obtain a first encoded vector; the second masked text is encoded to obtain a second encoded vector; the third masked text is encoded to obtain a third encoded vector. Based on the sentence type of each sentence in the first masked text, a first sentence class vector is determined; based on the sentence type of each sentence in the second masked text, a second sentence class vector is determined; based on the sentence type of each sentence in the third masked text, a third sentence class vector is determined. A third text vector is determined based on the first encoding vector and the first sentence class vector; a fourth text vector is determined based on the second encoding vector and the second sentence class vector; a fifth text vector is determined based on the third encoding vector and the third sentence class vector. The fourth index value is determined based on the third text vector, the fourth text vector, and the fifth text vector.

9. The method according to claim 8, characterized in that, The step of determining the fourth index value based on the third text vector, the fourth text vector, and the fifth text vector includes: A third similarity is determined based on the third text vector and the fifth text vector, and a fourth similarity is determined based on the fourth text vector and the fifth text vector; The fourth index value is determined based on the third similarity and the fourth similarity.

10. The method according to claim 2, characterized in that, The determination of the fifth index value based on the first word in the first text and the second word adjacent to the first word includes: Determine the first mapping point corresponding to the first word in the first coordinate system, and the second mapping point corresponding to the second word in the first coordinate system; Based on the coordinates of the first mapping point and the second mapping point, determine the sum of gradient changes between the first mapping point and the second mapping point; The fifth index value is determined based on the sum of the gradient changes.

11. The method according to any one of claims 2 to 10, characterized in that, The step of determining the second metric value based on the session response duration between the first object and the second object includes: Determine the first text length of the first conversation text and the second text length of the fourth conversation text in the first text; the fourth conversation text is the conversation text in the first text that belongs to the second object. The second metric value is determined based on the session response duration between the first object and the second object, the length of the first text, and the length of the second text.

12. An electronic device, characterized in that, include: Memory is used to store executable instructions for a computer; A processor, when executing computer-executable instructions stored in the memory, implements the data processing method according to any one of claims 1 to 11.

13. A computer-readable storage medium, characterized in that, The device stores computer-executable instructions, which, when executed by a processor, implement the data processing method according to any one of claims 1 to 11.

Images