Method, apparatus, device, and storage medium for evaluating a target task

Abstract

According to an embodiment of the present disclosure, a method, device, equipment and storage medium for evaluating a target task are provided. The method comprises: determining at least one dialogue set by dividing the multi-round dialogue data of a target object arranged in reverse chronological order; determining the corresponding evaluation information of the at least one dialogue set by using a trained machine learning model, the evaluation information at least indicating the operation characteristics and attitude change trend of the target object in the corresponding time range for the target evaluation task; determining the target evaluation information for the multi-round dialogue data by performing a weighted fusion operation on the corresponding evaluation information based on the corresponding time range; and determining the evaluation result for the target evaluation task based on the target evaluation information. Thus, by batch summarization and time sequence weighting, the behavior and willingness change of the user in different time ranges can be effectively captured, thereby improving the accuracy of the evaluation and the sensitivity to the time sequence trend.

Description

Technical Field

[0001] The exemplary embodiments disclosed herein generally relate to the field of information technology, and in particular to methods, apparatus, devices, and computer-readable storage media for evaluating target tasks. Background Technology

[0002] With the rapid development of machine learning technology, analyzing conversation records to assess user behavior and intent has become an important task. To accurately evaluate user behavior patterns and attitude trends over a period of time, it is often necessary to process this conversation data to support the evaluation needs of decision-making systems.

[0003] In related technologies, rule matching or keyword extraction methods are commonly used to analyze user dialogue records. However, these methods struggle to capture the dynamic trends of dialogue changes and often lack effective utilization of temporal information. For complex evaluation tasks, such as user loyalty prediction or risk warning, traditional methods lack the ability to integrate multi-turn dialogue information, making it difficult to achieve efficient and accurate analysis of changes in user behavior, thus limiting the accuracy of system evaluation results.

[0004] Therefore, a more comprehensive and accurate approach to assessing target tasks is needed to capture users' behavioral characteristics and attitude trends, thereby improving the accuracy and applicability of the assessment. Summary of the Invention

[0005] In a first aspect of this disclosure, a method for evaluating a target task is provided. The method includes: determining at least one dialogue set by partitioning multi-turn dialogue data for a target object, arranged in reverse chronological order, the dialogue set including at least one dialogue data within the same time frame; determining corresponding evaluation information for the at least one dialogue set using a trained machine learning model, the evaluation information at least indicating the target object's operational characteristics and attitude change trends for the target evaluation task within the corresponding time frame; determining target evaluation information for the multi-turn dialogue data by performing a weighted fusion operation on the corresponding evaluation information based on the corresponding time frame; and determining an evaluation result for the target evaluation task based on the target evaluation information.

[0006] In a second aspect of this disclosure, an apparatus for evaluating a target task is provided. The apparatus includes: a segmentation module configured to determine at least one dialogue set by segmenting multi-turn dialogue data for a target object in reverse chronological order, the dialogue set including at least one dialogue data within the same time range; an evaluation information determination module configured to determine corresponding evaluation information for the at least one dialogue set using a trained machine learning model, the evaluation information at least indicating the target object's operational characteristics and attitude change trends for the target evaluation task within the corresponding time range; a fusion module configured to determine target evaluation information for the multi-turn dialogue data by performing a weighted fusion operation on the corresponding evaluation information based on the corresponding time range; and an evaluation module configured to determine an evaluation result for the target evaluation task based on the target evaluation information.

[0007] In a third aspect of this disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. When executed by the at least one processing unit, the instructions cause the device to perform the method of the first aspect.

[0008] In a fourth aspect of this disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program that can be executed by a processor to implement the method of the first aspect.

[0009] In a fifth aspect of this disclosure, a computer program product is provided, which is tangibly stored in a computer storage medium and includes computer-executable instructions that, when executed by a device, cause the device to perform the method according to a first aspect of this disclosure.

[0010] It should be understood that the content described in this section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0011] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0012] Figure 1 A schematic diagram of an example environment in which embodiments of the present disclosure can be implemented is shown;

[0013] Figure 2 A flowchart illustrating a process for evaluating a target task according to some embodiments of the present disclosure is shown;

[0014] Figure 3A A schematic diagram illustrating the process of evaluating a target task according to some embodiments of this disclosure is shown;

[0015] Figure 3B A schematic diagram illustrating the data preprocessing process of some embodiments of this disclosure is shown;

[0016] Figure 3C A schematic diagram illustrating the batch summary process of some embodiments of this disclosure is shown;

[0017] Figure 3D A schematic diagram illustrating the timing weighting process of some embodiments of this disclosure is shown;

[0018] Figure 3E A schematic diagram illustrating the comprehensive evaluation process of some embodiments of this disclosure is shown;

[0019] Figure 4 A block diagram of an apparatus for evaluating a target task according to some embodiments of the present disclosure is shown; and

[0020] Figure 5 A block diagram of an electronic device capable of implementing one or more embodiments of the present disclosure is shown. Detailed Implementation

[0021] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0022] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0023] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0024] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0025] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0026] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0027] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below.

[0028] As briefly mentioned earlier, with the rapid development of machine learning technology, analyzing conversation records to assess user behavior and intent—such as customer service satisfaction analysis, user loyalty assessment, and risk monitoring and evaluation—has become an important task. To accurately assess user behavior patterns and attitude changes over a period of time, it is often necessary to process this conversation data to support the evaluation needs of decision-making systems.

[0029] Among the relevant technologies, the following are the main evaluation methods used to analyze user dialogue data to capture trends in user behavior and attitudes:

[0030] Rule-based keyword extraction methods analyze user attitudes by extracting relevant information from user conversations using a predefined keyword list. For example, it assesses repayment intention by identifying whether a user mentions keywords such as "repayment" or "delay." This method is simple and intuitive, but because it relies on predefined keywords, it struggles to adapt to diverse expressions and lacks flexibility.

[0031] Machine learning classification models: These models use traditional machine learning algorithms such as Support Vector Machines (SVM) and Random Forests to classify conversation content into different categories in order to evaluate the behavioral patterns and attitude trends of the target audience. These methods typically require large amounts of labeled data, and while they can provide basic predictions, they struggle to capture subtle changes and long-term trends in user attitudes.

[0032] One-way dialogue summarization techniques: These techniques use natural language processing models to summarize and analyze intent within a single dialogue. While focusing on the content of a single conversation, this approach ignores the user's dialogue behavior over time, thus failing to fully reflect long-term user behavior or changes in intent.

[0033] Sentiment analysis methods combine speech recognition technology to analyze the emotional state of a user's voice (such as anger, anxiety, etc.) to help assess the user's repayment attitude. Although sentiment analysis can enhance the understanding of user attitudes, its reliance on voice data limits its scope and it can only provide supplementary information on emotions.

[0034] While assessment methods in related technologies can analyze dialogue data of target individuals to some extent, they lack a holistic understanding of the context across multiple conversations, making it difficult to accurately grasp the true needs or purposes of the target user. For example, it is difficult to analyze multiple expressions made by a user at different times coherently. Furthermore, assessment methods in related technologies often fail to effectively distinguish the importance of recent and distant conversations. Because the target individual's expressions may be inconsistent across different periods, the analysis of a single conversation is unlikely to yield stable conclusions. For example, in assessing repayment willingness, recent conversations are usually more representative. Without a time-weighted mechanism, these methods will ignore the special importance of recent conversations, leading to inaccurate assessment results.

[0035] Therefore, in order to accurately assess changes in users' long-term behavior and intentions and overcome assessment bias caused by inconsistent expression, a more precise and efficient scheme for assessing target tasks is needed. This scheme should be able to more effectively capture the time trends and changes in user behavior, and through multi-turn dialogue processing, smooth out these short-term fluctuations and provide more accurate and stable assessment results.

[0036] In embodiments of this disclosure, a scheme for evaluating a target task is provided. According to embodiments of this disclosure, at least one dialogue set is determined by dividing multi-turn dialogue data for a target object into segments arranged in reverse chronological order. Each dialogue set includes at least one dialogue data segment within the same time frame. Using a trained machine learning model, corresponding evaluation information for the at least one dialogue set is determined. This evaluation information at least indicates the target object's operational characteristics and attitude change trends regarding the target evaluation task within the corresponding time frame. Target evaluation information for the multi-turn dialogue data is determined by performing a weighted fusion operation on the corresponding evaluation information based on the corresponding time frame. Finally, based on the target evaluation information, an evaluation result for the target evaluation task is determined.

[0037] Therefore, by summarizing multiple rounds of dialogue in batches and using time-series weighted evaluation, it is possible to effectively capture changes in the target audience's behavior and intentions at different times, especially by assigning higher weights to recent dialogue data, thereby improving the accuracy of the evaluation and its sensitivity to time-series trends. Furthermore, using a large language model to generate summary information for each batch improves the understanding of the target audience's complex expressions, while avoiding the limitations of related evaluation methods in terms of diverse expressions and long-term changes.

[0038] Figure 1 A schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented is shown. In environment 100, electronic device 110 acquires at least one round of dialogue data (collectively or individually referred to as dialogue data 120) for a target object. In some embodiments, electronic device 110 inputs the acquired dialogue data 120 into a trained machine learning model 130 to perform an evaluation operation on a target task. Using machine learning model 130, electronic device 110 can acquire an evaluation result 140 for a target evaluation task for the target object.

[0039] In environment 100, electronic device 110 can be any type of computing device, including terminal devices or server devices. Terminal devices can be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof. Server devices may include, for example, computing systems / servers, such as mainframes, edge computing nodes, computing devices in cloud environments, and so on.

[0040] It should be understood that the structure and function of environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure.

[0041] Figure 2 A flowchart of a process 200 for evaluating a target task according to some embodiments of the present disclosure is shown. Process 200 can be implemented in environment 100, for example, process 200 can be implemented at electronic device 110. In the following discussion, reference will be made to... Figure 1 To describe process 200.

[0042] In box 210, electronic device 110 determines at least one set of dialogues by dividing multi-turn dialogue data for a target object into reverse chronological order. The set of dialogues includes at least one set of dialogue data within the same time frame.

[0043] As an example, Figure 3A A schematic diagram of a process 300A for evaluating a target task according to some embodiments of the present disclosure is shown. Process 300A can be implemented at an electronic device 110.

[0044] In box 310, electronic device 110 acquires dialogue data of the target object. Dialogue data refers to multi-turn interaction data generated by the target object within a specific time range. The source of dialogue data can be either direct interaction between the target object and electronic device 110 (e.g., dialogue between a user and a customer service representative or intelligent customer service in a customer service system), or interaction content between the target object and other devices or systems (e.g., dialogue between a user and an intelligent assistant, other automated systems, or online applications).

[0045] In some embodiments, dialogue data may include various types of data such as text, voice, and images. For example, text messages entered by the target object through typing; voice interaction records between the target object and the system, which, after speech-to-text processing, can be used as input for text analysis or directly for emotion and tone analysis; and visual dialogue data where the target object provides feedback or information through images. This multimodal dialogue data can enrich the understanding of the target object's behavior, enabling the electronic device 110 to more accurately assess the target object's needs and emotions.

[0046] In box 320, electronic device 110 performs data preprocessing operations on multi-turn dialogue data for the target object.

[0047] In some embodiments, the electronic device 110 can perform data preprocessing, which involves dividing the multi-turn dialogue data of the target object into at least one dialogue set. By controlling the amount of dialogue data in each set, the dialogue data can be divided into more easily processed groups. This batch processing approach allows the electronic device 110 to analyze the target object's behavioral patterns and attitude changes in batches, reducing the processing burden of the model on the amount of data, thereby improving evaluation efficiency and response speed.

[0048] As an example, Figure 3B A schematic diagram of a data preprocessing procedure 320 according to some embodiments of the present disclosure is shown. Procedure 320 can be implemented at electronic device 110.

[0049] In box 321, after acquiring multi-turn dialogue data for the target object, the electronic device 110 can reverse the order of these dialogue data according to the time sequence of the interaction between the target object and the system.

[0050] In some embodiments, the electronic device 110 acquires multi-turn dialogue data corresponding to the target object and arranges the multi-turn dialogue data in reverse chronological order. This means that the most recent dialogue records are placed first, and the earlier records are placed later. Arranging the data in reverse chronological order allows the electronic device 110 to prioritize the most recent dialogue content, as it usually reflects the target object's current state or attitude more directly. This lays the foundation for subsequent time-weighted processing, enabling recent records to participate more clearly in the final evaluation.

[0051] In box 322, electronic device 110 determines the number of dialogue sets.

[0052] As an example, when determining the number of dialogue sets, i.e., the number of batches, the electronic device 110 can divide the total N dialogue data into M sets (batches) based on the number K of dialogue data included in each dialogue set. Here, M = ceil(N / K) represents rounding up the result of N divided by K to ensure that all dialogues are assigned to the corresponding set.

[0053] In box 323, electronic device 110 divides the multi-turn dialogue data corresponding to the target object.

[0054] In some embodiments, the electronic device 110 determines at least one dialogue set by dividing the multi-turn dialogue data into reverse-ordered segments based on the amount of multi-turn dialogue data and the amount of dialogue data included in at least one dialogue set.

[0055] As an example, electronic device 110 divides N dialogue data arranged in reverse order into M sets according to a step size (the number of dialogue data included in each dialogue set) K, with each dialogue set containing K dialogue data. If the number of dialogue data remaining after the division is less than K, then the last dialogue set can have fewer than K.

[0056] In this embodiment of the disclosure, by dividing the multi-turn dialogue data corresponding to the target object into dialogue sets with different processing batches, it can be ensured that all dialogue records are included in the analysis. Furthermore, this division method facilitates subsequent batch analysis, helping the model to more precisely identify behavioral patterns over different time periods.

[0057] Return to reference Figure 2 In box 220, electronic device 110 uses a trained machine learning model to determine corresponding evaluation information for at least one set of dialogues. The evaluation information at least indicates the target object's operational characteristics and attitude trends regarding the target evaluation task within a corresponding time frame.

[0058] Continue to refer to Figure 3A In box 330, electronic device 110 performs a batch summary operation.

[0059] In some embodiments, after dividing the multi-turn dialogue data corresponding to the target object into different dialogue sets in reverse chronological order, the electronic device 110 can analyze the dialogue data in each dialogue set through a machine learning model and generate corresponding evaluation information for each time batch.

[0060] Evaluation information refers to the summary information that the model outputs for each dialogue set, containing user behavior and attitudes within that time period. For example, evaluation information includes the target user's behavioral characteristics in the dialogue related to the target task; in a repayment scenario, this might reflect the user's clarity regarding their repayment intentions and whether they made specific commitments. In broader evaluation tasks, operational characteristics could be the user's expressed concerns, goals, or action intentions during the dialogue.

[0061] Evaluation information can also include changes in the target audience's mood or attitudes within the conversation data for that time period. For example, the model can analyze whether a user exhibits a shift from positive to negative, or from uncertainty to clear intent, throughout a series of conversations. Attitude change trends help identify user mood swings and psychological states during that time period.

[0062] As an example, Figure 3C A schematic diagram of a batch summary process 330 of some embodiments of the present disclosure is shown. Process 330 can be implemented at electronic device 110.

[0063] In box 331, electronic device 110 determines a prompt word template for guiding the model to determine the corresponding evaluation information.

[0064] As an example, a prompt template suitable for a target assessment task can be designed to guide machine learning models, such as Large Language Models (LLM), to extract key information from each dialogue set. For instance, a prompt template for assessing a user's willingness to repay a loan could include the following instructions: "Please summarize the user's willingness to repay and changes in attitude in the following dialogue, focusing on: 1) the specificity of the repayment commitment; 2) the trend of attitude changes; 3) factors that may affect repayment. Dialogue data: {dialogue content}."

[0065] In some embodiments, the electronic device 110 inputs dialogue data from at least one dialogue set within the same time frame into a large language model.

[0066] In box 332, the electronic device 110 summarizes the dialogue data in each dialogue set by calling the application programming interface (API) of the large language model according to the designed prompt template, in order to determine the corresponding evaluation information (summary results) for each dialogue set.

[0067] In box 333, electronic device 110 determines the corresponding evaluation information for each dialogue set.

[0068] In some embodiments, the electronic device 110 uses a large language model and a preset prompt word template to determine the operational characteristics and attitude change trends of the target object in response to the target evaluation task within a corresponding time range.

[0069] As an example, electronic device 110, based on the designed prompt template, calls the large language model API to summarize M dialogue sets respectively, obtaining M corresponding evaluation information (summary results). Specifically, the large language model will generate a summary based on the dialogue data included in each dialogue set, summarizing the user behavior, attitudes, and influencing factors in the dialogue sets of that time batch.

[0070] In this embodiment, by designing a prompt template and calling a large model, the evaluation information corresponding to each dialogue set is determined batch by batch, thereby ensuring the accurate extraction and processing of multi-turn dialogue data corresponding to the target object. This process not only makes the model output more targeted, but also preserves the temporal dimension of the target object's behavior and attitude, which helps in subsequent temporal weighted fusion and the generation of the final evaluation result, improving the accuracy and practicality of the evaluation task.

[0071] Return to reference Figure 2 In box 230, electronic device 110 determines the target evaluation information for multi-turn dialogue data by performing a weighted fusion operation on the corresponding evaluation information based on the corresponding time range.

[0072] In some embodiments, since the evaluation methods in related technologies often ignore the time factor of dialogue data, and thus cannot accurately distinguish the impact of recent dialogues and distant dialogues, the electronic device 110 can weight the summary results of dialogues corresponding to different time batches in chronological order to ensure that the most recent dialogue data has a higher weight in the evaluation results.

[0073] Return to reference Figure 3A In box 340, electronic device 110 performs a time-weighted operation on the corresponding evaluation information of at least one set of dialogues.

[0074] Figure 3DA schematic diagram of a timing-weighted process 340 according to some embodiments of the present disclosure is shown. Process 340 can be implemented at electronic device 110.

[0075] In box 341, electronic device 110 determines the weight of the corresponding evaluation information based on a time decay function, wherein the corresponding evaluation information of the dialogue set that is closer to the time of performing the target evaluation task has a higher weight.

[0076] As an example, electronic device 110 can assign weights to the summary results of different batches using a preset time decay function w(i):

[0077] w(i)=exp(-λ·i) (1)

[0078] Where i is the set index, indicating how close the time batch corresponding to this dialogue set is to the current time. λ is the decay coefficient, used to control the rate of weight decay. A larger λ value will accelerate decay, causing the corresponding evaluation information weight of dialogue sets that are far from the current time to decrease rapidly. A smaller λ value will slow down decay, ensuring that the corresponding evaluation information of dialogue sets that are far from the current time is not completely ignored.

[0079] In box 342, electronic device 110 performs a normalization operation on the weights of the corresponding evaluation information.

[0080] As an example, electronic device 110 adjusts the weights of the evaluation information corresponding to each dialogue set to a ratio that sums to 1 through normalization processing:

[0081] W(i)=w(i) / ∑w(i) (2)

[0082] Normalization ensures that the time-decayed weights are neither too large nor too small during the weighted fusion process, guaranteeing a reasonable comparison of the evaluation information for dialogue sets corresponding to different time batches. The sum of the normalized weights is 1, enabling the model to robustly integrate recent and long-term features without causing the evaluation results to deviate from reality due to abnormal weights.

[0083] In box 343, electronic device 110 obtains target evaluation information by performing a weighted fusion operation on the corresponding evaluation information based on the weights after normalization.

[0084] Target assessment information is a comprehensive result generated by weighted fusion of assessment information from various dialogue sets (i.e., summaries of the target object's operational characteristics and attitude changes over different time periods). Target assessment information represents the model's comprehensive understanding of the target object's behavioral and attitude changes throughout the assessment period; it is a dynamic, cross-time-period summary of the assessment result. It not only includes the overall trend of the target object's attitude changes but also demonstrates the overall impact of short- and long-term behavioral characteristics.

[0085] As an example, electronic device 110 obtains a comprehensive summary, namely the target evaluation information, by weighting and fusing the evaluation information corresponding to each time batch according to the weight W(i):

[0086] Final_summary=∑(W(i)*Summary(i)) (3)

[0087] Where i ranges from 1 to M (the number of dialogue sets). Since the weights are calculated and normalized based on a time decay function, the evaluation information corresponding to more recent dialogue sets has a greater impact on the fusion process.

[0088] In this embodiment, a time decay function is introduced for temporal weighting, enabling the model to dynamically adjust the impact of dialogue content across different time periods on the evaluation results. This ensures that the most recent dialogue data has a higher weight in the comprehensive evaluation. This approach effectively solves the problem of neglecting temporal differences in traditional evaluation methods, making the evaluation results more timely and accurately reflecting the target's current attitude and behavior. Simultaneously, the temporal weighted fusion retains the long-term influence of historical data, allowing the model to consider not only short-term changes in the target but also their long-term behavioral trends, thus achieving a comprehensive evaluation of the target's dynamic state. This fusion method improves the accuracy and relevance of the evaluation, providing more valuable support for subsequent decision-making.

[0089] Return to reference Figure 2 In box 240, electronic device 110 determines the evaluation result for the target evaluation task based on the target evaluation information.

[0090] In some embodiments, the electronic device 110 may use weighted dialogue summary information, i.e. target evaluation information, to comprehensively evaluate the target object through a large model, thereby determining the evaluation result for the target evaluation task.

[0091] During the weighted fusion process, the evaluation information corresponding to different time batches has been weighted according to the time decay function, with a focus on highlighting the impact of recent dialogue data, thus obtaining a comprehensive summary information, namely, target evaluation information. Therefore, electronic device 110 can comprehensively evaluate the trend of changes in the target object's behavior or intentions throughout the dialogue process based on the target evaluation information.

[0092] Return to reference Figure 3A In box 350, electronic device 110 performs a comprehensive evaluation operation based on the target evaluation information.

[0093] As an example, Figure 3EA schematic diagram of a comprehensive evaluation process 350 according to some embodiments of the present disclosure is shown. Process 350 can be implemented at electronic device 110.

[0094] In box 351, electronic device 110 determines prompt words for performing the target assessment task based on target assessment information.

[0095] In some embodiments, based on target assessment information (i.e., a comprehensive set of features integrating changes in user behavior and attitudes over different time periods), the electronic device 110 can determine a prompt to guide the model in generating assessment results. The content of the prompt can be customized according to the specific assessment task, enabling the model to focus on core assessment needs. For example, in a customer satisfaction assessment task, the prompt might guide the model to focus on assessment information such as overall customer satisfaction and changes in loyalty.

[0096] In box 352, electronic device 110 determines the evaluation result for the target evaluation task by inputting prompt words into a trained large language model.

[0097] In some embodiments, the electronic device 110 inputs the determined prompt into a large language model, enabling it to generate a final evaluation result based on the target evaluation information. For example, the large language model utilizes information from the prompt words to transform the target evaluation information into a task-related evaluation result, generating a structured and easy-to-understand evaluation report.

[0098] In this embodiment, prompt words guide the model to generate evaluation results, making the output evaluation content more targeted and accurate, directly focusing on the core needs of the current task. Simultaneously, the design of the prompt words can be flexibly adjusted according to the evaluation scenario, making the generated report applicable to various task requirements. This prompt-based output process also improves the response quality and interpretability of the large language model, providing decision-makers with accurate and in-depth evaluation insights and action recommendations.

[0099] Return to reference Figure 3A In box 360, electronic device 110 determines the evaluation results for the target evaluation task.

[0100] As an example, the assessment results for a target assessment task include a structured assessment report that indicates at least one of the following: the target's willingness to repay, trends in attitude change, and risk factors.

[0101] In some embodiments, after weighted fusion and large-scale model analysis of multi-turn dialogue data corresponding to the target object, the electronic device 110 can transform the final comprehensive evaluation result into a structured report format. This report not only provides user attitude and behavior scores in the target task (such as repayment willingness), but also provides an analysis of the trend of changes in user attitudes and points out risk factors that may affect user behavior.

[0102] As an example, the content of the assessment report can include a variety of structured information, such as: repayment willingness score, which is a quantitative assessment of the user's current repayment willingness, making it easier to intuitively understand the user's repayment attitude; attitude change trend, which is the analysis of changes in the user's attitude through dialogue data over multiple time periods; risk factors, which are external or internal factors that may affect the user's repayment, such as changes in financial situation, personal life events, etc.

[0103] In this embodiment, a structured assessment report is generated that includes the target audience's willingness to repay, attitude change trends, and risk factors. This allows complex dialogue analysis results to be presented in a clear structure, facilitating quick understanding and use. This structured report enhances the readability and usability of the assessment results, enabling decision-makers to formulate appropriate strategies more efficiently, thereby optimizing assessment effectiveness and user experience.

[0104] Figure 4 A block diagram of an apparatus 400 for evaluating a target task according to some embodiments of the present disclosure is shown. The apparatus 400 may be implemented as or include electronic device 110.

[0105] The apparatus 400 includes a segmentation module 410 configured to determine at least one dialogue set by segmenting multi-turn dialogue data for a target object in reverse chronological order, the dialogue set including at least one dialogue data within the same time range; an evaluation information determination module 420 configured to determine corresponding evaluation information for at least one dialogue set using a trained machine learning model, the evaluation information indicating at least the operational characteristics and attitude change trends of the target object for a target evaluation task within a corresponding time range; a fusion module 430 configured to determine target evaluation information for the multi-turn dialogue data by performing a weighted fusion operation on the corresponding evaluation information based on the corresponding time range; and an evaluation module 440 configured to determine an evaluation result for the target evaluation task based on the target evaluation information.

[0106] In some embodiments, the segmentation module 410 is further configured to acquire multi-turn dialogue data corresponding to the target object; arrange the multi-turn dialogue data in reverse chronological order; and determine at least one dialogue set by segmenting the reverse-ordered multi-turn dialogue data based on the number of multi-turn dialogue data and the number of dialogue data included in at least one dialogue set.

[0107] In some embodiments, the trained machine learning model includes a large language model.

[0108] In some embodiments, the evaluation information determination module 420 is further configured to input dialogue data from at least one dialogue set within the same time range into a large language model; and to use the large language model, based on a preset prompt word template, to determine the target object's operational characteristics and attitude change trends for the target evaluation task within the corresponding time range.

[0109] In some embodiments, the fusion module 430 is further configured to determine the weights of the corresponding evaluation information based on a time decay function, wherein the corresponding evaluation information of the dialogue set closer to the time of performing the target evaluation task has a higher weight; and to perform a normalization operation on the weights of the corresponding evaluation information; and to obtain the target evaluation information by performing a weighted fusion operation on the corresponding evaluation information based on the weights after the normalization operation. In some embodiments, the execution data indicates at least one of the following: the operation time, operation type, operation object, and operation path of the target operation.

[0110] In some embodiments, the evaluation module 440 is further configured to determine prompt words for performing the target evaluation task based on the target evaluation information; and to determine the evaluation result for the target evaluation task by inputting the prompt words into a trained large language model.

[0111] In some embodiments, the assessment results for the target assessment task include a structured assessment report, which indicates at least one of the following: the target's willingness to repay, trends in attitude change, and risk factors.

[0112] The modules included in device 400 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units can be implemented using software and / or firmware, such as machine-executable instructions stored on a storage medium. In addition to or as an alternative to machine-executable instructions, some or all of the units in device 400 can be implemented at least partially by one or more hardware logic components. By way of example, and not limitation, exemplary types of hardware logic components that can be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0113] Figure 5 A block diagram of an electronic device 500 in which one or more embodiments of the present disclosure may be implemented is shown. It should be understood that... Figure 5The electronic device 500 shown is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein.

[0114] like Figure 5 As shown, electronic device 500 is in the form of a general-purpose electronic device. Components of electronic device 500 may include, but are not limited to, one or more processors or processing units 510, memory 520, storage device 530, one or more communication units 540, one or more input devices 550, and one or more output devices 560. Processing unit 510 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 520. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 500.

[0115] Electronic device 500 typically includes multiple computer storage media. Such media can be any available media accessible to electronic device 500, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 520 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 530 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media capable of storing information and / or data and accessible within electronic device 500.

[0116] Electronic device 500 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 5 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 520 may include computer program product 525 having one or more program modules configured to perform various methods or actions of various embodiments of this disclosure.

[0117] Communication unit 540 enables communication with other electronic devices via a communication medium. Additionally, the functionality of components of electronic device 500 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, electronic device 500 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0118] Input device 550 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 560 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 500 can also communicate with one or more external devices (not shown) via communication unit 540 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 500, or with any device that enables electronic device 500 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0119] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores one or more computer instructions, wherein the one or more computer instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transient computer-readable medium and includes computer-executable instructions that are executed by a processor to implement the methods described above.

[0120] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0121] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0122] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0123] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0124] Various implementations of this disclosure have been described above. The foregoing description is exemplary and not exhaustive, nor is it limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the implementations disclosed herein.

Claims

1. A method for evaluating a target task, comprising: At least one dialogue set is determined by dividing the multi-turn dialogue data for the target object and arranged in reverse chronological order. The dialogue set includes at least one dialogue data within the same time range. Using a trained machine learning model, the corresponding evaluation information of the at least one dialogue set is determined, and the evaluation information at least indicates the operational characteristics and attitude change trends of the target object in response to the target evaluation task within a corresponding time range; The target evaluation information for the multi-turn dialogue data is determined by performing a weighted fusion operation on the corresponding evaluation information based on the corresponding time range. as well as Based on the target evaluation information, the evaluation result for the target evaluation task is determined.

2. The method of claim 1, wherein determining at least one set of dialogues comprises: Obtain the multi-turn dialogue data corresponding to the target object; Arrange the multi-turn dialogue data in reverse chronological order; as well as The at least one dialogue set is determined by dividing the multi-turn dialogue data into reverse-order segments based on the number of multi-turn dialogue data and the number of dialogue data included in the at least one dialogue set.

3. The method of claim 1, wherein the trained machine learning model comprises a large language model, and wherein determining the corresponding evaluation information for the at least one dialogue set comprises: Input dialogue data from the same time range included in the at least one dialogue set into the large language model; as well as Using the large language model and based on a preset prompt word template, the operational characteristics and attitude change trends of the target object in response to the target evaluation task within the corresponding time range are determined.

4. The method according to claim 1, wherein determining the target evaluation information includes: The weights of the corresponding evaluation information are determined based on the time decay function, wherein the corresponding evaluation information of the dialogue set that is closer to the time of performing the target evaluation task has a higher weight. as well as Perform a normalization operation on the weights of the corresponding evaluation information; as well as The target evaluation information is obtained by performing a weighted fusion operation on the corresponding evaluation information based on the weights after normalization.

5. The method of claim 1, wherein determining the evaluation result for the target evaluation task comprises: Based on the target evaluation information, determine the prompt words used to perform the target evaluation task; as well as The evaluation result for the target evaluation task is determined by inputting the cue words into a trained large language model.

6. The method of claim 1, wherein the evaluation result for the target evaluation task includes a structured evaluation report, the structured evaluation report indicating at least one of the following: The target group's willingness score, attitude change trend, and risk factors regarding repayment operations.

7. An apparatus for evaluating a target task, comprising: The segmentation module is configured to determine at least one dialogue set by segmenting multi-turn dialogue data for a target object in reverse chronological order, the dialogue set including at least one dialogue data within the same time range; An evaluation information determination module is configured to use a trained machine learning model to determine corresponding evaluation information for the at least one set of dialogues, the evaluation information indicating at least the operational characteristics and attitude change trends of the target object for the target evaluation task within a corresponding time range; The fusion module is configured to determine the target evaluation information for the multi-turn dialogue data by performing a weighted fusion operation on the corresponding evaluation information based on the corresponding time range. as well as The evaluation module is configured to determine the evaluation result for the target evaluation task based on the target evaluation information.

8. An electronic device, comprising: At least one processing unit; as well as At least one memory is coupled to at least one processing unit and stores instructions for execution by the at least one processing unit, which, when executed by the at least one processing unit, cause the electronic device to perform the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, the computer program being executable by a processor to implement the method according to any one of claims 1 to 6.

10. A computer program product tangibly stored in a computer storage medium and comprising computer-executable instructions that, when executed by a device, cause the device to perform the method according to any one of claims 1 to 6.

Images