Information pushing method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides an information pushing method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining correlation data having an association relationship with a user question when the user question is received; splicing the user problem and the correlation data to generate spliced data; generating initial reply information for the spliced data; determining an evaluation dimension, and calculating evaluation information for the user question, the correlation data and the initial reply information based on the evaluation dimension; when the evaluation information meets a preset condition, determining the initial reply information as target reply information, and pushing the target reply information to a user; a complete dialogue system framework is constructed, and the reply accuracy of the large language model in the intelligent customer service application is effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of information push, and in particular to an information push method, an information push device, an electronic device and a computer-readable storage medium. Background Art

[0002] In the field of intelligent customer service, large model technology has become a key tool for improving service efficiency and quality. They are trained through complex algorithms and large data sets to understand and respond to user queries. However, these large language models often face the problem of hallucination in practical applications, that is, the model sometimes generates information that is inconsistent with the facts. This is particularly prominent in scenarios that require a high degree of factual accuracy. If the generated responses contain factual errors, it will seriously affect the user's trust and satisfaction with the intelligent customer service system. Summary of the invention

[0003] The embodiments of the present invention provide an information push method, device, electronic device and computer-readable storage medium to overcome the above problems or at least partially solve the above problems.

[0004] The embodiment of the present invention discloses an information push method, including:

[0005] When receiving a user question, obtaining correlation data associated with the user question;

[0006] Splicing the user question and the correlation data to generate spliced ​​data;

[0007] generating initial response information for the spliced ​​data;

[0008] Determine an evaluation dimension, and based on the evaluation dimension, calculate evaluation information for the user question, the relevance data, and the initial reply information;

[0009] When the evaluation information meets the preset conditions, the initial reply information is determined as the target reply information, and the target reply information is pushed to the user.

[0010] Optionally, the step of obtaining correlation data associated with the user question includes:

[0011] Converting a database text of a preset database for storing reply content into a first text vector;

[0012] Convert the user question into a second text vector;

[0013] Calculating cosine similarity between the second text vector and the first text vector in the database one by one, and determining a similarity score between the second text vector and the first text vector;

[0014] sorting the similarity scores to generate a similarity score sequence;

[0015] Selecting a vector with a score greater than a preset threshold in the similarity score sequence as a target vector;

[0016] The text corresponding to the target vector is determined as the correlation data.

[0017] Optionally, the step of calculating the cosine similarity between the second text vector and the first text vector in the database one by one to determine the similarity score between the second text vector and the first text vector comprises:

[0018] Inputting the first text vector and the second text vector into Formula 1 to determine a similarity score between the second text vector and the first text vector;

[0019] The formula 1 is:

[0020]

[0021] Among them, D i is the first text vector, Q is the second text vector, S i is the similarity score.

[0022] Optionally, the step of generating initial reply information for the spliced ​​data includes:

[0023] Inputting the correlation data and the second text vector into Formula 2 to generate initial reply information for the spliced ​​data;

[0024] The formula 2 is:

[0025] f(D k∈K ,Q)=R0

[0026] Among them, f is the pre-trained large language model, D k∈K is the correlation data, and R0 is the initial reply information.

[0027] Optionally, the step of calculating evaluation information for the user question, the correlation data and the initial reply information includes:

[0028] Inputting the correlation data, the second text vector and the initial reply information into Formula 3 to obtain evaluation information; the evaluation information includes a multi-dimensional evaluation for the evaluation dimension and a quantitative score for the initial reply information;

[0029] The formula three is:

[0030] g(D k∈K ,Q,R0)=A0,s0

[0031] g is the evaluation model, A0 is the multi-dimensional evaluation, and s0 is the quantitative score.

[0032] Optionally, it also includes:

[0033] When the multi-dimensional evaluation and the quantitative score do not meet the preset conditions, updated reply information is generated by the large language model based on the multi-dimensional evaluation, the quantitative score, the relevance data, the second text vector and the initial reply information.

[0034] Optionally, it also includes:

[0035] A scoring threshold for the quantitative score is determined, and when the quantitative score is greater than or equal to the scoring threshold, the updated reply information is determined as the target reply information, and iterating the large language model is stopped.

[0036] The embodiment of the present invention also discloses an information push device, including:

[0037] A correlation data acquisition module, used for acquiring correlation data associated with the user question when receiving the user question;

[0038] A splicing data generating module, used for splicing the user question and the correlation data to generate splicing data;

[0039] An initial response information generating module, used for generating initial response information for the spliced ​​data;

[0040] An evaluation information calculation module, used to determine evaluation dimensions, and based on the evaluation dimensions, calculate evaluation information for the user question, the correlation data, and the initial reply information;

[0041] The target reply information pushing module is used to determine the initial reply information as the target reply information when the evaluation information meets the preset conditions, and push the target reply information to the user.

[0042] The embodiment of the present invention further discloses an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0043] The memory is used to store computer programs;

[0044] The processor is used to implement the method described in the embodiment of the present invention when executing the program stored in the memory.

[0045] The embodiment of the present invention further discloses a computer-readable storage medium having instructions stored thereon, which, when executed by one or more processors, enables the processors to execute the method described in the embodiment of the present invention.

[0046] The embodiments of the present invention include the following advantages:

[0047] The present mode example, when receiving a user question, obtains correlation data associated with the user question; splices the user question and the correlation data to generate spliced ​​data; generates initial reply information for the spliced ​​data; determines the evaluation dimension, and based on the evaluation dimension, calculates the evaluation information for the user question, the correlation data and the initial reply information; when the evaluation information meets the preset conditions, determines the initial reply information as the target reply information, and pushes the target reply information to the user; constructs a complete dialogue system framework, and realizes intelligent reply to user questions through retrieval, splicing, generation, evaluation, feedback and other links. This solution has the following advantages:

[0048] Strong scalability: The knowledge base, model structure and evaluation dimensions can be flexibly adjusted to adapt to different application scenarios.

[0049] Quality assurance: The quality of responses is ensured through multiple rounds of iterations and evaluation mechanisms.

[0050] User-Friendly: Provides accurate, relevant, and friendly responses, improving user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 is a flowchart of the steps of an information push method provided in an embodiment of the present invention;

[0052] Figure 2 It is a flowchart of an information push method provided in an embodiment of the present invention;

[0053] Figure 3 is a structural block diagram of an information push device provided in an embodiment of the present invention;

[0054] Figure 4 is a hardware structure block diagram of an electronic device provided in an embodiment of the present invention;

[0055] Figure 5 is a schematic diagram of a computer-readable medium provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0056] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0057] Large language models have become a shining star in the field of intelligent customer service. With powerful semantic understanding capabilities, they have significantly improved service efficiency and user experience. However, the model is prone to "hallucinations" when generating text, that is, generating content that does not match the facts. This is particularly prominent in customer service scenarios that require high accuracy. For example, when a user queries personal account information, if the model generates an incorrect response, it will seriously damage the user's trust in the service.

[0058] In order to alleviate the "hallucination" problem, the industry widely adopts retrieval-augmented generation (RAG) technology. RAG combines information from external knowledge bases with user queries to provide the model with richer context, thereby improving the accuracy of generated results. However, the RAG method still has limitations. The model's ability to understand and utilize retrieval results is a key factor affecting the generation effect. Existing models still have shortcomings in processing complex queries and associating multiple aspects of information, which limits the effectiveness of RAG in practical applications.

[0059] Therefore, it is urgent to explore more advanced technologies to improve the application effect of large language models in the field of intelligent customer service. This includes: enhancing the model's ability to understand external knowledge, optimizing the retrieval and generation process of RAG, and developing more effective evaluation indicators to measure the performance of the model in different scenarios. Only by continuously breaking through these technical bottlenecks can we truly achieve high quality and high efficiency of intelligent customer service.

[0060] Reference Figure 1 , shows a flow chart of the steps of an information push method provided in an embodiment of the present invention, which may specifically include the following steps:

[0061] Step 101, when a user question is received, obtaining correlation data associated with the user question;

[0062] Step 102, concatenating the user question and the correlation data to generate concatenated data;

[0063] Step 103, generating initial reply information for the spliced ​​data;

[0064] Step 104, determining evaluation dimensions, and calculating evaluation information for the user question, the correlation data, and the initial reply information based on the evaluation dimensions;

[0065] Step 105: When the evaluation information meets the preset conditions, the initial reply information is determined as the target reply information, and the target reply information is pushed to the user.

[0066] According to the embodiment of the present invention, correlation data associated with the user question can be obtained.

[0067] The purpose is to provide sufficient background information for the large model to help the model generate more accurate and relevant responses.

[0068] Implementation method:

[0069] Knowledge base retrieval: Retrieve relevant information fragments from a pre-built knowledge base based on the keywords or semantics in the user's question.

[0070] Vector database retrieval: Convert user questions and information in the knowledge base into vector representations, and use vector similarity calculation to find the information most similar to the user question.

[0071] External data source retrieval: Get real-time updated information from external data sources (such as search engines and databases) as needed.

[0072] For example, the user asks: "How much data is left on my mobile phone plan?";

[0073] The relevant data can be the following related information:

[0074] User's current package: unlimited package with a monthly fee of 88 yuan;

[0075] Package content: 30GB of high-speed traffic per month, unlimited speed after exceeding the limit;

[0076] Traffic used this month: 22GB;

[0077] Carryover traffic from last month: 15GB.

[0078] Beneficial effects: It enriches the model's input, improves the model's ability to understand user intent, and lays the foundation for generating high-quality responses.

[0079] In the embodiment of the present invention, the user question and the correlation data may be spliced ​​to generate spliced ​​data.

[0080] The purpose is to integrate user questions and related data into one context to provide a complete context for the model.

[0081] Implementation method:

[0082] Simple stitching: directly stitch user questions and related data together.

[0083] Template splicing: Use predefined templates to splice user questions and related data in a certain format.

[0084] Beneficial effect: It clarifies the focus of the model, helps the model focus on the core of the problem, and generates more accurate responses.

[0085] The embodiment of the present invention can generate initial reply information for the spliced ​​data.

[0086] The goal is to leverage the generative power of large models to generate initial responses given the context.

[0087] Implementation method:

[0088] Prompt Engineering: Design appropriate prompts to guide the model to generate high-quality responses.

[0089] Beam Search: Use search algorithms such as Beam Search to select the best response from multiple candidate responses.

[0090] Beneficial effect: It provides an initial response result, which provides a basis for subsequent evaluation and optimization.

[0091] In the embodiment of the present invention, the evaluation dimension can be determined, and based on the evaluation dimension, the evaluation information for the user question, the correlation data and the initial reply information can be calculated.

[0092] Its purpose is to evaluate the generated responses and determine whether their quality meets the requirements.

[0093] In a specific implementation, an embodiment of the present invention can introduce an independent evaluation model, and by providing the independent evaluation model with correlation data, user questions, and large model responses, splicing them in a specific format, perform multi-dimensional evaluation and quantitative scoring on the large model responses.

[0094] Define evaluation dimensions: Based on the specific application scenario, define a series of evaluation dimensions, which may include but are not limited to accuracy, relevance, fluency, completeness, etc.

[0095] Build a review model: Train a machine learning model to score the generated responses based on the given review dimensions.

[0096] Manual evaluation: Combined with manual evaluation, the evaluation results of the model are calibrated.

[0097] Beneficial effects: It provides quantitative evaluation indicators and provides a basis for subsequent decision-making.

[0098] In the embodiment of the present invention, when the evaluation information meets a preset condition, the initial reply information can be determined as target reply information, and the target reply information can be pushed to the user.

[0099] Its purpose is to send responses to users that meet quality requirements.

[0100] Implementation method:

[0101] Set threshold: Set an evaluation score threshold. When the evaluation score is higher than the threshold, the reply quality is considered qualified.

[0102] Feedback mechanism: Send the final response to the user and collect user feedback for continuous optimization of the model.

[0103] Beneficial effects: Ensure that users receive high-quality responses and improve user satisfaction.

[0104] Regarding the "hallucination" problem that may occur when large language models generate responses in the field of intelligent customer service, large language models are prone to generate information that is inconsistent with the facts when generating responses, which is the so-called "hallucination". This is mainly because the model may have data bias during the training process, lack a comprehensive understanding of the real world, or have logical errors during the generation process.

[0105] The embodiments of the present invention provide the model with rich context information by acquiring relevant data, which helps the model to understand the user's question more accurately and reduce the possibility of generating "hallucinations". By retrieving relevant data, the model can obtain knowledge closely related to the user's question, thereby generating more accurate and targeted responses.

[0106] By splicing data, user questions and related data are stitched together to form a complete context, providing a more comprehensive information input for the model. This helps the model better understand the semantics of the question and generate more coherent and reasonable responses.

[0107] By generating initial responses, the large model can generate initial responses based on the concatenated data. This enables the model to master the rules and knowledge of the language by learning a large amount of corpus and to generate text based on a given context.

[0108] The quality of the generated responses is evaluated through the evaluation information. The generated responses are scored by defining multiple evaluation dimensions, such as accuracy, relevance, fluency, etc. This helps to identify problems in the responses and provide a basis for subsequent optimization.

[0109] The target response is determined based on the evaluation results, and it is decided whether to send the generated response to the user as the final response. If the evaluation results meet the preset conditions, the generated response is considered to be of high quality and can be sent to the user; otherwise, further optimization or manual intervention can be performed.

[0110] The present mode example, when receiving a user question, obtains correlation data associated with the user question; splices the user question and the correlation data to generate spliced ​​data; generates initial reply information for the spliced ​​data; determines the evaluation dimension, and based on the evaluation dimension, calculates the evaluation information for the user question, the correlation data and the initial reply information; when the evaluation information meets the preset conditions, determines the initial reply information as the target reply information, and pushes the target reply information to the user; constructs a complete dialogue system framework, and realizes intelligent reply to user questions through retrieval, splicing, generation, evaluation, feedback and other links. This solution has the following advantages:

[0111] Strong scalability: The knowledge base, model structure and evaluation dimensions can be flexibly adjusted to adapt to different application scenarios.

[0112] Quality assurance: The quality of responses is ensured through multiple rounds of iterations and evaluation mechanisms.

[0113] User-Friendly: Provides accurate, relevant, and friendly responses, improving user experience.

[0114] Based on the above embodiment, a variant embodiment of the above embodiment is proposed. It should be noted that in order to make the description concise, only the differences from the above embodiment are described in the variant embodiment.

[0115] In an optional embodiment of the present invention, the step of obtaining correlation data associated with the user question includes:

[0116] Converting a database text of a preset database for storing reply content into a first text vector;

[0117] Convert the user question into a second text vector;

[0118] Calculating cosine similarity between the second text vector and the first text vector in the database one by one, and determining a similarity score between the second text vector and the first text vector;

[0119] sorting the similarity scores to generate a similarity score sequence;

[0120] Selecting a vector with a score greater than a preset threshold in the similarity score sequence as a target vector;

[0121] The text corresponding to the target vector is determined as the correlation data.

[0122] The embodiment of the present invention can convert text into a numerical representation that can be understood by a computer, that is, using a vector representation, and then determine the relevance between texts by calculating the similarity between vectors.

[0123] In a specific implementation, the embodiment of the present invention may convert the database text of the preset database for storing the reply content into the first text vector:

[0124] Text preprocessing: Perform preprocessing such as word segmentation, stop word removal, and word form restoration on the text in the database to remove irrelevant information and extract key words.

[0125] Vectorization: Convert the preprocessed text into vector representation. Common methods include:

[0126] Bag-of-Words model: The text is represented as a word frequency vector. The bag-of-words model is simple and easy to implement, but it ignores the word order information.

[0127] TF-IDF: Based on the bag-of-words model, it introduces term frequency-inverse document frequency to emphasize the importance of words.

[0128] Word Embedding: Map words into a dense vector space to capture the semantic and grammatical relationships between words, such as Word2Vec, GloVe, BERT, etc.

[0129] The embodiment of the present invention can also convert the user question into a second text vector:

[0130] Similar to the above, the user question is preprocessed and vectorized to obtain a vector representation of the user question.

[0131] By calculating the cosine value of the angle between two vectors, we can determine the similarity between them. The cosine similarity range is [-1,1], and the larger the value, the higher the similarity.

[0132] Calculation process: Calculate the cosine similarity between the vector of the user question and the vector of each text in the database one by one to obtain a series of similarity scores.

[0133] The calculated similarity scores are sorted in descending order to form a similarity score sequence.

[0134] Select the target vector:

[0135] Set threshold: A similarity threshold is set in advance, and only vectors with similarity scores higher than this threshold are considered relevant.

[0136] Filter target vectors: In the similarity score sequence, select vectors with similarity scores higher than the threshold as target vectors.

[0137] Determine the relevant data:

[0138] The text corresponding to the target vector is determined as data related to the user's question. This data will be used as a reference for generating responses later.

[0139] For example:

[0140] Suppose there is a database that stores a lot of answers to common questions about mobile phones. When a user asks "My mobile phone battery is not durable, what should I do?", the system will:

[0141] Convert the question “What should I do if my phone battery doesn’t last long?” into a second text vector.

[0142] The texts of all questions in the database are also converted into the first text vector.

[0143] Calculate the cosine similarity between the user question vector (second text vector) and the vector of each question in the database (first text vector).

[0144] Find the questions with the highest similarity, such as "Why do mobile phone batteries not last long?", "How to extend the life of mobile phone batteries?", etc.

[0145] The answers to these questions are provided to the big model as relevant data to help the model generate more accurate and relevant responses.

[0146] By converting text into vectors and calculating the similarity between vectors, we can efficiently find the most relevant parts of user questions from a large amount of text data. This method is not only applicable to intelligent customer service, but also to search engines, recommendation systems and other fields.

[0147] In an optional embodiment of the present invention, the step of calculating the cosine similarity between the second text vector and the first text vector in the database one by one and determining the similarity score between the second text vector and the first text vector comprises:

[0148] Inputting the first text vector and the second text vector into Formula 1 to determine a similarity score between the second text vector and the first text vector;

[0149] The formula 1 is:

[0150]

[0151] Among them, D i is the first text vector, Q is the second text vector, S i is the similarity score.

[0152] Formula 1 is the standard formula for calculating the cosine similarity between two vectors (i.e., text vectors).

[0153] in:

[0154] S i : The cosine similarity between vectors Q and Di, with a value range of [-1,1]. The closer it is to 1, the more similar the two vectors are; the closer it is to -1, the less similar the two vectors are; 0 means the two vectors are orthogonal, that is, completely unrelated.

[0155] Q: represents the query vector (i.e., the vector representation of the user’s question).

[0156] D i : Represents the vector representation of the i-th text in the database.

[0157] Q·D i : Represents the dot product of two vectors.

[0158] ||Q|| and ||D i || represents vectors Q and D respectively i The mold length.

[0159] Geometric meaning: Cosine similarity represents the cosine value of the angle between two vectors. If the two vectors are in the same direction, the cosine similarity is 1; if the two vectors are in opposite directions, the cosine similarity is -1; if the two vectors are orthogonal, the cosine similarity is 0. In the text vector space, the smaller the angle, the higher the semantic similarity between the two texts.

[0160] Not affected by vector length: Cosine similarity only cares about the directional relationship between vectors, not the size of the vectors. This makes it robust when dealing with texts of different lengths.

[0161] Simple calculation: The calculation formula of cosine similarity is simple and easy to understand, and the calculation efficiency is high.

[0162] Calculation process example:

[0163] Suppose we have transformed the two texts into the following two vectors:

[0164] Q = [0.5, 0.8, 0.2]

[0165] Di=[0.6,0.4,0.8]

[0166] The calculation process is as follows:

[0167] Calculate the vector dot product: Q·Di=0.5*0.6+0.8*0.4+0.2*0.8=0.7

[0168] Calculate the vector modulus: ||Q||=sqrt(0.5^2+0.8^2+0.2^2)≈0.94||Di||=sqrt(0.6^2+0.4^2+0.8^2)≈1

[0169] Calculate cosine similarity: Si = 0.7 / (0.94*1)≈0.74

[0170] Therefore, the cosine similarity of these two vectors is about 0.74, indicating that the two texts have a high similarity.

[0171] Application in text similarity calculation:

[0172] Information Retrieval: Based on cosine similarity, the most relevant text to the user query can be retrieved from a large amount of text.

[0173] Text clustering: Texts can be clustered into different categories based on the cosine similarity between texts.

[0174] Recommendation system: It can calculate the cosine similarity between the user preference vector and the item vector based on the user's historical behavior data, and recommend similar items to the user.

[0175] Question answering system: It can find the most appropriate answer by calculating the similarity between user questions and questions in the knowledge base.

[0176] Cosine similarity is a simple and effective method for calculating text similarity, which is widely used in the field of natural language processing. By converting text into vector representation and calculating the cosine similarity between vectors, efficient and accurate text similarity calculation can be achieved.

[0177] In an optional embodiment of the present invention, the step of generating initial reply information for the spliced ​​data includes:

[0178] Inputting the correlation data and the second text vector into Formula 2 to generate initial reply information for the spliced ​​data;

[0179] The formula 2 is:

[0180] f(D k∈K ,Q)=R0

[0181] Among them, f is the pre-trained large language model, D k∈K is the correlation data, and R0 is the initial reply information.

[0182] Formula 2 describes a text generation process based on a large language model. This process combines user questions (represented as vector Q) and related data (represented as vector D k∈K ) as input, through a pre-trained large language model f, to generate the initial response R0.

[0183] f: This is the core part, which represents a pre-trained large language model. This model has been trained on large-scale text data and has strong language understanding and generation capabilities.

[0184] D k∈K : Represents the text related to the user's question retrieved from the knowledge base, which is vectorized and used as one of the inputs of the model.

[0185] Q: A vector representation of the user’s question, which is used as the input of the model together with Dk.

[0186] R0: represents the initial response generated by the model.

[0187] The meaning of the formula is: user questions and related data are input into the pre-trained large language model together. The model will comprehensively consider these two inputs and generate the most likely response.

[0188] Large language models are usually based on the Transformer architecture, which uses the self-attention mechanism to capture long-range dependencies in the input sequence. During the training process, the model learns a large amount of text data, thus having the ability to generate text, translate languages, and write different types of creative content.

[0189] When model f receives input (user question and relevance data), it performs the following steps:

[0190] Encoding: Encode the input text (including user questions and relevance data) into a vector representation that the model can process.

[0191] Decoding: The model autoregressively generates the next word based on the encoded input until a complete sentence or paragraph is generated as the initial response.

[0192] Decoding process: During the decoding process, the model refers to the information in the input and the previously generated words to predict the next word.

[0193] The purpose of using a large language model is to:

[0194] Powerful generation capabilities: Large language models are able to generate diverse, high-quality texts covering a wide range of styles and topics.

[0195] Contextual understanding: Large language models are able to understand contextual information and generate responses that are relevant to the context.

[0196] Scalability: Large language models can continuously learn new knowledge and improve their generative capabilities.

[0197] Advantages of Formula 2:

[0198] End-to-end: The process from text vector to response generation is unified into one formula, which simplifies the structure of the model.

[0199] Interpretability: Although the internal mechanisms of large language models are very complex, through this formula, we can roughly understand how the model works.

[0200] Flexibility: The model’s structure, parameters, and training data can be adjusted to suit different tasks and scenarios.

[0201] Formula 2 describes a text generation process based on a large language model, which generates initial responses by inputting user questions and related data into a pre-trained large language model. This method fully utilizes the powerful capabilities of the large language model to generate high-quality and diverse responses, and has broad application prospects in the fields of intelligent customer service and machine translation.

[0202] In an optional embodiment of the present invention, the step of calculating evaluation information for the user question, the correlation data and the initial reply information includes:

[0203] Inputting the correlation data, the second text vector and the initial reply information into Formula 3 to obtain evaluation information; the evaluation information includes a multi-dimensional evaluation for the evaluation dimension and a quantitative score for the initial reply information;

[0204] The formula three is:

[0205] g(d k∈K ,Q,R0)=A0,s0

[0206] g is the evaluation model, A0 is the multi-dimensional evaluation, and s0 is the quantitative score.

[0207] Formula 3 describes an evaluation model g, which transforms the correlation data D k∈K , user question Q, and initial response R0 as input, and output multi-dimensional evaluation A0 and quantitative score s0.

[0208] g: This is an evaluation model, which is used to comprehensively evaluate the generated responses based on the input data.

[0209] D k∈K : Represents multiple texts related to the user's question retrieved from the knowledge base, which together constitute the relevance data.

[0210] Q: represents the user’s question, i.e., the query vector.

[0211] R0: represents the initial response generated by the model.

[0212] A0: represents the multi-dimensional evaluation of the response, such as accuracy, relevance, fluency, etc.

[0213] s0: represents the quantitative score of the response, usually a numerical value used to measure the overall quality of the response.

[0214] How the evaluation model works:

[0215] The evaluation model g is usually a machine learning model, such as a neural network, which learns from a large amount of training data to learn how to evaluate the generated responses based on the input data.

[0216] Training data: The training data contains a large number of question-answer pairs and the corresponding manually annotated evaluation scores. The model learns to map the input text to the corresponding evaluation scores by learning from this data.

[0217] Evaluation process: When the model receives new inputs (i.e. user questions, relevance data, and generated responses), it uses these inputs as features and feeds them into the model for prediction, resulting in a multi-dimensional evaluation vector and a quantitative score.

[0218] The significance of multi-dimensional evaluation and quantitative scoring:

[0219] Multi-dimensional evaluation: By evaluating on multiple dimensions, the quality of the response can be more comprehensively assessed. For example, a response may perform well in terms of accuracy but poorly in terms of fluency.

[0220] Quantitative scoring: Quantitative scoring can convert the evaluation results into a numerical value, which is convenient for comparison and sorting.

[0221] Advantages of Formula 3:

[0222] Comprehensive Assessment: Ability to consider multiple factors simultaneously and make a comprehensive assessment of the responses.

[0223] Customizable: Evaluation indicators can be customized by adjusting the model structure and training data.

[0224] Automation: It is possible to automatically evaluate the generated responses and improve efficiency.

[0225] Formula 3 describes a model for evaluating the quality of generated text. Through this model, the generated responses can be evaluated in a multi-dimensional and quantitative manner, so as to better control the generation process and improve the quality of the generated text.

[0226] In an optional embodiment of the present invention, it also includes:

[0227] When the multi-dimensional evaluation and the quantitative score do not meet the preset conditions, updated reply information is generated by the large language model based on the multi-dimensional evaluation, the quantitative score, the relevance data, the second text vector and the initial reply information.

[0228] Optionally, it also includes:

[0229] A scoring threshold for the quantitative score is determined, and when the quantitative score is greater than or equal to the scoring threshold, the updated reply information is determined as the target reply information, and iterating the large language model is stopped.

[0230] Exemplarily, the generation of update reply information can be implemented by formula four.

[0231] Formula 4:

[0232] f(D k∈K ,Q,R0,A0,s0)=R1

[0233] Formula 4 represents a function f, which outputs a new response R1 based on five input parameters. The meanings of these five parameters are as follows:

[0234] D k∈K : Multiple texts related to user questions retrieved from the knowledge base, namely, relevance data.

[0235] Q: Questions raised by users.

[0236] R0: The response initially generated by the large model.

[0237] A0: Multi-dimensional evaluation of R0, such as accuracy, relevance, fluency, etc.

[0238] s0: A quantitative score for R0, which is a numerical value indicating the overall quality of R0.

[0239] R1: Based on the above information, a new reply is generated after reflection and optimization, that is, the updated reply information.

[0240] Detailed Explanation of Reflection Optimization Process

[0241] a) Threshold mechanism:

[0242] Set threshold: Set a dynamic threshold τ, which can be adjusted according to different application scenarios and requirements for response quality.

[0243] Trigger condition: When the quantitative score s0 is lower than the threshold τ, it means that the quality of the initial response R0 does not meet the requirements, and the system will trigger the reflection mechanism.

[0244] b) Feedback signal construction:

[0245] Multi-dimensional information encoding: The multi-dimensional evaluation A0 and quantitative score s0 generated by the evaluation model are encoded into a structured feedback signal. This feedback signal can contain a variety of information, such as:

[0246] Which dimensions scored low?

[0247] How close is the overall score to the threshold?

[0248] Are there obvious errors of logic or fact?

[0249] The role of feedback signals: Feedback signals will provide the large model with detailed feedback on the initial response questions, helping the model to optimize in a targeted manner.

[0250] c) Restore and regenerate:

[0251] The big model receives the following information as input:

[0252] Original context (concatenated data): includes user question Q and related data D k∈K , initial response R0 and feedback signal.

[0253] Generate a new response: The big model will take this information into consideration and generate a new response R1. This new response should correct the problems in the initial response to a certain extent and be more in line with user expectations.

[0254] Formula 4 describes a closed-loop optimization process. After the big model generates the initial response, the system will evaluate the response. If the evaluation result is not ideal, the reflection mechanism will be triggered and the big model will optimize the response based on the feedback information. This process is iterated continuously until a response that meets the quality requirements is generated.

[0255] The advantages of using formula 4 to calculate the update reply information are:

[0256] Improve response quality: By introducing a reflection mechanism, the quality of generated responses can be effectively improved.

[0257] Strong adaptability: The dynamic threshold mechanism allows the system to flexibly adjust the requirements for response quality according to different scenarios.

[0258] Interpretability: Multi-dimensional evaluation can provide detailed feedback on the quality of responses, which helps to analyze and improve models.

[0259] The reflective optimization process described in Formula 4 provides an effective method for improving the quality of responses generated by large language models. By feeding the evaluation results back to the model and guiding the model to optimize, the performance of the system can be continuously improved.

[0260] Optionally, the embodiment of the present invention may also execute multiple reflection large model generation processes.

[0261] In a specific implementation, the final response may not only be given based on a single evaluation, but the generated responses may be evaluated and optimized multiple times until the preset quality standards are met.

[0262] The system continuously repeats the following steps: generating a reply, evaluating the reply, and generating a new reply based on the evaluation results until the termination condition is met.

[0263] Exemplarily, this can be achieved by formula five.

[0264] Formula 5:

[0265] f(D k∈K ,Q,R0,A0,s0,R1,A1,s1,…,R i ,A i ,s i )=R i+1

[0266] enter:

[0267] D k∈K : correlation data;

[0268] Q: User questions;

[0269] R0,R1,...,R i : previously generated responses;

[0270] A0,A1,...,A i : A multi-dimensional evaluation of previous responses;

[0271] S 0, S1,...,S i : Quantitative rating of previous responses;

[0272] Output:

[0273] R i+1 : The response generated for the i+1th time.

[0274] Formula 5 means: When the model generates a new response, it not only considers the current input (user question, relevance data), but also refers to all previously generated responses, evaluation results, and corresponding scores. This enables the model to learn from past mistakes and continuously improve the quality of responses.

[0275] Optimization objectives and termination conditions:

[0276] Optimization goal: maximize the rating of the evaluation model. In other words, the system will continuously adjust the generated responses to obtain higher evaluation scores.

[0277] Termination conditions:

[0278] The score exceeds the preset threshold τ: When the score of the generated reply exceeds the preset threshold, it means that the quality of the reply has met the requirements and the iteration can be stopped.

[0279] Reaching the maximum number of iterations N: If the score still does not reach the threshold after multiple iterations, the system will set a maximum number of iterations to avoid falling into an infinite loop.

[0280] Iterative optimization process:

[0281] Initial generation: Generate the first response R0 based on the user question and relevance data.

[0282] Evaluation: R0 is evaluated in multiple dimensions and quantitatively scored to obtain A0 and S0.

[0283] Judgment: If S0 exceeds the threshold τ, the iteration ends; otherwise, proceed to the next step.

[0284] Feedback: Provide information such as R0, A0, S0 as input to the model.

[0285] Generate new response: The model generates a new response R1 based on the new input.

[0286] Repeat the previous steps, and keep repeating the process of evaluation, judgment, feedback and generation until the termination condition is met.

[0287] Advantages and benefits:

[0288] Improve response quality: Through multiple iterations, the model can continuously learn and improve, generating more accurate, relevant, and fluent responses.

[0289] Strong adaptability: The system can adjust the threshold and maximum number of iterations according to different tasks and scenarios to meet different needs.

[0290] Interpretability: Multi-dimensional evaluation can provide detailed feedback on the quality of responses, which helps to analyze and improve models.

[0291] Rethinking the model generation process multiple times is an effective optimization method. Through continuous iteration and improvement, the quality of the responses generated by the model can be significantly improved. This method has broad application prospects in the field of natural language processing.

[0292] In order to enable those skilled in the art to better understand the embodiment of the present invention, an example is used below to illustrate the embodiment of the present invention.

[0293] The technical solution of this patented invention mainly involves four aspects: ① Generation based on a retrieval and recall large model; ② The evaluation model scores the generated responses; ③ Generation of the large model based on a reflection mechanism; ④ Multiple reflections on the large model generation process;

[0294] (I) Generation based on a retrieval and recall large model

[0295] The system first performs vector database modeling, using the BGE model to convert the text in the database into the first text vector representation. We use D i to represent the i-th vector in the database (0 ≤ i < N, where N is the number of text vectors in the database). Secondly, when the system receives the user's question, it uses the BGE model to convert the user's question into the second text vector representation, denoted as Q.

[0296] The system will calculate the cosine similarity between the second text vector and the first text vectors in the database one by one, so as to obtain the similarity score S i between the user query Q and the database vector D i , and the formula is as follows:

[0297]

[0298] After obtaining the similarity score S i , we sort the scores and select the texts corresponding to the K vectors with the largest scores as the relevant data D k∈K . Subsequently, the retrieved relevant data and the user query are concatenated in a specific template format to construct a structured context input. The constructed context input is passed to the trained large language model, and the model, based on the input, outputs an initial response reply through autoregressive decoding or sequence-to-sequence generation. We call the large language model that generates the reply f, and the following formula can be obtained, where R0 is the initial reply generated by the large model.

[0299] f(D k∈K ,Q) = R0

[0300] (II) The evaluation model scores the generated responses

[0301] An independent evaluation model is introduced, which receives three key inputs: the retrieved relevant data D k∈K , the user query Q, and the initial reply R0 generated by the large model. The evaluation model gives an analysis process and a quantitative score through multi-dimensional analysis. Specifically, it includes:

[0302] a) Design of judgment dimensions: including but not limited to factual accuracy, relevance, integrity, language fluency, etc.

[0303] b) Scoring mechanism: The score range is 0-10 points. The model first gives a natural language evaluation of each dimension, and then integrates the multi-dimensional evaluation into a single quantitative score.

[0304] c) Suggestions for improvement: Based on quantitative scoring and multi-dimensional evaluation, provide suggestions and directions for improvement.

[0305] Therefore, according to the above process, the evaluation model is called g, and the following formula can be obtained, where A0 and s0 represent the multi-dimensional evaluation and quantitative score given by the evaluation model to R0 respectively:

[0306] g(D k∈K ,Q,R0)=A0,s0

[0307] 3. Large Model Generation Based on Reflection Mechanism

[0308] The system decides whether to start the reflection optimization process based on the quantitative score of the evaluation model:

[0309] a) Threshold mechanism: A dynamic threshold τ (eg, τ=7) is set, and the reflection mechanism is triggered when the quantitative score is lower than τ.

[0310] b) Feedback signal construction: The multi-dimensional analysis results and quantitative scores of the evaluation model are encoded into structured feedback signals.

[0311] c) Response regeneration: The large model receives the original context, the initial generated response, and the feedback signal to produce an optimized response.

[0312] The following formula can be obtained, where R1 is the new response generated by the large model after reflection:

[0313] f(D k∈K ,Q,R0,A0,s0)=R1

[0314] (IV) Rethinking the large model generation process multiple times

[0315] The system enters an iterative optimization loop and repeats (ii) and (iii) until the termination condition is met:

[0316] a) Optimization objective: maximize the score of the evaluation model.

[0317] b) Termination condition: the score exceeds a preset threshold τ or reaches a maximum number of iterations N (eg, N=3).

[0318] The formula for the i-th reflection process can be expressed as:

[0319] f(D k∈K ,Q,R0,A0,s0,R1,A1,s1,…,R i ,A i ,s i)=R i+1

[0320] That is, the model needs to receive all model generation, reflection evaluation, and reflection scoring before the current reflection stage, so as to regenerate the response.

[0321] refer to Figure 2 , Figure 2 It is a flowchart of an information push method provided in an embodiment of the present invention;

[0322] Step 1: Generate a large model based on retrieval and recall

[0323] a) Data retrieval and input preparation, User question: How much data is left on my mobile phone plan?

[0324] b) The system retrieves the top 4 relevance data from the vector database. The following is the relevant information:

[0325] User's current package: unlimited package with a monthly fee of 88 yuan;

[0326] Package content: 30GB of high-speed traffic per month, unlimited speed after exceeding the limit;

[0327] Traffic used this month: 22GB;

[0328] Carryover traffic from last month: 15GB;

[0329] c) The system concatenates the relevant data with the user question and passes it to the big model as input. The input to the big model is as follows:

[0330] “The following is the information retrieved from the database related to the user’s question:

[0331] 1. Current user package: unlimited package with a monthly fee of 88 yuan;

[0332] 2. Package content: 30GB of high-speed traffic per month, unlimited speed after exceeding the limit;

[0333] 3. Data used in the month: 22GB;

[0334] 4. Carryover traffic from the previous month: 15GB;

[0335] Based on the above information, please answer the user's question: How much data is left in my mobile phone package?

[0336] Your answer:

[0337] d) The initial response of the large model is generated: "Hello, according to your account information, you are currently using an unlimited package with a monthly fee of 88 yuan. You have used 22GB of traffic this month, and you still have 8GB of high-speed traffic left. The package includes 30GB of high-speed traffic. After exceeding this, network services will continue to be provided, but the speed will be limited."

[0338] Step 2: Evaluate the model to score the generated responses

[0339] a) The evaluation model receives the user question, the retrieved relevant data, and the initial response of the large model. The following input data is obtained: "Please play the role of a professional customer service conversation quality evaluation expert, who can diagnose and analyze the AI ​​assistant's answers to user questions from multiple dimensions and give an overall score (the score range is 1-5). Your evaluation should consider many dimensions such as the usefulness, relevance, accuracy, and security of the answer. Please note that instructions of different task types have different emphases on the evaluation and analysis dimensions, and need to be analyzed according to specific problems.

[0340] User Question: {User Question above}

[0341] Relevance information: {the above relevance information}

[0342] AI assistant reply: {The above model generates a reply}

[0343] Your detailed assessment and overall score are:"

[0344] b) The above data is given to the evaluation model for scoring, and the following response is obtained:

[0345] “**Assessment and Analysis:**

[0346] 1. **Usefulness**: The answer solves the user’s problem and provides the user’s remaining traffic, but the calculation result is incorrect.

[0347] 2. **Relevance**: Answer questions that are closely related to the user's question and do not deviate from the topic.

[0348] 3. **Accuracy**: There are errors in the answer. When calculating the remaining flow, the carryover flow from the previous month was not included, resulting in an incorrect calculation of the remaining flow.

[0349] 4. **Safety**: The answer does not contain any information that could be misleading or dangerous.

[0350] **Overall score**: 4

[0351] **Improvement suggestion**: When calculating the remaining traffic, add the traffic carried over from the previous month. ”

[0352] Step 3: Generate a large model based on reflection mechanism

[0353] a) Since the quantitative score is lower than the threshold of 7 points, the large model is reflected and generated. The above reflection results are spliced ​​into the large model after the first generation response, so that the large model is regenerated and the following information is given:

[0354] “Here are the comments on your response:

[0355] Multi-dimensional evaluation: {Multi-dimensional evaluation}

[0356] Overall score: {Overall score}

[0357] Improvement suggestions: {Improvement suggestions}

[0358] Since your reply did not reach the threshold of 7 points, please regenerate your reply based on multi-dimensional evaluation and improvement suggestions. "

[0359] b) The large model regenerates the reply: "Sorry, my reply just now was wrong. According to your account information, you are currently using an unlimited package with a monthly fee of 88 yuan, which includes 30GB of high-speed traffic. In addition, you have 15GB of high-speed traffic carried over from last month. You have used 22GB of traffic this month, so there is 30GB+15GB-22GB=23GB of high-speed traffic left. After the traffic exceeds the limit, network services will continue to be provided, but the speed will be limited."

[0360] Step 4: Rethink the large model generation process multiple times

[0361] a) Give the latest reflection result to the evaluation model for evaluation and generate the following response:

[0362] “**Assessment and Analysis:**

[0363] 1. **Usefulness**: The answer solves the user's problem and provides correct calculation of the user's remaining traffic.

[0364] 2. **Relevance**: Answer questions that are closely related to the user's question and do not deviate from the topic.

[0365] 3. **Accuracy**: There are no calculation errors in the answer.

[0366] 4. **Safety**: The answer does not contain any information that could be misleading or dangerous.

[0367] **Overall rating**: 10

[0368] **Improvement suggestions**: None"

[0369] b) Since the score of the latest generated reply is higher than the threshold, the generated reply is returned to the user.

[0370] Through this example, we can see how this solution can improve the quality of responses from the intelligent customer service system in actual applications. If the quality of the initial response is not high enough, the reflection mechanism will be triggered, prompting the system to generate a more accurate and relevant response. This approach is particularly suitable for query scenarios that require a high degree of factual accuracy, such as bill inquiries, package details, etc.

[0371] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0372] Reference Figure 3 , shows a structural block diagram of an information push device provided in an embodiment of the present invention, which may specifically include the following modules:

[0373] The correlation data acquisition module 301 is used to acquire correlation data associated with the user question when receiving the user question;

[0374] A splicing data generating module 302 is used to splice the user question and the correlation data to generate splicing data;

[0375] An initial reply information generating module 303, used to generate initial reply information for the spliced ​​data;

[0376] An evaluation information calculation module 304 is used to determine evaluation dimensions, and based on the evaluation dimensions, calculate evaluation information for the user question, the correlation data, and the initial reply information;

[0377] The target reply information pushing module 305 is used to determine the initial reply information as the target reply information when the evaluation information meets the preset conditions, and push the target reply information to the user.

[0378] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0379] In addition, an embodiment of the present invention further provides an electronic device, such as Figure 4 As shown, it includes a processor 401, a communication interface 402, a memory 403 and a communication bus 404, wherein the processor 401, the communication interface 402, and the memory 403 communicate with each other through the communication bus 404.

[0380] Memory 403, used for storing computer programs;

[0381] The processor 401 is used to implement any of the information push methods described in the above embodiments when executing the program stored in the memory 403:

[0382] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0383] The communication interface is used for communication between the above terminal and other devices.

[0384] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0385] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0386] like Figure 5 As shown, in another embodiment provided by the present invention, a computer-readable storage medium 501 is also provided, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer executes the information push method described in the above embodiment.

[0387] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.

[0388] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the embodiments of the present invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0389] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0390] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0391] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0392] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0393] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, ROM, RAM, magnetic disks, or optical disks.

[0394] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. An information push method, characterized in that: include: When receiving a user question, obtaining correlation data associated with the user question; Splicing the user question and the correlation data to generate spliced ​​data; generating initial response information for the spliced ​​data; Determine an evaluation dimension, and based on the evaluation dimension, calculate evaluation information for the user question, the relevance data, and the initial reply information; When the evaluation information meets the preset conditions, the initial reply information is determined as the target reply information, and the target reply information is pushed to the user.

2. The method according to claim 1, characterized in that The step of obtaining correlation data associated with the user question includes: Converting a database text of a preset database for storing reply content into a first text vector; Convert the user question into a second text vector; Calculating cosine similarity between the second text vector and the first text vector in the database one by one, and determining a similarity score between the second text vector and the first text vector; sorting the similarity scores to generate a similarity score sequence; Selecting a vector with a score greater than a preset threshold in the similarity score sequence as a target vector; The text corresponding to the target vector is determined as the correlation data.

3. The method according to claim 2, characterized in that The step of calculating the cosine similarity between the second text vector and the first text vector in the database one by one to determine the similarity score between the second text vector and the first text vector comprises: Inputting the first text vector and the second text vector into Formula 1 to determine a similarity score between the second text vector and the first text vector; The formula 1 is: Among them, D i is the first text vector, Q is the second text vector, S i is the similarity score.

4. The method according to claim 3, characterized in that The step of generating initial reply information for the spliced ​​data comprises: Inputting the correlation data and the second text vector into Formula 2 to generate initial reply information for the spliced ​​data; The formula 2 is: f(D k∈K ,Q)=R0 Among them, f is the pre-trained large language model, D k∈K is the correlation data, and R0 is the initial reply information.

5. The method according to claim 4, characterized in that The step of calculating evaluation information for the user question, the correlation data and the initial reply information comprises: Inputting the correlation data, the second text vector and the initial reply information into Formula 3 to obtain evaluation information; the evaluation information includes a multi-dimensional evaluation for the evaluation dimension and a quantitative score for the initial reply information; The formula three is: g(D k∈K ,Q,R0)=A0,s0 g is the evaluation model, A0 is the multi-dimensional evaluation, and s0 is the quantitative score.

6. The method according to claim 5, characterized in that Also includes: When the multi-dimensional evaluation and the quantitative score do not meet the preset conditions, the updated reply information is generated by the large language model based on the multi-dimensional evaluation, the quantitative score, the relevance data, the second text vector and the initial reply information.

7. The method according to claim 6, characterized in that Also includes: A scoring threshold for the quantitative score is determined, and when the quantitative score is greater than or equal to the scoring threshold, the updated reply information is determined as the target reply information, and the iteration of the large language model is stopped.

8. An information push device, characterized in that: include: A correlation data acquisition module, used for acquiring correlation data associated with the user question when receiving the user question; A splicing data generating module, used for splicing the user question and the correlation data to generate splicing data; An initial response information generating module, used for generating initial response information for the spliced ​​data; An evaluation information calculation module, used to determine evaluation dimensions, and based on the evaluation dimensions, calculate evaluation information for the user question, the correlation data, and the initial reply information; The target reply information pushing module is used to determine the initial reply information as the target reply information when the evaluation information meets the preset conditions, and push the target reply information to the user.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; The memory is used to store computer programs; The processor is used to implement the method according to any one of claims 1 to 7 when executing the program stored in the memory.

10. A computer-readable storage medium having instructions stored thereon, which, when executed by one or more processors, cause the processors to perform the method according to any one of claims 1 to 7.

Images