Text generation method and device, equipment, storage medium and product

Abstract

The invention discloses a text generation method and device, equipment, a storage medium and a product. The method comprises the following steps: inputting a text vector of a text to be inquired into a text generation model, and carrying out encoding processing on the text vector by utilizing an encoder in the text generation model to obtain a first encoding vector and a second encoding vector; performing linear feature extraction on the first coding vector and the second coding vector to obtain a first fusion vector corresponding to the first coding vector and a second fusion vector corresponding to both the first coding vector and the second coding vector; performing nonlinear feature extraction on the first fusion vector and the first coding vector to obtain a first feature vector; performing cross fusion on the first feature vector and the second fusion vector to obtain a second feature vector; and decoding the second feature vector to obtain a reply text matched with the to-be-inquired text. According to the embodiment of the invention, the text generation accuracy can be improved when the text is generated.

Description

Technical Field

[0001] This application belongs to the field of computer text processing technology, and in particular relates to a text generation method, apparatus, device, storage medium and product. Background Technology

[0002] With the rapid development of ChatGPT and generative AI, conversational AI products and applications are becoming increasingly widespread. This trend has made improving the accuracy of conversational AI products a focus of industry attention. Against this backdrop, it is particularly important to use knowledge generation methods to feed back into the business knowledge base and assist AI products and applications in generalization and performance in business-side question-and-answer scenarios. This will ensure their reliability and robustness in practical applications, contribute to the development of conversational AI products and applications, and enhance overall product competitiveness.

[0003] Most existing short text knowledge generation technologies employ large-scale pre-trained generative models using deep learning techniques. These models learn feature representations by utilizing large-scale data, thereby completing the text generation strategy.

[0004] However, existing short text knowledge generation methods may not be able to fully understand contextual information, resulting in a lack of coherence and relevance in the generated text, i.e. a lack of contextual understanding ability, leading to low accuracy of the generated text. Summary of the Invention

[0005] This application provides a text generation method, apparatus, device, storage medium, and product to solve the problem of low accuracy in generated text.

[0006] In a first aspect, embodiments of this application provide a text generation method, the method comprising:

[0007] The text vector of the text to be queried is input into the text generation model, and the encoder in the text generation model is used to encode the text vector to obtain the first encoded vector and the second encoded vector.

[0008] Using the cross-fusion attention unit in the text generation model, linear feature extraction is performed on the first encoding vector and the second encoding vector to obtain a first fusion vector corresponding to the first encoding vector and a second fusion vector corresponding to both the first encoding vector and the second encoding vector.

[0009] Using the feedforward neural network in the text generation model, nonlinear feature extraction is performed on the first fusion vector and the first encoding vector to obtain the first feature vector;

[0010] Using the feedforward neural network and the encoder, the first feature vector and the second fusion vector are cross-fused to obtain the second feature vector;

[0011] Using the text generation model, the second feature vector is decoded to obtain the response text that matches the text to be queried.

[0012] Secondly, embodiments of this application provide a text generation apparatus, the apparatus comprising:

[0013] The input module is used to input the text vector of the text to be queried into the text generation model, and to encode the text vector using the encoder in the text generation model to obtain a first encoded vector and a second encoded vector.

[0014] The first extraction module is used to perform linear feature extraction on the first encoding vector and the second encoding vector using the cross-fusion attention unit in the text generation model, so as to obtain a first fusion vector corresponding to the first encoding vector and a second fusion vector corresponding to both the first encoding vector and the second encoding vector.

[0015] The second extraction module is used to perform nonlinear feature extraction on the first fusion vector and the first encoding vector using the feedforward neural network in the text generation model to obtain the first feature vector.

[0016] The fusion module is used to cross-fuse the first feature vector and the second fusion vector using the feedforward neural network and the encoder to obtain the second feature vector.

[0017] The decoding module is used to decode the second feature vector using the text generation model to obtain a response text that matches the text to be queried.

[0018] Thirdly, embodiments of this application provide a terminal device, the device including: a processor and a memory storing computer program instructions;

[0019] When the processor executes computer program instructions, it implements a text generation method as described in the first aspect.

[0020] Fourthly, embodiments of this application provide a computer storage medium on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the text generation method as described in the first aspect is implemented.

[0021] Fifthly, embodiments of this application provide a computer program product in which instructions, when executed by a processor of an electronic device, cause the electronic device to perform the text generation method as described in the first aspect.

[0022] The text generation method provided in this application encodes text vectors using an encoder to generate a first encoded vector and a second encoded vector, which helps the text generation model capture text information from different dimensions or levels. By cross-fusion attention units, linear feature extraction is performed on the encoded vectors to generate a fused vector. This cross-fusion approach not only considers the features of a single encoded vector but also integrates information between different vectors, enhancing the model's ability to capture the inherent relationships within the text. Simultaneously, the combination of linear and nonlinear feature extraction (using a feedforward neural network) further enriches the hierarchy and depth of features, improving the model's expressive power. This innovatively enhances the model's ability to capture and express short text features, strengthens contextual understanding, and solves the problems of information sparsity and insufficient contextual understanding faced by existing technologies when processing short texts. This results in improved performance for short text knowledge generation tasks, with more accurate and higher-quality semantic understanding of the generated text knowledge. Attached Figure Description

[0023] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a flowchart illustrating the text generation method provided in an embodiment of this application;

[0025] Figure 2 This is a schematic diagram of the dual attention cross-fusion enhancement mechanism in a text generation model;

[0026] Figure 3 Overall architecture diagram of the text generation model;

[0027] Figure 4 This is a schematic diagram of the structure of the text generation device provided in the embodiments of this application;

[0028] Figure 5 This is a schematic diagram of the structure of the terminal device provided in the embodiments of this application. Detailed Implementation

[0029] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0030] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0031] To address the problems of the prior art, embodiments of this application provide a text generation method, apparatus, device, storage medium, and product.

[0032] The text generation method provided in the embodiments of this application will be described below.

[0033] Figure 1 A flowchart illustrating a text generation method provided in one embodiment of this application is shown. Figure 1 As shown, the method includes:

[0034] Step 101: Input the text vector of the text to be queried into the text generation model, and use the encoder in the text generation model to encode the text vector to obtain the first encoded vector and the second encoded vector.

[0035] In this embodiment, the query text can be first vectorized to obtain the corresponding text vector, and then the text vector can be input into the text generation model. For example... Figure 2 and Figure 3 As shown, Figure 2 This is a schematic diagram of the dual attention cross-fusion enhancement mechanism in text generation models. Figure 3 The overall architecture diagram of the text generation model. This text generation model has two encoders, namely the first encoder and the second encoder, i.e. Figure 2Encoder 1 and Encoder 2 in the example.

[0036] Specifically, in one embodiment, step 101 includes:

[0037] The text vector is input into the text generation model, and the first encoder is used to process the text vector to obtain the first encoded vector;

[0038] The second encoder is used to process the first encoded vector to obtain the second encoded vector.

[0039] In this embodiment, the text vector first needs to be encoded using a first encoder to obtain a first encoded vector (i.e., Figure 2 The first encoded vector (B1) is then input into the second encoder for encoding processing to obtain the second encoded vector (D2).

[0040] In this embodiment, by introducing two independent encoders to process text vectors, this method achieves multi-level and multi-angle parsing of text information. This not only deepens the understanding of the text content but also captures complex relationships and abstract concepts between texts at a higher level. Since the second encoder is based on the first encoder, it can utilize the information already extracted from the first encoded vector to perform more refined and in-depth analysis, thereby generating richer and more accurate second encoded vectors. Through this hierarchical encoding approach, the model can capture text information more comprehensively, improving the depth and breadth of the model's understanding of the text content.

[0041] Step 102: Using the cross-fusion attention unit in the text generation model, perform linear feature extraction on the first encoding vector and the second encoding vector to obtain a first fusion vector corresponding to the first encoding vector and a second fusion vector corresponding to both the first encoding vector and the second encoding vector.

[0042] In this embodiment, the cross-fusion attention unit includes a first self-attention unit and a second self-attention unit. Figure 2 In this context, "Attention" is used to represent the concept.

[0043] In one embodiment, step 102 includes:

[0044] The first self-attention unit is used to extract features from the first encoded vector to obtain the first fusion vector;

[0045] The first and second encoded vectors are fused to obtain the third encoded vector;

[0046] The second self-attention unit is used to extract features from the third encoding vector to obtain the second fusion vector.

[0047] In this embodiment, the first encoding vector B1 output by the first encoder flows to two self-attention units. On one hand, B1 undergoes attention processing to generate a first fusion vector SelfAttention1, which then flows back to the first encoder; on the other hand, B1 is fused with the second encoding vector D1 output by the second encoder and flows to the second self-attention unit, where it undergoes attention processing to generate a second fusion vector SelfAttention2. The formula is as follows:

[0048] SelfAttention1=Attention(B1)

[0049] SelfAttention2=Attention(D1+B1)

[0050] The fusion method for vectors B1(x1,x2,x3,…,xn) and D1(y1,y2,y3,…,yn) is additive fusion, that is, the two vectors are added element by element to obtain the fused third encoded vector. The formula is as follows:

[0051] D1+B1=(x1+y1,x2+y2,x3+y3,…,x n +y n )

[0052] In this embodiment, a dual attention mechanism is introduced into the model, which can capture information from text vectors from multiple angles. At the same time, the dual attention units are fused and parameters are passed between them. Through linear processing, the model’s ability to capture and express short text features is innovatively enhanced, and the ability to understand the context is improved. This solves the problems of information sparsity and insufficient context understanding faced by existing technologies when processing short texts.

[0053] Step 103: Using the feedforward neural network in the text generation model, perform nonlinear feature extraction on the first fusion vector and the first encoding vector to obtain the first feature vector;

[0054] Specifically, in one embodiment, step 103 includes:

[0055] The first fusion vector and the first encoding vector are fused to obtain a third fusion vector;

[0056] The feedforward neural network is used to perform nonlinear feature extraction on the third fusion vector to obtain the first feature vector.

[0057] In this embodiment, since matrix calculations in step 102 can only capture linear relationships between vectors, a feed-forward network (FFN) is introduced to perform nonlinear transformations and feature extraction on the input features based on multiple fully connected layers. The SelfAttention1, flowing to the first encoder, fuses the first encoded vector B1 output by the first encoder to obtain a third fused vector. After processing by the FFN, the first feature vector FFN1 is output and transmitted to the second encoder. The formula is as follows:

[0058] FFN1 = FFN(SelfAttention1 + B1)

[0059] In this embodiment, the FFN consists of an input layer, three hidden layers, and an output layer. Taking the first hidden layer as an example, the output is as shown in the formula:

[0060] f(x1) = w 1 x1+b 1

[0061] Where x1 is the output vector, w 1 As the weight, b 1 As a bias, function f uses the sigmoid activation function.

[0062] It should be noted that, for ease of illustration, Figure 2 The two encoders 1 are the same encoder, that is, both are the first encoder, and the B2 output from encoder 1 is the same as B1, both being the first encoded vector. The two encoders 2 are the same encoder, that is, both are the second encoder.

[0063] In this embodiment, by fusing the first fusion vector and the first encoding vector to obtain the third fusion vector, information from different processing stages can be integrated. The first encoding vector may contain the basic semantic and structural information of the text, while the first fusion vector may contain richer contextual dependencies after advanced processing such as self-attention mechanisms. The fusion of the two allows the third fusion vector to more comprehensively represent the features of the text. By using a feedforward neural network to perform nonlinear feature extraction on the third fusion vector, the most valuable and representative features of the text to be queried can be extracted. Using these features can improve the model's generalization ability when processing unseen text.

[0064] Step 104: Using the feedforward neural network and the encoder, the first feature vector and the second fusion vector are cross-fused to obtain the second feature vector;

[0065] In one embodiment, step 104 specifically includes:

[0066] The first feature vector is processed using the second encoder to obtain the third feature vector;

[0067] The third feature vector and the second fusion vector are fused to obtain a fourth fusion vector;

[0068] The feedforward neural network is used to process the fourth fusion vector to obtain the second feature vector.

[0069] In this embodiment, the first feature vector FFN1 is first processed by the second encoder, and the resulting vector is the third feature vector D2, as shown in the following formula:

[0070] D2 = Encoder2(FFN1)

[0071] After adding and fusing D2 with the second fusion vector SelfAttention2, a fourth fusion vector is obtained. This fourth fusion vector is then processed by a feedforward neural network to finally output the second feature vector FFN2. The formula is as follows:

[0072] FFN2 = FFN(D2 + SelfAttention2)

[0073] In this embodiment, a third fusion vector is obtained by fusing the first fusion vector and the first encoding vector, and a feedforward neural network is used to extract nonlinear features from it to obtain the first feature vector. In this process, the fusion process integrates information from different processing stages, enhancing the comprehensiveness and richness of the features; nonlinear feature extraction can capture more complex and abstract patterns in text data, improving the model's ability to understand text content.

[0074] Step 105: Using the text generation model, decode the second feature vector to obtain the response text that matches the text to be queried.

[0075] In this embodiment, as Figure 3 As shown, the text generation model also includes a first decoder (decoder 1) and a second decoder (decoder 2). After obtaining the second feature vector, the feedforward neural network transmits the second feature vector to different layers of each decoder in the form of residual connections, so that the second feature vector is transmitted forward without loss.

[0076] Among them, residual connections introduce a cross-layer short-circuit connection method to optimize deep neural networks. The second feature vector FFN2 bypasses some layers and is added to and fused with the output vectors of these layers before being fed into the attention modules of the first and second decoders.

[0077] Thus, by fusing the outputs of two self-attention units and the results of cross-fusing parameter-passing attention units, the model's ability to capture textual semantic features has been enhanced.

[0078] In this embodiment, the text vector is encoded by an encoder to generate a first encoded vector and a second encoded vector, which helps the text generation model capture text information from different dimensions or levels. By cross-fusing attention units, linear feature extraction is performed on the encoded vectors to generate a fused vector. This cross-fusion approach not only considers the features of a single encoded vector but also integrates information between different vectors, enhancing the model's ability to capture the inherent relationships within the text. Simultaneously, the combination of linear and nonlinear feature extraction (using a feedforward neural network) further enriches the hierarchy and depth of features, improving the model's expressive power. This innovative approach enhances the model's ability to capture and express short text features, strengthens contextual understanding, and solves the problems of information sparsity and insufficient contextual understanding faced by existing technologies when processing short texts. This results in improved performance for short text knowledge generation tasks, with more accurate and higher-quality semantic understanding of the generated text knowledge.

[0079] In one embodiment of this application, before inputting the text vector of the text to be queried into a text generation model, and using the encoder in the text generation model to encode the text vector to obtain a first encoded vector and a second encoded vector, the method further includes:

[0080] Obtain multiple similar questions that have a similarity greater than a first threshold with the standard question;

[0081] The multiple similar problems are preprocessed to obtain a first training set, which includes the preprocessed multiple similar problems.

[0082] The first training set is used to train the model to be trained, and the first model is obtained.

[0083] Add a perturbation to each of the multiple similar problems to obtain a second training set;

[0084] The first model is trained using the second training set to obtain the text generation model.

[0085] In this embodiment, the aforementioned standard question refers to a question with a standard format in the source system knowledge base corresponding to the text generation model. When obtaining similar questions, the standard question can be used as an index seed to index similar questions from public data of web search engines (such as Baidu Knows, Wikipedia, etc.). After preprocessing, these questions are organized into a set of similar questions. Then, a high threshold is set through similarity retrieval (BM25, vector retrieval based on Word2Vec, etc.) to filter out data that exceeds the threshold. The system format is then converted to obtain multiple similar questions.

[0086] Preprocessing specifically includes filtering abnormal characters, deduplication, formatting, word segmentation, and vectorization. After the above processing, the first training set is obtained. As shown in Table 1, Table 1 is the first training set obtained after preprocessing, where the left side represents the standard set and the right side represents the first training set.

[0087]

[0088] Table 1, First Training Set

[0089] After obtaining the first training set, the model to be trained can be trained using the first training set to obtain the first model, where the model to be trained can be SimBERT.

[0090] After obtaining the first model, in order to enhance the model's adaptability to changes in the input text and improve its robustness, robustness training is also required.

[0091] Specifically, perturbations can be added to each similar problem to obtain a second training set. Perturbation types include word perturbation, synonym repetition, and non-fluency perturbation, and one or more types can be selected based on the actual situation. 1) Word perturbation: Four perturbation methods are set based on word granularity or word granularity: First, random word insertion to create redundancy; second, random word deletion to create omissions; third, synonym replacement or slot replacement, replacing words with near-synonyms, homophones, or similar-looking characters; fourth, swapping the order of words with the same or similar attributes to create word order changes. Through these methods, text training is generated that can resist noise perturbations and adapt to linguistic diversity. 2) Synonymous paraphrasing perturbation: describing a sentence again using different expressions. 3) Non-fluency perturbation: simulating the user's colloquial expression characteristics, adding user-specific colloquial descriptions, inappropriate pauses, etc., to sentences, such as "Why can't the camera on the wall in my living room connect to the internet?". Through the above methods, a new robust training set is finally generated that simulates a real question-and-answer scenario with various disturbances and noises.

[0092] Table 2 shows text examples implementing four perturbation methods at the word level.

[0093]

[0094] Table 2 shows text examples implementing four perturbation methods at the word level.

[0095] After obtaining the second training set, the first model was iteratively trained using optimizers such as Adam and SGD. Cross-validation and other methods were used to evaluate the model's performance. Simultaneously, some SimBERT hyperparameters, such as the number of attention heads, hidden layer size, and Dropout rate, were adjusted. Combining the first and second training sets improved the model's performance in real-world, complex user environments, enhancing the accuracy and reliability of the generated text. Real-time training and monitoring were performed using a negative cross-entropy loss function. Fine-tuning ended when the loss function converged, resulting in the final text training model.

[0096] The cross-entropy lossy function evaluates the inconsistency between the probability distribution of the predicted next word and the true distribution in each generation step, as shown in the following formula.

[0097]

[0098] Where C is the number of categories, y i It is the i-th encoding of the real label, p i It is the probability distribution predicted by the model.

[0099] In this embodiment, firstly, multiple questions with high similarity to the standard question are acquired as training data. This ensures that the model is exposed to inputs closely related to real-world application scenarios from the initial training stage, which helps improve the model's generalization ability. Next, these questions are preprocessed to construct a high-quality first training set, laying a solid foundation for the model's initial training. Furthermore, a second training set is constructed by adding perturbations to similar questions. This not only enhances the model's robustness but also enables it to better cope with various variations that may occur in real-world applications.

[0100] In one embodiment of this application, after training the first model using the second training set to obtain the text generation model, the method further includes:

[0101] The standard question is processed using the encoder in the text generation model to obtain a sentence vector;

[0102] Unidentified user questions whose similarity to the sentence vector is greater than a second threshold are identified as related knowledge;

[0103] The associated knowledge, the standard question, and the multiple similar questions corresponding to the standard question are stored as a knowledge set.

[0104] In this embodiment, the generation ratio of standard questions to similar questions is 1:n. Similar questions are sorted according to their similarity, and the top k data points are used to supplement the question-and-answer corpus. Specific values ​​can be adjusted according to actual circumstances. For example, in the algorithm evaluation experiment, n is set to 100, k to 10, and data alignment and annotation are performed to generate a knowledge set.

[0105] Knowledge generation based on unidentified user questions: First, based on the model's encoder, "sentence vectors" are generated from standard questions in the question-answering knowledge base of the training system, and an index is built and saved to an index file. Then, similarity retrieval is performed on the input unidentified user questions. Sentences with similarity greater than a certain threshold are tagged and marked as "related knowledge," while others are marked as "unrelated knowledge." Relevant knowledge is then added to the generated knowledge set. Here, unidentified user questions refer to questions input by users that are not recognized by the text generation model.

[0106] In this embodiment, the encoder processes the standard question to obtain sentence vectors, and identifies related knowledge (i.e., unidentified but semantically similar user questions) based on sentence vector similarity. This step not only enhances the model's semantic understanding ability but also achieves effective knowledge expansion and accumulation. Integrating related knowledge, standard questions, and similar questions into a knowledge set not only facilitates subsequent knowledge management and application but also improves the model's response speed and accuracy when handling similar questions.

[0107] In one embodiment of this application, after storing the associated knowledge, the standard question, and the plurality of similar questions corresponding to the standard question as a knowledge set, the method further includes:

[0108] The associated knowledge, the standard question, and the similar question in the knowledge set are sequentially input into the text generation model to obtain answers corresponding to the associated knowledge, the standard question, and the similar question, respectively.

[0109] The response time of the text generation model in generating the answer, the accuracy of the answer, and the filtering status of sensitive words in the knowledge set are obtained.

[0110] The quantitative analysis results of the text generation model are determined using the response time, accuracy, and sensitive word blocking.

[0111] In this embodiment, the generated knowledge set is input into the text generation model one by one for dialogue simulation, and the quantitative results of the generated knowledge are evaluated in conjunction with the actual business situation, as well as the knowledge generation quality recommendation table based on qualitative evaluation.

[0112] The quantitative results quality analysis includes evaluation indicators such as average response time, question-answering accuracy, sensitive word blocking effectiveness, and system robustness.

[0113] The formula for calculating the quantitative results is defined as follows:

[0114] Average response time: Where TN represents the response time for each round, which is the time difference between sending the question request and receiving the response, and N represents the number of simulated single-round question and answer sessions.

[0115] Question and answer accuracy: The ratio of the number of correct answers (A) to the total number of responses (Q).

[0116] Sensitive word blocking score: The ratio of the number of sensitive words M to the total number of words W in the sensitive word database used to correctly identify sensitive words.

[0117] As an example, Table 3 presents a quantitative analysis example of this evaluation scheme.

[0118]

[0119]

[0120] Table 3 provides an example of quantitative analysis of the scheme.

[0121] Robustness analysis assesses the system's ability to resist robustness issues from three dimensions: linguistic diversity, colloquial features, and noise perturbation, using a perturbation training dataset. The correlation between evaluation metrics and perturbation types is shown in Table 4.

[0122] ability Language diversity colloquial features noise disturbance Word perturbation √ √ Synonyms √ Unsmooth perturbation √

[0123] Table 4. Relationship between robustness assessment indicators and disturbance type

[0124] In this embodiment, firstly, by using related knowledge, standard questions, and similar questions as input, the application scenarios of the text generation model are further expanded, enhancing the model's practicality and flexibility. Secondly, by monitoring the model's response time, accuracy, and sensitive word blocking, a comprehensive evaluation of the model's performance is achieved, helping to promptly identify and optimize any problems. This not only improves the user experience but also enhances the model's reliability and security.

[0125] In one embodiment of this application, after storing the associated knowledge, the standard question, and the plurality of similar questions corresponding to the standard question as a knowledge set, the method further includes:

[0126] Obtain the actual responses from the text generation model under a preset business topic;

[0127] The knowledge hit rate of the text generation model under a preset business topic is determined by using the actual responses and preset standard answers. The knowledge hit rate is used to represent the qualitative analysis results of the text generation model.

[0128] In this embodiment, the qualitative evaluation specifically includes knowledge hit radar charts for each business topic, correlation analysis between actual responses and standard answers, etc.

[0129] Create a knowledge hit radar chart for each business topic to show the knowledge hit rate. Label the percentage or score for each topic on the radar chart. Compare the knowledge hit rates for different topics with the standard answers to determine relevance.

[0130] The Pearson correlation coefficient was used to quantify the correlation between genuine responses and standard answers. The correlation coefficient between genuine responses and standard answers for each topic was calculated, and a histogram of correlation coefficients was created.

[0131] The above indicators are categorized, integrated, and visualized to output a knowledge generation recommendation table containing both quantitative and qualitative analysis results.

[0132] In this embodiment, by continuously improving the knowledge hit rate, it can be ensured that the responses generated by the model are closer to the user's real needs, reducing misunderstandings and biases, and enhancing user satisfaction.

[0133] like Figure 4 As shown in the figure, this application embodiment also provides a text generation device 400, the device comprising:

[0134] The input module 401 is used to input the text vector of the text to be queried into the text generation model, and use the encoder in the text generation model to encode the text vector to obtain a first encoded vector and a second encoded vector.

[0135] The first extraction module 402 is used to perform linear feature extraction on the first encoding vector and the second encoding vector using the cross-fusion attention unit in the text generation model, to obtain a first fusion vector corresponding to the first encoding vector and a second fusion vector corresponding to both the first encoding vector and the second encoding vector.

[0136] The second extraction module 403 is used to perform nonlinear feature extraction on the first fusion vector and the first encoding vector using the feedforward neural network in the text generation model to obtain the first feature vector.

[0137] The fusion module 404 is used to cross-fuse the first feature vector and the second fusion vector using the feedforward neural network and the encoder to obtain the second feature vector.

[0138] The decoding module 405 is used to decode the second feature vector using the text generation model to obtain a response text that matches the text to be queried.

[0139] Optionally, input module 401 includes:

[0140] The first input submodule is used to input the text vector into the text generation model and process the text vector using the first encoder to obtain the first encoded vector;

[0141] The first processing submodule is used to process the first encoded vector using the second encoder to obtain the second encoded vector.

[0142] Optionally, the fusion module 404 includes:

[0143] The second processing submodule is used to process the first feature vector using the second encoder to obtain the third feature vector;

[0144] The first fusion submodule is used to fuse the third feature vector and the second fusion vector to obtain a fourth fusion vector;

[0145] The second processing submodule is used to process the fourth fusion vector using the feedforward neural network to obtain the second feature vector.

[0146] Optionally, the first extraction module 402 includes:

[0147] The first extraction submodule is used to extract features from the first encoding vector using the first self-attention unit to obtain a first fusion vector;

[0148] The second fusion submodule is used to fuse the first encoding vector and the second encoding vector to obtain the third encoding vector;

[0149] The second extraction submodule is used to extract features from the third encoding vector using the second self-attention unit to obtain the second fusion vector.

[0150] Optionally, the second extraction module 403 includes:

[0151] The third fusion submodule is used to fuse the first fusion vector and the first encoding vector to obtain the third fusion vector;

[0152] The third extraction submodule is used to perform nonlinear feature extraction on the third fusion vector using the feedforward neural network to obtain the first feature vector.

[0153] Optionally, the text generation device 400 further includes:

[0154] The first acquisition module is used to acquire multiple similar questions that have a similarity greater than a first threshold with the standard question;

[0155] The preprocessing module is used to preprocess the multiple similar problems to obtain a first training set, wherein the first training set includes the preprocessed multiple similar problems;

[0156] The first training module is used to train the model to be trained using the first training set to obtain the first model;

[0157] A module is added to each of the multiple similar problems to obtain a second training set;

[0158] The second training module is used to train the first model using the second training set to obtain the text generation model.

[0159] Optionally, the text generation device 400 further includes:

[0160] The processing module is used to process the standard question using the encoder in the text generation model to obtain sentence vectors;

[0161] The first determining module is used to determine unidentified user questions whose similarity to the sentence vector is greater than a second threshold as related knowledge;

[0162] The storage module is used to store the associated knowledge, the standard question, and the multiple similar questions corresponding to the standard question as a knowledge set.

[0163] Optionally, the text generation device 400 further includes:

[0164] The second input module is used to sequentially input the associated knowledge, the standard question, and the similar question from the knowledge set into the text generation model to obtain answers corresponding to the associated knowledge, the standard question, and the similar question, respectively.

[0165] The second acquisition module is used to acquire the response time of the text generation model in generating the answer, the accuracy of the answer, and the sensitive word blocking status in the knowledge set;

[0166] The second determining module is used to determine the quantitative analysis results of the text generation model by utilizing the response time, the accuracy, and the sensitive word blocking status.

[0167] Optionally, the text generation device 400 further includes:

[0168] The third acquisition module is used to acquire the actual responses of the text generation model under a preset business topic;

[0169] The third determining module is used to determine the knowledge hit rate of the text generation model under a preset business topic using the real responses and preset standard answers. The knowledge hit rate is used to represent the qualitative analysis results of the text generation model.

[0170] It should be noted that the text generation device 400 is a device corresponding to the above-described text generation method. All implementation methods in the above method embodiments are applicable to the embodiments of this device and can achieve the same technical effect.

[0171] Figure 5 A schematic diagram of the hardware structure of the terminal device provided in an embodiment of this application is shown.

[0172] The terminal device may include a processor 501 and a memory 502 storing computer program instructions.

[0173] Specifically, the processor 501 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0174] Memory 502 may include mass storage for data or instructions. For example, and not limitingly, memory 502 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 502 may include removable or non-removable (or fixed) media. Where appropriate, memory 502 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, memory 502 is non-volatile solid-state memory.

[0175] In certain embodiments, memory 502 may include read-only memory (ROM), random access memory (RAM), disk storage media device, optical storage media device, flash memory device, electrical, optical, or other physical / tangible memory storage device. Thus, generally, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to one aspect of this disclosure.

[0176] The processor 501 implements any of the text generation methods described in the above embodiments by reading and executing computer program instructions stored in the memory 502.

[0177] In one example, the terminal device may further include a communication interface 505 and a bus 510. Wherein, as... Figure 5 As shown, the processor 501, memory 502, and communication interface 505 are connected through bus 510 and complete communication with each other.

[0178] The communication interface 505 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0179] Bus 510 includes hardware, software, or both, that couples components of an online data traffic metering device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 510 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, any suitable bus or interconnect is contemplated herein.

[0180] Furthermore, in conjunction with the text generation methods in the above embodiments, this application embodiment can provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the text generation methods in the above embodiments.

[0181] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0182] The functional blocks shown in the above block diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0183] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0184] The aspects of this disclosure have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable text generation apparatus to create a machine such that these instructions, executable via the processor of the computer or other programmable text generation apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by special-purpose hardware performing the specified functions or actions, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0185] The above are merely specific embodiments of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A text generation method, characterized in that, The method includes: The text vector of the text to be queried is input into the text generation model, and the encoder in the text generation model is used to encode the text vector to obtain the first encoded vector and the second encoded vector. Using the cross-fusion attention unit in the text generation model, linear feature extraction is performed on the first encoding vector and the second encoding vector to obtain a first fusion vector corresponding to the first encoding vector and a second fusion vector corresponding to both the first encoding vector and the second encoding vector. Using the feedforward neural network in the text generation model, nonlinear feature extraction is performed on the first fusion vector and the first encoding vector to obtain the first feature vector; Using the feedforward neural network and the encoder, the first feature vector and the second fusion vector are cross-fused to obtain the second feature vector; Using the text generation model, the second feature vector is decoded to obtain the response text that matches the text to be queried.

2. The text generation method as described in claim 1, characterized in that, The encoder includes a first encoder and a second encoder; The step of inputting the text vector of the text to be queried into the text generation model, and using the encoder in the text generation model to encode the text vector to obtain a first encoded vector and a second encoded vector, includes: The text vector is input into the text generation model, and the first encoder is used to process the text vector to obtain the first encoded vector; The second encoder is used to process the first encoded vector to obtain the second encoded vector.

3. The text generation method as described in claim 2, characterized in that, The step of cross-fusing the first feature vector and the second fusion vector using the feedforward neural network and the encoder to obtain the second feature vector includes: The first feature vector is processed using the second encoder to obtain the third feature vector; The third feature vector and the second fusion vector are fused to obtain a fourth fusion vector; The feedforward neural network is used to process the fourth fusion vector to obtain the second feature vector.

4. The text generation method as described in claim 1, characterized in that, The cross-fusion attention unit includes a first self-attention unit and a second self-attention unit; The step of using the cross-fusion attention unit in the text generation model to perform linear feature extraction on the first and second encoding vectors to obtain a first fusion vector corresponding to the first encoding vector and a second fusion vector corresponding to both the first and second encoding vectors includes: The first self-attention unit is used to extract features from the first encoded vector to obtain the first fusion vector; The first and second encoded vectors are fused to obtain the third encoded vector; The second self-attention unit is used to extract features from the third encoding vector to obtain the second fusion vector.

5. The text generation method as described in claim 1, characterized in that, The step of using the feedforward neural network in the text generation model to perform nonlinear feature extraction on the first fusion vector and the first encoded vector to obtain the first feature vector includes: The first fusion vector and the first encoding vector are fused to obtain the third fusion vector; The feedforward neural network is used to perform nonlinear feature extraction on the third fusion vector to obtain the first feature vector.

6. The text generation method as described in claim 1, characterized in that, Before inputting the text vector of the text to be queried into the text generation model, and using the encoder in the text generation model to encode the text vector to obtain the first encoded vector and the second encoded vector, the method further includes: Obtain multiple similar questions that have a similarity greater than a first threshold with the standard question; The multiple similar problems are preprocessed to obtain a first training set, which includes the preprocessed multiple similar problems. The first training set is used to train the model to be trained, and the first model is obtained. Add a perturbation to each of the multiple similar problems to obtain a second training set; The first model is trained using the second training set to obtain the text generation model.

7. The text generation method as described in claim 6, characterized in that, The preprocessing of the multiple similar problems to obtain the first training set includes: Data cleaning and deduplication are performed on the multiple similar problems to obtain the first training set.

8. The text generation method as described in claim 6, characterized in that, After training the first model using the second training set to obtain the text generation model, the method further includes: The standard question is processed using the encoder in the text generation model to obtain a sentence vector; Unidentified user questions whose similarity to the sentence vector is greater than a second threshold are identified as related knowledge; The associated knowledge, the standard question, and the multiple similar questions corresponding to the standard question are stored as a knowledge set.

9. The text generation method as described in claim 8, characterized in that, After storing the associated knowledge, the standard question, and the multiple similar questions corresponding to the standard question as a knowledge set, the method further includes: The associated knowledge, the standard question, and the similar question in the knowledge set are sequentially input into the text generation model to obtain answers corresponding to the associated knowledge, the standard question, and the similar question, respectively. The response time of the text generation model in generating the answer, the accuracy of the answer, and the filtering status of sensitive words in the knowledge set are obtained. The quantitative analysis results of the text generation model are determined using the response time, accuracy, and sensitive word blocking.

10. The text generation method as described in claim 9, characterized in that, After storing the associated knowledge, the standard question, and the multiple similar questions corresponding to the standard question as a knowledge set, the method further includes: Obtain the actual responses from the text generation model under a preset business topic; The knowledge hit rate of the text generation model under a preset business topic is determined by using the actual responses and preset standard answers. The knowledge hit rate is used to represent the qualitative analysis results of the text generation model.

11. A text generation device, characterized in that, The device includes: The input module is used to input the text vector of the text to be queried into the text generation model, and to encode the text vector using the encoder in the text generation model to obtain a first encoded vector and a second encoded vector. The first extraction module is used to perform linear feature extraction on the first encoding vector and the second encoding vector using the cross-fusion attention unit in the text generation model, so as to obtain a first fusion vector corresponding to the first encoding vector and a second fusion vector corresponding to both the first encoding vector and the second encoding vector. The second extraction module is used to perform nonlinear feature extraction on the first fusion vector and the first encoding vector using the feedforward neural network in the text generation model to obtain the first feature vector. The fusion module is used to cross-fuse the first feature vector and the second fusion vector using the feedforward neural network and the encoder to obtain the second feature vector. The decoding module is used to decode the second feature vector using the text generation model to obtain a response text that matches the text to be queried.

12. A terminal device, characterized in that, The device includes: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, it implements the text generation method as described in any one of claims 1-10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the text generation method as described in any one of claims 1-10.

14. A computer program product, characterized in that, The computer product includes a computer program that, when executed by a processor, implements the text generation method according to any one of claims 1-10.

Images