Text generation method and device, electronic equipment and storage medium

Abstract

The application provides a text generation method and device, electronic equipment and storage medium, and belongs to the technical field of artificial intelligence. The method comprises the following steps: acquiring an entity in an input text, and extracting serialized information corresponding to the entity in a knowledge graph, the serialized information being used for representing attribute information of the entity; fusing the serialized information and the input text to obtain fusion data; and determining a feedback text matched with the input text based on the fusion data. The method reduces interference information caused by knowledge solidification, reduces the possibility of knowledge loss, improves the analysis accuracy of the input text, and makes the obtained feedback text more fluent.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular relates to a text generation method and device, an electronic device, and a storage medium. BACKGROUND

[0002] Natural Language Generation (NLG) refers to automatically generating understandable natural language text by using artificial intelligence and linguistics. With the development of technology, natural language generation technology is widely applied in various scenarios, such as translation, automatic title, and merchant copy generation. Traditional natural language generation technology generally generates based on templates, such as selecting a corresponding generation template according to an input text and generating a feedback text corresponding to the input text based on the generation template. With the development of large-scale pre-training models, the generation method based on pre-training models has gradually become the mainstream of natural language generation technology.

[0003] Conventionally, when generating a feedback text by using a pre-training model, an input text is input into the pre-training model for recognition analysis to generate a feedback text matched with the input text. This method has the problem of knowledge solidification, which reduces the fluency of the text. In addition, there are problems such as knowledge loss or knowledge interference in the recognition process of the text, which reduces the recognition accuracy of the text, thereby reducing the matching of the feedback text and the input text, resulting in an inaccurate generated feedback text. Therefore, in the field of natural language generation, how to effectively and accurately generate a feedback text for an input text has become one of the hot issues in current research. SUMMARY

[0004] The present application provides a text generation method, device, electronic device, and storage medium to accurately generate a feedback text for an input text.

[0005] In a first aspect, the present application provides a text generation method, which includes: obtaining an entity in an input text and extracting serialized information corresponding to the entity in a knowledge graph, the serialized information being used to represent attribute information of the entity; fusing the serialized information with the input text to obtain fusion data; and determining a feedback text matched with the input text based on the fusion data.

[0006] In a second aspect, the present application provides a text generation apparatus, comprising: an obtaining module configured to obtain an entity in an input text, and extract serialized information corresponding to the entity in a knowledge graph, the serialized information being used to represent attribute information of the entity; a fusion module configured to fuse the serialized information with the input text to obtain fusion data; and a processing module configured to determine a feedback text matched with the input text based on the fusion data.

[0007] In a third aspect, the present application provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores one or more computer programs executable by the at least one processor, and the one or more computer programs are executed by the at least one processor to enable the at least one processor to perform the text generation method described above.

[0008] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the text generation method described above.

[0009] The embodiments provided by the present application generate a feedback text matched with an input text based on a knowledge graph. Specifically, an entity in the input text is obtained, and then serialized information corresponding to the entity in the knowledge graph is extracted. The serialized information is fused with the input text to obtain fusion data. Further, a feedback text matched with the input text is determined by analyzing the fusion data. In the above text generation process, the fusion data is obtained by fusing the serialized information with the input text. Thus, the fusion data can reflect the entity in the input text and the attribute information of each entity. When analyzing the fusion data, the attribute information of the entity is considered to reduce information interference irrelevant to the attribute information of the entity, and the possibility of knowledge loss is reduced. Thus, the feedback text can accurately reflect the text information corresponding to the fusion data, so that the feedback text is more accurate and has higher fluency.

[0010] It should be understood that the content described in this section is not intended to identify key or important features of the embodiments of the present application, nor is it used to limit the scope of the present application. Other features of the present application will become apparent through the following description. BRIEF DESCRIPTION OF DRAWINGS

[0011] The accompanying drawings are included to provide a further understanding of the present application, and constitute a part of the specification, illustrate embodiments of the present application, and are used to explain the present application, and do not constitute a limitation of the present application. The above and other features and advantages will become more apparent to those skilled in the art from the detailed description of the exemplary embodiments with reference to the accompanying drawings, in which:

[0012] Figure 1 A composition block diagram of a feedback text generation device based on a knowledge graph provided by the related art.

[0013] Figure 2 A flowchart of a text generation method provided by the embodiment of the present application.

[0014] Figure 3 A schematic diagram of an entity recognition model provided by the embodiment of the present application.

[0015] Figure 4 A schematic diagram of a text recognition network model provided by the embodiment of the present application.

[0016] Figure 5 A schematic diagram of determining an encoding feature sequence vector by using an attention mechanism in the text recognition network model provided by the embodiment of the present application.

[0017] Figure 6 A schematic diagram of fusing a text and its corresponding attention encoding provided by the embodiment of the present application.

[0018] Figure 7 A schematic diagram of fusing a text and its corresponding attention encoding provided by the embodiment of the present application.

[0019] Figure 8 A schematic diagram of a professional knowledge graph provided by the embodiment of the present application.

[0020] Figure 9 A composition block diagram of a text generation device provided by the embodiment of the present application.

[0021] Figure 10 A composition block diagram of an electronic device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0022] In order for those skilled in the art to better understand the technical solutions of the present application, the exemplary embodiments of the present application are described below in conjunction with the accompanying drawings, which include various details of the embodiments of the present application to help understanding, and should be considered only as exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Also, in order to be clear and concise, the description in the following description omits the description of well-known functions and structures.

[0023] In the case of no conflict, the embodiments of the present application and the features in the embodiments can be combined with each other.

[0024] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0025] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application. As used herein, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, they specify the presence of features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof. Words such as “connected” or “linked” are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect.

[0026] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this application, and will not be interpreted as having an idealized or overly formal meaning, unless expressly so defined herein.

[0027] In the library and information science field, a knowledge graph, also known as a knowledge domain visualization or knowledge domain mapping map, is a series of various graphics that display the development process and structural relationships of knowledge. It uses visualization techniques to describe knowledge resources and their carriers, and to mine, analyze, construct, draw, and display the interrelationships between knowledge and knowledge resources. Knowledge graphs combine theories and methods from applied mathematics, computer graphics, information visualization techniques, and information science, along with quantitative citation analysis and co-occurrence analysis methods, to visually represent the core structure, development history, cutting-edge fields, and overall knowledge architecture of a discipline, enabling the integration of content from multiple disciplines.

[0028] Knowledge graphs can be used to analyze input text and obtain feedback text that matches it. The device used to generate this feedback text is called a feedback text generation device. In some related technologies, the feedback text generation device can input the input text into a pre-trained knowledge graph-based model for recognition and analysis to generate feedback text that matches the input text. For example, Figure 1 A block diagram illustrating the components of a knowledge graph-based feedback text generation device for related technologies. (Example:) Figure 1 As shown, the feedback text generation device includes an encoder model and a decoder model.

[0029] Among them, the encoder model is as follows Figure 1As shown in the left part, the encoder model adopts a model structure of a multi-graph convolutional network (MGCN). The input data is sequentially processed by MGCN layer 1, MGCN layer 2, …, and MGCN layer n. Each MGCN layer includes multiple graph encoders that encode the input data to obtain encoded graph embedding data. The multiple encoded graph embedding data are input to an aggregation layer for aggregation processing to generate aggregated data corresponding to each MGCN layer. For example, first aggregated data corresponding to MGCN layer 1, second aggregated data corresponding to MGCN layer 2, …, and n-th aggregated data corresponding to MGCN layer n. The first aggregated data, the second aggregated data, …, and the n-th aggregated data are all output to Figure 1 the adder shown in the right part for addition processing. The aggregated data after addition is output as graph embedding data, which can be represented in the form of a vector (e.g., the graph embedding data includes a word embedding vector). The word embedding vector in the graph embedding data is output to Figure 1 the decoder model shown in the right part. n represents the number of MGCN layers, and n is an integer greater than or equal to 1.

[0030] The decoder model is shown in the right part. The decoder model includes multiple long short-term memory (LSTM) layers (e.g., LSTM layer 1, LSTM layer 2, …, and LSTM layer m, where m represents the number of LSTM layers, and m is an integer greater than or equal to 1) and a cross-attention processing layer. The word embedding vector output by the encoder model is input to LSTM layer 1. After processing by multiple LSTM layers, the output result of LSTM layer m is obtained. The output result and the word embedding vector output by the encoder model are input to the cross-attention processing layer for processing to obtain the encoding of the feedback text. Finally, the encoding of the feedback text is converted into the feedback text. Figure 1

[0031] However, the multi-layer LSTM has low computational efficiency and generates text slowly. When the multi-layer LSTM processes the word embedding vector, it can only represent the graph layer structure corresponding to the word embedding vector, but cannot accurately represent the semantic representation (e.g., the close relationship between the generated feedback text and the topic) corresponding to the graph embedding data. This results in knowledge loss or knowledge interference in the obtained feedback text, reducing the recognition accuracy of the text. ​

[0032] Therefore, the embodiments of the present application provide a text generation method and device, electronic equipment and storage medium, which are used for improving the accuracy of text recognition. The text generation method can be executed by a terminal device or a server, etc. The terminal device can be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The method can be realized by a processor calling computer readable program instructions stored in a memory. The server can be a stand-alone physical server, a server cluster composed of multiple servers, or a cloud server capable of cloud computing.

[0033] In a first aspect, the embodiments of the present application provide a text generation method. Figure 2 A flowchart of the text generation method provided by the embodiments of the present application is shown in FIG. 1. As shown in the figure, the text generation method includes but is not limited to the following steps: Figure 2

[0034] In step S201, an entity in an input text is obtained, and serialized information corresponding to the entity is extracted in a knowledge graph.

[0035] In different application scenarios, the input text is different. For example, in an insurance consultation scenario, the input text can be "a child accident insurance". For another example, in a product sales scenario, the input text can be "how much is this car sold for", etc. The serialized information is used to represent the attribute information of the entity.

[0036] For example, if the entity is a motor vehicle, the serialized information corresponding to the motor vehicle is (motor vehicle, contains, motorcycle); if the entity is motor vehicle insurance, the serialized information corresponding to the motor vehicle insurance is (motor vehicle insurance, object, motor vehicle).

[0037] The knowledge graph is a visualized processing of a knowledge domain, thereby obtaining a corresponding knowledge field mapping map. In actual application, the knowledge graph can be used to describe various entities and concepts, as well as the relationship between them, and can be regarded as a semantic network. In some optional implementation manners, the basic component unit of the knowledge graph is a relationship triple of "entity-relation-entity", and the value pair information of the entity and its related attributes, the entities are connected with each other through the relationship, thereby constituting a networked knowledge structure.

[0038] ​An entity in the knowledge graph is an instantiation of a concept. For example, "person" is a concept in the ontology box, and the concept also specifies related attribute information (such as the "gender" of a person). For example, Xiaoming is a specific person, called an instance, Xiaoming's gender, Xiaoming, and the ontology concept "person" embodying Xiaoming, and related attribute information, are called an entity.

[0039] A relationship in the knowledge graph is a relationship between entities, such as a relationship between entities under the concept of "person", which includes parent-child relationships, parent-child relationships, and the like. The attribute in the knowledge graph refers to the inherent attribute of the entity and the concept to which the entity belongs, such as gender, age, and the like.

[0040] It should be noted that the above description of entities in the knowledge graph and the relationship between entities is only an example, and the embodiments of the present application do not limit this.

[0041] In some optional embodiments, the obtaining of the entity in the input text in step S201 comprises: inputting the input text into an entity recognition model to recognize and obtain the entity. The entity recognition model is a model constructed based on entities and relationships between entities in data of a preset field.

[0042] It should be noted that the data of the preset field includes at least one of historical consultation session record information, consultation product information, and consultation service information; the input text includes consultation text, and the consultation text is used to record the consultation information of the user's demand.

[0043] For example, Figure 3 A schematic diagram of an entity recognition model provided by an embodiment of the present application. As shown in Figure 3 The entity recognition model includes a first encoding network and a word classification network. The first encoding network includes a pre-trained language model, a word embedding layer, a segment embedding layer, and a position embedding layer. The pre-trained language model can be implemented using a language recognition model with a double attention mechanism (such as the chinese-roberta-wwm-ext model).

[0044] It should be noted that the role of the word embedding layer is to map human language into a geometric space, and the data processed by the word embedding layer is a low-dimensional floating-point number vector (i.e. a dense vector, as opposed to a sparse vector). The segment embedding layer is used to distinguish which sentence each word belongs to, for example, a label of 0 for sentence A and a label of 1 for sentence B. The position embedding layer is used to retain the position encoding information of each word in the text; the position vector of each word is randomly initialized first, and then the position vector containing the position information of each word is obtained through subsequent model training.

[0045] Pooling in character classification networks can be achieved by using a 768-dimensional * 768-dimensional dense layer with a tanh activation function, followed by a 768-dimensional * 65-dimensional dense layer with a normalized exponential function (e.g., softmax function).

[0046] like Figure 3 As shown, assuming the input text is "a child's accident insurance policy", the input text is first fed into the word embedding layer for word embedding vector encoding. The segment embedding layer's processing result is "0000000". The corresponding position embedding layer encodes the position of each word in the input text, with each character's position code being "0123456". By fusing the information from the word embedding layer, segment embedding layer, and position embedding layer and inputting it into the pre-trained language model for processing, the output of the first encoding network can be obtained. This output serves as the input to the character classification network, which divides the input into different categories, such as O category (i.e., other), B-ins category (i.e., the start (Begin) position of the insurance entity), and I-ins category (i.e., the middle (Inter) character information of the entity in the insurance scenario). During the training of the entity recognition model, multiple sets of labeled data can be collected (e.g., 30,000 labeled data sets, including 32 entity classes). These labeled data can originate from open-source datasets or from real-world application scenarios, such as document data from insurance consultation scenarios (e.g., online chat logs, insurance product documents, and contract templates). Then, according to a preset ratio, the multiple sets of labeled data are divided into training set data (90% of the total data volume), validation set data (5% of the total data volume), and test set data (5% of the total data volume).

[0047] After determining the training set data, the entity recognition model is trained as follows: The text in the training set data and its corresponding sequence labels are encoded to obtain training text encoding sequences and training label encoding sequences. Then, the training text encoding sequences are input into the initial entity recognition model to identify entities and obtain sequence prediction vectors. The average cross-entropy loss between the training label encoding sequences and the sequence prediction vectors is calculated, and the network gradient of the entity recognition model is calculated based on the average cross-entropy loss. The network gradient is then updated using an optimizer (e.g., an optimizer using the weighted adaptive moment estimation (AdamW) algorithm). The above steps are repeated until the evaluation metric of the obtained entity recognition model on the validation set data reaches the target metric, thus obtaining the entity recognition model.

[0048] The sequence label corresponding to the text is used to represent the annotation position of the entity in the text. The verification set data includes the text and the sequence label corresponding to the text. By encoding the text in the verification set data, a verification text encoding sequence can be obtained. The sequence label of the text in the verification set data is encoded to obtain a verification label encoding sequence.

[0049] For example, during the training process, after performing the above loop 400 times, a verification process is performed. The verification process is to input the verification text encoding sequence in the verification set data into the entity recognition model for processing to obtain a label sequence prediction result, and then calculate the difference between the label sequence prediction result and the verification label encoding sequence in the verification set data, and take the difference as an evaluation index. The learning rate of the optimizer can be set to 1*e -5 , and the weight decay of the optimizer can be set to 1*e -2 .

[0050] After obtaining the entities in the input text through the entity recognition model, the relationship between the obtained entities can be further determined. In specific implementation, any one or more of the following methods can be used to determine the relationship between the entities in the input text:

[0051] 1) Manually identify the relationship between entities. For some high-frequency special entities, manually add the relationship between entities. For example, manually establish the following relationship between “Xueping insurance” and “student insurance”: (“Xueping insurance” (product), synonymous, “student insurance” (product)).

[0052] 2) Identify the relationship between entities through pre-set matching rules. For example, a relationship regular template is pre-set, and the relationship regular template is used to match the entities in the input text. If it is determined that there are two matched entities, it is determined that there is a relationship between the two entities. For example, using the template “<product> does not bear insurance liability under <condition>”, the entities in the input text are matched, and it is determined that “Xueping insurance does not bear insurance liability during the waiting period” is found, and the following relationship can be established: (“Xueping insurance” (product), “exemption” (relationship), “during the waiting period” (condition)).

[0053] 3) Input the input text containing at least two entities into a professional knowledge recognition model for processing, and predict the relationship information between the at least two entities through the professional knowledge recognition model.

[0054] 4) Decompose the composite entity into a main entity and an attribute entity, and connect them through an attribute relationship. For example, the composite entity “7-seat motor vehicle” is decomposed, and the following relationship can be established: (“motor vehicle” (object), attribute (relationship), “7-seat” (seat number)).

[0055] In some optional embodiments, the extracting, in the step S201, the serialized information corresponding to the entity in the knowledge graph can be implemented in the following manner: taking the entity as an index, extracting the relation triple corresponding to the entity in the knowledge graph, and serializing the relation triple to obtain the serialized information.

[0056] For example, based on the identified entity, the knowledge graph is searched to obtain a head entity-relation-tail relation triple corresponding to the entity; and the relation triple is serialized to obtain the serialized information.

[0057] Through the above method, the entity in the input text and the relationship between the entities can be determined. In some specific implementations, the Translating Embedding (TransE) method can also be used to supplement the relationship between the entities in the knowledge graph to enrich the relationship between the entities.

[0058] The principle of the TransE method is as follows: for a relation triple (h, l, t), where h represents the head entity, t represents the tail entity, and l represents the relationship between the head entity and the tail entity. The above three parameters have the following relationship: t ≈ h + l, that is, t should be as close to h + l as possible; conversely, if h, l and t cannot form a relation triple, the above three parameters should be as far away from each other as possible.

[0059] The TransE model trained by the TransE method can predict whether there is a relationship between different entities. For example, the to-be-predicted head entity-relation-to-be-predicted tail entity is input into the TransE model for prediction. When the output prediction obtains a relationship that is closer to the meaning represented by the to-be-predicted head entity and the to-be-predicted tail entity, it is determined that the to-be-predicted head entity-relation-to-be-predicted tail entity can form a relation triple.

[0060] In step S202, the serialized information is fused with the input text to obtain fused data.

[0061] For example, the serialized information is merged with the input text, or the serialized information is spliced with the input text, etc., so as to obtain more complete fused data for understanding the input text.

[0062] In some optional embodiments, the step of fusing the serialization information with the input text to obtain the fused data in the step S202 can be implemented in the following manner: inserting the serialization information corresponding to the entity into a preset position in the input text to obtain a fused text; determining attention mask information based on the fused text, the attention mask information being used to distinguish the input text and the serialization information; and generating the fused data based on the attention mask information and the fused text.

[0063] The preset position includes a tail of the entity and / or a tail of demand information corresponding to the entity. The demand information is information representing a service that the entity needs to handle. The attention mask information can include a first attention mask corresponding to the input text and a second attention mask corresponding to the serialization information. Different attention masks can be used to distinguish the input text and the serialization information, so that the fused data is more conducive to subsequent analysis.

[0064] For example, if the entity is determined to be a "motorcycle", the serialization information can be a relation triple (motor vehicle, contains, motorcycle); if the entity is determined to be "motorcycle insurance", the serialization information corresponding thereto can be a relation triple (motor vehicle insurance, object, motor vehicle). The above serialization information can reflect the category of the entity and the relationship between the entity and other entities (for example, the relationship between motor vehicle insurance and motor vehicle, or the relationship between motor vehicle and motorcycle, etc.).

[0065] For another example, the serialization information is inserted into the tail of the entity (i.e., the tail of the motorcycle) in the input text (for example, "is there motorcycle insurance"), to obtain the following fused text: "is there motorcycle (motor vehicle contains motorcycle, the insurance object of motor vehicle insurance is motor vehicle) insurance".

[0066] The serialization information is inserted into the tail of the demand information corresponding to the entity (i.e., the tail of the "insurance" of the motorcycle) in the input text, to obtain the following fused text: "is there motorcycle insurance (the insurance object of motor vehicle insurance is motor vehicle, motor vehicle contains motorcycle)".

[0067] By inserting the serialization information of each entity into the position related to the entity (for example, the position related to the entity includes the tail of the entity and / or the tail of the demand information corresponding to the entity, etc.) in the input text, the obtained fused data can clearly reflect the information related to the entity, facilitating the understanding of the input text.

[0068] In the step S203, a feedback text matched with the input text is determined based on the fused data.

[0069] For example, the fused data is input into a text recognition network model for processing to obtain feedback text that matches the input text. The text recognition network model is a language recognition model based on a dual-attention mechanism.

[0070] It should be noted that the text recognition network model in this application can be a text recognition model based on the dual attention mechanism, or other text recognition models. This application does not limit the text recognition network model, as long as it can recognize the input text and obtain the feedback text that matches the input text.

[0071] The following is a detailed description of the text recognition network model based on the dual attention mechanism in the embodiments of this application. Figure 4 This is a schematic diagram of a text recognition network model provided in an embodiment of this application. Figure 4 As shown, the text recognition network model includes a text encoding layer, a text decoding layer, and a pooling layer (not shown in the figure).

[0072] The pooling layer can be implemented using a 768*768 dimensional dense layer with a softmax function. The text encoding layer can be implemented using the autoregressive language model BART. The pre-trained language model can be implemented using a language recognition model based on a dual attention mechanism (such as the chinese-roberta-wwm-ext model), which can improve the accuracy of text analysis.

[0073] like Figure 4 As shown, the input text for the text encoding layer is: "This is the input"; the input text for the text decoding layer is: <bos>This is the goal, among others, <bos>represents a start symbol; the output text of the text decoding layer is: "This is the target <eos>"wherein <eos>End of file.

[0074] It should be noted that the training data of the text recognition network model can be derived from online historical dialogue data. For example, 50,000 high-quality question and answer pairs are filtered from online historical dialogue data, and the filtered data is divided into training set data (accounting for 95% of the total data amount) and test set data (accounting for 5% of the total data amount). The high-quality question and answer pairs need to meet the following conditions: the user question is a real business consultation question; the corresponding answer content is not a greeting, polite language, etc.; the answer content needs to meet the requirements of business knowledge; the answer content needs to match the theme in the user question; and the answer content can solve the user question.

[0075] Further, the text recognition network model can be trained in the following manner:

[0076] The question in the obtained test set data is taken as the input text of the text encoding layer, and the answer content corresponding to the question output in the training process is taken as the prediction target (i.e., the output text of the text decoding layer); the input parameters and the prediction target are encoded respectively to obtain the input sequence and the target sequence; then the input sequence and the target sequence are input to the initial text recognition network model for training to obtain a multi-dimensional language identification vector sequence (such as a coding feature vector sequence with 21128 dimensions); further, the average cross-entropy loss between the multi-dimensional language identification vector sequence and the target sequence is calculated, and based on the average cross-entropy loss, the network gradient of the text recognition network model is calculated, the Adam optimizer is used to minimize the average cross-entropy loss, and the weights of the text recognition network model are updated; the above steps are repeated until the sequence output by the obtained text recognition network model is the target sequence, thereby obtaining the trained entity recognition model.

[0077] Among them, the learning rate of the Adam optimizer with weight decay function can be set to 3*e -5 , and the weight decay value of the Adam optimizer is set to 1*e -2 .

[0078] It should be noted that during the training process, the weight value corresponding to the current model is saved after executing the above loop 1000 times; the above steps are repeated until 10000 loops are completed. Further, based on the weight values corresponding to the saved 10 models, the Adam optimizer is adjusted to update the weight values of the text recognition network model, and then the test set data is input to the text recognition network model with updated weight values for testing until the sequence output by the obtained text recognition network model is the target sequence.

[0079] Among them, the generated feedback text can also be artificially evaluated, and the weight value corresponding to the feedback text with the optimal evaluation result is selected as the weight value of the text recognition network model.

[0080] In some optional embodiments, the determining of the feedback text matched with the input text based on the fusion data in step S203 can be implemented in the following manner:

[0081] inputting the fusion data into a text encoding layer of a text recognition network model to obtain an encoding feature sequence vector; traversing an initial decoding sequence list, and inputting the encoding feature sequence vector and each initial encoding sequence in the initial decoding sequence list into a text decoding layer of the text recognition network model respectively to obtain a candidate decoding sequence list; traversing the candidate decoding sequence list, and moving a candidate decoding sequence with an end identifier and a score corresponding to the candidate decoding sequence to an output decoding sequence list; and determining the feedback text according to a size relationship between the scores of each output decoding sequence in the output decoding sequence list and a highest score in the candidate decoding sequence list.

[0082] The text encoding layer is a network layer encoded based on a double attention mechanism. The end identifier is an identifier representing that the candidate decoding sequence is an end sequence. The initial decoding list includes a plurality of initial encoding sequences and scores corresponding to each initial encoding sequence. The candidate decoding sequence list includes a plurality of candidate decoding sequences and scores corresponding to each candidate decoding sequence.

[0083] It should be noted that, in the process of attention calculation, the text encoding layer receives two different attention masks as input, and then calculates and obtains an attention distribution corresponding to the attention mask based on the different attention masks, and then performs a summation operation on the different attention distributions to obtain a target attention distribution. Then, the input text is processed according to the target attention distribution, which can reflect the entity in the input text and the attention distribution corresponding to the serialized information corresponding to the entity, so that the obtained encoding feature sequence vector can more accurately reflect the features of the fusion data.

[0084] In some optional embodiments, the attention mask information includes a first attention mask and a second attention mask; and the mask positions of the first attention mask and the second attention mask are different.

[0085] inputting the fusion data into a text encoding layer of a text recognition network model to obtain an encoding feature sequence vector, including: inputting the first attention mask into the text encoding layer to obtain a first attention distribution; inputting the second attention mask into the text encoding layer to obtain a second attention distribution; determining a target attention distribution according to the first attention distribution and the second attention distribution; and encoding the fusion data based on the target attention distribution to obtain the encoding feature sequence vector.

[0086] Figure 5 A schematic diagram of determining an encoding feature sequence vector by using an attention mechanism in a text recognition network model is provided for the embodiments of the present application. As shown in FIG. 1, the text recognition network model includes a text encoding layer and a text decoding layer. Figure 5 As shown in the figure, the fusion text is "How to buy a motorcycle (motor vehicles include motorcycles)", the corresponding first attention mask is "1110000000000111", and the first attention value is determined based on the first attention mask; the second attention mask is "0001111111111000", and the second attention value is determined based on the second attention mask. Through two different attention masks, "How to buy a motorcycle" and "(motor vehicles include motorcycles)" can be quickly and accurately distinguished to obtain the difference between the above two texts, and further, the attention value is calculated for different texts.

[0087] For example, the i-th layer representation vector sequence corresponding to the first attention value and the i-th layer representation vector sequence corresponding to the second attention value are added, thereby obtaining the attention value of the i-th layer corresponding to the fusion text. Wherein, i is an integer greater than or equal to 1.

[0088] By determining different attention values through different attention masks, and further processing the fusion text based on different attention values, the feature information of the fusion text can be more accurately represented, and the representation accuracy of the fusion text is improved.

[0089] For a question text (such as "Is there motorcycle insurance?"), the entity "motorcycle" is identified through an entity recognition model; then, based on the entity "motorcycle", the knowledge graph is searched to obtain the following relationship triplets: (motor vehicles, include, motorcycles) and (motor vehicle insurance, object, motor vehicles). Through the above relationship triplets, the following information can be determined: "motor vehicles include motorcycles" and "the insurance object of motor vehicle insurance is motor vehicles".

[0090] Further, the above two relationship triplets are inserted into the question text to obtain a fusion text.

[0091] For example, both of the two relationship triplets can be inserted at the tail of the entity "motorcycle". Figure 6 A schematic diagram of a fusion text and its corresponding attention encoding provided by an embodiment of the present application.

[0092] As Figure 6 shown, the fusion text is "Is there motorcycle (motor vehicles include motorcycles, the insurance object of motor vehicle insurance is motor vehicles) insurance", the corresponding first attention mask is "111111000000000000000000000000011"; the second attention mask is "000000111111111111111111111111100".

[0093] For example, the information represented by the relation triple (motor vehicle, contains, motorcycle) (i.e., "motor vehicle contains motorcycle") can be inserted into the tail of the entity "motorcycle"; then, the information represented by the relation triple (motor vehicle insurance, object, motor vehicle) (i.e., "the insurance object of motor vehicle insurance is motor vehicle") can be inserted into the tail of "insurance", thereby obtaining the fusion text.

[0094] Figure 7 A schematic diagram of a fusion text and its corresponding attention encoding provided by an embodiment of the present application is shown in FIG. 1. As shown in FIG. 1, the fusion text is "is there motorcycle (motor vehicle contains motorcycle) insurance (the insurance object of motor vehicle insurance is motor vehicle)", the corresponding first attention mask is "111111000000000011000000000000000", and the corresponding second attention mask is "000000111111111100111111111111111". Figure 7

[0095] By performing operations on the different attention masks described above (e.g., sum operation, or operation, and operation, etc. on the first attention mask and the second attention), the corresponding target attention mask can be obtained. Then, the target attention mask is used to process the fusion text, and the encoded feature sequence vector is obtained, so that the encoded feature sequence vector can more accurately reflect the features of the fusion text.

[0096] In some optional embodiments, the initial decoding sequence list includes a start identifier, K initial decoding sequences, and scores corresponding to the K initial decoding sequences, where K is an integer greater than or equal to 1.

[0097] The initial decoding sequence list is traversed, and the encoded feature sequence vector and each initial decoding sequence in the initial decoding sequence list are input into the text decoding layer of the text recognition network model for processing, to obtain a candidate decoding sequence list. The following methods can be used to achieve this: the encoded feature sequence vector and each initial decoding sequence in the initial decoding sequence list are input into the text decoding layer for decoding, to obtain a prediction probability table; N candidate prediction words are added to the tail of each initial decoding sequence in the initial decoding sequence list, to obtain a to-be-processed decoding sequence list; the prediction probability value of each candidate prediction word in each to-be-processed decoding sequence is multiplied by the score of the initial decoding sequence corresponding to the to-be-processed decoding sequence, to obtain a score corresponding to each to-be-processed decoding sequence; the first K sequences in the to-be-processed decoding sequence list whose scores are greater than a preset threshold are taken as candidate decoding sequences, to obtain the candidate decoding sequence list.

[0098] ​The prediction probability table includes N candidate prediction words whose prediction probability values ​​are greater than a preset probability threshold, and the prediction probability value corresponding to each candidate prediction word, where N is an integer less than or equal to K. The list of decoding sequences to be processed includes N*K decoding sequences to be processed.

[0099] For example, let N be 2 and K be 3. The prediction probability table includes two candidate prediction words whose prediction probability values ​​are greater than a preset probability threshold. The initial decoding sequence table includes three initial decoding sequences and their corresponding scores. Two candidate prediction words (e.g., candidate prediction words A / B) are added to the end of each initial decoding sequence to obtain a decoding sequence table to be processed (e.g., sequence 1-A, sequence 1-B; sequence 2-A, sequence 2-B; sequence 3-A, sequence 3-B). The probability values ​​of the three candidate prediction words are multiplied by the scores of their corresponding decoding sequences to obtain a decoding sequence table to be processed, which includes 2*3 decoding sequences to be processed. The top three sequences with scores greater than a preset threshold in the decoding sequence table to be processed (including six decoding sequences to be processed) are selected as candidate decoding sequences to obtain a candidate decoding sequence table.

[0100] In some optional implementations, the feedback text is determined based on the relationship between the scores of each output decoding sequence in the output decoding sequence table and the highest score in the candidate decoding sequence table, including:

[0101] If the highest score in the output decoding sequence table is greater than or equal to the highest score in the candidate decoding sequence table, the decoding sequence corresponding to the highest score in the output decoding sequence table is processed to obtain the feedback text; otherwise, the candidate decoding sequence table is used as the initial decoding sequence table, and the execution of traversing the initial decoding sequence table and subsequent steps is triggered until the feedback text is obtained.

[0102] For example, the feedback text can be generated using the following method: Figure 6 or Figure 7 The fused text and its corresponding attention mask are input into the text encoding layer of the text recognition network model for processing, obtaining an encoded feature sequence vector to reflect the semantic features of the fused text; based on the starting identifier... <bos>Initialize a plurality of decoding sequences (e.g., 5 decoding sequences 1 / 2 / 3 / 4 / 5), and assign an initial score to each decoding sequence (e.g., the initial scores of the 5 decoding sequences are 1 / 1 / 1 / 1 / 1), and construct an initial decoding sequence list based on the above 5 decoding sequences and their corresponding initial scores.

[0103] Then, traverse the initial decoding sequence list, and input the encoded feature sequence vector and each sequence in the initial decoding sequence list into the text decoding layer for processing respectively, to obtain a candidate decoding sequence list.

[0104] For example, input the encoded feature sequence vector and each sequence in the initial decoding sequence list into the decoding network for decoding respectively, to obtain an initial decoding sequence list and a prediction probability table, wherein the prediction probability table includes 5 candidate prediction words with a prediction probability value greater than a preset probability threshold, and the initial decoding sequence list includes 5 initial decoding sequences and their corresponding scores; add the 5 candidate prediction words (e.g., candidate prediction words A / B / C / D / E) to the tail of each initial decoding sequence respectively, to obtain a to-be-processed decoding sequence list (e.g., sequence 1-A, sequence 1-B, sequence 1-C, sequence 1-D, sequence 1-E; sequence 2-A, sequence 2-B, sequence 2-C, sequence 2-D, sequence 2-E; sequence 3-A, sequence 3-B, sequence 3-C, sequence 3-D, sequence 3-E; sequence 4-A, sequence 4-B, sequence 4-C, sequence 4-D, sequence 4-E; sequence 5-A, sequence 5-B, sequence 5-C, sequence 5-D, sequence 5-E); multiply the prediction probability value of the candidate prediction word in each to-be-processed decoding sequence with the score of the initial decoding sequence corresponding to the to-be-processed decoding sequence, to obtain the score corresponding to each to-be-processed decoding sequence; take the top 5 sequences in the to-be-processed decoding sequence list (which includes 5*5 to-be-processed decoding sequences in total) with a score greater than a preset threshold as candidate decoding sequences, to obtain a candidate decoding sequence list.

[0105] Further, traverse the candidate decoding sequence list, and move the candidate decoding sequence with the end identifier and its corresponding score to an output decoding sequence list; and compare the scores of each sequence in the output decoding sequence list and the candidate decoding sequence list.

[0106] In a case where it is determined that the highest score in the output decoding sequence list is greater than or equal to the highest score in the candidate decoding sequence list, process the decoding sequence corresponding to the highest score in the output decoding sequence list, to obtain a feedback text (for example, perform word decoding on the decoding sequence corresponding to the highest score in the output decoding sequence list, to obtain a decoded text (i.e., the feedback text)); otherwise, take the candidate decoding sequence list as the initial decoding sequence list, and trigger the execution of the steps of traversing the initial decoding sequence list and the subsequent steps until the feedback text is obtained.

[0107] Through the above text processing based on the double attention mechanism, the interference information in the input text can be reduced, and the semantics of the input text is clearer. Then, through the analysis of the entities in the input text based on the professional knowledge and common sense knowledge in the knowledge graph, the obtained feedback text can be more easily understood, and the generated feedback text is more fluent.

[0108] In the embodiments of the present application, the knowledge graph-based feedback text matching the input text is obtained. Specifically, the entities in the input text are obtained, and then the serialized information corresponding to the entities is extracted from the knowledge graph. The serialized information is fused with the input text to obtain fusion data. Further, the feedback text matching the input text is determined by analyzing the fusion data. The fusion data is obtained by fusing the serialized information with the input text. The fusion data can reflect the entities in the input text and the attribute information of each entity. By considering the attribute information of the entities when analyzing the fusion data, the information interference unrelated to the attribute information of the entities can be reduced, and the possibility of knowledge loss can be reduced. The feedback text can accurately reflect the text information corresponding to the fusion data, so that the feedback text is more accurate and fluent.

[0109] In some optional embodiments, the knowledge graph includes a professional knowledge graph and a common sense knowledge graph. The generation step of the knowledge graph includes: identifying entities and relationship information between entities obtained from data in a preset field, and constructing a professional knowledge graph based on the entities and the relationship information between the entities; mining entities and relationship information between entities based on the professional knowledge graph, and establishing a common sense knowledge graph based on the mined entities and relationship information between the entities.

[0110] For example, a professional knowledge recognition model is used to identify data in a preset field to obtain entities and relationship information between entities, and a professional knowledge graph is constructed based on the obtained entities and relationship information between entities. Further, the professional knowledge graph is mined for entities and relationship information between entities based on a common sense knowledge recognition model, so as to establish a common sense knowledge graph based on the mined entities and relationship information between entities.

[0111] In some specific implementations, the professional knowledge recognition model is a model for predicting and classifying entities and relationship information between entities in a preset field, and the training text includes at least two entities. The common sense knowledge recognition model is a model obtained by training the entities and relationship information between entities in the professional knowledge graph as input parameters.

[0112] The professional knowledge recognition model and the common sense knowledge recognition model are used to determine different knowledge graphs respectively, so that the content in the knowledge graph is more abundant, the possibility of knowledge loss is reduced, and the subsequent identification of the input text is facilitated, thereby improving the identification accuracy of the input text.

[0113] For example, Figure 8 A schematic diagram of a professional knowledge graph is provided for an embodiment of the present application. As shown in Figure 8 The professional knowledge graph includes a second encoding network, a relationship classification network, and a pooling layer (not shown in the figure).

[0114] The pooling method used by the pooling layer can be a dense layer of 768*768 with a tanh activation function, followed by a dense layer of 768*8 with a softmax function. The pre-trained language model in the second encoding network can be implemented using the chinese-roberta-wwm-ext model.

[0115] As shown in Figure 8 The word embedding layer inputs "children [B-ins] children [I-ins] intention [I-ins] outside [I-ins] risk [I-ins]", the corresponding segment embedding layer is "0000000000", and the corresponding position embedding layer is "0123456789". By inputting the above information into the pre-trained language model for processing, the output result of the second encoding network, i.e. the input data of the relationship classification network, can be obtained. Then, the relationship classification network determines the corresponding relationship of the input data (e.g. "children's accidental insurance" represents insurance entities, etc.).

[0116] In the process of training the professional knowledge recognition model, the training set data is consistent with the entity recognition model, i.e. a plurality of labeled data (e.g. 30,000 labeled data, including 8 types of relationship) can be collected.

[0117] It should be noted that different labeled data can be derived from open source data or from document data in insurance consulting scenarios (e.g. online dialogue logs, insurance product documents, and contract template data, etc.). Further, the plurality of labeled data is divided into training set data (90% of the total data amount), validation set data (5% of the total data amount), and test set data (5% of the total data amount) according to a preset proportion.

[0118] After the training set data is determined, the professional knowledge recognition model is trained in the following manner: the text in the training set data and the sequence label corresponding to the text are encoded to obtain a training text encoding sequence and a training label encoding sequence corresponding to the training set data; then, the training text encoding sequence is input into the initial professional knowledge recognition model to identify the relationship between entities and obtain a relationship prediction vector; the cross-entropy loss between the training label encoding sequence and the relationship prediction vector is calculated, the network gradient of the professional knowledge recognition model is calculated based on the cross-entropy loss, and the network gradient is updated by an optimizer (such as an AdamW optimizer).

[0119] The validation set data includes: text and a sequence label corresponding to the text. By encoding the text in the validation set data, a validation text encoding sequence can be obtained. The sequence label of the text in the validation set data is encoded to obtain a validation label encoding sequence.

[0120] For example, during the training process, the validation process is performed once every 800 times of the above-mentioned loop. The validation process is to input the validation text encoding sequence in the validation set data into the professional knowledge recognition model for processing to obtain a label sequence prediction result, and then calculate the difference between the label sequence prediction result and the validation label encoding sequence in the validation set data, and take the difference as an evaluation index. The learning rate of the optimizer can be set to 1*e -5 , and the weight decay of the optimizer can be set to 1*e -2 .

[0121] In some optional embodiments, the entity and the relationship information between entities in the data of the preset field are identified, and a professional knowledge graph is constructed based on the entity and the relationship information between entities, including:

[0122] A plurality of to-be-processed texts in a preset field are obtained; each candidate entity pair in the to-be-processed text is marked to obtain a to-be-identified text; the to-be-identified text is input into the professional knowledge recognition model to predict the relationship information between two entities in each candidate entity pair; the maximum relationship probability distribution value is selected from a plurality of relationship probability distribution values of each candidate entity pair, and the predicted relationship corresponding to the maximum relationship probability distribution value is taken as the relationship between two entities in the corresponding candidate entity pair; and the professional knowledge graph is determined based on the relationship between two entities in each candidate entity pair.

[0123] Each to-be-processed text includes at least two entities, and each entity forms a candidate entity pair with any other entity. The relationship information between the two entities in each candidate entity pair includes a plurality of predicted relationships and corresponding relationship probability distribution values.

[0124] For example, the to-be-processed text is "purchase school insurance needs to meet the age of 3 years old or below 18 years old", which includes two entities, namely, an insurance entity ("school insurance") and a condition entity ("age of 3 years old or below 18 years old"). The insurance entity and the condition entity are further labeled respectively, thereby obtaining a to-be-identified text: "purchase school [B-ins] insurance [I-ins] needs to meet the age [B-con] of 3 [I-con] years old [I-con] or below [I-con] 18 [I-con] years old [I-con]".

[0125] Wherein, [B-ins] represents the start position of the insurance entity, [I-ins] represents the Inter word information of the insurance entity; [B-con] represents the start position of the condition entity, and [I-con] represents other word information in the condition entity.

[0126] It should be noted that, by labeling each candidate entity pair in the to-be-processed text to obtain the to-be-identified text, more text can be obtained, and the data is more abundant.

[0127] Further, by inputting the to-be-identified text into a professional knowledge recognition model, the relationship information between the two entities in each candidate entity pair is predicted, wherein the relationship information between the two entities in each candidate entity pair includes a plurality of predicted relationships and corresponding relationship probability distribution values.

[0128] For example, the relationship information between the two entities in a certain candidate entity pair includes a first predicted relationship with a relationship probability distribution value of 5%, a second predicted relationship with a relationship probability distribution value of 10%, a third predicted relationship with a relationship probability distribution value of 25%, and a fourth predicted relationship with a relationship probability distribution value of 40%.

[0129] Then, from the plurality of relationship probability distribution values of each candidate entity pair (i.e., from 5%, 10%, 25%, 40%, etc.), the predicted relationship corresponding to the maximum relationship probability distribution value (i.e., the fourth predicted relationship corresponding to 40%) is selected as the relationship between the two entities in the corresponding candidate entity pair, which can effectively identify the relationship between different entities and strengthen the association between different entities.

[0130] In some optional embodiments, the professional knowledge graph is mined to obtain entity and relationship information between entities, and a common knowledge graph is established based on the mined entity and relationship information between entities, including: taking the entities in the professional knowledge graph as an entity candidate set, and obtaining a relationship candidate set; combining each entity in the entity candidate set with each relationship in the relationship candidate set to obtain a search string corresponding to each entity; searching in a preset database according to the search string corresponding to each entity to obtain a candidate relationship sentence corresponding to each entity; identifying a relationship triple corresponding to each entity based on the candidate relationship sentence corresponding to each entity; and determining the common knowledge graph based on the relationship triples corresponding to multiple entities. The relationship triple includes a head entity, relationship information between entities, and a tail entity. The preset database includes multiple entities and relationship sentences related to the entities.

[0131] For example, the relationships in the relationship candidate set are combined with the entities in the entity candidate set randomly to obtain a search string corresponding to the entity. Alternatively, each entity in the entity candidate set is combined with each relationship in the relationship candidate set to obtain a search string corresponding to the entity. The latter can make the obtained search string more abundant.

[0132] For example, the candidate relationship sentence can be obtained in the following manner: first, sample part of the types of entities in the professional knowledge graph as an entity candidate set; then, extract at least one entity from the entity candidate set, and select at least one relationship from the relationship candidate set, combine the at least one entity and the at least one relationship to obtain a search string, and search in a preset database according to the search string corresponding to each entity to obtain a candidate relationship sentence corresponding to each entity.

[0133] For example, the object-type entity "motor vehicle" can be combined with the three relationships "include", "is a kind of", and "also known as" to obtain the following search strings: "motor vehicle includes", "motor vehicle is a kind of", and "motor vehicle also known as", and the search strings are input into a preset database (for example, a database storing entities, relationship sentences related to the entities (or relationship information between entities)) respectively to obtain search results, and the first 50 results in the search results are extracted; further, the first 50 results in the search results can be parsed and de-duplicated to obtain a candidate relationship sentence.

[0134] In some optional embodiments, the candidate relationship sentence that can meet the search demand of the search string can also be obtained by manual annotation.

[0135] In an implementation manner, the identifying of the relation triple corresponding to each entity based on the candidate relation sentence corresponding to each entity and the determining of the common sense knowledge graph based on the relation triple corresponding to each entity can comprise: inputting the candidate relation sentence into a common sense knowledge recognition model to obtain a predicted tail entity, then constructing a predicted relation triple (such as searching for an entity-searching relation-predicted tail entity), and adding the predicted relation triple to the common sense knowledge graph to enrich the common sense knowledge graph.

[0136] The common sense knowledge model can be trained based on a plurality of relation sentences, and the plurality of relation sentences can be the candidate relation sentences determined for each entity as described above. For example, five thousand candidate relation sentences are obtained, and the candidate relation sentences are divided into training set data (80% of the total data amount), verification set data (10% of the total data amount), and test set data (10% of the total data amount). The common sense knowledge recognition model can be trained by using the same training method as the entity recognition model to obtain a common sense knowledge recognition model meeting the use requirement.

[0137] Taking the entity "motor vehicle" and the relation "contains" as an example, a search string "motor vehicle contains" can be constructed. Then, the "motor vehicle contains" is input into a preset database (such as a search application database provided by a search server) to search and obtain the following candidate relation sentences: "motor vehicle includes cars and car trains, motorcycles and light motorcycles, tractor transport units, wheeled special mechanical vehicles, and electric vehicles, etc.". Then, the candidate relation sentences are input into the common sense knowledge recognition model to obtain the predicted tail entities: "car", "car train", "motorcycle", "light motorcycle", "tractor transport unit", "wheeled special mechanical vehicle", and "electric vehicle". Thus, the following predicted relation triples can be constructed: motor vehicle-contains-car, motor vehicle-contains-car train, motor vehicle-contains-motorcycle, motor vehicle-contains-light motorcycle, motor vehicle-contains-tractor transport unit, motor vehicle-contains-wheeled special mechanical vehicle, and motor vehicle-contains-electric vehicle.

[0138] The second aspect of the embodiment of the present application provides a text generation device. Figure 9 The text generation device provided by the embodiment of the present application is shown in FIG. 9. As shown in FIG. 9, the text generation device 900 comprises: Figure 9

[0139] The obtaining module 901 is configured to obtain an entity in the input text and extract serialized information corresponding to the entity in the knowledge graph, the serialized information being used to represent attribute information of the entity.

[0140] The fusion module 902 is configured to fuse the serialized information with the input text to obtain fusion data. ​

[0141] The processing module 903 is configured to determine the feedback text matched with the input text based on the fusion data.

[0142] In some optional embodiments, the fusion module 902 is further configured to insert the serialized information corresponding to the entity into a preset position in the input text to obtain the fusion text; determine the attention mask information based on the fusion text, the attention mask information being used to distinguish the input text and the serialized information; and generate the fusion data based on the attention mask information and the fusion text.

[0143] The preset position includes a tail of the entity and / or a tail of the demand information corresponding to the entity, and the demand information is business information representing a service to be processed by the entity.

[0144] In some optional embodiments, the processing module 903 is further configured to input the fusion data into a text encoding layer of the text recognition network model to obtain an encoding feature sequence vector, the text encoding layer being a network layer encoded based on a double attention mechanism; traverse an initial decoding sequence list, and input the encoding feature sequence vector and each initial decoding sequence in the initial decoding sequence list into a text decoding layer of the text recognition network model for processing to obtain a candidate decoding sequence list, the initial decoding sequence list including a plurality of initial decoding sequences and a score corresponding to each initial decoding sequence, and the candidate decoding sequence list including a plurality of candidate decoding sequences and a score corresponding to each candidate decoding sequence; traverse the candidate decoding sequence list, and move a candidate decoding sequence with an end identifier and a score corresponding to the candidate decoding sequence to an output decoding sequence list, the end identifier being an identifier representing that the candidate decoding sequence is an end sequence; and determine the feedback text according to a size relationship between the scores of each output decoding sequence in the output decoding sequence list and a highest score in the candidate decoding sequence list.

[0145] In some optional embodiments, the attention mask information includes a first attention mask and a second attention mask, the first attention mask and the second attention mask having different mask positions; and when the processing module 903 performs the step of inputting the fusion data into the text encoding layer of the text recognition network model to obtain the encoding feature sequence vector, the processing module 903 performs the following steps: inputs the first attention mask into the text encoding layer for processing to obtain a first attention distribution; inputs the second attention mask into the text encoding layer for processing to obtain a second attention distribution; determines a target attention distribution according to the first attention distribution and the second attention distribution; and encodes the fusion data based on the target attention distribution to obtain the encoding feature sequence vector.

[0146] In some optional embodiments, the initial decoding sequence list includes a start identifier, K initial decoding sequences and scores corresponding to the K initial decoding sequences, K being an integer greater than or equal to 1.

[0147] The processing module 903 performs the following steps when traversing the initial decoding sequence list, respectively inputting the encoded feature sequence vector and each initial decoding sequence in the initial decoding sequence list into the text decoding layer of the text recognition network model to process, and obtaining the candidate decoding sequence list:

[0148] respectively inputting the encoded feature sequence vector and each initial decoding sequence in the initial decoding sequence list into the text decoding layer to decode, obtaining a prediction probability table, wherein the prediction probability table includes N candidate prediction words with a prediction probability value greater than a preset probability threshold and a prediction probability value corresponding to each candidate prediction word, and N is an integer less than or equal to K; respectively adding the N candidate prediction words to the tail of each initial decoding sequence in the initial decoding sequence list to obtain a to-be-processed decoding sequence list, and the to-be-processed decoding sequence list includes N*K to-be-processed decoding sequences; multiplying the prediction probability value of each candidate prediction word in each to-be-processed decoding sequence with the score of the initial decoding sequence corresponding thereto to obtain a score corresponding to each to-be-processed decoding sequence; taking the top K sequences in the to-be-processed decoding sequence list with a score greater than a preset threshold as candidate decoding sequences to obtain the candidate decoding sequence list.

[0149] In some optional embodiments, the processing module 903 performs the following steps when determining the feedback text according to the size relationship between the score of each output decoding sequence in the output decoding sequence list and the highest score in the candidate decoding sequence list:

[0150] In a case where the highest score in the output decoding sequence list is greater than or equal to the highest score in the candidate decoding sequence list, processing the decoding sequence corresponding to the highest score in the output decoding sequence list to obtain the feedback text; otherwise, taking the candidate decoding sequence list as the initial decoding sequence list, and triggering the steps of traversing the initial decoding sequence list and the subsequent steps until the feedback text is obtained.

[0151] In some optional embodiments, the acquisition module 901 performs the following steps when acquiring the entity in the input text: inputting the input text into an entity recognition model to recognize and obtain the entity.

[0152] The entity recognition model is a model constructed based on entities and relationships between entities in preset domain data; the preset domain data includes at least one of historical consultation session record information, consultation product information, and consultation service information; and the input text includes consultation text, and the consultation text is used to record consultation information of user demand.

[0153] In some optional embodiments, the knowledge graph comprises a professional knowledge graph and a common sense knowledge graph, and the text generation apparatus 900 further comprises a knowledge graph determination module configured to identify entity and relationship information between entities obtained from data in a preset field, and construct a professional knowledge graph based on the entity and relationship information between entities; mine the entity and relationship information between entities based on the professional knowledge graph, and establish a common sense knowledge graph based on the mined entity and relationship information between entities.

[0154] In some optional embodiments, the data in the preset field comprises a plurality of to-be-processed texts; when identifying the entity and relationship information between entities obtained from data in a preset field, and constructing a professional knowledge graph based on the entity and relationship information between entities, the knowledge graph determination module performs the following steps: obtaining the plurality of to-be-processed texts in the preset field; marking each candidate entity pair in the to-be-processed text to obtain a to-be-identified text; inputting the to-be-identified text into a professional knowledge identification model to predict relationship information between two entities in each candidate entity pair; selecting a maximum relationship probability distribution value from a plurality of relationship probability distribution values of each candidate entity pair, and taking a predicted relationship corresponding to the maximum relationship probability distribution value as a relationship between two entities in the corresponding candidate entity pair; determining the professional knowledge graph based on the relationship between two entities in each candidate entity pair.

[0155] In some optional embodiments, the data in the preset field comprises a plurality of to-be-processed texts; when identifying the entity and relationship information between entities obtained from data in a preset field, and constructing a professional knowledge graph based on the entity and relationship information between entities, the knowledge graph determination module performs the following steps: obtaining the plurality of to-be-processed texts in the preset field; marking each candidate entity pair in the to-be-processed text to obtain a to-be-identified text; inputting the to-be-identified text into a professional knowledge identification model to predict relationship information between two entities in each candidate entity pair; selecting a maximum relationship probability distribution value from a plurality of relationship probability distribution values of each candidate entity pair, and taking a predicted relationship corresponding to the maximum relationship probability distribution value as a relationship between two entities in the corresponding candidate entity pair; determining the professional knowledge graph based on the relationship between two entities in each candidate entity pair.

[0156] In some optional embodiments, when mining the entity and relationship information between entities based on the professional knowledge graph, and establishing a common sense knowledge graph based on the mined entity and relationship information between entities, the knowledge graph determination module performs the following steps: taking the entities in the professional knowledge graph as an entity candidate set, and obtaining a relationship candidate set; combining each entity in the entity candidate set with each relationship in the relationship candidate set to obtain a search string corresponding to each entity; searching in a preset database according to the search string corresponding to each entity to obtain a candidate relationship sentence corresponding to each entity, the preset database comprising a plurality of entities and relationship sentences related to the entities; identifying a relationship triple corresponding to each entity based on the candidate relationship sentence corresponding to each entity, wherein the relationship triple comprises a head entity, relationship information between entities, and a tail entity; determining the common sense knowledge graph based on the relationship triples corresponding to a plurality of entities.

[0157] In this embodiment, the acquisition module generates a matched feedback text for the input text based on the knowledge graph. Specifically, an entity in the input text is acquired, and then the serialized information corresponding to the entity is extracted in the knowledge graph. The fusion module is used to fuse the serialized information and the input text to obtain fusion data. Further, the processing module analyzes the fusion data to determine the feedback text matched with the input text. In the above text generation process, the fusion data is obtained by fusing the serialized information and the input text. In this way, the fusion data can reflect the entities and attribute information of each entity in the input text. When analyzing the fusion data, the attribute information of the entity is considered, which can reduce the information interference irrelevant to the attribute information of the entity, and reduce the possibility of knowledge loss. The feedback text can accurately reflect the text information corresponding to the fusion data, so that the feedback text is more accurate and has higher fluency.

[0158] It can be understood that the above-mentioned various method embodiments mentioned in the present application can be combined with each other to form combined embodiments without violating the principle logic. Limited by the length, the present application will not be described again. Those skilled in the art can understand that in the above-mentioned method of the specific embodiment, the specific execution order of each step should be determined according to its function and possible internal logic.

[0159] The third aspect of the embodiments of the present application provides an electronic device and a computer readable storage medium, which can be used to implement any one of the text generation methods in the embodiments of the present application. The corresponding technical solutions and descriptions are described in the method part and are not described again.

[0160] Figure 10 A composition block diagram of an electronic device provided by the embodiments of the present application is provided.

[0161] As Figure 10 shown, the embodiments of the present application provide an electronic device, which includes at least one processor 1001, at least one memory 1002, and one or more I / O interfaces 1003 connected between the processor 1001 and the memory 1002. The memory 1002 stores one or more computer programs executable by the at least one processor 1001. The one or more computer programs are executed by the at least one processor 1001 to enable the at least one processor 1001 to perform the above-mentioned text generation method.

[0162] The embodiments of the present application also provide a computer readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the above-mentioned text generation method. The computer readable storage medium can be a volatile or non-volatile computer readable storage medium.

[0163] The embodiment of the present application further provides a computer program product, comprising computer readable code or a nonvolatile computer readable storage medium carrying computer readable code, when the computer readable code is run in a processor of an electronic device, the processor in the electronic device performs the text generation method described above.

[0164] Those of ordinary skill in the art understand that all or some of the steps in the methods disclosed above, the functions of the modules / units in the systems and devices can be implemented as software, firmware, hardware and appropriate combinations thereof. In the hardware implementation, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, one physical component can have multiple functions, or one function or step can be performed by several physical components in cooperation. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit. Such software can be distributed on a computer readable storage medium, which can include computer storage media (or non-transitory media) and communication media (or transitory media).

[0165] As known to those of ordinary skill in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable program instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM), static random access memory (SRAM), flash memory or other memory technology, portable compact disc read only memory (CD-ROM), digital versatile disc (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. Furthermore, it is known to those of ordinary skill in the art that communication media typically includes computer readable program instructions, data structures, program modules or other data in modulated data signals such as carrier waves or other transport mechanisms, and can include any information delivery medium.

[0166] Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0167] Computer readable program instructions for carrying out operations of the present application can be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions can execute entirely on the user's computing / processing device, partly on the user's computing / processing device, as a stand-alone software package, partly on the user's computing / processing device and partly on a remote computing / processing device or entirely on the remote computing / processing device or server. In the latter scenario, the remote computing / processing device can be connected to the user's computing / processing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computing / processing device, for example, through the Internet using an Internet Service Provider. In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present application.

[0168] The computer program product described herein can be embodied in a specifically designed high-speed and / or digital signal processing computer, such as a chip set comprising specific circuitry for commencing execution of computer readable program instructions, a processing circuit configured to execute a software program, a hardware logic circuit comprising a number of logic elements configured to execute computer readable program instructions, or any combination thereof. In one alternative, the computer program product is specifically configured for the task at hand, for example, the computer program product is a software development kit (SDK). In another alternative, the computer program product is a general purpose computer, which is configured to execute computer readable program instructions to perform the task at hand. In another alternative, the computer program product is a software program, which is configured to operate in, use, or communicate with the computer program product.

[0169] The computer program product described herein can be embodied in a specifically designed high-speed and / or digital signal processing computer, such as a chip set comprising specific circuitry for commencing execution of computer readable program instructions, a processing circuit configured to execute a software program, a hardware logic circuit comprising a number of logic elements configured to execute computer readable program instructions, or any combination thereof. In one alternative, the computer program product is specifically configured for the task at hand, for example, the computer program product is a software development kit (SDK). In another alternative, the computer program product is a general purpose computer, which is configured to execute computer readable program instructions to perform the task at hand. In another alternative, the computer program product is a software program, which is configured to operate in, use, or communicate with the computer program product.

[0170] These computer readable program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can include a non-transitory computer readable storage medium that can be a computer- readable storage medium having no data storage cycles that change state. The instructions can be executed by one or more processors of a computer, to cause a series of operational elements or steps to be performed on the computer to produce a computer implemented process; such that the instructions, which execute via one or more computer program product, implement a computer implemented process for performing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0171] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational elements or steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer implemented process such that the instructions that execute on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0172] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational elements or steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer implemented process such that the instructions that execute on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0173] Example embodiments have been disclosed and although specific terms are employed, they are used in a generic and descriptive sense only and not for purposes of limitation. In some instances, it will be apparent to those skilled in the art that features, characteristics or elements described with respect to a particular embodiment can be used, alone or in combination, with other embodiments unless specifically recited otherwise. Accordingly, various modifications, alterations, and improvements will readily occur to those skilled in the art with the foregoing description. Accordingly, the present application is not intended to be limited by the method of description set forth herein, but is to be accorded the full scope of the claims appended hereto.< / bos> < / eos> < / eos> < / bos> < / bos>

Claims

1. A text generation method characterized by, The method comprises the following steps: acquiring an entity in input text and extracting serialized information corresponding to the entity in a knowledge graph, the serialized information being used to represent attribute information of the entity; fusing the serialized information and the input text based on attention mask information to obtain fusion data, the attention mask information being used to distinguish the input text and the serialized information; determining feedback text matched with the input text based on the fusion data by using a double attention mechanism.

2. The method of claim 1, wherein, The step of fusing the serialized information and the input text to obtain fusion data comprises the following steps: inserting the serialized information corresponding to the entity into a preset position in the input text to obtain fusion text, the preset position comprising a tail of the entity and / or a tail of demand information corresponding to the entity, the demand information being used to represent business information to be processed by the entity; generating the fusion data based on the attention mask information and the fusion text.

3. The method of claim 2, wherein, The step of determining feedback text matched with the input text based on the fusion data by using a double attention mechanism comprises the following steps: inputting the fusion data into a text encoding layer of a text recognition network model to obtain an encoding feature sequence vector, the text encoding layer being a network layer encoded based on a double attention mechanism; traversing an initial decoding sequence list, inputting the encoding feature sequence vector and each initial encoding sequence in the initial decoding sequence list into a text decoding layer of the text recognition network model to obtain a candidate decoding sequence list, the initial decoding sequence list comprising multiple initial encoding sequences and scores corresponding to each initial encoding sequence, the candidate decoding sequence list comprising multiple candidate decoding sequences and scores corresponding to each candidate decoding sequence; traversing the candidate decoding sequence list, moving a candidate decoding sequence with an end identifier and a score corresponding to the candidate decoding sequence to an output decoding sequence list, the end identifier being an identifier used to represent that the candidate decoding sequence is an end sequence; determining the feedback text according to a size relationship between scores of each output decoding sequence in the output decoding sequence list and a highest score in the candidate decoding sequence list.

4. The method of claim 3, wherein, The attention mask information comprises a first attention mask and a second attention mask, the mask positions of the first attention mask and the second attention mask being different; The step of inputting the fusion data into a text encoding layer of a text recognition network model to obtain an encoding feature sequence vector comprises the following steps: inputting the first attention mask into the text encoding layer to obtain a first attention distribution; inputting the second attention mask into the text encoding layer to obtain a second attention distribution; determining a target attention distribution according to the first attention distribution and the second attention distribution; encoding the fusion data based on the target attention distribution to obtain the encoding feature sequence vector.

5. The method of claim 3, wherein, The initial decoding sequence list comprises a start identifier, K initial decoding sequences and scores corresponding to the initial decoding sequences, K being an integer greater than or equal to 1. The initial decoding sequence list is traversed, and the encoding feature sequence vector and each initial encoding sequence in the initial decoding sequence list are respectively input into a text decoding layer of the text recognition network model for processing to obtain a candidate decoding sequence list, including: The encoding feature sequence vector and each initial decoding sequence in the initial decoding sequence list are respectively input into the text decoding layer for decoding to obtain a prediction probability table, wherein the prediction probability table includes N candidate prediction words with a prediction probability value greater than a preset probability threshold and a prediction probability value corresponding to each candidate prediction word, and N is an integer less than or equal to K; The N candidate prediction words are respectively added to the tail of each initial decoding sequence in the initial decoding sequence list to obtain a to-be-processed decoding sequence list, and the to-be-processed decoding sequence list includes N*K to-be-processed decoding sequences; The prediction probability value of the candidate prediction word in each to-be-processed decoding sequence is multiplied by the score of the corresponding initial decoding sequence to obtain a score corresponding to each to-be-processed decoding sequence; The top K sequences in the to-be-processed decoding sequence list with a score greater than a preset threshold are taken as candidate decoding sequences to obtain the candidate decoding sequence list.

6. The method according to any one of claims 3 to 5, characterized in that, The feedback text is determined according to the size relationship between the scores of each output decoding sequence in the output decoding sequence list and the highest score in the candidate decoding sequence list, including: In a case where the highest score in the output decoding sequence list is greater than or equal to the highest score in the candidate decoding sequence list, the decoding sequence corresponding to the highest score in the output decoding sequence list is processed to obtain the feedback text; Otherwise, the candidate decoding sequence list is taken as the initial decoding sequence list, and the steps of traversing the initial decoding sequence list and the subsequent steps are triggered to be executed until the feedback text is obtained.

7. The method of claim 1, wherein, The entity in the input text is obtained, including: The input text is input into an entity recognition model for recognition to obtain the entity; The entity recognition model is a model constructed based on entities and relationships between entities in preset field data; the preset field data includes at least one of historical consultation session record information, consultation product information, and consultation service information; and the input text includes a consultation text used to record consultation information of user demand.

8. The method of claim 1, wherein, The knowledge graph includes a professional knowledge graph and a common sense knowledge graph, and the generation steps of the knowledge graph include: Identifying entities and relationship information between entities obtained from the preset field data, and constructing the professional knowledge graph based on the entities and the relationship information between the entities; Based on the professional knowledge graph, the relationship information between the entities is mined, and the common sense knowledge graph is established based on the mined entities and the relationship information between the entities.

9. The method of claim 8, wherein, The preset field data includes a plurality of to-be-processed texts; and the entities and the relationship information between the entities obtained from the preset field data are identified, and the professional knowledge graph is constructed based on the entities and the relationship information between the entities, including: obtaining a plurality of to-be-processed texts in the preset field, each to-be-processed text comprising at least two entities, and each entity and any other entity forming a candidate entity pair; labeling each candidate entity pair in the to-be-processed text to obtain a to-be-identified text; inputting the to-be-identified text into a professional knowledge identification model to predict relationship information between two entities in each candidate entity pair, wherein the relationship information between two entities in each candidate entity pair comprises a plurality of predicted relationships and corresponding relationship probability distribution values; selecting a maximum relationship probability distribution value from the plurality of relationship probability distribution values of each candidate entity pair, and taking a predicted relationship corresponding to the maximum relationship probability distribution value as the relationship between two entities in the corresponding candidate entity pair; determining the professional knowledge graph based on the relationship between two entities in each candidate entity pair.

10. The method of claim 8, wherein, The professional knowledge graph is mined based on the entity and the relationship information between the entities, and the common sense knowledge graph is established based on the mined entity and the relationship information between the entities, comprising: taking the entity in the professional knowledge graph as an entity candidate set and obtaining a relationship candidate set; combining each entity in the entity candidate set with each relationship in the relationship candidate set to obtain a search string corresponding to each entity; searching in a preset database according to the search string corresponding to each entity to obtain a candidate relationship sentence corresponding to each entity, wherein the preset database comprises a plurality of entities and relationship sentences related to the entities; identifying a relationship triple corresponding to each entity pair based on the candidate relationship sentence corresponding to each entity, wherein the relationship triple comprises a head entity, relationship information between entities, and a tail entity; determining the common sense knowledge graph based on the relationship triple corresponding to a plurality of entities.

11. A text generating apparatus characterized by comprising: comprising: an acquisition module configured to acquire an entity in an input text and extract serialized information corresponding to the entity in a knowledge graph, the serialized information being used to represent attribute information of the entity; a fusion module configured to fuse the serialized information and the input text based on attention mask information to obtain fusion data, the attention mask information being used to distinguish the input text and the serialized information; a processing module configured to determine a feedback text matched with the input text based on the fusion data using a double-attention mechanism.

12. An electronic device, comprising: comprising: at least one processor; and a memory connected in communication with the at least one processor; wherein the memory stores one or more computer programs executable by the at least one processor, and the one or more computer programs are executed by the at least one processor to enable the at least one processor to perform the text generation method of any one of claims 1-10.

13. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by the processor, implements the text generation method of any one of claims 1-10. The computer program, when executed by the processor, implements the text generation method of any one of claims 1-10.

Images