Entity generation model training method, entity generation method and device

Abstract

The invention discloses a training method of an entity generation model and an entity generation method and device, and relates to the field of computers, in particular to the field of artificial intelligence such as deep learning and large models. According to the specific implementation scheme, an input sequence corresponding to any main body in a target domain is obtained; wherein the input sequence comprises a sample text and a to-be-extracted field; encoding the input sequence by adopting a base model of a target domain to obtain an encoding vector; performing linear transformation on the coding vector to obtain a scoring matrix corresponding to the to-be-extracted field; wherein each element in the scoring matrix represents that any two characters in the input sequence are scores of the entity boundary of the to-be-extracted field; decoding the coding vector by adopting a base model to obtain an entity extraction result; and training the base model according to the scoring matrix and the entity extraction result to obtain a trained entity generation model corresponding to any subject.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to the field of artificial intelligence such as deep learning and large models, specifically to a training method for an entity generation model, an entity generation method, and an apparatus. Background Technology

[0002] As business scale grows across industries, a large number of business documents require element parsing to extract entities for use by business systems. Entity extraction has been widely applied in multiple fields, such as digital domains like financial information services, internet services, and information security, as well as physical domains like industrial manufacturing and energy and power. Summary of the Invention

[0003] This application provides a training method for an entity generation model, an entity generation method, and an apparatus. The specific solution is as follows: According to one aspect of this application, a method for training an entity generation model is provided, comprising: Obtain the input sequence corresponding to any subject in the target domain; wherein, the input sequence includes sample text and fields to be extracted; Using the base model of the target domain, the input sequence is encoded to obtain an encoded vector; The encoding vector is linearly transformed to obtain the scoring matrix corresponding to the field to be extracted; wherein each element in the scoring matrix represents the score of any two characters in the input sequence as the entity boundary of the field to be extracted, and the entity boundary refers to the start and end positions of the string associated with the field to be extracted in the input sequence. Using the aforementioned base model, the encoded vector is decoded to obtain entity extraction results; Based on the scoring matrix and the entity extraction results, the base model is trained to obtain the trained entity generation model corresponding to any subject.

[0004] According to another aspect of this application, an entity generation method is provided, comprising: Obtain the input sequence corresponding to any subject in the target domain; wherein, the input sequence includes the text to be parsed and the fields to be extracted; Based on the input sequence, an entity extraction result is generated using the entity generation model corresponding to any of the subjects; wherein the entity generation model is trained using the method described in one of the above embodiments.

[0005] According to another aspect of this application, a training apparatus for an entity generation model is provided, comprising: An acquisition module is used to acquire an input sequence corresponding to any subject in the target domain; wherein, the input sequence includes sample text and fields to be extracted; The encoding module is used to encode the input sequence using the base model of the target domain to obtain an encoding vector; The transformation module is used to perform a linear transformation on the encoding vector to obtain the scoring matrix corresponding to the field to be extracted; wherein, each element in the scoring matrix represents the score of any two characters in the input sequence as the entity boundary of the field to be extracted, and the entity boundary refers to the start and end positions of the string associated with the field to be extracted in the input sequence. The decoding module is used to decode the encoded vector using the base model to obtain the entity extraction result; The training module is used to train the base model based on the scoring matrix and the entity extraction results to obtain the trained entity generation model corresponding to any subject.

[0006] According to another aspect of this application, an entity generation apparatus is provided, comprising: The acquisition module is used to acquire the input sequence corresponding to any subject in the target domain; wherein, the input sequence includes the text to be parsed and the fields to be extracted; The generation module is used to generate entity extraction results based on the input sequence and using the entity generation model corresponding to any subject; wherein the entity generation model is trained using the method described in one of the above embodiments.

[0007] According to another aspect of this application, an electronic device is provided, comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method described in the above embodiments.

[0008] According to another aspect of this application, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to perform the method described in the above embodiments.

[0009] According to another aspect of this application, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the method described in the above embodiments.

[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0011] The accompanying drawings are provided for a better understanding of this solution and do not constitute a limitation of this application. Wherein: Figure 1 A flowchart illustrating a training method for an entity generation model provided in an embodiment of this application; Figure 2 A schematic diagram of a scoring matrix provided in an embodiment of this application; Figure 3 A flowchart illustrating a training method for an entity generation model provided in another embodiment of this application; Figure 4 A flowchart illustrating a training method for an entity generation model provided in another embodiment of this application; Figure 5 A flowchart illustrating an entity generation method provided in an embodiment of this application; Figure 6 A schematic diagram of the structure of a training device for an entity generation model provided in an embodiment of this application; Figure 7 This is a schematic diagram of the structure of an entity generation apparatus provided in an embodiment of this application; Figure 8 This is a block diagram of an electronic device used to implement the training method of the entity generation model in the embodiments of this application. Detailed Implementation

[0012] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of this application, including various details to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0013] It should be noted that the acquisition, storage, use, and processing of data in this application comply with the relevant provisions of national laws and regulations and do not violate public order and good morals.

[0014] The following description, with reference to the accompanying drawings, describes a training method, entity generation method, apparatus, electronic device, and storage medium for an entity generation model according to embodiments of this application.

[0015] Figure 1 This is a flowchart illustrating a training method for an entity generation model provided in an embodiment of this application.

[0016] The training method for the entity generation model in this application embodiment can be executed by the device in this application embodiment, which can be configured in an electronic device.

[0017] Among them, electronic devices can be any device with computing capabilities, such as personal computers, mobile terminals, servers, etc. Mobile terminals can be hardware devices with various operating systems, touch screens and / or displays, such as in-vehicle devices, mobile phones, tablets, personal digital assistants, wearable devices, etc.

[0018] like Figure 1 As shown, the training method for this entity generation model includes: Step 101: Obtain the input sequence corresponding to any subject in the target domain.

[0019] In this application, the target field can be any field, such as finance, education, healthcare, law, e-commerce, industrial manufacturing, energy and electricity, etc.

[0020] Since different entities in the same domain may extract different types of entities, such as Bank A, Securities B, and Trust C in the financial field, and Middle School A and Early Childhood Education Research Institution B in the education field, different entities in the same domain may extract different types of entities. Therefore, for the same domain, corresponding entity generation models can be trained according to different entities to extract entities from documents related to the entities.

[0021] In this application, for any subject in the target domain, the input sequence of that subject can be obtained.

[0022] The input sequence may include sample text corresponding to the subject, fields to be extracted, etc. For example, there may be one or more fields to be extracted corresponding to the subject, which is not limited.

[0023] For example, the input sequence can be obtained by filling the entity extraction prompt template corresponding to the entity with the sample text corresponding to the entity. The entity extraction prompt template may include the field to be extracted, entity extraction task information, etc. Then the input sequence can be used to prompt the base model of the target domain to perform the entity extraction task in order to extract the entity corresponding to the field to be extracted from the sample text.

[0024] For example, the base model for the target domain can be a large model, or it can be obtained by supervised full-parameter training of a pre-trained language model based on business data from multiple subjects in the target domain.

[0025] For example, business data of banks, securities, funds, trusts, insurance, etc. in the financial field can be used to fine-tune a 7B pre-trained language model to obtain a financial base model, that is, the base model in the financial field.

[0026] Step 102: Use the base model of the target domain to encode the input sequence to obtain an encoded vector.

[0027] In this application, the encoder in the base model of the target domain can be used to encode the input sequence to obtain an encoded vector.

[0028] Step 103: Linearly transform the encoded vector to obtain a scoring matrix corresponding to the fields to be extracted.

[0029] In the generative entity extraction task, the model outputs entities in text form. However, due to the free generation characteristic of the decoding process, the model often fails to align with the original text, resulting in entity boundary drift.

[0030] Based on this, in order to enhance entity boundary determination, in this application, the encoded vector output by the encoding layer can be linearly transformed to obtain a scoring matrix corresponding to the fields to be extracted, where each element in the scoring matrix represents the score that any two characters in the input sequence are the entity boundaries of the fields to be extracted.

[0031] Among them, the entity boundary of the field to be extracted can refer to the start position and end position of the string associated with the field to be extracted in the input sequence. Or, the entity boundary of the field to be extracted can also be understood as the boundary of the field to be extracted and its corresponding entity in the input sequence in the input sequence.

[0032] For example, Figure 2 in the input sequence is "***, the coupon rate data for 5 years is 3.45%.", assuming the field to be extracted is the coupon rate data, any character in this input sequence may be the start position of the entity boundary of this field to be extracted, and any character in this input sequence may be the end position of the entity boundary. Then, any two characters in the input sequence can be combined as the entity boundary, and a scoring matrix can be obtained according to the scores of any two characters as the entity boundary.

[0033] Since the actual start position of the entity boundary of the coupon rate data in the input sequence "***, the coupon rate data for 5 years is 3.45%." is "票" and the end position is "%", the learning goal is that the score of the character "票" as the start position of the entity boundary and the character "%" as the end position of the entity boundary in the scoring matrix approaches 1, that is, Figure 2The score for the position where "1" is located is close to 1, and the score for other characters in the input sequence that are entity boundaries is close to 0. For example, the score for the character "5" as the start position of the entity boundary and the character "%" as the end position of the entity boundary should be close to 0.

[0034] For example, if there are multiple fields to be extracted, each field has a corresponding scoring matrix, and the scoring matrices of different fields to be extracted have the same size. For instance, if the length of the input sequence is M, then the scoring matrix is ​​an M*M matrix.

[0035] Therefore, by using a scoring matrix, in addition to character-level generation, the position of entity boundaries is further constrained, reducing issues such as too many or too few characters and entity boundary drift from the source.

[0036] Step 104: Using the base model, the encoded vector is decoded to obtain the entity extraction result.

[0037] In this application, a decoder in the base model can be used to decode the encoded vector to obtain the entity extraction result.

[0038] The entity extraction result may include the field to be extracted and the field value of the field to be extracted. Here, the field value of the field to be extracted is the entity extracted from the sample text.

[0039] For example, the encoded vector output by the encoding layer of the base model of the target domain can be input into the decoder, which outputs the extracted entities character by character.

[0040] For example, for single-entity tasks, the base model can directly generate a single entity; for multi-entity tasks, the base model can generate multiple entity results in a serialized form; for negative input samples, the base model generates an output of "no corresponding entity" to ensure robustness.

[0041] For example, the output of the base model can be in the form of natural language answers or standardized structured fields, which can be easily used directly in the business system of any entity.

[0042] Step 105: Based on the scoring matrix and entity extraction results, train the base model to obtain the trained entity generation model corresponding to any subject.

[0043] In this application, the boundary loss can be determined based on the scoring matrix, the generation loss can be determined based on the entity extraction results, and the base model can be trained based on the boundary loss and the generation loss to obtain the trained entity generation model corresponding to any subject.

[0044] For example, the boundary loss and the generation loss can be weighted to obtain a sum of losses. Based on the sum of losses, the parameters of the base model can be adjusted, and the base model with adjusted parameters can continue to be trained until the training termination condition is met, thus obtaining the entity generation model.

[0045] The training termination condition can be reaching a preset number of training sessions, or the loss being less than a preset threshold, or other conditions, which are not limited.

[0046] For example, a trained entity generation model corresponding to any subject can be used to extract entities from the business data of any subject.

[0047] In this embodiment, a base model for the target domain is used to encode the input sequence corresponding to any subject in the target domain, obtaining an encoding vector. The encoding vector is then linearly transformed to obtain a scoring matrix for the field to be extracted. The base model is then used to decode the encoding vector to obtain the entity extraction result. Based on the scoring matrix and the entity extraction result, the base model is trained to obtain the entity generation model corresponding to that subject. Thus, by introducing a scoring matrix to predict the entity boundaries of the field to be extracted, entity boundary awareness is introduced into generative information extraction. In addition to character-level generation, the position of entity boundaries is further constrained, enhancing the accuracy of entity boundary discrimination, reducing model drift at entity boundaries, keeping the entity extraction result aligned with the original text, and improving the accuracy of entity extraction.

[0048] Figure 3 This is a flowchart illustrating a training method for an entity generation model provided in another embodiment of this application.

[0049] like Figure 3 As shown, the training method for this entity generation model includes: Step 301: Obtain the input sequence corresponding to any subject in the target domain.

[0050] Step 302: Using the base model of the target domain, the input sequence is encoded to obtain the encoded vector.

[0051] In this application, steps 301-302 can be implemented in any of the embodiments of this application, so they will not be described in detail here.

[0052] Step 303: Perform a linear transformation on the encoding vector to obtain the first sequence vector and the second sequence vector corresponding to the field to be extracted.

[0053] The first sequence vector and the second sequence vector corresponding to the field to be extracted can be used to identify the entity corresponding to the field to be extracted in the input sequence.

[0054] If there are multiple fields to be extracted, the first sequence vector and the second sequence vector corresponding to each field to be extracted can be obtained.

[0055] In this application, an additional boundary auxiliary head network can be used outside the base model to perform a linear transformation on the encoded vector, obtaining the first sequence vector and the second sequence vector corresponding to the fields to be extracted. The boundary auxiliary head network can be understood as a simplified multi-head attention mechanism network, with the number of attention mechanism heads being the same as the number of fields to be extracted.

[0056] In some embodiments, a boundary auxiliary head network can be used to obtain a first sequence vector based on the encoded vector and the first weight parameters corresponding to the field to be extracted, and to obtain a second sequence vector based on the encoded vector and the second weight parameters corresponding to the field to be extracted.

[0057] Among them, the first weight parameter and the second weight parameter corresponding to the field to be extracted are learnable parameters.

[0058] For example, a first sequence vector can be obtained by multiplying each element in the encoding vector with the first weight parameter corresponding to the field to be extracted, and a second sequence vector can be obtained by multiplying each element in the encoding vector with the second weight parameter corresponding to the field to be extracted.

[0059] If there are multiple fields to be extracted, each field to be extracted has a corresponding first weight parameter and second weight parameter.

[0060] As an example, the encoded vector is The encoding vector can be linearly transformed using the following formulas (1)-(2): (1) (2) in, Indicates the first The first sequence vector corresponding to the field to be extracted One element; Indicates the first The second sequence vector corresponding to the field to be extracted One element; Indicates the first The first weight parameter of the field to be extracted; Indicates the first The second weight parameter of the fields to be extracted; Represents the first in the encoded vector One element; and It is a constant.

[0061] Therefore, the first sequence vector can be obtained according to formula (1). According to formula (2), the second sequence vector can be obtained. .

[0062] Therefore, for any field to be extracted, a first sequence vector is obtained by using each element in the encoding vector and the first weight parameter of the field to be extracted, and a second sequence vector is obtained by using each element in the encoding vector and the second weight parameter of the field to be extracted. Thus, by using different weight parameters to transform the encoding vector for the same field to be extracted, two sequence vectors can be obtained, which can improve the accuracy of scoring and thus improve the accuracy of entity boundary discrimination.

[0063] Step 304: Obtain the scoring matrix by multiplying any element in the first sequence vector with any element in the second sequence vector.

[0064] In this application, a boundary auxiliary head network can be used to multiply each element of the first sequence vector with each element of the second sequence vector to obtain a scoring matrix. Thus, the boundary auxiliary head network outputs the scoring matrix corresponding to the field to be extracted. If there are multiple fields to be extracted, the boundary auxiliary head network outputs the scoring matrices corresponding to each of the multiple fields.

[0065] As an example, combining the above formulas (1)-(2), the scoring matrix can be calculated using the following formula (3): (3) in, Represents the first in the input sequence The character and the first Each character represents a score indicating the start and end positions of the entity boundary of the field to be extracted.

[0066] In this embodiment, if there are multiple fields to be extracted, the encoding vectors can be input into the attention layers corresponding to the fields to be extracted in the boundary auxiliary head network, so as to obtain the scoring matrix corresponding to the fields to be extracted through linear transformation.

[0067] Step 305: Using the base model, the encoded vector is decoded to obtain the entity extraction result.

[0068] Step 306: Based on the scoring matrix and entity extraction results, train the base model to obtain the trained entity generation model corresponding to any subject.

[0069] In this application, steps 305-306 can be implemented in any of the embodiments of this application, so they will not be described in detail here.

[0070] In this embodiment, a first sequence vector and a second sequence vector are obtained by performing a linear transformation on the encoding vector to identify the entity corresponding to the field to be extracted in the input sequence. Then, the scoring matrix corresponding to the field to be extracted is obtained by multiplying any element of the first sequence vector and any element of the second sequence vector. Thus, by performing a linear transformation on the encoding vector to obtain two sequence vectors corresponding to the field to be extracted, and obtaining the score for any two characters belonging to the entity boundary based on the two sequence vectors, the accuracy of each score in the scoring matrix can be improved, thereby improving the accuracy of entity boundary discrimination of the field to be extracted.

[0071] Figure 4 This is a flowchart illustrating a training method for an entity generation model provided in another embodiment of this application.

[0072] like Figure 4 As shown, the training method for this entity generation model includes: Step 401: Obtain the input sequence corresponding to any subject in the target domain.

[0073] Step 402: Using the base model of the target domain, the input sequence is encoded to obtain the encoded vector.

[0074] Step 403: Perform a linear transformation on the encoding vector to obtain the scoring matrix corresponding to the field to be extracted.

[0075] Step 404: Using the base model, the encoded vector is decoded to obtain the entity extraction result.

[0076] In this application, steps 401-404 can be implemented in any of the embodiments of this application, so they will not be described in detail here.

[0077] Step 405: Based on the scoring matrix and entity extraction results, adjust the parameters of the initial parsing head in the base model to obtain the target parsing head corresponding to any subject.

[0078] The initial parsing head can refer to a linear layer in the base model of the target domain. For example, the initial parsing head can include some linear layers in the base model, or it can include all linear layers in the base model.

[0079] For example, the parameters of the linear layer may include the query matrix (i.e., the Q matrix), the value matrix (i.e., the V matrix), the key matrix (i.e., the K matrix), etc.

[0080] The query matrix can be used to control the direction of attention, determining which positions the model "focuses on" in the input sequence. Fine-tuning the query matrix can directly affect the model's attention distribution to different positions, making it more focused on task-related information.

[0081] The value matrix can be used to adjust the content representation. The value matrix stores the actual aggregated information content. Fine-tuning the value matrix can change the model's representation of information at the points of interest, directly affecting the quality of the output content.

[0082] In this application, since fine-tuning the query matrix and value matrix is ​​nearly as effective as full fine-tuning for language generation tasks, the parameters of the initial parsing head can include the query matrix and value matrix of the linear layer in the base model of the target domain, in order to reduce the amount of parameter tuning, computation, and training efficiency while ensuring model performance. In other words, the query matrix and value matrix of the linear layer in the base model can be adjusted based on the scoring matrix and entity extraction results.

[0083] For example, the parameters of the initial parsing header may include the query matrix and value matrix of some linear layers in the base model, or the parameters of the initial parsing header may include the query matrix and value matrix of all linear layers in the base model, and this application does not limit this.

[0084] In this application, during the process of adjusting the query matrix and value matrix of the linear layers in the base model, other parameters in the base model other than the query matrix and value matrix of these linear layers can be frozen until the training is completed.

[0085] In some embodiments, the generation loss can be determined by cross-entropy based on the entity extraction results and the entity labels corresponding to the fields to be extracted. The boundary loss can be determined by cross-entropy based on the scoring matrix and the position information of the entity labels in the input sequence. Then, the parameters of the initial parser head can be adjusted based on the generation loss and the boundary loss to obtain the parameter-adjusted parser head. The adjusted parser head can be trained until the training termination condition is met to obtain the target parser head.

[0086] The training termination condition can be reaching a preset number of training sessions, or the loss being less than a preset threshold, or other conditions, which are not limited.

[0087] For example, the true entity boundary corresponding to the field to be extracted can be determined based on the position information of the entity label in the input sequence. Then, based on the true entity boundary, the training objective of each element in the scoring matrix can be determined. Thus, the boundary loss can be determined by cross-entropy based on the scoring matrix and the training objective of each element in the scoring matrix.

[0088] Since the end position of the entity boundary in the input sequence is after the start position, we do not need to consider the scores of the positions in the lower triangular matrix of the scoring matrix where the end position index is less than the start position index. Therefore, we can first mask the scores of these positions in the scoring matrix, and then determine the boundary loss based on the elements in the upper triangular matrix of the scoring matrix.

[0089] for example, Figure 2 In the diagram, the vertical axis represents the start boundary, and the horizontal axis represents the end boundary. The characters in the input sequence "***, 5-year coupon rate data 3.45%." serve as the start and end boundaries. When calculating the boundary loss, the elements below the diagonal can be masked, and the boundary loss can be determined only based on the elements above the diagonal.

[0090] If there are multiple fields to be extracted, and each field has a corresponding boundary loss, then the parameters of the initial parsing header can be adjusted according to the generation loss and the boundary losses corresponding to the multiple fields to be extracted.

[0091] Therefore, by introducing the entity labels of the field to be extracted as a benchmark to determine the generation loss, and by introducing the position information of the entity labels in the input sequence as a benchmark to determine the boundary loss, the accuracy of the generation loss and the boundary loss can be improved. Then, training the initial parsing head based on the generation loss and the boundary loss can improve the accuracy of character generation and the accuracy of the model in judging entity boundaries, thereby improving the accuracy of entity extraction.

[0092] Generative information extraction models are prone to character-level errors and fictitious elements during sequence generation, leading to deviations in semantic consistency between the output and the real text. For example, if the amount is 2.28, the extracted value is 2.18.

[0093] Therefore, to ensure semantic consistency, in some embodiments, the alignment loss can be determined based on the entity labels corresponding to the fields to be extracted and the generated entities corresponding to the fields to be extracted in the entity extraction results. Then, the parameters of the initial parsing head are adjusted based on the generation loss, boundary loss, and alignment loss to obtain the target parsing head. Thus, by introducing the alignment loss to train the initial parsing head in the base model, the semantic consistency between the entity extraction results and the sample text can be improved.

[0094] The generated entity corresponding to the field to be extracted is the entity corresponding to the field to be extracted from the input sequence.

[0095] For example, the generation loss, boundary loss, and alignment loss can be weighted to obtain the total loss, and the parameters of the initial parsing head can be adjusted based on the total loss.

[0096] Edit distance is typically used to measure the difference between two text sequences. However, the minimum operation in edit distance, such as insertion, deletion, and replacement, is non-differentiable and cannot be used for model training.

[0097] Based on this, for example, the editing operation cost of converting the generated entity into an entity label can be determined, and the editing operation cost can be smoothed and fused to obtain the processed operation cost. Based on the processed operation cost, the target editing distance between the generated entity and the entity label can be determined, and the alignment loss can be determined based on the target editing distance.

[0098] The cost of editing operations can refer to the time spent editing a string (e.g., before generating an entity). Converting one string (e.g., the first character of an entity label) into another string (e.g., the first character of the entity label) The cost of performing editing operations, such as insertion, deletion, and replacement, during the process of converting a string into another string (e.g., 1 character). For example, the cost of an editing operation can be a scalar value reflecting the cost required to perform the editing operation when converting a string into another string. Therefore, the cost of editing operations in converting an entity into an entity tag can refer to the cost incurred in performing the editing operation during the process of converting an entity into an entity tag.

[0099] For example, the cost of editing operations may include the cost of insertion operations, the cost of deletion operations, the cost of replacement operations, etc.

[0100] For example, the generated entities have a total of The entity tag contains [number] characters. The cost of the insertion operation to convert an entity into an entity tag, which generates the first character, can refer to the cost of inserting the entity tag in the first character. The cost of inserting a character and the cost of deleting an entity into an entity tag can refer to the cost of deleting the first character of the generated entity. The cost of replacing the generated entity with an entity label, where the generated entity is replaced by the first character, can refer to the cost of replacing the first character. The cost of each character.

[0101] For example, a dynamic programming method can be used to determine the editing operation cost of converting generated entities into entity labels through recursion, and the editing operation cost in the recursion process can also be smoothly integrated.

[0102] For example, the SoftMin function can be used to smoothly blend the editing costs throughout the recursive process.

[0103] For example, the editing operation cost of converting a generated entity into an entity label can be determined based on the processed operation cost and the matching cost between the last character of the generated entity and the last character of the entity label. This cost is also the minimum cost of converting a generated entity into an entity label, and the minimum cost of converting a generated entity into an entity label is determined as the target edit distance.

[0104] As an example, the following formula (4) is used to calculate the first step of generating the entity. The character and the entity label before Edit distance between characters That is, before the entity is generated Before converting one character into an entity label Minimum cost for each character: (4) in, Represents the first entity generated The first character and the entity tag The matching cost for each character can be a preset scalar value, for example, if the match is... =0, if not a match It equals 2; The definition is: ,in It's a temperature parameter. smaller The closer it is to the min function; Indicates the unit cost of the deletion operation; Indicates the preceding part of the entity generation Before converting one character into an entity label Minimum cost for 1 character; Indicates the deletion of the first generated entity. The cost of each character, that is, the cost of generating the entity. Before converting characters into entity tags The cost of deleting one character; Indicates the unit cost of the insertion operation; Indicates the preceding part of the entity generation Before converting one character into an entity label Minimum cost for 1 character; Indicates the first instance of the entity label The cost of inserting a character at each position, which is also the cost of generating the entity before... Before converting characters into entity tags The cost of inserting one character; Indicates the unit cost of the replacement operation; Indicates the preceding part of the entity generation Before converting one character into an entity label Minimum cost for 1 character; Indicates the replacement of the generated entity's first... The cost of each character, that is, the cost of generating the entity. Before converting characters into entity tags The cost of replacing a character.

[0105] The unit cost of the deletion operation can be a predefined scalar value that reflects the cost required to perform the deletion operation; the unit cost of the insertion operation can be a predefined scalar value that reflects the cost required to perform the insertion operation; and the unit cost of the replacement operation can be a predefined scalar value that reflects the cost required to perform the replacement operation.

[0106] It is understandable that the cost of editing operations that convert entities into entity tags includes the cost of deletion operations. The cost of insertion operations and the cost of replacement operations ( Therefore, the processed cost of the editing operation of converting the generated entity into an entity label, after smoothing and merging, is: .

[0107] Therefore, when the characters in the generated entity do not match the characters in the entity label, the replacement cost is weighted by similarity instead of being directly calculated as 1. This results in the gradient signal being "gradually corrected" rather than a complete error judgment based on cross-entropy. Furthermore, for cases with missing characters, not only are paths that can be directly deleted considered, but paths that can be inserted with missing characters are also included. The SoftMin function can assign weights to candidate paths, allowing the model to still obtain the gradient signal of the characters that should be filled in. This enables the model to obtain fine-grained feedback in a continuously differentiable path space and gradually converge to the correct entity output, rather than receiving only a single penalty.

[0108] As an example, generate entities Entity label Using the above formula (4) from , The process begins recursively, ultimately yielding the target edit distance between the generated entity and its label. The alignment loss can be calculated using the following formula (5): (5) in, It is a normalization constant, such as the length of the entity label. Or That is, the maximum of the length of the entity tag and the length of the actual tag.

[0109] Therefore, by smoothing and fusing the cost of editing operations, a distance metric that is smooth and differentiable for insertion, deletion, and replacement operations can be obtained. This metric is learnable and trainable, and compared to cross-entropy loss, it can provide finer-grained gradient signals for misspellings and omissions, thereby improving the model's stability in entity extraction.

[0110] In addition, while adjusting the initial parsing head according to the total loss, the first weight parameter and the second weight parameter corresponding to the field to be extracted can be adjusted according to the total loss, so as to learn the first sequence vector and the second sequence vector corresponding to the field to be extracted, so that the score of the real entity boundary in the scoring matrix is ​​closer to 1, and the others are closer to 0.

[0111] Step 406: Obtain the entity generation model based on the target parsing header.

[0112] In this application, the parameters of the target parsing header can be combined with other frozen parameters in the base station model of the target domain, excluding the parameters in the target parsing header, to obtain an entity generation model.

[0113] In this application, for different subjects in the target domain, a parsing head corresponding to each subject can be obtained, supporting entity substitution and differentiated parsing for different subjects. Then, in the inference application stage, the parsing head matching the subject can be loaded and combined with the base model of the target domain to obtain the entity generation model corresponding to that subject, which can effectively solve the extraction confusion problem caused by semantic opposition between different subjects.

[0114] In this embodiment, the query matrix and value matrix of the linear layer in the base model are adjusted according to the scoring matrix and entity extraction results. Thus, through two-layer linear projection, the computational and parameter overhead can be reduced and the training efficiency can be improved while ensuring model performance.

[0115] Figure 5 This is a flowchart illustrating an entity generation method provided in an embodiment of this application.

[0116] like Figure 5 As shown, the entity generation method includes: Step 501: Obtain the input sequence corresponding to any subject in the target domain.

[0117] The input sequence includes the text to be parsed and the fields to be extracted.

[0118] For example, the input sequence can be obtained by filling in the entity extraction prompt template corresponding to the subject with the text to be parsed. The entity extraction prompt template may include the field to be extracted, entity extraction task information, etc. Then the input sequence can be used to prompt the entity generation model corresponding to the subject to perform the entity extraction task in order to extract the entity corresponding to the field to be extracted from the text to be parsed.

[0119] Step 502: Based on the input sequence, generate entity extraction results using the entity generation model corresponding to any subject.

[0120] In this application, the entity generation model corresponding to any subject can be trained using the training method described in any of the above embodiments. The entity generation model corresponding to the subject can be used to process the input sequence to extract the entity of the field to be extracted from the input sequence and obtain the entity extraction result.

[0121] For example, the entity generation model corresponding to this subject can encode the input sequence to obtain an encoding vector, and then decode the encoding vector to obtain the entity extraction result.

[0122] The entity extraction results may include the field to be extracted, the entity to be extracted corresponding to the field to be extracted, etc.

[0123] For example, the output of entity extraction results can be in the form of natural language answers or standardized structured fields, which can be easily used directly in the business system of the subject.

[0124] Optionally, the encoded vector can be input into the trained boundary auxiliary head network for linear transformation to obtain the scoring matrix corresponding to the field to be extracted. The entity boundary of the field to be extracted in the input sequence is determined based on the scoring matrix. The entity extraction result is verified based on the entity boundary to determine whether the entity extracted by the model is correct. If it is incorrect, the entity generation model is used to continue generating until the generated entity extraction result passes the verification.

[0125] In this embodiment, by generating entity extraction results based on the input sequence using an entity generation model corresponding to any subject trained by the above training method, it is possible to ensure that the entity extraction results are aligned with the original text, thereby improving the accuracy of the entity extraction results.

[0126] To implement the above embodiments, this application also proposes a training device for an entity generation model. Figure 6 This is a schematic diagram of the structure of a training device for an entity generation model provided in an embodiment of this application.

[0127] like Figure 6 As shown, the training device 600 for the entity generation model includes: The acquisition module 610 is used to acquire an input sequence corresponding to any subject in the target domain; wherein, the input sequence includes sample text and fields to be extracted; The encoding module 620 is used to encode the input sequence using the base model of the target domain to obtain an encoding vector; Transformation module 630 is used to perform a linear transformation on the encoding vector to obtain a scoring matrix corresponding to the field to be extracted; wherein, each element in the scoring matrix represents the score of any two characters in the input sequence as the entity boundary of the field to be extracted, and the entity boundary refers to the start and end positions of the string associated with the field to be extracted in the input sequence. The decoding module 640 is used to decode the encoded vector using the base model to obtain entity extraction results; The training module 650 is used to train the base model based on the scoring matrix and the entity extraction results to obtain the trained entity generation model corresponding to any subject.

[0128] Optionally, the transformation module 630 is used for: The encoded vector is linearly transformed to obtain a first sequence vector and a second sequence vector corresponding to the field to be extracted; wherein the first sequence vector and the second sequence vector are used to identify the entity corresponding to the field to be extracted in the input sequence; The scoring matrix is ​​obtained by multiplying any element in the first sequence vector and any element in the second sequence vector.

[0129] Optionally, the transformation module 630 is used for: The first sequence vector is obtained by multiplying each element in the encoded vector with the first weight parameter corresponding to the field to be extracted. The second sequence vector is obtained by multiplying each element in the encoded vector with the second weight parameter corresponding to the field to be extracted.

[0130] Optionally, training module 650 is used for: Based on the scoring matrix and the entity extraction results, the parameters of the initial parsing head in the base model are adjusted to obtain the target parsing head corresponding to any entity; wherein, the parameters of the initial parsing head include the query matrix and value matrix of the linear layer in the base model; The entity generation model is obtained based on the target parsing header.

[0131] Optionally, training module 650 is used for: Based on the entity extraction results and the entity labels corresponding to the fields to be extracted, the generation loss is determined; The boundary loss is determined based on the scoring matrix and the position information of the entity labels in the input sequence; Based on the generation loss and the boundary loss, the parameters of the initial parsing head are adjusted to obtain the target parsing head.

[0132] Optionally, training module 650 is used for: Based on the entity label corresponding to the field to be extracted and the generated entity corresponding to the field to be extracted in the entity extraction result, determine the alignment loss; Based on the generation loss, the boundary loss, and the alignment loss, the parameters of the initial parsing header are adjusted to obtain the target parsing header. Optionally, training module 650 is used for: Determine the cost of the editing operation for converting the generated entity into the entity label; The cost of the editing operation is smoothed and fused to obtain the processed cost; Based on the processed operation cost, determine the target edit distance between the generated entity and the entity label; The alignment loss is determined based on the target edit distance.

[0133] It should be noted that the explanation of the aforementioned training method embodiment for the entity generation model also applies to the training device of the entity generation model in this embodiment, and therefore will not be repeated here.

[0134] In this embodiment, a base model for the target domain is used to encode the input sequence corresponding to any subject in the target domain, obtaining an encoding vector. The encoding vector is then linearly transformed to obtain a scoring matrix for the field to be extracted. The base model is then used to decode the encoding vector to obtain the entity extraction result. Based on the scoring matrix and the entity extraction result, the base model is trained to obtain the entity generation model corresponding to that subject. Thus, by introducing a scoring matrix to predict the entity boundaries of the field to be extracted, entity boundary awareness is introduced into generative information extraction. In addition to character-level generation, the position of entity boundaries is further constrained, enhancing the accuracy of entity boundary discrimination, reducing model drift at entity boundaries, keeping the entity extraction result aligned with the original text, and improving the accuracy of entity extraction.

[0135] To implement the above embodiments, this application also proposes an entity generation apparatus. Figure 7 This is a schematic diagram of the structure of an entity generation device provided in an embodiment of this application.

[0136] like Figure 7 As shown, the entity generation apparatus 700 includes: The acquisition module 710 is used to acquire an input sequence corresponding to any subject in the target domain; wherein, the input sequence includes the text to be parsed and the fields to be extracted; The generation module 720 is used to generate entity extraction results based on the input sequence and using the entity generation model corresponding to any subject; wherein the entity generation model is trained using the method described in any of the above embodiments.

[0137] It should be noted that the explanation of the aforementioned training method embodiment for the entity generation model also applies to the training device of the entity generation model in this embodiment, and therefore will not be repeated here.

[0138] In this embodiment, by generating entity extraction results based on the input sequence using an entity generation model corresponding to any subject trained by the above training method, it is possible to ensure that the entity extraction results are aligned with the original text, thereby improving the accuracy of the entity extraction results.

[0139] According to embodiments of this application, this application also provides an electronic device, a readable storage medium, and a computer program product.

[0140] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of this application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the application described and / or claimed herein.

[0141] like Figure 8 As shown, device 800 includes a computing unit 801, which can perform various appropriate actions and processes based on a computer program stored in ROM (Read-Only Memory) 802 or a computer program loaded from storage unit 808 into RAM (Random Access Memory) 803. RAM 803 can also store various programs and data required for the operation of device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via bus 804. I / O (Input / Output) interface 805 is also connected to bus 804.

[0142] Multiple components in device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of monitors, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0143] The computing unit 801 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, CPUs (Central Processing Units), GPUs (Graphics Processing Units), various special-purpose AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, DSPs (Digital Signal Processors), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as the training method for an entity generation model. For example, in some embodiments, the training method for an entity generation model can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the training method for an entity generation model described above can be performed. Alternatively, in other embodiments, computing unit 801 may be configured to perform a training method for an entity generation model by any other suitable means (e.g., by means of firmware).

[0144] It should be noted that the electronic device used to implement the entity generation method of this application embodiment has a similar structure to the electronic device in the figure, so it will not be described again here.

[0145] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, FPGAs (Field Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), ASSPs (Application-Specific Standard Products), SOCs (System-on-Chips), CPLDs (Complex Programmable Logic Devices), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0146] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0147] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, RAM, ROM, EPROM (Electrically Programmable Read-Only Memory) or flash memory, optical fiber, CD-ROM (Compact Disc Read-Only Memory), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0148] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0149] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include LANs (Local Area Networks), WANs (Wide Area Networks), the Internet, and blockchain networks.

[0150] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is established by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service ecosystem, addressing the shortcomings of traditional physical hosts and VPS (Virtual Private Server) services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.

[0151] According to embodiments of this application, this application also provides a computer program product, which, when an instruction processor in the computer program product is executed, performs the training method for the entity generation model or the entity generation method proposed in the above embodiments of this application.

[0152] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this application can be achieved, and this is not limited herein.

[0153] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A training method for an entity generation model, comprising: Obtain the input sequence corresponding to any subject in the target domain; wherein, the input sequence includes sample text and fields to be extracted; Using the base model of the target domain, the input sequence is encoded to obtain an encoded vector; The encoding vector is linearly transformed to obtain the scoring matrix corresponding to the field to be extracted; wherein each element in the scoring matrix represents the score of any two characters in the input sequence as the entity boundary of the field to be extracted, and the entity boundary refers to the start and end positions of the string associated with the field to be extracted in the input sequence. Using the aforementioned base model, the encoded vector is decoded to obtain entity extraction results; Based on the scoring matrix and the entity extraction results, the base model is trained to obtain the trained entity generation model corresponding to any subject.

2. The method as described in claim 1, wherein, The step of performing a linear transformation on the encoded vector to obtain the scoring matrix corresponding to the field to be extracted includes: The encoded vector is linearly transformed to obtain a first sequence vector and a second sequence vector corresponding to the field to be extracted; wherein the first sequence vector and the second sequence vector are used to identify the entity corresponding to the field to be extracted in the input sequence; The scoring matrix is ​​obtained by multiplying any element in the first sequence vector and any element in the second sequence vector.

3. The method as described in claim 2, wherein, The linear transformation of the encoded vector to obtain the first sequence vector and the second sequence vector corresponding to the field to be extracted includes: The first sequence vector is obtained by multiplying each element in the encoded vector with the first weight parameter corresponding to the field to be extracted. The second sequence vector is obtained by multiplying each element in the encoded vector with the second weight parameter corresponding to the field to be extracted.

4. The method of claim 1, wherein, The step of training the base model based on the scoring matrix and the entity extraction results to obtain the trained entity generation model corresponding to any subject includes: Based on the scoring matrix and the entity extraction results, the parameters of the initial parsing head in the base model are adjusted to obtain the target parsing head corresponding to any entity; wherein, the parameters of the initial parsing head include the query matrix and value matrix of the linear layer in the base model; The entity generation model is obtained based on the target parsing header.

5. The method of claim 4, wherein, The step of adjusting the parameters of the initial parsing head in the base model based on the scoring matrix and the entity extraction result to obtain the target parsing head corresponding to any entity includes: Based on the entity extraction results and the entity labels corresponding to the fields to be extracted, the generation loss is determined; The boundary loss is determined based on the scoring matrix and the position information of the entity labels in the input sequence; Based on the generation loss and the boundary loss, the parameters of the initial parsing head are adjusted to obtain the target parsing head.

6. The method of claim 5, wherein, The step of adjusting the parameters of the initial parsing head based on the generation loss and the boundary loss to obtain the target parsing head includes: Based on the entity label corresponding to the field to be extracted and the generated entity corresponding to the field to be extracted in the entity extraction result, determine the alignment loss; The parameters of the initial parsing header are adjusted based on the generation loss, the boundary loss, and the alignment loss to obtain the target parsing header.

7. The method of claim 6, wherein, The step of determining the alignment loss based on the entity label corresponding to the field to be extracted and the generated entity corresponding to the field to be extracted in the entity extraction result includes: Determine the cost of the editing operation for converting the generated entity into the entity label; The cost of the editing operation is smoothed and fused to obtain the processed cost; Based on the processed operation cost, determine the target edit distance between the generated entity and the entity label; The alignment loss is determined based on the target edit distance.

8. An entity generation method, comprising: Obtain the input sequence corresponding to any subject in the target domain; wherein, the input sequence includes the text to be parsed and the fields to be extracted; Based on the input sequence, an entity extraction result is generated using the entity generation model corresponding to any of the subjects; wherein the entity generation model is trained using the method described in any one of claims 1-7.

9. A training device for an entity generation model, comprising: An acquisition module is used to acquire an input sequence corresponding to any subject in the target domain; wherein, the input sequence includes sample text and fields to be extracted; The encoding module is used to encode the input sequence using the base model of the target domain to obtain an encoding vector; The transformation module is used to perform a linear transformation on the encoding vector to obtain the scoring matrix corresponding to the field to be extracted; wherein, each element in the scoring matrix represents the score of any two characters in the input sequence as the entity boundary of the field to be extracted, and the entity boundary refers to the start and end positions of the string associated with the field to be extracted in the input sequence. The decoding module is used to decode the encoded vector using the base model to obtain the entity extraction result; The training module is used to train the base model based on the scoring matrix and the entity extraction results to obtain the trained entity generation model corresponding to any subject.

10. An entity generation apparatus, comprising: The acquisition module is used to acquire the input sequence corresponding to any subject in the target domain; wherein, the input sequence includes the text to be parsed and the fields to be extracted; The generation module is used to generate entity extraction results based on the input sequence and using the entity generation model corresponding to any one of the subjects; wherein the entity generation model is trained using the method described in any one of claims 1-7.

11. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-8.

12. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-8.

13. A computer program product comprising a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1-8.

Images