Method, device, electronic device and storage medium for extracting entity content

Abstract

The present application relates to the field of artificial intelligence technology, and discloses a method, device, electronic device and storage medium for extracting entity content, the method comprising: obtaining grouping information of N entities; determining an extraction order corresponding to each group based on the grouping information; concatenating a target text with an identification text corresponding to each entity in the N entities to obtain an input text; encoding the input text to obtain an encoded latent state sequence corresponding to the input text; performing M-way parallel decoding on the encoded latent state sequence corresponding to the input text according to the extraction order based on a masked attention mechanism to obtain M-way output text; the masked attention mechanism is used to mask the attention score of non-related text corresponding to a to-be-decoded word relative to the to-be-decoded word; the present scheme extracts entity content of multiple entities in a serial-parallel manner, thereby improving the efficiency of entity content extraction.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and more specifically, to a method, device, electronic device, and storage medium for extracting entity content. Background Art

[0002] The OCR text obtained by performing OCR (Optical Character Recognition) on the text in the image may contain errors or disordered information. Therefore, in order to facilitate subsequent processing, the OCR text needs to be structured, that is, the OCR text is converted into text with structural information, such as into the form of Key-value pairs. The process of structuring the OCR text is the process of extracting entity content from each entity, where the key can be regarded as the entity and the value is the entity content corresponding to the entity. Figure 1 The upper side shows a schematic diagram of the OCR text. In order to convert the OCR text into the form of key-value pairs, the related art is a single word decoding method, that is, one time step decoding obtains one word, Figure 1 In the lower part of , the part before “:” represents the Key, such as “name”, etc.; the part after “:” represents the Value.

[0003] Since the related art decodes one word per time step during entity content extraction, which is equivalent to serial decoding, when the entity content of multiple entities needs to be extracted, this method takes a long time to extract the entity content, and the efficiency of entity content extraction is low. Summary of the Invention

[0004] In view of the above problems, the embodiments of the present application propose a method, device, electronic device and storage medium for extracting entity content to improve the above problems.

[0005] According to one aspect of an embodiment of the present application, a method for entity extraction is provided, the method comprising: obtaining grouping information of N entities, the grouping information indicating that the N entities are divided into M groups and entities included in each group, wherein M and N are positive integers, 2≤M<N; the sum of text length thresholds corresponding to entities belonging to the same group is not greater than a target length threshold; determining an extraction order corresponding to each decoding path in M-path decoding according to the grouping information, wherein one decoding path is used to extract entity content corresponding to the entity included in a group; concatenating a target text with an identification text corresponding to each entity in the N entities to obtain an input text; encoding the input text to obtain an encoded hidden state sequence corresponding to the input text; based on a masked attention mechanism, decoding the encoded hidden state sequence corresponding to the input text in parallel according to the extraction order to obtain M-path output texts, wherein one output text path includes entity content corresponding to all entities belonging to a group in the target text, and the text length of the entity content corresponding to each entity does not exceed the text length threshold corresponding to the entity.

[0006] According to one aspect of an embodiment of the present application, a device for extracting entity content is provided, comprising: an acquisition module for acquiring grouping information of N entities, the grouping information indicating the division of the N entities into M groups and the entities included in each group, wherein M and N are positive integers, 2≤M<N; the sum of text length thresholds corresponding to entities belonging to the same group is not greater than a target length threshold; a sequence determination module for determining, based on the grouping information, an extraction order corresponding to each decoding path in M ​​decoding paths, wherein one decoding path is used to extract entity content corresponding to the entity included in a group; a splicing module for splicing a target text with an identification text corresponding to each entity in the N entities to obtain an input text; an encoding module for encoding the input text to obtain an encoded hidden state sequence corresponding to the input text; and a parallel decoding module for performing parallel decoding on the encoded hidden state sequence corresponding to the input text according to the extraction order based on a masked attention mechanism to obtain M output texts, wherein one output text path includes entity content corresponding to all entities belonging to a group in the target text, and the text length of the entity content corresponding to each entity does not exceed the text length threshold corresponding to the entity.

[0007] According to one aspect of an embodiment of the present application, an electronic device is provided, including: a processor; a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the method for extracting entity content as described above is implemented.

[0008] According to one aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor, the method for extracting entity content as described above is implemented.

[0009] According to one aspect of an embodiment of the present application, a computer program product is provided, including computer instructions, which, when executed by a processor, implement the method for extracting entity content as described above.

[0010] In this application, N entities are divided into M groups according to a text length threshold, where N>M. This indicates that there is at least one group containing two entities. Thus, during entity content extraction, entities in different groups are decoded in parallel, while entities in the same group are decoded serially, thereby achieving serial-parallel entity content extraction. Compared to the serial entity content extraction method in the related art, where one word is decoded per time step, the method of this application can decode M words per time step, significantly shortening the time spent extracting entity content from multiple entities and thereby improving the efficiency of entity content extraction.

[0011] In addition, since the masked attention mechanism is used to mask the attention score of a word to be decoded with the non-related text corresponding to the word to be decoded during the decoding process, the feature information of the non-related text corresponding to the word to be decoded is not paid attention to in the process of decoding to determine the word to be decoded, but only the feature information of the relevant text corresponding to the word to be decoded is paid attention to. Therefore, the mutual dependence between words can be avoided in the serial and parallel decoding process, thereby ensuring the accuracy of the entity content extracted for multiple entities. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0013] Figure 1 It is a schematic diagram of structuring OCR text in related technologies.

[0014] Figure 2 A schematic diagram shows an application scenario in which the technical solution of the embodiments of the present application can be applied.

[0015] Figure 3 The figure is a flowchart of a method for extracting entity content according to an embodiment of the present application.

[0016] Figure 4 A schematic diagram of a business license is shown as an example.

[0017] Figure 5 A- Figure 5 C is a schematic diagram illustrating the effect of attention according to an embodiment of the present application.

[0018] Figure 6 FIG. 2 is a flowchart of step 240 according to an embodiment of the present application.

[0019] Figure 7 FIG. 6 is a flowchart of step 630 according to an embodiment of the present application.

[0020] Figure 8 FIG. 7 is a flowchart of step 720 according to an embodiment of the present application.

[0021] Figure 9 2 is a schematic diagram of the structure of a decoder sub-network according to an embodiment of the present application.

[0022] Figure 10 Schematic diagram of the structure of an encoder sub-network according to an embodiment of the present application.

[0023] Figure 11 This is a flowchart of extracting entity content according to an embodiment of the present application.

[0024] Figure 12 FIG. 6 is a flowchart of step 630 according to another embodiment of the present application.

[0025] Figure 13 This is a flowchart of extracting entity content according to another embodiment of the present application.

[0026] Figure 14 This is a flowchart of extracting entity content according to another embodiment of the present application.

[0027] Figure 15 It is a block diagram of an apparatus for extracting entity content according to an embodiment of the present application.

[0028] Figure 16 A schematic diagram of the structure of a computer system suitable for implementing an electronic device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0029] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this application will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art.

[0030] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner.In the following description, many specific details are provided so as to provide a full understanding of the embodiments of the present application. However, it will be appreciated by those skilled in the art that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps etc. can be adopted. In other cases, known methods, devices, implementations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0031] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically separate entities. That is, these functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0032] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, while others may be combined or partially combined. Therefore, the actual execution order may vary depending on the actual situation.

[0033] It should be noted that the term "plurality" used in this document refers to two or more. "And / or" describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A alone, A and B together, or B alone. The character " / " generally indicates an "or" relationship between the associated objects.

[0034] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0035] Natural language processing (NLP) is a key area of ​​research in computer science and artificial intelligence. It studies the theories and methods that enable effective communication between humans and computers using natural language. Natural language processing (NLP) integrates linguistics, computer science, and mathematics. Therefore, research in this field involves natural language—the language we use in everyday life—and is closely linked to the study of linguistics. Natural language processing technologies typically include text processing, semantic understanding, machine translation, robotic question answering, and knowledge graphs.

[0036] Figure 2 The following is a schematic diagram showing an application scenario in which the technical solution of the embodiment of the present application can be applied. Figure 2 As shown, the application scenario includes a terminal 210 and a server 220. The terminal 210 establishes a communication connection with the server 220 via a wired network or a wireless network. The terminal 210 can be a smartphone, a tablet computer, a laptop computer, a desktop computer, a self-service terminal, an in-vehicle terminal, or other electronic device that can interact with a user, and is not specifically limited here.

[0037] The server 220 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0038] The terminal 210 may display an interactive interface, based on which the user may trigger the selection of a target text, and the terminal 210 may extract entity content corresponding to each of the N entities from the target text.

[0039] Furthermore, based on the interactive interface in the terminal 210, the user can also select entities that need to be extracted from the target text, and further set the text length threshold corresponding to each selected entity, and send content extraction configuration information to the server 220. The content extraction configuration information is used to indicate the entity identifier of the selected entity that needs to be extracted (for example, the entity name of the entity) and the text length threshold set for the selected entity. On this basis, the server combines the content extraction configuration information to extract the entity content corresponding to each entity from the target text according to the method of the present application.

[0040] Of course, in other embodiments, the content extraction configuration information may also be pre-stored in the server 220 , so that after receiving the target text, the server 220 extracts the entity content corresponding to each entity from the target text according to the content extraction configuration information.

[0041] In other embodiments, the target text may also be pre-stored in the server 210, and a text length threshold corresponding to the entity to be subjected to entity content extraction may be pre-stored. The terminal 210 may send entity extraction indication information to the server 210, and the entity extraction indication information may include a text identifier corresponding to the target text and an entity identifier corresponding to the entity to be subjected to entity content extraction. Thus, the server 210 determines the text length threshold corresponding to the entity indicated by the entity extraction indication information according to the entity extraction prompt information, and determines the target text according to the text identifier indicated by the entity extraction indication information. Thereafter, based on the determined text length threshold corresponding to each entity and the target text, the entity content corresponding to each entity is extracted from the target text according to the method of the present application.

[0042] In other embodiments, the method of the present application can be applied to structured processing of OCR text. In this application scenario, the target text can be the OCR text obtained by performing OCR (Optical Character Recognition) on the text in the image. Since the obtained OCR text is simply lines of text, it is not possible to quickly determine which text corresponds to which entity. Therefore, according to the method of the present application, the entity that needs to be extracted from the entity content can be specified, and the text length threshold corresponding to the entity can be set. Then, according to the specified multiple entities and the corresponding text length threshold, the OCR text is processed according to the method of the present application to obtain the entity content corresponding to each entity. On this basis, the entity and the entity content corresponding to the entity are converted into the form of a Key-Value pair, where the Key represents the entity and the Value represents the entity content (which can also be understood as an attribute value). In other words, through the method of the present application, it is possible to achieve structured processing of the OCR text and convert the OCR text into the form of a Key-Value pair, thereby facilitating subsequent data processing.

[0043] It is worth mentioning that the method of the present application is not limited to being executed by the server 220. The method of the present application can also be executed by the terminal 210 with sufficient processing capabilities, and can also be executed interactively by the system composed of the server 220 and the terminal 210.

[0044] The following is a detailed description of the implementation details of the technical solution of the embodiment of the present application:

[0045] Figure 3This is a flow chart of a method for extracting entity content according to an embodiment of the present application. The method can be executed by an electronic device with processing capabilities, such as a server, a terminal, or a system composed of a server and a terminal, etc., which is not specifically limited here. Figure 3 As shown, the method includes at least steps 310 to 350, which are described in detail as follows:

[0046] Step 310: Obtain grouping information of N entities, where the grouping information indicates that the N entities are divided into M groups and the entities included in each group, where M and N are positive integers, 2≤M<N; and the sum of text length thresholds corresponding to entities belonging to the same group is not greater than the target length threshold.

[0047] N entities are entities whose entity content is to be extracted from the target text, where N≥3. The target text is the text from which the entity content to be extracted comes. In other words, the entity content extracted in this disclosure is extracted from the target text. Therefore, any text that needs to extract entity content can be regarded as the target text in this application. Among them, extracting the entity content corresponding to the entity from the target text in this application can also be understood as extracting the attribute value corresponding to the entity from the target text.

[0048] In some application scenarios, the target text may be text obtained by performing OCR on an image, wherein the image may be an image of a business license, an image of an invoice, an image of a train ticket, an image of a document (such as an ID card or a bank card), etc. Figure 4 A schematic diagram of a business license is shown as an example. Figure 4 As shown, the unified social credit code, license number, name, type, legal representative, business scope, registered capital, establishment date, business term, domicile, registration authority, etc. shown in the business license can be regarded as an entity, and the content shown after the entity is the entity content corresponding to the entity.

[0049] In other embodiments, the target text may also be any given piece of text. For example, the target text may be an introduction text. When a question text is received, an identification text representing the entity being asked is extracted from the question text. Then, the entity content of the entity indicated by the question text may be extracted from the introduction text according to the method of the present application. For example, if the target text is an introduction text of a scenic spot, and if three question texts are received, where the entity indicated by one question text is the closing time of the scenic spot, the entity indicated by another question text is the opening time of the scenic spot, and the entity indicated by another question text is the construction time of the scenic spot, then for these three entities, the entity content corresponding to the three entities may be extracted from the introduction text of the scenic spot according to the method of the present application.

[0050] Therefore, the method of the present application can be used to provide automatic question-answering services for multiple users. In this way, it is possible to provide answers to multiple questions at once, thereby improving the efficiency of the automatic text question-answering service. In this case, the N entities to be extracted can be extracted by keywords from multiple question texts, or they can be determined by user triggering selection.

[0051] In different application scenarios, the entities that need to be extracted from the target text may be different. Therefore, multiple entities that need to be extracted can be set in advance, or multiple entities that need to be extracted can be determined in real time based on received questions.

[0052] The text length threshold corresponding to the entity is used to limit the maximum length of the corresponding entity content, that is, the length of the entity content extracted from the target text does not exceed the corresponding text length threshold.

[0053] It is understandable that the text length thresholds corresponding to different entities can be the same or different, and can be set according to actual needs. For example, if entity extraction is performed from the OCR recognition text corresponding to the business license, the text length thresholds set for the legal representative and the unified social credit code can be different.

[0054] In some embodiments, before step 310, the method further includes: obtaining text length thresholds set for N entities respectively; taking the maximum value of the text length thresholds corresponding to the N entities as the target length threshold; dividing the N entities into M groups according to the text length threshold and the target length threshold corresponding to each entity to obtain grouping information.

[0055] The entity to be extracted from the target text may be selected by the user, and similarly, the text length threshold corresponding to the entity may also be customized by the user. In other embodiments, the entity to be extracted may also be pre-set and stored, and the text length threshold corresponding to the entity may also be pre-set and stored.

[0056] The text length threshold corresponding to the entity can be set according to actual needs. After obtaining the text length threshold corresponding to each entity in the N entities, the N entities are grouped based on the text length threshold to obtain grouping information.

[0057] The following describes the entity grouping process in conjunction with a specific embodiment. Table 1 shows the text length thresholds set for entities in the OCR recognition results of the business license image. Based on the text length thresholds corresponding to each entity in Table 1, it can be seen that the maximum length threshold is 100, and the target length threshold can be determined to be 100. On this basis, based on the target text length threshold and the text length thresholds corresponding to each entity shown in Table 1, the 10 entities in Table 1 can be divided into 3 groups.

[0058] Table 1

[0059]

[0060]

[0061] Table 2 shows the grouping results of the 10 entities in Table 1 according to the target length threshold of 100. As shown in Table 2, the 10 entities in Table 1 are divided into 3 groups, where the entities in group 1 include: unified social credit code, certificate number, name, type, legal representative, and registered capital; the entities in group 2 include business scope; and the entities in group 3 include establishment date, business term, and residence.

[0062] Table 2

[0063]

[0064] In some embodiments, during the grouping process, grouping can be performed based on a target length threshold, that is, the sum of the text length thresholds corresponding to entities in the same group is made as close to the target length threshold as possible, so that the number of groups can be reduced.

[0065] In other embodiments, grouping can be performed based on a given target length threshold and the total number of groups M, with the goal of ensuring that the total number of groups is M and the sum of the text length thresholds corresponding to entities in the same group does not exceed the target length threshold.

[0066] It is understandable that grouping N entities based on the text length threshold corresponding to each entity in the N entities so that the sum of the text length thresholds corresponding to entities in the same group is no greater than the target length threshold may not result in a unique grouping result.

[0067] In other embodiments, based on the maximum value of the length thresholds corresponding to the N entities, the result of adding the maximum text length threshold and the set length value can be used as the target length threshold. By adding the maximum text length threshold and the specified length value, a certain length space is left for entity content extraction.

[0068] In some embodiments, a mapping relationship between text types and content extraction indication information can be pre-set, wherein the content extraction indication information corresponding to a text type is used to indicate the entity identifier of the entity that needs to perform entity content extraction in the target text corresponding to the text type and the text length threshold corresponding to each entity, wherein the text type can be set according to actual needs, for example, the text type can include business license OCR text, invoice OCR text, train ticket OCR text, identity card OCR text, scenic spot introduction text, business introduction text, etc., which are not specifically limited here. On this basis, the user can select the text type to which the target text belongs, thereby, according to the text type selected by the user and the mapping relationship between the text type and the content extraction indication information, the content extraction indication information corresponding to the text type selected by the user can be determined, and the entities can be grouped according to the determined content extraction indication information.

[0069] In other embodiments, a mapping relationship between text types and grouping information may be pre-set, wherein the grouping information corresponding to a text type indicates the grouping results of entities for which entity content extraction is required for a target text of that text type. Based on this, after determining the text type to which the target text belongs, the grouping information corresponding to the text type to which the target text belongs may be obtained.

[0070] In this application, since N entities are divided into M groups, M < N, it means that there is at least one group, and the group includes two entities. In this application, an entity belongs to only one group.

[0071] Step 320: Determine the extraction order corresponding to each group according to the group information.

[0072] The extraction order for a decoding path indicates the order in which all entities in the corresponding group are extracted. For example, if a group contains entities such as establishment date, business term, and address, the extraction order for that group can be set to: establishment date → business term → address.

[0073] When a group includes multiple entities, the extraction order corresponding to the group can be specified according to actual needs and is not specifically limited here. Of course, it can also be determined randomly and is not specifically limited here.

[0074] Step 330 : Concatenate the target text with the identification text corresponding to each of the N entities to obtain the input text.

[0075] The identification text corresponding to the entity can be the entity name of the entity. In this application, the target text is spliced ​​with the recognition text corresponding to N entities, and the identification text corresponding to the entity is used to indicate the entity that needs to be extracted, so as to facilitate the targeted extraction of the entity content corresponding to the entity in the subsequent decoding process.

[0076] In some embodiments, the target text and the identification text corresponding to the entity can be spliced ​​according to the grouping results of N entities. Specifically, in the process of splicing the target text and the identification text corresponding to the entity, the identification texts corresponding to the entities belonging to the same group can be spliced ​​adjacently, that is, the identification texts corresponding to the entities in the same group are adjacent in position in the input text. For example, if the target text is XXXX, the entities in one group include establishment date, business term, and residence, and the other group includes the entity of business scope, then the input text can be: XXXX establishment date business term residence business scope. It can be seen that in the input text, the entity names of the three entities "establishment date, business term, residence" are adjacent in position.

[0077] In other embodiments, the identification texts corresponding to entities in the same group in the input text may not be located adjacent to each other, and the specific splicing position may be set according to actual needs.

[0078] Step 340: Encode the input text to obtain an encoded hidden state sequence corresponding to the input text.

[0079] In some embodiments, the input text may be encoded using an encoder network to obtain an encoded hidden state sequence corresponding to the input text, wherein the encoder network may be constructed using at least one of a convolutional neural network, a recurrent neural network, a fully connected neural network, a feedforward neural network, etc. In other embodiments, the encoder network may be an encoder in a Transformer model (converter network model).

[0080] The encoded latent state sequence is a vectorized description of the text features of the input text, that is, the encoded latent state sequence indicates the text features of the input text, and the text features of the input text at least include the semantic features of the input text.

[0081] Step 350: Based on the masked attention mechanism, the encoded latent state sequence corresponding to the input text is decoded in parallel in M ​​ways in the extraction order to obtain M ways of output text, where one way of output text includes the entity content corresponding to all entities belonging to a group in the target text, and the text length of the entity content corresponding to each entity does not exceed the text length threshold corresponding to the entity.

[0082] Among them, the masked attention mechanism is used to mask the attention score of the non-relevant text corresponding to the word to be decoded relative to the word to be decoded. The non-relevant text corresponding to the word to be decoded includes the identification text corresponding to other entities except the target entity and the decoded words corresponding to other entities except the target entity. The target entity refers to the entity corresponding to the word to be decoded.

[0083] Specifically, a decoder network can be used to decode the encoded latent state sequence corresponding to the input text in parallel based on a masked attention mechanism. The decoder network can be constructed using one or more neural networks, such as convolutional neural networks, recurrent neural networks, fully connected neural networks, (unidirectional or bidirectional) long short-term memory networks, feedforward neural networks, pooling neural networks, etc., without specific limitation herein. In one specific embodiment, the decoder network can be the decoder in a Transformer model (converter network model).

[0084] During the decoding process, decoding is performed time-step by time-step, and M-way decoding is performed in parallel, that is, at each time-step, one word in each of the M entities (i.e., the target entity hereinafter) is decoded in parallel. In this application, the current word to be decoded is referred to as the word to be decoded. Therefore, at each time-step, there are M words to be decoded, and different words to be decoded corresponding to the same time-step correspond to different entities. Each time-step can be decoded to obtain the feature information corresponding to the M words to be decoded (i.e., the decoding latent state vector hereinafter). Since the extraction order of the entity content corresponding to the entities in the same group is predetermined, during the decoding process, the decoding output is performed according to the extraction order of the corresponding group, and the output text of the decoding corresponding to each group is obtained. Afterwards, according to the determined extraction order and the text length threshold corresponding to each entity, the entity content corresponding to each entity in the group corresponding to the output text is obtained from the output text.

[0085] Masking refers to masking certain values ​​so that they have no effect when the parameters are updated, which is equivalent to ignoring the existence of the masked positions. In this application, at each time step, the attention score of the non-related text corresponding to each word to be decoded is masked to the word to be decoded. Therefore, the word to be decoded is determined not by relying on the feature information of the non-related text corresponding to the word to be decoded, but by relying on the feature information of the related text corresponding to the word to be decoded. Therefore, the dependency between words during the parallel decoding process can be avoided and the accuracy of parallel decoding can be prevented.

[0086] For example, if a group includes entity 1, entity 2 and entity 3, the extraction order corresponding to the group is entity 1 → entity 2 → entity 3, where the text length threshold corresponding to entity 1 is 2, the text length threshold corresponding to entity 2 is 3, and the text length threshold corresponding to entity 3 is 2. The target length threshold is 7, and the output text corresponding to the group includes 10 words. Assuming that the output text is: Q1Q2Q3Q4Q5Q6Q7 (where Qi represents the i-th word in the corresponding output text), then press According to the extraction order corresponding to the group and the text length threshold corresponding to each entity, first extract the text length threshold (i.e. 2) words corresponding to entity 1 from the output text of this path, and obtain Q1Q2, then Q1Q2 is the entity content corresponding to entity 1; similarly, then extract the text length threshold (i.e. 3) words corresponding to entity 2 from the remaining text in the output text of this path, and obtain Q3Q4Q5, then Q3Q4Q5 is the entity content corresponding to entity 2; finally, use the remaining text in the output text of this path as the entity content corresponding to entity 3.

[0087] In this application, a masked attention mechanism is used to mask the non-relevant text corresponding to a word to be decoded relative to the attention score of the word to be decoded. Thus, in the process of decoding the word to be decoded, the information of the words in the non-relevant text corresponding to the word to be decoded is not paid attention to, but only the relevant text corresponding to the word to be decoded is paid attention to. Thus, the non-relevant text corresponding to the word to be decoded is prevented from affecting the decoding accuracy of the word to be decoded, thereby ensuring the accuracy of the extracted entity content. It is understandable that since different words to be decoded belong to different entities at each time step, the non-relevant text corresponding to different words to be decoded is also different at each time step.

[0088] For a word to be decoded, the relevant text corresponding to the word to be decoded includes a target text, an identification text of an entity corresponding to the word to be decoded, and a decoded word for the entity corresponding to the word to be decoded.

[0089] In a specific embodiment, the non-relevant text corresponding to each to-be-decoded word can be determined based on the to-be-decoded word at each time step. The attention score of each to-be-decoded word for the input information at each time step is first calculated using an attention mechanism. Then, based on the non-relevant text corresponding to the to-be-decoded word at the current time step, the attention score of the non-relevant text corresponding to the to-be-decoded word for the to-be-decoded word is masked out, while the attention score of the relevant text corresponding to the to-be-decoded word for the to-be-decoded word is retained. Specifically, the attention score of each word in the input information at each time step for each to-be-decoded word can be determined using a single-head attention mechanism or a multi-head attention mechanism.

[0090] The input information at each time step includes the encoded latent state sequence corresponding to the input text (the encoded latent state sequence reflects the semantic information of the target text and the semantic information of the identification text corresponding to each entity), and the decoded input information at the current time step. The decoded input information at the current time step includes at least the decoded latent state vector of the decoded word determined by decoding before the current time step. It is understood that during the decoding process, after determining the decoded latent state vector of a word, the decoded latent state vector can be classified to determine the word corresponding to the decoded latent state vector. Therefore, if a decoded latent state vector is determined, it is equivalent to determining the word represented by the decoded latent state vector.

[0091] In this application, N entities are divided into M groups according to a text length threshold, where N>M. This indicates that there is at least one group containing two entities. Thus, during entity content extraction, entities in different groups are decoded in parallel, while entities in the same group are decoded serially, thereby achieving serial-parallel entity content extraction. Compared to the related art method of performing entity content extraction serially, decoding one word per time step, the method of this application can significantly shorten the time spent extracting entity content for multiple entities, thereby improving the efficiency of entity content extraction.

[0092] In addition, since the masked attention mechanism is used to mask the attention score of a word to be decoded with the non-related text corresponding to the word to be decoded during the decoding process, the feature information of the non-related text corresponding to the word to be decoded is not paid attention to in the process of decoding to determine the word to be decoded, but only the feature information of the relevant text corresponding to the word to be decoded is paid attention to. Therefore, the mutual dependence between words can be avoided in the serial and parallel decoding process, thereby ensuring the accuracy of the entity content extracted for multiple entities.

[0093] Another method for extracting entity content exists in the related art, namely, dividing N entities into N groups, and performing N decoding steps during the decoding process. During the N decoding process, the number of steps in each decoding step is determined by the maximum text length threshold. For the decoding of the path corresponding to the entity whose text length threshold is less than the maximum text length threshold, the number of decoding steps required is also the same as the number of steps in the decoding of the path corresponding to the entity corresponding to the maximum text length threshold. In this process, except for the decoding of the path corresponding to the entity corresponding to the maximum text length threshold, there are invalid decoding processes in the other decoding paths, that is, redundant calculation processes. If there are two entities with a large difference in text length threshold among the N entities, invalid decoding will be performed more frequently, thus wasting computing resources.

[0094] For example, for the 10 entities in Table 1, the maximum text length threshold is 100. If the 10 entities are divided into 10 groups (i.e., one entity is one group) and decoded in parallel, 10 words are decoded in each time step, and 100 time steps are decoded to obtain the entity content corresponding to the 10 entities in Table 1. However, if the method of the present application is adopted, the 10 entities are divided into 3 groups. In this way, 3 words are decoded in each time step, and 100 time steps are decoded. It can be seen that although the total number of time steps required for the decoding process is the same, the number of words decoded in each time step is reduced. In this way, the probability of invalid decoding can be reduced, the amount of redundant calculations can be reduced, and the amount of calculation can be saved.

[0095] The method of the present application is applied in the case where there are entities with large differences in text length thresholds among multiple entities. Based on the text length threshold, N entities are divided into M groups, so that the sum of the text lengths of the entity contents corresponding to all entities in the same group is basically close to the target length threshold. In this way, the computational complexity of entity content extraction can be greatly reduced, and redundant computational complexity can be reduced.

[0096] Figure 5 A- Figure 5 C is a schematic diagram of the attention effect in extracting entity content from the target text according to an embodiment of the present application. In this embodiment, the entities to be subjected to entity content extraction include 4 entities (entities K0, K10, K11, K12). If the text length threshold corresponding to entity K0 is set to 10, the text length threshold corresponding to entity K10 is set to 2, the text length threshold corresponding to entity K11 is set to 4, and the text length threshold corresponding to entity K12 is set to 3; then according to the text length threshold corresponding to each entity, the 4 entities can be divided into two groups, group 1 includes one entity, namely entity K0; group 2 includes two entities, namely entity K10, entity K11 and entity K12. If the extraction order corresponding to group 2 is set from first to last, it is entity K10 → entity K11 → entity K12.

[0097] In this implementation, since the number of groups is 2 (i.e., M = 2), each decoding time step yields two decoded latent state vectors corresponding to the word to be decoded. Therefore, to ensure that two decoded latent state vectors are obtained per decoding time step instead of just one, two placeholders are input to the decoder network at each time step. Assume that t = 1 is the first time step of the decoding process.

[0098] like Figure 5As shown in Figure A, at the first time step (i.e., t=1), two placeholders (i.e., mask01 and mask11) are input to the decoder network. These two placeholders are used to indicate that two words of information (decoded latent state vectors) need to be decoded at this time step, which is equivalent to instructing the decoder network to perform two decodings in parallel. According to the extraction order corresponding to group 1 and the extraction order corresponding to group 2, at the first time step, mask01 corresponds to the first character of the entity content corresponding to entity K0, and mask11 corresponds to the first character of the entity content corresponding to entity K10.

[0099] In this case, since it is the first time step, there are no decoded words for each target entity. Therefore, for entity K0 as the target entity, at the first time step, the non-relevant text corresponding to entity K0 includes the identification text corresponding to entity K10, the identification text corresponding to entity K11, and the identification text corresponding to entity K12. In this case, the attention score of mask01 for the non-relevant text corresponding to entity K0 at the first time step is masked out, and only the attention score of mask01 for the target text and the identification text corresponding to entity K0 is retained.

[0100] Similarly, at the first time step, for entity K10 as the target entity, at the first time step, the non-relevant text corresponding to entity K10 includes the identification text corresponding to entity K0, the identification text corresponding to entity K11, and the identification text corresponding to entity K12. The attention scores of the non-relevant text corresponding to entity K10 on mask11 at the first time step are masked out, and only the attention scores of the target text and the identification text corresponding to entity K10 on mask11 are retained. The schematic diagram of the object that pays attention to the placeholders mask01 and mask11 at the first time step is as follows: Figure 5 As shown by the arrow A, Figure 5 The part without an arrow in A indicates that the attention score of the corresponding text on the placeholder is not paid attention to.

[0101] By decoding at time step t=1, we can obtain the decoding latent state vector O01 of the first character in the entity content corresponding to entity K0 and the decoding latent state vector O11 of the first character in the entity content corresponding to entity K10.

[0102] Then, at time step 2 (i.e., t=2), the two decoded hidden state vectors (O01 and O11) obtained at time step 1 and two placeholders (mask02 and mask12) are input to the decoder network. Similarly, the placeholders mask02 and mask12 indicate that the decoder network needs to perform two parallel decoding passes at this time step and output two decoded hidden state vectors.

[0103] At the second time step, the non-relevant text corresponding to entity K0 includes the identification text corresponding to entity K10, the identification text corresponding to entity K11, the identification text corresponding to entity K12, and the decoding hidden state vector O11 of the first character corresponding to entity K10. Then, the attention scores of the identification text corresponding to entity K10, the identification text corresponding to entity K11, the identification text corresponding to entity K12, and the decoding hidden state vector O11 of the first character corresponding to entity K10 on mask02 are masked and only the attention scores of the target text, the identification text corresponding to entity K0, and the decoding hidden state vector O01 of the decoded word corresponding to entity K0 on mask02 are retained. The schematic diagram of the corresponding attention score effect is as follows Figure 5 As shown in B.

[0104] Similarly, at the second time step, the non-relevant text corresponding to entity K10 includes the identification text corresponding to entity K0, the identification text corresponding to entity K11, the identification text corresponding to entity K12, and the decoding hidden state vector O01 of the first character corresponding to entity K0. The attention scores of the identification text corresponding to entity K0, the identification text corresponding to entity K11, the identification text corresponding to entity K12, and the decoding hidden state vector O01 of the first character corresponding to entity K0 on mask12 are masked and processed, and only the attention scores of the target text, the identification text corresponding to entity K10, and the decoding hidden state vector O11 of the decoded word corresponding to entity K10 on mask12 are retained. The schematic diagram of the corresponding attention score effect is as follows Figure 5 As shown in B.

[0105] Through decoding in the second time step, the decoded hidden state vector O02 of the second word in the entity content corresponding to entity K0 and the decoded hidden state vector O12 of the second word in the entity content corresponding to entity K10 can be obtained.

[0106] Afterwards, at the third time step (i.e., t=3), since the decoded latent state vectors corresponding to the two words in the entity content corresponding to entity K10 were output at the first and second time steps respectively, and the text length threshold corresponding to entity K10 is 2, it is equivalent to that entity K10 has been decoded. According to the extraction order corresponding to group 2, the entity content corresponding to entity K11 is extracted at the third time step. Similarly, since the text length threshold corresponding to entity K0 is 10, at the third time step, the entity content corresponding to entity K0 is continued to be extracted for group 1.

[0107] At the 3rd time step, the four decoded latent vectors (i.e., O01, O11, O02, and O12) obtained at the 1st and 2nd time steps and two placeholders (mask03 and mask13) are input to the decoder network.

[0108] At the third time step, for entity K0, the non-related text corresponding to entity K0 includes the identification text corresponding to entity K10, the identification text corresponding to entity K11, the identification text corresponding to entity K12, and the decoded word corresponding to entity K10 (i.e., the first and second words in the entity content corresponding to entity K10. Of course, in the decoding process, the two words are represented by the two decoding hidden state vectors (O11, O12)). Then the attention score of mask03 of the identification text corresponding to entity K10, the identification text corresponding to entity K11, the identification text corresponding to entity K12, and the decoded word corresponding to entity K10 is masked out, and only the attention score of the target text, the identification text corresponding to entity K0, and the decoded word corresponding to entity K0 (similarly, the decoded word corresponding to entity K0 at the second time step is represented by the decoding hidden state vectors O01 and O01) to mask03 is retained. The schematic diagram of the corresponding attention score effect is shown as follows: Figure 5 As shown in C.

[0109] Similarly, at the third time step, for entity K11, the non-related text corresponding to entity K11 includes the identification text corresponding to entity K0, the identification text corresponding to entity K10, the identification text corresponding to entity K12, the decoded word corresponding to entity K10, and the decoded word corresponding to entity K0. Therefore, the attention score mask of the identification text corresponding to entity K0, the identification text corresponding to entity K10, the identification text corresponding to entity K12, the decoded word corresponding to entity K10, and the decoded word corresponding to entity K0 to the placeholder mask13 is processed, and only the attention score of the target text and the identification text corresponding to entity K11 to the placeholder mask13 is retained. The schematic diagram of the corresponding attention score effect is as follows Figure 5 C. The decoding process in the subsequent time steps is similar to the above process and will not be repeated here.

[0110] Through the above process, it can be seen that at each time step, the decoding latent state vectors of multiple words can be obtained by decoding, and the masked attention mechanism avoids the dependence of the non-correlated words to be decoded on the words to be decoded, thereby ensuring the accuracy and precision of the decoding and the accuracy of the entity content extracted for each entity. Moreover, since parallel decoding is performed at each time step, the decoding latent state vectors of multiple words are obtained at each time step. Compared with decoding one word at each time step, the method of this application greatly improves the decoding speed. In other words, it improves the speed of extracting entity content.

[0111] In some embodiments, as Figure 6 As shown, step 240 includes:

[0112] Step 610 , determining the target entities corresponding to the M to-be-decoded words to be decoded at the t-th time step according to the extraction order, where different to-be-decoded words correspond to different target entities; wherein t is a positive integer.

[0113] As described above, the decoding process is divided into M parallel decoding paths. Therefore, at each time step, one decoding path is used to obtain the decoding of a word in the entity content corresponding to an entity. Therefore, in this scheme, M words to be decoded can be decoded at the tth time step, and the M words to be decoded correspond to different entities.

[0114] The word to be decoded at the tth time step refers to the word corresponding to the latent state vector to be decoded and obtained at the tth time step. For example, if the decoded latent state vector corresponding to the third word in the entity content corresponding to entity 1 needs to be decoded and obtained at the tth time step, then the third word in the entity content corresponding to entity 1 is the word to be decoded to be decoded at the tth time step. In this application, for ease of description, the entity corresponding to the current word to be decoded is referred to as the target entity. It is understandable that at different time steps, the target entity for the same group can be the same or different.

[0115] In some embodiments, before step 610, the method further includes: determining a time step interval based on a target length threshold; the time step interval is 1 to Q, where Q is a target length threshold; dividing the time step interval into a target number of time step sub-intervals according to the extraction order corresponding to the group based on the target number of entities included in a group and the text length threshold corresponding to the entities included in a group, wherein a time step sub-interval corresponds to an entity in the corresponding group, and in a group, except for the last entity in the extraction order, the interval length of the time step sub-interval corresponding to other entities is the corresponding text length threshold; in this embodiment, step 610 includes: for each group in the M groups, in the time step sub-interval corresponding to the group, determining the time step sub-interval where t is located, and taking the entity corresponding to the time step sub-interval where t is located as the target entity of the corresponding group at the tth time step.

[0116] In this embodiment, the time step intervals corresponding to different groups are the same, all ranging from 1 to Q. It is understandable that, because the sum of the text length thresholds corresponding to all entities in a group may be less than the target length threshold, in this case, according to the extraction order corresponding to the group, the interval lengths of the time step subintervals corresponding to the entities except the last entity in the extraction order are equal to the corresponding text length thresholds, and the interval length of the time step subinterval corresponding to the last entity in the extraction order is not less than the text length threshold corresponding to the entity. It is understandable that the time step subintervals obtained by dividing the time step interval corresponding to a group are continuous.

[0117] continue Figure 5 In the example of embodiment A-5C, the target length threshold Q = 10, and the four entities are divided into two groups, where Group 1 includes one entity, namely entity K0 (its corresponding text length threshold is 10); Group 2 includes two entities, namely entity K10 (its corresponding text length threshold is 2), entity K11 (its corresponding text length threshold is 4), and entity K12 (its corresponding text length threshold is 3). The extraction order corresponding to Group 2 is entity K10 → entity K11 → entity K12.

[0118] On this basis, the time step interval corresponding to group 1 is determined to be 1 to 10. Similarly, the time step interval corresponding to group 2 is also 1 to 10. Since group 1 only includes one entity, there is no need to divide the time step interval corresponding to group 1 into time step sub-intervals.

[0119] For group 2, the corresponding extraction order is entity K10 → entity K11 → entity K12. The time step interval 1 to 10 is divided into three time step sub-intervals according to the text length threshold corresponding to entity K10, the text length threshold corresponding to entity K11, and the text length threshold corresponding to entity K12, namely time step sub-intervals 1 to 2 corresponding to entity K10, time step sub-intervals 3 to 6 corresponding to entity K11, and time step sub-intervals 7 to 10 corresponding to entity K12. It can be seen that the last entity in the extraction order corresponding to group 2, namely entity K12, has a corresponding time step sub-interval 7 to 10 with an interval length of 4, which is greater than the text length threshold 3 corresponding to entity K12.

[0120] Based on the above time step sub-interval division, for group 1, the words to be decoded in the 1st to 10th time steps are all words in the entity content corresponding to entity K0; for group 2, the words to be decoded in the 1st time step and the words to be decoded in the 2nd time step are words in the entity content corresponding to entity K10, the words to be decoded in the 3rd, 4th, 5th and 6th time steps are words in the entity content corresponding to entity K11, and the words to be decoded in the 7th, 8th and 9th time steps are words in the entity content corresponding to entity K12.

[0121] In step 620 , the embedding vectors corresponding to the M placeholders are concatenated with the decoded hidden state vector before the t-th time step to obtain the input sequence corresponding to the decoder network at the t-th time step.

[0122] Because the target entities corresponding to the decoded words at different time steps in different groups are different, in this application, the decoded latent state vectors before the tth time step are not limited to the decoded latent state vectors corresponding to the M target entities at the tth time step, but also include the decoded latent state vectors corresponding to other entities other than the M target entities before the tth time step. The embedding vector corresponding to the placeholder is a vectorized representation of the placeholder.

[0123] For example, in Figure 5 In the implementation corresponding to C, the target entities at the third time step include entities K0 and K11, and the decoded latent state vectors before the third time step include the decoded latent state vectors O01 and O02 obtained for entity K0, and the decoded latent state vectors O11 and O12 obtained for entity K10. As above, the M placeholders are used to indicate that M decoded latent state vectors need to be decoded at this time step. In a specific embodiment, the placeholders can be embedded to obtain the encoding vectors corresponding to the placeholders.

[0124] In step 630, the decoder network performs parallel decoding based on the masked attention mechanism using the input sequence corresponding to the t-th time step and the encoded hidden state sequence corresponding to the input text to obtain a decoded hidden state sequence at the t-th time step. The decoded hidden state sequence includes M decoding hidden state vectors, and one decoding hidden state vector is used to determine a word to be decoded.

[0125] The input sequence corresponding to the t-th time step indicates the decoded hidden state vector corresponding to the decoded word before the t-th time step and the information of the M placeholders input at the t-th time step.

[0126] In step 630, after the decoder network determines the attention scores of the input sequence corresponding to the t-th time step and the encoded hidden state sequence corresponding to the input text for each placeholder, since a placeholder represents a current word to be decoded, the non-relevant text corresponding to the word to be decoded at the t-th time step is determined, and the attention scores of the corresponding placeholders of the non-relevant text are masked. Then, the attention scores that have not been masked at the t-th time step are used to determine the decoding hidden state vectors corresponding to the M words to be decoded at the t-th time step.

[0127] Repeat the above process. If the number of decoding hidden state vectors obtained by one decoding path reaches the target length threshold, or the decoding hidden state vector representing the end symbol is obtained in all decoding paths, there is no need to further call the decoder network to continue decoding.

[0128] In step 640, the characters corresponding to the decoded latent state vectors obtained by the same decoding path are combined in chronological order to obtain M output texts.

[0129] In a specific embodiment, after obtaining each decoded latent state vector, a linear transformation can be performed on the decoded latent state vector using a fully connected layer. The resulting vectors are then classified using a classification function in a classification layer to determine the corresponding word. This processing is performed on all decoded latent state vectors in a decoding path, following the chronological order of the decoded latent state vectors (i.e., the order in which the decoded latent state vectors were output within the same decoding path). This process can be used to determine the corresponding output text for each decoding path.

[0130] In some embodiments, the decoder network includes a cascade of the first-stage decoder sub-network to the P-th-stage decoder sub-network, where P is a positive integer; in this embodiment, Figure 7 As shown, step 630 includes:

[0131] Step 710: Obtain the K-level decoding input sequence of the K-th level decoder sub-network at the t-th time step; wherein K is a positive integer, 1≤K≤P; when K=1, the K-level decoding input sequence at the t-th time step is a sequence obtained by processing the input sequence corresponding to the t-th time step; when K>1, the K-level decoding input sequence at the t-th time step is the K-1-level decoding output sequence output by the K-1-th level decoder sub-network at the t-th time step.

[0132] In step 720, the K-th level decoder sub-network performs parallel decoding based on the K-th level decoding input sequence at the t-th time step and the encoded hidden state sequence corresponding to the input text based on the masked attention mechanism, and obtains the K-th level decoding output sequence of the K-th level decoder sub-network at the t-th time step.

[0133] Step 730: If K < P, the K-level decoding output sequence of the K-th level decoder sub-network at the t-th time step is used as the K+1-level decoding input sequence corresponding to the K+1-th level decoder sub-network at the t-th time step, and is decoded by the K+1-th level decoder sub-network.

[0134] Step 740: If K=P, the K-level decoding output sequence of the K-th level decoder sub-network at the t-th time step is used as the decoding hidden state sequence at the t-th time step.

[0135] The network structures of the decoder sub-networks at different levels from the first to the P-th decoder sub-networks can be the same or different, and are not specifically limited here. In a specific embodiment, the decoder sub-networks at each level in the decoder network can be decoders in a Transformer model (transformer network model).

[0136] The input sequence corresponding to the t-th time step is processed by position encoding and segment encoding of the embedding vectors of each placeholder at the t-th time step and the decoded hidden state vector before the t-th time step, and then generating a level 1 decoding input sequence based on the embedding vectors, position encoding (Position embedding) and segment encoding (Segment embedding) of each placeholder, and the decoded hidden state vector, the position encoding and segment encoding corresponding to the decoded hidden state vector. Among them, the position encoding is used for the position in the input sequence corresponding to the t-th time step, and the segment encoding is used to indicate whether it belongs to the entity content to be output. In the present application, since the decoded hidden state vector and the embedding vector of the placeholder are both used to represent the words in the entity content, at the t-th time step, the segment encoding corresponding to the decoded hidden state vector and the segment encoding corresponding to the embedding vector of the placeholder are the same, and both are encodings indicating that they belong to the entity content to be output.

[0137] In this embodiment, for the convenience of description, the decoding input sequence input to the K-th level decoder sub-network at the t-th time step is referred to as the K-th decoding input sequence, and the decoding output sequence output by the K-th level decoder sub-network at the t-th time step is referred to as the K-th decoding output sequence.

[0138] The K-th level decoder sub-network can first determine an attention matrix based on the K-th level decoded input sequence and the encoded hidden state sequence corresponding to the input text at time step t, using an attention mechanism. This attention matrix is ​​used to indicate the attention of existing information on the current word to be decoded, where the existing information includes the target text, the identification text corresponding to each of the N entities, and the decoded hidden state vector obtained before time step t. It can be understood that in each level of the decoder sub-network, the information of the target text and the identification text corresponding to the N entities is represented by the encoded hidden state sequence of the input text.

[0139] Then, based on the M to-be-decoded characters corresponding to time step t, the non-relevant text corresponding to each to-be-decoded character is determined, and the attention score of each character in the non-relevant text to the to-be-decoded character is determined as the attention score that needs to be masked. Then, based on the determined attention score that needs to be masked, the attention at the corresponding position in the attention matrix is ​​masked, so that the K-th level decoder sub-network does not pay attention to the attention score of each character in the non-relevant text corresponding to the to-be-decoded character in the subsequent processing process.

[0140] Each level of decoder sub-network decodes step by step according to the above process to obtain the decoded output sequence output by the corresponding level decoder sub-network. If the current output decoded output sequence is not the last level decoder sub-network in the decoder network, the decoded output sequence is input to the next level decoder sub-network, and the next level decoder sub-network continues to perform decoding processing according to the input decoded output sequence and the encoded hidden state sequence corresponding to the input text; conversely, if the current output decoded output sequence is the last level decoder sub-network in the decoder network, the decoded output sequence is used as the decoded hidden state sequence at the tth time step.

[0141] In some embodiments, the K-th level decoder sub-network includes a K-th level mask attention layer and a K-th level decoding processing layer; in this embodiment, Figure 8 As shown, step 720 includes:

[0142] In step 810, the K-th level masked attention layer determines the K-th level attention matrix corresponding to the t-th time step based on the K-th level decoded input sequence and the encoded hidden state sequence corresponding to the input text.

[0143] In this application, each level of encoder sub-network in the encoder network includes a masked attention layer. The masked attention layer in the K-th level encoder sub-network is referred to as the K-th level masked attention layer. Correspondingly, the attention matrix determined by the K-th level masked attention layer is referred to as the K-level attention matrix. It is understandable that at different time steps, due to the different inputs of the decoder network, the attention matrices determined by the K-th level masked attention layer at different time steps are also different.

[0144] The K-level attention matrix corresponding to the t-th time step indicates the attention scores of each word in the input text, each word in the identification text corresponding to the entity, and the decoded state latent vector before the t-th time step to each placeholder at the t-th time step (that is, the word to be decoded at the t-th time step).

[0145] In a specific embodiment, the K-th level masked attention layer can be based on a single-head attention mechanism or a multi-head attention mechanism to determine the K-th level attention matrix corresponding to the t-th time step according to the K-th level decoding input sequence and the encoded hidden state sequence corresponding to the input text.

[0146] In step 820, the K-th level masked attention layer determines the K-level masked attention matrix corresponding to the t-th time step based on the K-level attention matrix corresponding to the t-th time step and the mask matrix corresponding to the t-th time step; the mask matrix corresponding to the t-th time step indicates the attention scores that need to be masked and the attention scores that do not need to be masked in the K-level attention matrix corresponding to the t-th time step.

[0147] In this application, for the convenience of description, the masked attention layer in the K-th level decoder subnetwork is referred to as the K-th level masked attention layer, and the other neural network layers after the K-th level masked attention layer in the K-th level decoder subnetwork are referred to as the K-th level decoding processing layer, wherein the neural network layer after the masked attention layer in the decoder subnetwork can be one layer or multiple layers, for example, it can be a fully connected network layer, a convolutional network layer, a feedforward network layer, etc., which is not specifically limited here.

[0148] It is understandable that the decoded input sequences of masked attention layers at different levels at the same time step are different. Therefore, the attention matrices determined by masked attention layers at different levels at the same time step may be different.

[0149] In some embodiments, at the t-th time step, the decoder networks at each level can share the mask matrix corresponding to the t-th time step determined by the above process, without having to repeatedly determine the mask matrix corresponding to the t-th time step. In this embodiment, before step 820, the mask matrix corresponding to the t-th time step can be determined by the following process: for the target entity corresponding to each to-be-decoded word at the t-th time step, the non-related text corresponding to the to-be-decoded word at the t-th time step is determined in the target text and the decoded words determined before the t-th time step; the mask matrix corresponding to the t-th time step is determined based on the corresponding positions of the words in the non-related text corresponding to the to-be-decoded word at the t-th time step in the K-th level decoding input sequence and the K-th level encoding hidden state sequence.

[0150] In some embodiments, a K-level attention matrix corresponding to the t-th time step and a mask matrix corresponding to the t-th time step may be matrix-added, and the resulting matrix may be used as the K-level mask attention matrix corresponding to the t-th time step. In the mask matrix corresponding to the t-th time step, the attention score of each character in the non-related text corresponding to the to-be-decoded character to the to-be-decoded character is set to a first specified number, such as negative infinity, and the attention score of each character in the related text corresponding to the to-be-decoded character to the to-be-decoded character is set to a second specified number, such as 0.

[0151] In this way, after the addition, the attention score of the relevant text of a word to be decoded in the K-level masked attention matrix for the word to be decoded is the same as that in the K-level attention matrix, and the attention score of the non-relevant text corresponding to a word to be decoded in the K-level masked attention matrix for the word to be decoded is reset to negative infinity. In the subsequent processing process, the attention score of the non-relevant text corresponding to the word to be decoded for the word to be decoded can be converted to 0 through normalization and other methods. In this way, the purpose of determining the decoding hidden state vector of the word to be decoded without using the information of the non-relevant text corresponding to the word to be decoded can be achieved.

[0152] In other embodiments, after determining the non-relevant text corresponding to each word to be decoded at the t-th time step, the attention score of the word in the relevant text corresponding to the word to be decoded to the word to be decoded can be set to 0 in the mask matrix corresponding to the t-th time step, and the attention score of the word in the non-relevant text corresponding to the word to be decoded to the word to be decoded can be set to the opposite of the attention score of the corresponding position in the K-level attention matrix. In this way, after matrix addition of the K-level attention matrix corresponding to the t-th time step and the mask matrix corresponding to the t-th time step, in the resulting K-level mask attention matrix, the attention score of the relevant text of a word to be decoded to the word to be decoded is the same as that in the K-level attention matrix, and the attention score of the non-relevant text of a word to be decoded to the word to be decoded is 0.

[0153] In the Transformer model, a sequence mask matrix is ​​used to prevent the model from seeing future information. That is, for a sequence, at time step t, the decoded output can only depend on the output before time t, and cannot depend on the output after time t. The specific method is: generate a lower triangular mask (attentionmask) matrix with all upper triangular values ​​​​zero. Applying this matrix to each sequence can make time t only see the values ​​before time t. Specifically in this solution, based on this method, the mask matrix at each time step can be determined in a similar way, so that in the mask matrix, the word in the non-related text corresponding to a word to be decoded has an attention score of zero for the word to be decoded.

[0154] In step 830, the K-level decoding processing layer performs processing based on the K-level mask attention matrix corresponding to the t-th time step, the K-th level decoding input sequence, and the encoded hidden state sequence corresponding to the input text to obtain the K-level decoding output sequence of the K-th level decoder subnetwork at the t-th time step.

[0155] In some embodiments, the K-level decoding processing layer can be weighted according to the attention score indicated by the K-level masked attention matrix corresponding to the t-th time step, and the vector representations corresponding to all words in the K-th level decoding input sequence and the encoded hidden state sequence corresponding to the input text are processed to obtain the K-level decoding output sequence of the K-th level decoder sub-network at the t-th time step.

[0156] In some embodiments, if the decoder subnetwork is a decoder in a Transformer model, the K-level decoding processing layer may include a first add and normalize layer, a feedforward network layer, and a second add and normalize layer sequentially cascaded with a mask attention layer.

[0157] Figure 9: is a schematic diagram of the structure of the decoder sub-network according to an embodiment of the present application. In this embodiment, the decoder sub-network is the decoder in the Transformer model. Figure 9 As shown, the decoder subnetwork includes a cascaded masked attention layer, a first add & normalize layer, a first feed forward network layer, and a second add & normalize layer, wherein the decoder subnetwork also includes an identity mapping from the input of the masked attention layer to the first add & normalize layer, and an identity mapping from the input of the first feed forward network layer to the second add & normalize layer (identity mapping is also called residual connection). In this embodiment, the masked attention layer is a multi-head masked attention layer (Masked Multi-Head Attention). Through the residual connection, the shallow layer information is applied to the deep layer calculation to avoid the problems of gradient disappearance and gradient explosion.

[0158] In some embodiments, based on Figure 7 In a corresponding embodiment, the encoded hidden state sequence corresponding to the input text can be obtained by encoding the input text through a multi-level encoder sub-network. Specifically, in this embodiment, step 230 includes: obtaining the R-level encoded input sequence corresponding to the R-th level encoder sub-network, wherein when R=1, the R-level encoded input sequence is the sequence obtained by processing the input text; when R>1, the R-level encoded input sequence is the R-1-level encoded output sequence output by the R-1-th level encoder sub-network; 1≤R≤S, S is the total number of encoder sub-networks; R and S are positive integers; the R-th level encoder sub-network encodes the R-level encoded input sequence and outputs the R-level encoded output sequence; if R<S, the R-level encoded output sequence is used as the R+1-level encoded input sequence corresponding to the R+1-th level encoder sub-network, and the R+1-th level encoder sub-network encodes the R+1-level encoded input sequence; if R=S, the R-level encoded output sequence is used as the encoded hidden state sequence corresponding to the input text.

[0159] In this embodiment, for the convenience of description, the encoding input sequence input into the R-th level encoder sub-network is referred to as the R-level encoding input sequence, and the encoding output sequence output by the R-th level encoder sub-network is referred to as the R-level encoding output sequence.

[0160] In this embodiment, a multi-stage encoder sub-network performs feature extraction on the first-stage encoded input sequence, thereby obtaining an encoded latent state sequence that accurately reflects the semantic information of the input text. Similarly, the structures of the encoder sub-networks at different levels can be the same or different, and are not specifically limited here.

[0161] In some embodiments, each character in the input text can be subjected to token embedding, position encoding, and segment encoding. The embedding vectors, position encoding, and segment encoding corresponding to each character in the input text are then superimposed and combined to obtain a Level 1 encoding input sequence corresponding to the input text. Similarly, the position encoding corresponding to a character is used to indicate the position of the character in the input text, and the segment encoding is used to indicate whether the character belongs to the entity content to be output.

[0162] The encoder subnetwork can be constructed using one or more neural networks, such as at least one of a convolutional neural network, a fully connected neural network, a recurrent neural network, and a feedforward neural network. The structures of encoder subnetworks at different levels can be the same or different. In some specific embodiments, the encoder subnetwork can be an encoder in a Transformer model.

[0163] Figure 10 : is a schematic diagram of the structure of the encoder sub-network according to an embodiment of the present application. In this embodiment, the encoder sub-network is the encoder in the Transformer model (converter network model). Figure 10 As shown, the encoder sub-network includes a sequentially cascaded attention layer, a third summation and normalization layer, a second feedforward network layer, and a fourth summation and normalization layer. The encoder sub-network also includes an identity mapping from the input of the attention layer to the third summation and normalization layer, and an identity mapping from the input of the second feedforward network layer to the fourth summation and normalization layer. The attention layer can calculate the corresponding attention score based on a single-head attention mechanism or a multi-head attention mechanism.

[0164] Figure 11 FIG. 1 is a flow chart of extracting entity content according to an embodiment of the present application. Figure 11 As shown, the encoder network 1110 includes a multi-stage encoder sub-network, and the decoder network 1120 includes a multi-stage decoder sub-network. Figure 11 As shown, it is assumed that the entity to be extracted is Figure 5 A-5C correspond to the entities in the embodiment, and their corresponding grouping results are also as shown above. Figure 11 In the example, mask is abbreviated as m, that is, mask00 is m01.

[0165] like Figure 11As shown in the figure, the token embeddings corresponding to each character in the input text and the position of each character in the input text are positionally encoded (position embeddings), and each character is segmented (segment embedding). After superimposing the token embeddings, position encoding, and segment encoding corresponding to each character, the resulting level 1 encoding input sequence is input into the first-level encoder sub-network. After that, each level of the encoder sub-network performs encoding processing based on the level 1 encoding input sequence, and the S-level encoding output sequence output by the last level of the encoder sub-network (i.e., the S-level encoder sub-network) is used as the encoded hidden state sequence corresponding to the input text.

[0166] Afterwards, the encoded hidden state sequence corresponding to the input text is input into the decoder sub-networks at each level. The decoder sub-networks at each level use the encoded hidden state sequence corresponding to the input text and the decoded output sequence output by the previous decoder sub-network to perform decoding processing step by step, and obtain M decoding hidden state vectors at each time step.

[0167] For the decoder network, at each time step, after determining the input sequence corresponding to the decoder network at time step t, position encoding and segment encoding are performed accordingly to obtain a level 1 decoding input sequence. After that, the decoder sub-networks at each level perform decoding processing step by step to obtain M decoding hidden state vectors corresponding to the time step. Figure 11 In , u is the target text length threshold.

[0168] In this embodiment, the total number (or total number of levels) of the encoder sub-networks and the total number (or total number of levels) of the decoder sub-networks may be the same or different, and may be set according to actual needs and is not specifically limited here.

[0169] In the above example, the information used by the decoder sub-networks at different levels from the encoder network is the same. In other embodiments, the information used by the encoder sub-networks at different levels from the encoder network may also be different.

[0170] Specifically, in this embodiment, the decoder network includes a cascade of the first-level decoder subnetwork to the P-th-level decoder subnetwork, where P is a positive integer; the information from the encoder network used by different levels of encoder subnetworks may be different. Specifically, in this embodiment, the encoding hidden state sequence includes the encoding hidden state sequence from the 1st level to the Pth level; in this embodiment, if Figure 12 As shown, step 630 includes:

[0171] Step 1210: Obtain the K-level decoding input sequence corresponding to the K-th level decoder subnetwork; wherein K is a positive integer, 1≤K≤P; when K=1, the K-level decoding input sequence is a sequence obtained by processing the input sequence corresponding to the t-th time step; when K>1, the K-level decoding input sequence is the K-1-th level output sequence of the K-1-th level decoder subnetwork at the t-th time step.

[0172] In step 1220, the K-th level decoder sub-network performs decoding processing based on the K-th level decoding input sequence and the K-th level encoding hidden state sequence based on the masked attention mechanism to obtain the K-th level decoding output sequence of the K-th level decoder sub-network at the t-th time step.

[0173] In some embodiments, the K-th level decoder sub-network includes a K-th level masked attention layer and a K-th level decoding processing layer; in this embodiment, step 1220 includes: the K-th level masked attention layer determines the K-th level attention matrix corresponding to the t time step based on the K-th level decoding input sequence and the K-level encoding hidden state sequence; the K-th level masked attention layer determines the K-level masked attention matrix corresponding to the t time step based on the K-level attention matrix corresponding to the t time step and the mask matrix corresponding to the t time step; the mask matrix corresponding to the t time step indicates the attention scores that need to be masked and the attention scores that do not need to be masked in the K-level attention matrix corresponding to the t time step; the K-level decoding processing layer processes the K-level masked attention matrix corresponding to the t time step, the K-th level decoding input sequence and the encoding hidden state sequence corresponding to the input text to obtain the K-level decoding output sequence of the K-th level decoder sub-network at the t time step.

[0174] The specific process of calculating the K-level attention matrix, the mask matrix corresponding to the t-th time step, and the K-level attention matrix corresponding to the t-th time step is described above and will not be repeated here.

[0175] In step 1230, if K < P, the K-level decoding output sequence at the t-th time step is used as the K+1-level decoding input sequence corresponding to the K+1-th level decoder sub-network, and the K+1-level decoder sub-network continues to perform decoding processing.

[0176] Step 1240: If K=P, the K-level decoding output sequence at the t-th time step is used as the decoding hidden state sequence at the t-th time step.

[0177] In this embodiment, during the decoding process, the information used from the decoder network is not only the feature information output by the last-level decoder sub-network in the decoder network, but also includes the feature information output by the intermediate-level decoder sub-network. Therefore, the decoding process utilizes more feature information, thereby further improving the decoding accuracy.

[0178] In some embodiments, based on Figure 12 In the illustrated embodiment, step 230 includes obtaining a T-level encoding input sequence corresponding to the T-th level encoder sub-network, where 1≤T≤P-1. When T=1, the T-level encoding input sequence is the 1-level encoding hidden state sequence obtained by embedding the input text; when T>1, the T-level encoding input sequence is the T-level encoding hidden state sequence output by the T-1-th level encoder network. The T-th level encoder sub-network encodes the T-level encoding input sequence to obtain the T+1-level encoding hidden state sequence output by the T-th level encoder sub-network. If T<P-1, the T+1-level encoding hidden state sequence is used as the T+1-level encoding input sequence corresponding to the T+1-th level encoder sub-network and is encoded by the T+1-th level encoder sub-network. If T=P-1, the 1-level encoding hidden state sequence to the P-level encoding hidden state sequence are used as the encoding hidden state sequence corresponding to the input text.

[0179] In this embodiment, since the sequence obtained by embedding the input text is used as the first-level encoded hidden state sequence, the total number of encoder sub-networks is one less than the total number of decoder sub-networks. The structures of the encoder sub-networks at each level can be the same or different. For example, the structures of the encoder sub-networks at each level can be as follows: Figure 10 shown.

[0180] Figure 13 This is a flowchart of extracting entity content according to another embodiment of the present application. Figure 11 In this embodiment, the feature information from the encoder used by the decoder sub-networks at each level is different. In this embodiment, the feature information from the encoder used by the first-level decoder sub-network is the first-level encoded input sequence of the first-level encoder sub-network. It can also be understood that the feature information from the encoder used by the K-th level encoder sub-network is the K-th level encoded input sequence of the K-th level encoder sub-network.

[0181] In another embodiment, the total number of levels of the encoder sub-network may be set equal to the total number of levels of the decoder sub-network, so that the information from the encoder used by the first-level decoder sub-network is the first-level encoded output sequence of the first-level encoder sub-network. Figure 14 This is a flowchart of extracting entity content according to another embodiment of the present application. Figure 13 In this embodiment, the feature information from the encoder used by the K-th level encoder sub-network is the K-th level encoded output sequence output by the K-th level encoder sub-network.

[0182] In the above embodiment, the encoder subnetwork can be an encoder in a Transformer model, and the decoder subnetwork can be a decoder in a Transformer model. By using the Transformer model to extract entity content according to the method of the present application, the entity content corresponding to multiple entities can be extracted in parallel, which greatly improves the efficiency of entity content extraction. Moreover, there is no length limit for the text length of the entity content, and it can be widely used in scenarios of entity content extraction, such as OCR text structuring (certificates, documents, bills, etc.), machine question answering and other application scenarios. Due to the use of a masked attention mechanism, in the parallel decoding process, the attention score of the non-related text corresponding to the word to be decoded is masked relative to the word to be decoded, so that the feature information of the word to be decoded is only related to the feature information of the related text of the word to be decoded, and will not be affected by the feature information of the non-related text corresponding to the word to be decoded. Therefore, the entity content corresponding to each entity extracted according to the method of the present application is completely consistent with the entity content decoded serially word by word, so that there is no need to adjust the training process of the Transformer model.

[0183] The method of the present application can be applied to scenarios where entity content extraction is performed in the cloud. By adopting the method of the present application, the user's usage delay can be greatly reduced, the user experience of cloud products can be improved, and redundant computing can be reduced. Therefore, the hardware cost can be greatly reduced and hardware cost overhead can be saved.

[0184] The following describes an embodiment of the device of the present application, which can be used to perform the method described in the above embodiment of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the above method embodiment of the present application.

[0185] Figure 15 is a block diagram of an apparatus for extracting entity content according to an embodiment of the present application, such as Figure 15As shown, the device for extracting entity content includes: an acquisition module 1510, which is used to obtain grouping information of N entities, the grouping information indicates that the N entities are divided into M groups and the entities included in each group, wherein M and N are positive integers, 2≤M<N; the sum of the text length thresholds corresponding to entities belonging to the same group is not greater than the target length threshold; a sequence determination module 1520, which is used to determine the extraction order corresponding to each group according to the grouping information; a splicing module 1530, which is used to splice the target text with the identification text corresponding to each entity in the N entities to obtain an input text; an encoding module 1540, which is used to encode the input text to obtain an encoded hidden state sequence corresponding to the input text; a parallel decoding module 1550, which is used to perform M-way parallel decoding on the encoded hidden state sequence corresponding to the input text according to the extraction order based on the masked attention mechanism to obtain M-way output text, wherein one output text includes the entity content corresponding to all entities belonging to a group in the target text, and the text length of the entity content corresponding to each entity does not exceed the text length threshold corresponding to the entity.

[0186] In some embodiments, the parallel decoding module 1550 includes: a target entity determination unit, which is used to determine the target entities corresponding to the M to-be-decoded characters to be decoded at the t time step according to the extraction order, and different to-be-decoded characters correspond to different target entities; wherein t is a positive integer; a first splicing unit, which is used to splice the embedding vectors corresponding to the M placeholders with the decoded hidden state vector before the t time step to obtain the input sequence corresponding to the decoder network at the t time step; a parallel decoding unit, which is used by the decoder network to use the input sequence corresponding to the t time step and the encoded hidden state corresponding to the input text based on the masked attention mechanism. The sequence is decoded in parallel to obtain a decoding hidden state sequence at the t-th time step, where the decoding hidden state sequence includes M decoding hidden state vectors, and one decoding hidden state vector is used to determine a word to be decoded; the masked attention mechanism is used to mask the attention score of the non-relevant text corresponding to the word to be decoded relative to the word to be decoded, and the non-relevant text corresponding to the word to be decoded includes the identification text corresponding to other entities except the target entity and the decoded words corresponding to other entities except the target entity; the combination unit is used to combine the words corresponding to the M decoding hidden state vectors obtained at each time step in chronological order to obtain M output texts.

[0187] In some embodiments, the device for extracting entity content further includes: a time step interval determination module, which is used to determine the time step interval according to a target length threshold; the time step interval is 1 to Q, and Q is the target length threshold; a time step sub-interval division module, which is used to divide the time step interval into a target number of time step sub-intervals according to the extraction order corresponding to the group based on the target number of entities included in a group and the text length threshold corresponding to the entities included in a group, wherein a time step sub-interval corresponds to an entity in the corresponding group, and the interval length of a time step sub-interval is the text length threshold corresponding to the corresponding entity; in this embodiment, the target entity determination unit is further configured to: for each group in the M groups, determine the time step sub-interval where t is located in the time step sub-interval corresponding to the group, and use the entity corresponding to the time step sub-interval where t is located as the target entity of the corresponding group at the tth time step.

[0188] In some embodiments, the decoder network includes a 1st-level decoder subnetwork to a P-th-level decoder subnetwork cascaded in sequence, where P is a positive integer; in this embodiment, the parallel decoding unit includes: a first acquisition unit, configured to acquire a K-level decoding input sequence of the K-th-level decoder subnetwork at the t-th time step; wherein K is a positive integer, 1≤K≤P; when K=1, the K-level decoding input sequence at the t-th time step is a sequence obtained by processing the input sequence corresponding to the t-th time step; when K>1, the K-level decoding input sequence at the t-th time step is a K-1-level decoding output sequence output by the K-1-th-level decoder subnetwork at the t-th time step; the first decoding unit is configured to obtain a K-level decoding input sequence of the K-th-level decoder subnetwork at the t-th time step; The decoder subnetwork is based on the masked attention mechanism, and performs parallel decoding according to the K-level decoding input sequence at the t-th time step and the encoded hidden state sequence corresponding to the input text to obtain the K-level decoding output sequence of the K-th decoder subnetwork at the t-th time step; the first continued decoding unit is used to, if K<P, use the K-level decoding output sequence of the K-th decoder subnetwork at the t-th time step as the K+1-level decoding input sequence corresponding to the K+1-th decoder subnetwork at the t-th time step, and decode it by the K+1-th decoder subnetwork; the first determination unit is used to, if K=P, use the K-level decoding output sequence of the K-th decoder subnetwork at the t-th time step as the decoding hidden state sequence at the t-th time step.

[0189] In some embodiments, the encoding module includes: a second acquisition unit, used to obtain the R-level encoding input sequence corresponding to the R-th level encoder sub-network, wherein, when R=1, the R-level encoding input sequence is a sequence obtained by processing the input text; when R>1, the R-level encoding input sequence is the R-1-level encoding output sequence output by the R-1-th level encoder sub-network; 1≤R≤S, S is the total number of encoder sub-networks; R and S are positive integers; a first encoding unit, used to encode the R-level encoding input sequence by the R-th level encoder sub-network and output the R-level encoding output sequence; a first continued encoding unit, used to, if R<S, use the R-level encoding output sequence as the R+1-level encoding input sequence corresponding to the R+1-th level encoder sub-network, and use the R+1-th level encoder sub-network to encode the R+1-level encoding input sequence; a second determination unit, used to, if R=S, use the R-level encoding output sequence as the encoding hidden state sequence corresponding to the input text.

[0190] In other embodiments, the decoder network includes a 1st-level decoder subnetwork to a Pth-level decoder subnetwork cascaded in sequence, where P is a positive integer; the encoded hidden state sequence includes a 1st-level to Pth-level encoded hidden state sequence; in this embodiment, the parallel decoding unit includes: a third acquisition unit for acquiring a K-level decoding input sequence corresponding to the K-th-level decoder subnetwork; wherein K is a positive integer, 1≤K≤P; when K=1, the K-level decoding input sequence is a sequence obtained by processing the input sequence corresponding to the t-th time step; when K>1, the K-level decoding input sequence is the K-1-level output sequence of the K-1-th-level decoder subnetwork at the t-th time step. ; The second decoding unit is used for the K-th level decoder sub-network to perform decoding processing based on the masked attention mechanism, according to the K-th level decoding input sequence and the K-th level encoding hidden state sequence, to obtain the K-th level decoding output sequence of the K-th level decoder sub-network at the t-th time step; the second continued decoding unit is used for, if K<P, using the K-th level decoding output sequence at the t-th time step as the K+1-th level decoding input sequence corresponding to the K+1-th level decoder sub-network, and continuing the decoding processing by the K+1-th level decoder sub-network; the third determination unit is used for, if K=P, using the K-th level decoding output sequence at the t-th time step as the decoding hidden state sequence at the t-th time step.

[0191] In some embodiments, the encoding module includes: a fourth acquisition unit, used to obtain a T-level encoding input sequence corresponding to the T-th level encoder sub-network, wherein 1≤T≤P-1, when T=1, the T-level encoding input sequence is a 1-level encoding hidden state sequence obtained by embedding the input text; when T>1, the T-level encoding input sequence is a T-level encoding hidden state sequence output by the T-1-th level encoder network; a second encoding unit, used to encode the T-level encoding input sequence by the T-th level encoder sub-network to obtain a T+1-level encoding hidden state sequence output by the T-th level encoder sub-network; a second continuing encoding unit, used to, if T<P-1, use the T+1-level encoding hidden state sequence as the T+1-level encoding input sequence corresponding to the T+1-th level encoder sub-network, and encode it by the T+1-th level encoder sub-network; a fourth determining unit, used to, if T=P-1, use the 1-level encoding hidden state sequence to the P-level encoding hidden state sequence as the encoding hidden state sequence corresponding to the input text.

[0192] In some embodiments, the K-th level decoder subnetwork includes a K-th level masked attention layer and a K-th level decoding processing layer; in this embodiment, the second decoding unit includes: an attention matrix determination unit, which is used by the K-th level masked attention layer to determine the K-level attention matrix corresponding to the t time step according to the K-th level decoding input sequence and the K-level encoding hidden state sequence; a masked attention matrix determination unit, which is used by the K-th level masked attention layer to determine the K-level masked attention matrix corresponding to the t time step according to the K-level attention matrix corresponding to the t time step and the mask matrix corresponding to the t time step; the mask matrix corresponding to the t time step indicates the attention scores that need to be masked and the attention scores that do not need to be masked in the K-level attention matrix corresponding to the t time step; a processing unit, which is used by the K-level decoding processing layer to process the K-level masked attention matrix corresponding to the t time step, the K-th level decoding input sequence and the encoding hidden state sequence corresponding to the input text to obtain the K-level decoding output sequence of the K-th level decoder subnetwork at the t time step.

[0193] In some embodiments, the device for extracting entity content further includes: a non-relevant text determination module, which is used to determine, for the target entity corresponding to each to-be-decoded word at the t-th time step, the non-relevant text corresponding to the word to be decoded at the t-th time step in the target text and the decoded words determined before the t-th time step; a mask matrix determination module, which is used to determine the mask matrix corresponding to the t-th time step based on the corresponding positions of the words in the non-relevant text corresponding to the word to be decoded at the t-th time step in the K-th level decoding input sequence and the K-th level encoding hidden state sequence.

[0194] In some embodiments, the device for extracting entity content also includes: a text length threshold acquisition module, used to obtain the text length thresholds set for N entities respectively; a target length threshold determination module, used to take the maximum value of the text length thresholds corresponding to the N entities as the target length threshold; a grouping module, used to divide the N entities into M groups according to the text length threshold and target length threshold corresponding to each entity, to obtain grouping information.

[0195] Figure 16 The following is a schematic diagram showing the structure of a computer system suitable for implementing an electronic device according to an embodiment of the present application. Figure 16 The computer system 1600 of the electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present application.

[0196] like Figure 16 As shown, computer system 1600 includes a central processing unit (CPU) 1601, which can perform various appropriate actions and processes according to the program stored in read-only memory (ROM) 1602 or the program loaded from storage part 1608 into random access memory (RAM) 1603, such as executing the method in the above embodiment. Various programs and data required for system operation are also stored in RAM 1603. CPU 1601, ROM 1602 and RAM 1603 are connected to each other via bus 1604. Input / output (I / O) interface 1605 is also connected to bus 1604.

[0197] The following components are connected to the I / O interface 1605: an input section 1606 including a keyboard, a mouse, and the like; an output section 1607 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 1608 including a hard disk; and a communication section 1609 including a network interface card such as a LAN (Local Area Network) card or a modem. The communication section 1609 performs communication processing via a network such as the Internet. A drive 1610 is also connected to the I / O interface 1605 as needed. Removable media 1611, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 1610 as needed, so that computer programs read from the removable media can be installed in the storage section 1608 as needed.

[0198] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 1609, and / or installed from a removable medium 1611. When the computer program is executed by the central processing unit (CPU) 1601, the various functions defined in the system of the present application are executed.

[0199] It should be noted that the computer-readable medium shown in the embodiments of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0200] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. Among them, each box in the flowchart or block diagram can represent a module, program segment, or part of the code, and the above-mentioned module, program segment, or part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0201] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. In some cases, the names of these units do not constitute limitations on the units themselves.

[0202] As another aspect, the present application further provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments, or may exist independently without being incorporated into the electronic device. The computer-readable storage medium carries computer-readable instructions, and when the computer-readable instructions are executed by a processor, the method of any of the above embodiments is implemented.

[0203] According to one aspect of the present application, an electronic device is also provided, which includes: a processor; a memory, wherein computer-readable instructions are stored in the memory, and when the computer-readable instructions are executed by the processor, the method in any of the above embodiments is implemented.

[0204] According to one aspect of an embodiment of the present application, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the method of any of the above embodiments.

[0205] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiment of the application, the features and functions of two or more modules or units described above can be concretized in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.

[0206] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the embodiments of the present application.

[0207] Those skilled in the art will readily conceive of other embodiments of the present application after considering the specification and practicing the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of this application and include common knowledge or customary techniques in the art that are not disclosed herein.

[0208] It should be understood that the present application is not limited to the exact structures described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A method for extracting entity content, characterized in that: include: Obtaining grouping information of N entities, the grouping information indicating that the N entities are divided into M groups and entities included in each group, where M and N are positive integers, 2≤M<N; and a sum of text length thresholds corresponding to entities belonging to the same group is not greater than a target length threshold; Determining an extraction order corresponding to each group according to the grouping information; Concatenate the target text with the identification text corresponding to each of the N entities to obtain the input text; Encoding the input text to obtain an encoded latent state sequence corresponding to the input text; Based on the masked attention mechanism, the encoded hidden state sequence corresponding to the input text is decoded in M ​​parallel ways according to the extraction order to obtain M output texts, wherein one output text includes the entity contents corresponding to all entities belonging to a group in the target text, and the text length of the entity content corresponding to each entity does not exceed the text length threshold corresponding to the entity.

2. The method according to claim 1, characterized in that The masked attention mechanism is based on which the encoded hidden state sequence corresponding to the input text is decoded in parallel in M ​​ways according to the extraction order to obtain M ways of output text, including: Determine the target entities corresponding to the M to-be-decoded words to be decoded at the t-th time step according to the extraction order, where different to-be-decoded words correspond to different target entities; wherein t is a positive integer; Concatenate the embedding vectors corresponding to the M placeholders with the decoded hidden state vector before the tth time step to obtain the input sequence corresponding to the decoder network at the tth time step; The decoder network performs parallel decoding based on a masked attention mechanism using an input sequence corresponding to the tth time step and an encoded hidden state sequence corresponding to the input text to obtain a decoded hidden state sequence at the tth time step, wherein the decoded hidden state sequence includes M decoded hidden state vectors, and each decoded hidden state vector is used to determine a word to be decoded; wherein the masked attention mechanism is used to mask the attention score of non-relevant text corresponding to the word to be decoded relative to the word to be decoded, and the non-relevant text corresponding to a word to be decoded includes identification text corresponding to entities other than the target entity and decoded words corresponding to entities other than the target entity; The words corresponding to the decoded hidden state vectors obtained by the same decoding path are combined in chronological order to obtain M-path output texts.

3. The method according to claim 2, characterized in that Before determining the target entities respectively corresponding to the M to-be-decoded words to be decoded at the t-th time step according to the extraction order, the method further includes: Determine a time step interval according to the target length threshold; the time step interval is 1 to Q, where Q is the target length threshold; According to the target number of entities included in a group and the text length threshold corresponding to the entities included in a group, the time step interval is divided into a target number of time step sub-intervals according to the extraction order corresponding to the group, wherein a time step sub-interval corresponds to an entity in the corresponding group, and the interval length of the time step sub-interval corresponding to the entities in the group except the last entity in the extraction order is the corresponding text length threshold; The determining, according to the extraction order, target entities corresponding to the M to-be-decoded words to be decoded at the t-th time step, includes: For each of the M groups, in the time step subinterval corresponding to the group, the time step subinterval where t is located is determined, and the entity corresponding to the time step subinterval where t is located is used as the target entity of the corresponding group at the tth time step.

4. The method according to claim 2, characterized in that The decoder network includes a 1st-stage decoder subnetwork to a P-th-stage decoder subnetwork cascaded in sequence, where P is a positive integer; The decoder network performs parallel decoding based on a masked attention mechanism using an input sequence corresponding to the t time step and an encoded hidden state sequence corresponding to the input text to obtain a decoded hidden state sequence at the t time step, including: Obtaining a K-level decoding input sequence of the K-th level decoder subnetwork at the t-th time step; wherein K is a positive integer, 1≤K≤P; when K=1, the K-level decoding input sequence at the t-th time step is a sequence obtained by processing the input sequence corresponding to the t-th time step; when K>1, the K-level decoding input sequence at the t-th time step is the K-1-level decoding output sequence output by the K-1-th level decoder subnetwork at the t-th time step; The K-th level decoder sub-network performs parallel decoding based on the K-th level decoding input sequence at the t-th time step and the encoded hidden state sequence corresponding to the input text based on the masked attention mechanism, thereby obtaining the K-th level decoding output sequence of the K-th level decoder sub-network at the t-th time step; If K<P, the K-level decoding output sequence of the K-th level decoder sub-network at the t-th time step is used as the K+1-level decoding input sequence corresponding to the K+1-th level decoder sub-network at the t-th time step, and is decoded by the K+1-th level decoder sub-network; if K=P, the K-level decoding output sequence of the K-th level decoder sub-network at the t-th time step is used as the decoding hidden state sequence at the t-th time step.

5. The method according to claim 4, characterized in that The encoding process of the input text to obtain an encoded latent state sequence corresponding to the input text includes: Obtain an R-level encoding input sequence corresponding to the R-th level encoder sub-network, wherein, when R=1, the R-level encoding input sequence is a sequence obtained by processing the input text; when R>1, the R-level encoding input sequence is an R-1-level encoding output sequence output by the R-1-th level encoder sub-network; 1≤R≤S, S is the total number of encoder sub-networks; R and S are positive integers; The R-th level encoder sub-network performs encoding processing on the R-level encoded input sequence and outputs an R-level encoded output sequence; If R<S, the R-level encoding output sequence is used as the R+1-level encoding input sequence corresponding to the R+1-level encoder sub-network, and the R+1-level encoder sub-network encodes the R+1-level encoding input sequence; if R=S, the R-level encoding output sequence is used as the encoding hidden state sequence corresponding to the input text.

6. The method according to claim 2, characterized in that The decoder network includes a 1st-level decoder subnetwork to a Pth-level decoder subnetwork cascaded in sequence, where P is a positive integer; the encoding hidden state sequence includes a 1st-level to Pth-level encoding hidden state sequence; The decoder network performs parallel decoding based on a masked attention mechanism using an input sequence corresponding to the t time step and an encoded hidden state sequence corresponding to the input text to obtain a decoded hidden state sequence at the t time step, including: Obtain a K-level decoding input sequence corresponding to the K-th level decoder subnetwork; wherein K is a positive integer, 1≤K≤P; when K=1, the K-level decoding input sequence is a sequence obtained by processing the input sequence corresponding to the t-th time step; when K>1, the K-level decoding input sequence is the K-1-th level output sequence of the K-1-th level decoder subnetwork at the t-th time step; The K-th level decoder sub-network performs decoding processing based on the K-th level decoding input sequence and the K-th level encoding hidden state sequence based on the masked attention mechanism to obtain the K-th level decoding output sequence of the K-th level decoder sub-network at the t-th time step; If K<P, the K-level decoding output sequence at the t-th time step is used as the K+1-level decoding input sequence corresponding to the K+1-th level decoder sub-network, and the K+1-level decoder sub-network continues to perform decoding processing; if K=P, the K-level decoding output sequence at the t-th time step is used as the decoding hidden state sequence at the t-th time step.

7. The method according to claim 6, characterized in that The encoding process of the input text to obtain an encoded latent state sequence corresponding to the input text includes: Obtain a T-level encoding input sequence corresponding to the T-th level encoder subnetwork, where 1≤T≤P-1. When T=1, the T-level encoding input sequence is a 1-level encoding hidden state sequence obtained by embedding the input text; when T>1, the T-level encoding input sequence is a T-level encoding hidden state sequence output by the T-1-th level encoder network; The T-th level encoder sub-network performs encoding processing on the T-th level encoding input sequence to obtain a T+1-th level encoding hidden state sequence output by the T-th level encoder sub-network; If T<P-1, the T+1-level encoded hidden state sequence is used as the T+1-level encoded input sequence corresponding to the T+1-level encoder sub-network, and is encoded by the T+1-level encoder sub-network; if T=P-1, the 1-level encoded hidden state sequence to the P-level encoded hidden state sequence are used as the encoded hidden state sequence corresponding to the input text.

8. The method according to claim 6, characterized in that The K-th level decoder sub-network includes a K-th level mask attention layer and a K-th level decoding processing layer; The K-th level decoder sub-network performs decoding processing based on the K-th level decoding input sequence and the K-th level encoding hidden state sequence based on the masked attention mechanism to obtain the K-th level decoding output sequence of the K-th level decoder sub-network at the t-th time step, including: The K-th level masked attention layer determines the K-th level attention matrix corresponding to the t-th time step according to the K-th level decoded input sequence and the K-th level encoded hidden state sequence; The K-th level masked attention layer determines the K-level masked attention matrix corresponding to the t-th time step according to the K-level attention matrix corresponding to the t-th time step and the mask matrix corresponding to the t-th time step; the mask matrix corresponding to the t-th time step indicates the attention scores that need to be masked and the attention scores that do not need to be masked in the K-level attention matrix corresponding to the t-th time step; The K-level decoding processing layer performs processing according to the K-level masked attention matrix corresponding to the t-th time step, the K-th level decoding input sequence and the encoded hidden state sequence corresponding to the input text to obtain the K-level decoding output sequence of the K-th level decoder subnetwork at the t-th time step.

9. The method according to claim 8, characterized in that Before determining, by the K-th level masked attention layer, the K-th level masked attention matrix corresponding to the t-th time step based on the K-th level attention matrix corresponding to the t-th time step and the mask matrix corresponding to the t-th time step, the method further includes: For the target entity corresponding to each to-be-decoded word at the t-th time step, determine the non-related text corresponding to the to-be-decoded word at the t-th time step from the target text and the decoded words determined before the t-th time step; The mask matrix corresponding to the t time step is determined according to the corresponding positions of the words in the non-related text corresponding to the to-be-decoded words in the t step in the K-th level decoding input sequence and the K-level encoding hidden state sequence.

10. The method according to claim 1, characterized in that Before obtaining the grouping information of the N entities, the method further includes: Get the text length thresholds set for each of N entities; The maximum value among the text length thresholds corresponding to the N entities is used as the target length threshold; According to the text length threshold corresponding to each entity and the target length threshold, the N entities are divided into M groups to obtain the group information.

11. A device for extracting entity content, characterized in that: include: an acquisition module, configured to acquire grouping information of N entities, wherein the grouping information indicates that the N entities are divided into M groups and the entities included in each group, wherein M and N are positive integers, 2≤M<N; and the sum of text length thresholds corresponding to entities belonging to the same group is not greater than a target length threshold; An order determination module, configured to determine an extraction order corresponding to each group according to the grouping information; a concatenation module, configured to concatenate the target text with the identification text corresponding to each of the N entities to obtain an input text; An encoding module is used to encode the input text to obtain an encoded latent state sequence corresponding to the input text; A parallel decoding module is used to perform M-way parallel decoding on the encoded hidden state sequence corresponding to the input text in the extraction order based on the masked attention mechanism to obtain M-way output text, wherein one output text includes the entity content corresponding to all entities belonging to a group in the target text, and the text length of the entity content corresponding to each entity does not exceed the text length threshold corresponding to the entity.

12. An electronic device, characterized in that: include: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 10 is implemented.

13. A computer-readable storage medium having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by a processor, the method according to any one of claims 1 to 10 is implemented.

14. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images