A method of text processing, a method of entity recognition, and an apparatus

Abstract

The application discloses a text processing method, an entity recognition method and device, and is used for reducing the training cost of a model for recognizing entities in a specific field. The scheme provided by the application can be applied to any application scenario related to entity recognition. The text processing method comprises the following steps: obtaining a text to be processed. A semantic network is constructed according to the text to be processed. The semantic network comprises nodes and directed edges. The nodes are used for representing words in the text to be processed. The directed edges are used for connecting the nodes representing adjacent words in the text to be processed. The weight of the directed edge is used for indicating the probability of adjacent appearance of the adjacent words in the text to be processed. A network for recognizing target entities is obtained according to the semantic network and a background knowledge network. The background knowledge network is used for indicating the constituent elements of the target entities and the corresponding relationship between the constituent elements and the target nodes in the semantic network.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of named entity recognition, and particularly relates to a text processing method, an entity recognition method and device. BACKGROUND

[0002] Artificial intelligence (AI) is to use a digital computer or a machine controlled by a digital computer to simulate, extend and expand human intelligence, perceive environment, acquire knowledge and use the knowledge to obtain the best results. In other words, artificial intelligence is a branch of computer science, which attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making. The research in the field of artificial intelligence includes robots, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, AI basic theory, etc.

[0003] Named entity recognition, also known as entity recognition, is a basic task in natural language processing, and has a very wide range of applications. Named entity generally refers to an entity in a text that has a specific meaning or strong reference, usually including names, place names, organization names, dates and times, and proper nouns. The training of an entity recognition model often requires the use of a large-scale manually annotated dataset. However, for some specific fields (such as the financial field and the legal field), the cost of obtaining a manually annotated dataset is high. Therefore, how to reduce the training cost of an entity recognition model in a specific field needs to be solved urgently. SUMMARY

[0004] The embodiments of the present application provide a text processing method, an entity recognition method and device to reduce the training cost of an entity recognition model in a specific field.

[0005] To achieve the above object, the embodiments of the present application provide the following technical solutions:

[0006] In a first aspect, an embodiment of the present application provides a text processing method, including: a text processing device can acquire a to-be-processed text according to the requirement of entity recognition. The to-be-processed text in the embodiment of the present application refers to a text including a target entity. For example, if the entity of "contract validity period" is to be recognized, different types of contract texts can be used as the to-be-processed text, such as a large number of contracts including the contract validity period searched from the Internet, such as a rental contract text, a labor contract text, and the like. For another example, if the entity of "judgment result" is to be recognized, different types of judgment results can be used as the to-be-processed text, such as judgment documents including the judgment result searched from a website related to the judgment result. A semantic network is constructed according to the to-be-processed text, the semantic network is used to describe the context semantic environment of a word representing a target element, and is used to indicate the probability of adjacent words in the context of each word representing the target element. Specifically, the semantic network includes nodes and directed edges, the nodes are used to represent words in the to-be-processed text, the directed edges are used to connect nodes representing adjacent words in the to-be-processed text, and the weight of the directed edge is used to indicate the probability of adjacent occurrence of adjacent words in the to-be-processed text. A network used for recognizing the target entity is acquired according to the semantic network and a background knowledge network. For example, the nodes of the words representing the target element in the semantic network can be linked with the target element in the background knowledge network to acquire the network used for recognizing the target entity, and the words representing the target element are acquired by inputting the to-be-processed text into a general entity recognition model. The type of the general entity recognition model can be selected according to the type of the general entity to which the target element belongs, so that the general entity recognition model is used to recognize words of the same entity type as the target element. The to-be-processed text can be used as the input of the general entity recognition model to recognize the words of the target element from the to-be-processed text. The background knowledge network is used to indicate the target entity represented by the target element. The target element is also referred to as a component element in the present application, and both represent the same meaning. Thus, the recognition network of the target entity is constructed according to the background knowledge network and the semantic network. As can be seen from the first aspect, the embodiment of the present application does not need to use a large amount of to-be-processed texts in which the position of the target entity is labeled as training samples to train a model for recognizing the target entity. When the target element obtained after concept decomposition belongs to a general entity, the words representing the target element can be recognized from the training text by using an existing general entity recognition model. Then, the words representing the target element and the target element in the background knowledge network are established, and the network used for recognizing the target entity can be constructed. The embodiment of the present application only needs to use un-labeled texts including the target entity as training samples to acquire a model for recognizing the target entity, thereby reducing the cost of model training.

[0007] In a possible implementation manner of the first aspect, the to-be-processed text is traversed. A sub-semantic network is constructed according to the currently traversed to-be-processed text, the sub-semantic network including a second node and a second directed edge, the second node being used to represent a word in the currently traversed to-be-processed text, the second directed edge being used to connect a second source node and a second destination node, the word represented by the second source node and the word represented by the second destination node being adjacent words in the currently traversed to-be-processed text, and a weight of the second directed edge being obtained according to a probability of adjacent occurrence of the adjacent words in the currently traversed to-be-processed text. The plurality of sub-semantic networks are merged according to a first rule to construct a semantic network, the first rule including merging nodes representing the same word in the plurality of sub-semantic networks and merging the same directed edges in the plurality of sub-semantic networks, and the weight of the merged directed edge being positively correlated with the number of the same directed edges in the plurality of sub-semantic networks. In this implementation manner, a specific manner of constructing the semantic network is given, and the diversity of the scheme is increased.

[0008] In a possible implementation manner of the first aspect, the method further includes: performing updating processing on the semantic network according to a second rule to obtain an updated semantic network, the second rule including deleting low-weight directed edges in the semantic network and nodes not connected by the directed edges, the low-weight directed edge being a directed edge in the semantic network with a weight lower than a preset threshold. The recognition network is constructed according to the updated semantic network and the background knowledge network, for example, linking the node representing the word of the target element in the updated semantic network to the target element in the background knowledge network. In this implementation manner, the semantic network is optimized, and the semantic network is more concise.

[0009] In a possible implementation manner of the first aspect, the method further includes: replacing the node representing the word of the target element in the updated semantic network with a preset node. In this implementation manner, a general recognition network is provided, and when a specific entity recognition task is performed, the target entity can be recognized even if the word of the target element is inconsistent with the word of the target element in the training phase.

[0010] In a possible implementation manner of the first aspect, the method further includes: performing multiple clustering processes on the nodes included in the semantic network with each preset node as an initial clustering center until each node in the semantic network is added to a clustering community with each preset node as a clustering center, wherein any one of the multiple clustering processes includes: obtaining an increment of the modularity of each clustering community obtained after a previous clustering process if a to-be-processed node is added to the clustering community, the to-be-processed node being any one node in the semantic network except the preset nodes; and adding the to-be-processed node to the clustering community with the largest increment of the modularity to obtain each clustering community obtained after the current clustering process. In this implementation manner, the nodes included in the semantic network are clustered to obtain multiple clustering communities, and the clustering communities can better assist a technical expert in the field to determine which target element a node representing a word of a target element corresponds to a target entity.

[0011] In a possible implementation manner of the first aspect, the method further includes: receiving an instruction, the instruction being used to indicate a component element of the target entity and a corresponding relationship between the component element and a target node in the semantic network; and obtaining the background knowledge network according to the instruction.

[0012] In a possible implementation manner of the first aspect, the target element belongs to at least one of a general entity of time, a person name, a place name, an organization name, a date and time, or a proper noun.

[0013] In a second aspect, an embodiment of the present application provides a method for entity recognition, comprising: obtaining a to-be-processed text. At least one candidate path is constructed according to the to-be-processed text, each candidate path comprising a node and a directed edge, the node of the candidate path being used to represent a word in the to-be-processed text, the directed edge of the candidate path being used to connect the nodes representing adjacent words in the to-be-processed text, each candidate path comprising at least one target node, the word represented by the target node being obtained by inputting the to-be-processed text into a general entity recognition model, the general entity recognition model being used to recognize words belonging to the same entity type as the constituent elements of a target entity. The target entity in the to-be-processed text is obtained according to the target nodes included in a target path in the candidate path and a recognition network, the target path being the candidate path with the largest weight in the candidate paths, the weight of the target path being determined according to the weights of the directed edges included in the target path, the weight of the directed edge included in the target path being obtained according to the recognition network, the recognition network being obtained according to a semantic network and a background knowledge network, the semantic network being obtained according to training text, the semantic network comprising nodes and directed edges, the nodes of the semantic network being used to represent words in the training text, the directed edges of the semantic network being used to connect the nodes representing adjacent words in the training text, the weight of the directed edge of the semantic network being used to indicate the probability of adjacent appearance of the adjacent words in the training text, the background knowledge network being used to indicate the constituent elements of the target entity and the corresponding relationship between the constituent elements and the target nodes in the semantic network. As can be seen from the second aspect, the embodiment of the present application can complete the task of proposing a specific domain entity from a large amount of unstructured and unlabeled text, and can greatly reduce the cost and work efficiency of manual extraction.

[0014] In a possible implementation manner of the second aspect, the semantic network is obtained by merging processing a plurality of sub-semantic networks according to a first rule, the first rule comprising merging processing the nodes representing the same word in the plurality of sub-semantic networks and merging processing the target directed edges in the plurality of sub-semantic networks, the weight of the target directed edge being positively correlated with the number of the target directed edges in the plurality of sub-semantic networks, the target directed edge being the directed edge connecting the nodes representing the same adjacent word in the plurality of sub-semantic networks, and the sub-semantic network being constructed according to the currently traversed training text.

[0015] In a possible implementation manner of the second aspect, the recognition network is obtained according to the updated semantic network and the background knowledge network, the updated semantic network being obtained by updating processing the semantic network according to a second rule, the second rule comprising deleting the low-weight directed edges in the semantic network and the nodes not connected by the directed edges, and the low-weight directed edge being the directed edge with a weight lower than a preset threshold in the semantic network.

[0016] In a possible implementation manner of the second aspect, the target node in the updated semantic network is replaced by a preset node.

[0017] In a possible implementation manner of the second aspect, the semantic network comprises a plurality of aggregated communities, each of the plurality of aggregated communities is a cluster center with each preset node as an initial cluster center, and the nodes comprised in the semantic network are subjected to a plurality of clustering processes until each node in the semantic network is added to a cluster community with each preset node as a cluster center, wherein any one of the plurality of clustering processes comprises: obtaining an increment of a modularity of each aggregated community obtained after a previous clustering process if a to-be-processed node is added to each aggregated community obtained after the previous clustering process, the to-be-processed node being any one node in the semantic network except the preset nodes. The to-be-processed node is added to an aggregated community with the largest increment of the modularity to obtain each aggregated community obtained after the current clustering process.

[0018] In a possible implementation manner of the second aspect, the background knowledge network is obtained according to received instructions, the instructions being used to indicate a constituent element of the target entity and a corresponding relationship between the constituent element and a target node in the semantic network.

[0019] In a possible implementation manner of the second aspect, the target element belongs to at least one of a general entity of time, a general entity of a person name, a general entity of a place name, a general entity of an organization name, a general entity of a date and time, or a general entity of a proper noun.

[0020] In a possible implementation manner of the third aspect, the semantic network comprises a plurality of aggregated communities, each of the plurality of aggregated communities is a cluster center with each preset node as an initial cluster center, and the nodes comprised in the semantic network are subjected to a plurality of clustering processes until each node in the semantic network is added to a cluster community with each preset node as a cluster center, wherein any one of the plurality of clustering processes comprises: obtaining an increment of a modularity of each aggregated community obtained after a previous clustering process if a to-be-processed node is added to each aggregated community obtained after the previous clustering process, the to-be-processed node being any one node in the semantic network except the preset nodes. The to-be-processed node is added to an aggregated community with the largest increment of the modularity to obtain each aggregated community obtained after the current clustering process.

[0021] In a possible implementation manner of the third aspect, the semantic network constructing module is specifically configured to: traverse the to-be-processed text; construct a sub semantic network according to the to-be-processed text currently traversed, the sub semantic network including a second node and a second directed edge, the second node being used to represent a word in the to-be-processed text currently traversed, the second directed edge being used to connect a second source node and a second destination node, the word represented by the second source node and the word represented by the second destination node being adjacent words in the to-be-processed text currently traversed, and a weight of the second directed edge being obtained according to a probability of adjacent occurrence of the adjacent words in the to-be-processed text currently traversed; and perform merging processing on a plurality of the sub semantic networks according to a first rule to construct the semantic network, the first rule including performing merging processing on nodes representing the same word in the plurality of the sub semantic networks, and performing merging processing on the same directed edge in the plurality of the sub semantic networks, and the weight of the merged directed edge being positively correlated with a number of the same directed edge in the plurality of the sub semantic networks.

[0022] In a possible implementation manner of the third aspect, the method further includes an updating module configured to perform updating processing on the semantic network according to a second rule to obtain an updated semantic network, the second rule including deleting a low-weight directed edge in the semantic network and the nodes not connected by the directed edge, the low-weight directed edge being the directed edge with a weight lower than a preset threshold in the semantic network.

[0023] In a possible implementation manner of the third aspect, the method further includes a replacing module configured to replace the target node in the updated semantic network with a preset node.

[0024] In a possible implementation manner of the third aspect, the method further includes a clustering module configured to: perform multiple times of clustering processing on the nodes included in the semantic network with each preset node as an initial clustering center until each node in the semantic network is added to a clustering community with each preset node as a clustering center, wherein any one of the multiple times of clustering processing includes: obtaining an increment of a modularity of each clustering community obtained after last clustering processing if a to-be-processed node is added to the clustering community, the to-be-processed node being any one of the nodes in the semantic network except the preset nodes; and adding the to-be-processed node to the clustering community with the largest increment of the modularity to obtain each clustering community obtained after the current clustering processing.

[0025] In a possible implementation manner of the third aspect, the method further includes a receiving module configured to receive an instruction, the instruction being used to indicate a constituent element of the target entity and a correspondence between the constituent element and a target node in the semantic network.

[0026] In a possible implementation manner of the third aspect, the target element belongs to at least one general entity of time, person name, place name, organization name, date and time, or proper noun.

[0027] The fourth aspect provides a device for entity recognition, including: an acquisition module configured to acquire a to-be-processed text; a construction module configured to construct at least one candidate path according to the to-be-processed text, each candidate path including nodes and directed edges, wherein the nodes of the candidate path are used to represent words in the to-be-processed text, the directed edges of the candidate path are used to connect the nodes representing adjacent words in the to-be-processed text, and each candidate path includes at least one target node, the word represented by the target node is acquired by inputting the to-be-processed text into a general entity recognition model, and the general entity recognition model is used to identify words belonging to the same entity type as a constituent element of a target entity; and an identification module configured to acquire the target entity in the to-be-processed text according to the target nodes included in a target path in the candidate paths and an identification network, the target path being a candidate path with the largest weight in the candidate paths, the weight of the target path being determined according to the weights of the directed edges included in the target path, and the weights of the directed edges included in the target path being acquired according to the identification network, the identification network being acquired according to a semantic network and a background knowledge network, the semantic network being acquired according to training text, the semantic network including nodes and directed edges, the nodes of the semantic network being used to represent words in the training text, the directed edges of the semantic network being used to connect the nodes representing adjacent words in the training text, the weights of the directed edges of the semantic network being used to indicate the probability of adjacent appearance of the adjacent words in the training text, the background knowledge network being used to indicate constituent elements of the target entity, and the corresponding relationship between the constituent elements and target nodes in the semantic network.

[0028] In a possible implementation manner of the fourth aspect, the semantic network is acquired by performing merging processing on a plurality of sub semantic networks according to a first rule, the first rule including performing merging processing on nodes representing the same word in the plurality of sub semantic networks and performing merging processing on target directed edges in the plurality of sub semantic networks, the weight of the target directed edge being positively correlated with the number of target directed edges in the plurality of sub semantic networks, the target directed edge being a directed edge connecting the same adjacent words in the plurality of sub semantic networks, and the sub semantic network being constructed according to the currently traversed training text.

[0029] In a possible implementation manner of the fourth aspect, the identification network is obtained according to the updated semantic network and the background knowledge network, for example, is obtained after linking a node representing a target element in the updated semantic network with the target element in the background knowledge network, the updated semantic network is obtained after updating the semantic network according to a second rule, and the second rule includes deleting low-weight directed edges in the semantic network and nodes not connected by the directed edges, and the low-weight directed edge is a directed edge with a weight lower than a preset threshold in the semantic network.

[0030] In a possible implementation manner of the fourth aspect, the target node in the identification network is replaced by a preset node.

[0031] In a possible implementation manner of the fourth aspect, the semantic network includes a plurality of aggregated communities, the plurality of aggregated communities are obtained by performing a plurality of clustering processes on nodes included in the semantic network, taking each preset node as an initial clustering center, and each node in the semantic network is added to a clustering community with a preset node as a clustering center, wherein any one of the plurality of clustering processes includes: obtaining an increment of a modularity of each aggregated community obtained after a previous clustering process if a to-be-processed node is added to the aggregated community, and the to-be-processed node is any one node in the semantic network except the preset node. The to-be-processed node is added to an aggregated community with the largest increment of the modularity to obtain each aggregated community obtained after the current clustering process.

[0032] In a possible implementation manner of the fourth aspect, the identification network is obtained according to the received instruction, and the instruction is used to indicate a composition element of the target entity and a correspondence between the composition element and the target node in the semantic network.

[0033] In a possible implementation manner of the fourth aspect, the target element belongs to at least one of the following general entities: time, personal name, place name, organization name, date and time, or proper noun.

[0034] In the fifth aspect, an embodiment of the present application provides a text processing apparatus, including a memory configured to store computer readable instructions. The apparatus further includes a processor coupled to the memory and configured to execute the computer readable instructions in the memory to perform the method described in the first aspect or any possible implementation manner of the first aspect.

[0035] In the sixth aspect, an embodiment of the present application provides an entity identification apparatus, including a memory configured to store computer readable instructions. The apparatus further includes a processor coupled to the memory and configured to execute the computer readable instructions in the memory to perform the method described in the second aspect or any possible implementation manner of the second aspect.

[0036] In a seventh aspect, an embodiment of the present application provides a computer readable storage medium, including instructions, when the instructions are run on a computer device, causing the computer device to perform the method described in the first aspect or any possible implementation of the first aspect.

[0037] In an eighth aspect, an embodiment of the present application provides a computer readable storage medium, including instructions, when the instructions are run on a computer device, causing the computer device to perform the method described in the second aspect or any possible implementation of the second aspect.

[0038] In a ninth aspect, an embodiment of the present application provides a chip, coupled with a memory, for executing a program stored in the memory, to perform the method described in the first aspect or any possible implementation of the first aspect.

[0039] In a tenth aspect, an embodiment of the present application provides a chip, coupled with a memory, for executing a program stored in the memory, to perform the method described in the second aspect or any possible implementation of the second aspect.

[0040] In an eleventh aspect, an embodiment of the present application provides a computer program product including instructions, when the instructions are run on a computer, causing the computer to perform the method described in the first aspect or any possible implementation of the first aspect.

[0041] In a twelfth aspect, an embodiment of the present application provides a computer program product including instructions, when the instructions are run on a computer, causing the computer to perform the method described in the second aspect or any possible implementation of the second aspect.

[0042] The beneficial effects brought by the third aspect, the fifth aspect, the seventh aspect, the ninth aspect, the eleventh aspect and various implementations of each aspect of the present application can refer to the beneficial effects brought by the first aspect and various implementations of the first aspect.

[0043] The beneficial effects brought by the fourth aspect, the sixth aspect, the eighth aspect, the tenth aspect, the twelfth aspect and various implementations of each aspect of the present application can refer to the beneficial effects brought by the second aspect and various implementations of the second aspect.

[0044] The scheme provided by the embodiment of the application performs concept decomposition on entities in a specific field, and represents the entities in the specific field through target elements. When the target elements belong to general entities, a general entity recognition model can be used to identify the positions of the words corresponding to the target elements in the text to be processed, i.e., to identify the instance words corresponding to the general entities in the text to be processed. The context of the instance words in the text to be processed is counted, and technical experts in the field can correspond the instance words to the target elements. According to the decomposition relationship between the target elements and the entities in the specific field, appropriate instance words can be selected from the instance words to form the text corresponding to the entities in the specific field. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 A schematic diagram of an acquisition system for performing an entity recognition task model;

[0046] Figure 2a A schematic diagram of concept decomposition on entities in a specific field;

[0047] Figure 2b A schematic diagram of concept decomposition on entities in a specific field;

[0048] Figure 3 A schematic diagram of background knowledge network;

[0049] Figure 4 A schematic diagram of background knowledge network;

[0050] Figure 5 A schematic diagram of a text processing method provided by the embodiment of the application;

[0051] Figure 6a A schematic diagram of a text processing method provided by the embodiment of the application;

[0052] Figure 6b A schematic diagram of another text processing method provided by the embodiment of the application;

[0053] Figure 7 A schematic diagram of another text processing method provided by the embodiment of the application;

[0054] Figure 8 A schematic diagram of another text processing method provided by the embodiment of the application;

[0055] Figure 9 A schematic diagram of another text processing method provided by the embodiment of the application;

[0056] Figure 10 A schematic diagram of another text processing method provided by the embodiment of the application;

[0057] Figure 11 A scenario diagram of another text processing method provided by an embodiment of the present application is shown in FIG. 3.

[0058] Figure 12 A scenario diagram of another text processing method provided by an embodiment of the present application is shown in FIG. 3.

[0059] Figure 13 A flow diagram of an entity recognition method provided by an embodiment of the present application is shown in FIG. 4.

[0060] Figure 14 A scenario diagram of an entity recognition method provided by an embodiment of the present application is shown in FIG. 5.

[0061] Figure 15 A structure diagram of a computer device provided by an embodiment of the present application is shown in FIG. 6.

[0062] Figure 16 A structure diagram of another computer device provided by an embodiment of the present application is shown in FIG. 7. DETAILED DESCRIPTION

[0063] The embodiments of the present application are described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments. Those skilled in the art can know that with the development of technology and the emergence of new scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0064] The present application provides a text processing method, an entity recognition method and a device. The network obtained by the text processing method provided by the present application can be used to recognize entities in a specific field, and improve the accuracy of entity recognition in the specific field.

[0065] An entity (also referred to as a named entity) can be understood as a collection of a certain type of things. The certain type of things can be a specific person or an abstract concept. For example, the certain type of things is Zhang San, Li Si and Wang Er, which can be considered as a collection of the type of things of names, and the name can be regarded as an entity. For example, the certain type of things is Shijiazhuang, Shenzhen and Beijing, which can be considered as a collection of the type of things of place names, and the place name can be regarded as an entity. For example, the two concepts of contract start time and contract end time belong to the constituent elements of contract validity period, and the contract validity period can be regarded as an entity.

[0066] General entities can be understood as common entities. These typically include names of people, places, organizations, quantities, dates and times, proper nouns, etc. Existing technologies have mature entity recognition models that can accurately identify these entities. In other words, existing technologies readily provide a large amount of training text labeled with the locations of certain types of entities, and use this training text to train entity recognition models, enabling the trained models to accurately identify these entities. This application refers to these entities as general entities, and the model that identifies these entities as a general entity recognition model.

[0067] Entity recognition is a fundamental task in natural language processing. It refers to the process of identifying instances corresponding to a specified entity from text. Instances in text can take various forms, typically words or phrases. For example, if the entity is time, a specific instance could be the phrase "March 25th" or the word "today." This application does not limit the specific form of an instance. This application sometimes refers to the specific form of an entity as an instance or instance word, or simply as an entity. For ease of explanation, this application sometimes refers to identifying instances corresponding to a specified entity from text, simply as identifying entities from text. Those skilled in the art should understand this expression, and it will not be repeated hereafter.

[0068] Named entity recognition (NER) models (also known as named entity recognition systems) can extract specified entities from unstructured text. By automatically extracting necessary information from massive amounts of unstructured text using NER models, we can effectively reduce manual labor, improve work efficiency, and minimize human error.

[0069] like Figure 1 The diagram shown is a schematic of the system for acquiring the model for performing entity recognition tasks. Figure 1As shown, the obtaining system for performing the entity recognition task model comprises an execution device 210, a training device 220, and a database 230. In the training phase, the database 230 stores training texts, each of which is labeled with the positions of each entity to be recognized in each training text. Generally, these original training texts need to be manually labeled by human beings. The training device 220 generates a model / rule 201 and iteratively trains the model / rule 201 using a set of training data in the database to obtain a mature model / rule 201. It should be noted that the database 230 can be in any form of storage medium and is not limited to a traditional database. When performing the entity recognition task according to the mature model / rule 201, the execution device 210 can call the model / rule 201, take the text to be recognized as the input of the mature model / rule 201, and obtain the entity to be recognized in the text to be recognized according to the output of the mature model / rule 201. In some embodiments of the present application, for example Figure 1 In the above description, the “user” can directly interact with the execution device 210, that is, the execution device 210 and the client device are integrated in the same device. As an example, in some application scenarios, the execution device 210 can represent a terminal device, and when performing the entity recognition task according to the mature model / rule 201, the execution device 210 can take the text input by the user as the input of the mature model / rule 201. In some application scenarios, the user input can also be voice data or image data, and the execution device 210 can convert the voice data or image data into text data and take the text data as the input of the mature model / rule 201.

[0070] The execution device 210 can also be in other forms, which are not listed here. In some scenarios, the execution device 210 and the client device can be separate devices, and the execution device 210 is configured with an input / output interface to interact with the client device. The “user” can input at least one task to the execution device 210 through the input / output interface of the client device, and the execution device 210 returns the processing result to the client device through the input / output interface.

[0071] The performance of the entity recognition task model is closely related to the number of training texts, which can be reflected in that the more the number of training texts, the more iterations of training can be performed on the model / rules 201. In turn, the model / rules 201 can better learn how to identify the to-be-identified entity in the text according to the position of the to-be-identified entity manually annotated in the training text. However, for some texts in specific fields (such as the financial field and the legal field), the training samples annotated with entity positions in the text are limited, and it is difficult to train an entity recognition model with excellent performance using these few training samples. In addition, annotating texts in these fields may require a large amount of manpower and material resources, such as to-be-identified entities that may be distributed in different positions of unstructured texts, which may require professionals in these fields to participate in annotation. Therefore, how to improve the performance of the entity recognition task model so that the entity recognition task model also has good effects on the recognition of entities in these specific field texts is a problem to be solved.

[0072] To solve the above problems, the present application provides a text processing method and an entity recognition method. Through the text processing method provided by the present application, a network for identifying entities in a specific field can be obtained without relying on a large number of training samples. For the network obtained by the text processing method, the present application also provides a method for performing entity recognition according to the network. The process of obtaining the network and the process of applying the network to perform entity recognition are described below.

[0073] I. Obtaining an entity recognition network

[0074] When obtaining a network for identifying entities in a specific field, the present application utilizes a model that can identify general entities. In order to utilize the model that can identify general entities, the present application provides a scheme that conceptually disassembles entities in a specific field, and represents the entities in a specific field by a plurality of elements. When the plurality of elements all belong to general entities, the scheme provided by the present application can utilize the model that can identify general entities.

[0075] Among them, the network is also referred to as the model by the present application, and both represent the same meaning. General entities usually include names, place names, organization names, dates and times, quantities, and proper nouns. Training texts for the model that can identify general entities are easier to obtain. Sufficient training samples enable the model that can identify general entities to have good performance. The present application also refers to entities in a specific field as target entities, which means entities that a user hopes to identify from a text. For example, a user hopes to identify the entity “contract validity period” from a text, and the “contract validity period” is the target entity.

[0076] The following will combine a specific example to explain how to conceptually disassemble entities in a specific field to represent entities in a specific field by multiple elements. Referring to Figure 2a , assume that the target entity is "contract validity period", the elements obtained by conceptually disassembling the "contract validity period" entity can be "contract start date" and "contract end date", that is, the concept of "contract validity period" can include the two elements of "contract start date" and "contract end date". Both "contract start date" and "contract end date" belong to the general entity "time". At this point, "contract validity period" is disassembled into multiple elements, and the multiple elements belong to the general entity, so the identification of the "contract validity period" entity can use the model for identifying the "time" entity. How to use the model for identifying the "time" entity will be described later, which will not be described here. It should be noted that the target entity can be conceptually disassembled in many different ways. For example, as shown in Figure 2b , the "contract validity period" entity can also be disassembled into "contract start date" and "contract duration". The "contract start date" element belongs to the general entity "time", and the "contract duration" belongs to the general entity "quantity". Therefore, the identification of the "contract validity period" entity can use the model for identifying the "time" entity, and can also use the model for identifying the "quantity" entity. How to use the model for identifying the "time" entity and how to use the model for identifying the "quantity" entity will be described later, which will not be described here. In order to fully utilize the model for identifying general entities, the target entity can be reasonably conceptually disassembled in as many ways as possible, and the experience of experts can be used to participate in the conceptual disassembly process of the target entity by experts in a specific field.

[0077] The above example of conceptually disassembling the "contract validity period" entity obtains elements that belong to general entities through disassembly. It should be noted that for some complex entities, the target entity may need to be conceptually disassembled multiple times, and the number of times of conceptually disassembling the target entity by the embodiments of the present application is not limited. The disassembly method of the target concept can be determined according to the actual problem. In addition, it should be noted that for some target entities, it can be difficult to exhaust all conceptually disassembled ways to make the elements of the disassembled target entities belong to general entities, and the target entities in the embodiments of the present application do not include this type of target entities.

[0078] After conceptually decomposing the target entity using various decomposition methods, the target entity can be represented by multiple sets of general entities, thus forming an element network of the target entity. This application also refers to the element network as a background knowledge network, with both meaning the same thing. The following section combines... Figure 3 and Figure 4 Please provide an explanation. For example... Figure 3 As shown, two conceptual decomposition methods for a target entity are illustrated. In practical applications, there may be multiple conceptual decomposition methods for a target entity. This application does not limit this approach; only two decomposition methods are used as examples. Figure 3 As shown, one way to decompose the concept is to break down the target entity into element 1 and element 2, and further break down element 1 into element 1.1 and element 1.2. Element 1.1 belongs to general entity 1, while elements 1.2 and 2 belong to general entity 2; in other words, elements 1.2 and 2 belong to the same general entity. Another way to decompose the concept is to break down the target entity into element 2 and element 3, where element 3 belongs to general entity 3. To better understand the background knowledge network, we will continue to use the entity "contract validity period" as an example for explanation. Figure 4 As shown, the process of decomposing the entity "contract validity period" has been explained in the example above, and will not be repeated here. Only one possible background knowledge network of the entity "contract validity period" will be shown.

[0079] After conceptually decomposing the target entity, it can be represented by multiple general entities. For general entities, mature entity recognition models already exist that can effectively identify them. These mature entity recognition models can be used to identify the location of each element in the text to be processed, and the target entity can be obtained using these locations. The following section explains how to obtain the target entity using the locations of these elements in the text to be processed, using a specific implementation method.

[0080] See Figure 5 This is a flowchart illustrating a text processing method provided in the application embodiment.

[0081] like Figure 5 As shown in the embodiment of this application, a text processing method includes the following steps:

[0082] 501. Obtain the text to be processed.

[0083] The to-be-processed text can be obtained according to the requirement of entity recognition. The to-be-processed text in the embodiment of the present application refers to a text including a target entity. For example, if the entity of "contract validity period" is to be recognized, different types of contract texts can be used as to-be-processed texts, such as a large number of contracts including contract validity periods searched from the Internet, such as rental contract texts, labor contract texts, etc. For another example, if the entity of "judgment result" is to be recognized, different types of judgment results can be used as to-be-processed texts, such as judgment documents including judgment results searched from websites related to judgment results. It should be noted that the embodiment of the present application does not limit the manner of obtaining the to-be-processed text.

[0084] In addition, it should be noted that the to-be-processed text in the embodiment of the present application can be an unlabeled text. In other words, the to-be-processed text in the embodiment of the present application does not need to mark the position of the target entity in the to-be-processed text. Therefore, the difficulty of obtaining training texts can be greatly reduced, and the human and material resources required for obtaining training texts in the training stage can be reduced.

[0085] 502, taking the to-be-processed text as an input of a general entity recognition model to recognize general entities from the to-be-processed text.

[0086] The general entity recognition model is selected according to the general entities constituting the target entity. For example, if the target entity is "contract validity period", the general entities constituting the target entity include "contract" and "validity period". Therefore, the general entity recognition model that can recognize "contract" and "validity period" is selected. Figure 3 Taking the background knowledge network of the target entity shown as an example, the general entities constituting the target entity include general entity 1, general entity 2, and general entity 3. Therefore, the general entity recognition model that can recognize general entity 1, general entity 2, and general entity 3 is selected. It should be noted that the general entity recognition model can include one or more, and the embodiment of the present application does not limit this.

[0087] For example, in one possible implementation, a general entity recognition model can be used to recognize the general entities constituting the target entity in the to-be-processed text. Specifically, when there is a mature model that can recognize each general entity constituting the target entity, the embodiment of the present application can directly use it. When the model that can recognize each general entity constituting the target entity cannot be directly obtained, a model that can recognize each general entity constituting the target entity can also be trained. For example, corresponding to the target entity of "contract validity period", a model that can recognize "contract" and "validity period" can be trained. Figure 3In the illustrated example, a general entity recognition model can be trained, which can recognize general entity 1, general entity 2, and general entity 3 at the same time. As to how to train a model that can recognize each general entity that constitutes a target entity, those skilled in the art can adopt different manners according to actual needs, and the embodiments of the present application do not limit this. Compared with the difficulty and cost of obtaining training text labeled with a target entity, the difficulty and cost of obtaining training text of each general entity that constitutes a target entity will be greatly reduced.

[0088] In addition, in a possible implementation, the general entities that constitute a target entity in the to-be-processed text can be recognized by multiple general entity recognition models. For example, general entity recognition model 1 can recognize general entity 1, general entity recognition model 2 can recognize general entity 2, and general entity recognition model 3 can recognize general entity 3. In the example, the to-be-processed text can be input into entity recognition model 1 to recognize general entity 1 in the to-be-processed text, the output of entity recognition model 1 can be input into entity recognition model 2 to recognize general entity 2 in the to-be-processed text, and the output of entity recognition model 2 can be input into entity recognition model 3 to recognize general entity 3 in the to-be-processed text. In this way, the recognition of general entity 1, general entity 2, and general entity 3 in the to-be-processed text is completed. It should be noted that in the above example, the to-be-processed text is input into general entity recognition model 1 first, then input into general entity recognition model 2, and then input into general entity recognition model 3. This order is only for the convenience of description, and is intended to indicate that the to-be-processed text can be processed by multiple general entity recognition models to recognize each general entity that constitutes a target entity in the to-be-processed text, and is not limited.

[0089] The selection of the reference entity is determined according to the general entity to which the element obtained after the conceptualization of the target entity belongs. For example, still taking the above target entity “contract has a valid period” as an example, the elements obtained after the conceptualization of the target entity “contract valid period” can include “contract start date”, “contract end date”, and “contract duration”. Among them, “contract start date” and “contract end date” belong to the general entity “time”, and “contract duration” belongs to the general entity “quantity”. Therefore, it can be determined that the reference entity includes “time” and “quantity”. The entity recognition model can include one entity recognition model that can recognize both the entity “time” and the entity “quantity”. Or the entity recognition model can include two entity recognition models, one of which recognizes the entity “time” and the other of which recognizes the entity “quantity”. The number of entity recognition models is not limited by the embodiments of the present application.

[0090] For better understanding of the embodiments of the present application, the embodiments of the present application are further illustrated in combination with the examples shown in the following. Figure 4 The general entity constituting the "contract validity period" includes time and quantity. Then, a general entity recognition model for recognizing the two entities of time and quantity is selected. For example, a model for recognizing both the entities of time and quantity can be selected; or two general entity recognition models can be selected, one of which is for recognizing the entity of time and the other of which is for recognizing the entity of quantity.

[0091] The instance words of the same entity in the text to be processed are represented by the same label. It is assumed that the general entity recognition model for recognizing the entity of time is selected. The text to be processed is taken as the input of the general entity recognition model to recognize the instance words of the entity of time from the text to be processed. As shown in Figure 6a , the recognition results of the instance words of the entity of time in a training sample are shown. As shown in Figure 6b , a schematic diagram for replacing the recognized instance words of the entity of time with the label date is shown. It should be noted that the date label herein is only illustrative and does not represent the limitation of the embodiments of the present application, and the embodiments of the present application do not limit the representation of the label. It should be noted that different labels are used to replace the instances of different entities. For example, the instance words of the entity of quantity are replaced by num, and the like. The instance words in the embodiments of the present application represent specific words, phrases or labels, which will not be repeated herein.

[0092] The text to be processed replaced by the label is processed to obtain the text to be processed represented by words (or phrases). It should be noted that the word is expanded in the present application, that is, the word includes the word and the phrase, which will not be repeated in the present application. In one possible implementation, the text to be processed can be processed based on a professional dictionary in the art to obtain the text to be processed represented by words. It should be noted that the skilled person in the art can use any possible implementation to process the text to be processed to obtain the text to be processed represented by words, and the embodiments of the present application do not limit it. For example, according to the Ngram method, the fixed phrase or word in the text to be processed can be identified according to the phrase cohesion degree and the freedom degree outside the phrase, and then the text to be processed is processed. In one possible implementation, after the text to be processed is processed, the stop words in the text to be processed after the processing can be deleted. For example, some words such as mood words, auxiliary words and the like have no meaning for the context environment description of the general entity.

[0093] 503. According to each general entity in the text to be processed, a semantic network is constructed.

[0094] By executing step 502, the position of each general entity constituting the target entity in the text to be processed can be obtained. According to the context of the general entity in the text to be processed, a semantic network can be constructed. The semantic network is used to describe the context semantic environment of each general entity, and is used to indicate the probability of adjacent words in the context of each general entity. According to the context of the general entity, the technical expert in the art can determine which element of the target entity the general entity corresponds to. For example, the words with high frequency in the context of the general entity mean that these words usually appear synchronously with the general entity in the text to be processed, which means that the correlation between these words and the general entity is great. The technical expert in the art is familiar with which words will often appear in the context of each element constituting the target entity. When the words with high frequency in the context of a general entity are obtained, the technical expert in the art can associate the general entity with the element of the target entity.

[0095] After obtaining the text to be processed represented by words, the frequency of each word or adjacent word in the context of the general entity can be counted to obtain the words strongly related to the general entity. The technical expert can determine which element of the target entity the general entity belongs to according to the strongly related words.

[0096] In a possible implementation, the to-be-processed text represented by the words (hereinafter referred to as the to-be-processed text) can be traversed. Specifically, the to-be-processed text can be traversed in units of sentences, in units of a fixed number of words, or in units of paragraphs, or in units of each training text, and the embodiments of the present application do not limit this. A reference path is constructed according to the currently traversed to-be-processed text, the reference path includes nodes and directed edges, the nodes are used to represent different words in the currently traversed to-be-processed text, and the directed edges are used to connect adjacent words in the currently traversed to-be-processed text, and the directed edges are used to represent the order of different words in the to-be-processed text. According to the first rule, the plurality of reference paths are merged to obtain at least one path, and the first rule indicates that if the adjacent words in the currently traversed to-be-processed text appear in the historically traversed to-be-processed text, the target directed edge of the currently constructed reference path is merged with the target directed edge of the historically constructed reference path, the target directed edge is used to connect the repeatedly appearing adjacent words, and the weight of the directed edge connecting the adjacent words is used to represent the probability of the repeatedly appearing adjacent words. In order to better understand this process, a specific example is used to illustrate this. As shown in Figure 7 , it is assumed that the currently traversed to-be-processed text is text 1, and text 1 is the text obtained after the word segmentation processing. According to the words in text 1 and the order of the words in the text before the word segmentation processing, a reference path 1 is constructed. It should be noted that the word segmentation processing does not change the order of the words in the to-be-processed text. The reference path can be constructed according to the order of the words in the text after the word segmentation processing. For example, the first word in text 1 is the agreement, the second word is both parties, the third word is signed, the fourth word is date, the fifth word is effective, the sixth word is date, and the seventh word is terminated. According to text 1, a reference path 1 is constructed, the reference path 1 includes 7 nodes, which are used to represent the 7 words in text 1 in turn, each directed edge is used to connect two adjacent words, and each directed edge can be used to indicate the order of the 7 words in text 1, and the initial weight of each directed edge is 1. The reference path 1 can be understood with reference to Figure 7 . It is assumed that the next traversed to-be-processed text is text 2, the first word in text 2 is date, the second word is effective, the third word is date, and the fourth word is terminated. According to the method of constructing the reference path 1, a reference path 2 can be constructed. Since the reference path 1 and the reference path 2 have the same directed edges, the same directed edges are merged, and the weight on the directed edge is increased. As shown in Figure 7As shown in the diagram, the directed edges connecting the "date" node and "effective" in reference path 1 and reference path 2 are duplicates, as are the directed edges connecting the "effective" node and "date" and the directed edge connecting the "date" node and "termination". Therefore, these three directed edges are merged, and their weights are increased accordingly. The weight of a directed edge represents the probability that adjacent words appear repeatedly. For example, the weight of the directed edge connecting the "sign" node and the "date" node is 1, and the weight of the directed edge connecting the "date" node and the "effective" node is 2, indicating that the probability of the adjacent words "date" and "effective" appearing repeatedly is greater than the probability of "sign" and "date" appearing repeatedly.

[0097] In one possible implementation, after traversing all the text to be processed, identical nodes in the path, except for common entities, can be merged to optimize the semantic network. For example... Figure 8 As shown, assuming that after traversing all the text to be processed and completing the merging of directed edges, the following is obtained: Figure 8 The examples show paths 1 and 2. Besides the common entity, paths 1 and 2 share the same nodes: "Contract," "Effective," and "Terminated." Therefore, the common nodes among these three nodes in paths 1 and 2 are merged, resulting in the following merged path: Figure 8 As shown.

[0098] In one possible implementation, the semantic network can be further optimized. Since not every instance word of a general entity identified in the text to be processed may be associated with an element that makes up the target entity, some identified instance words of general entities may be instance words of general entities unrelated to the elements that make up the target entity. For example, see [link to documentation]. Figure 9 Assuming the target entity is the contract validity period, the concept decomposition process of contract validity period has already been illustrated above. This will be further explained using the above example. After decomposition, a general entity recognition model for identifying the entities of time and quantity was selected. The text to be processed is used as input to this general entity recognition model to identify various instance words of the entity of time and various instance words of the entity of quantity from the text, such as... Figure 6aAs shown, the recognition results of the instance words of the entity of time and the instance words of the entity of quantity in a training sample are shown. It can be found that the instance word "888" is used to describe a street, and the instance word "1000" is used to describe a building area, which are irrelevant to the disassembled elements of the contract validity period. Therefore, in order to reduce the instance words of the general entity irrelevant to the disassembled elements in the semantic network, further optimization processing needs to be performed on the semantic network. Since the embodiment of the present application selects the text including the target entity when selecting the text to be processed. Therefore, in a large number of texts to be processed, the number of instance words of the general entity related to the elements constituting the target entity is usually much more than the number of instance words of the general entity irrelevant to the elements constituting the target entity. Therefore, the context of the instance words of the general entity related to the elements constituting the target entity is richer. In other words, the number of adjacent words of the instance words of the general entity related to the elements constituting the target entity is more, that is, the probability that the adjacent words include the instance words of the general entity related to the elements constituting the target entity is higher than the probability that the adjacent words include the instance words of the general entity irrelevant to the elements constituting the target entity.

[0099] Based on this idea, the semantic network can be further optimized. For example, in a possible implementation, if the sum of the weights of all directed edges directly connecting the instance words of the general entity is lower than a preset threshold, the node corresponding to the instance words of the general entity is deleted from the semantic network. In addition, in a possible implementation, other nodes closely related to the deleted instance words of the general entity are also deleted from the semantic network. The other nodes closely related to the deleted instance words of the general entity can be set in different ways. For example, assuming that the deleted instance words of the general entity are instance word 1, and the semantic network further includes other instance words of the general entity, such as instance word 2 and instance word 3. If, after deleting instance word 1 from the semantic network, a certain word node cannot be directly or indirectly connected to the nodes of other instance words of the general entity, it is considered that the word node is closely related to instance word 1 and should be deleted. In order to better understand this embodiment, the following is described in combination with an example. Figure 10 Figure 10 is the semantic network to be optimized, in order to distinguish different instance words, here Figure 10 ​Each of the example words in the semantic network is numbered, respectively date1, date2, date3 and date4. Assuming that the sum of the weights of all directed edges directly connecting the example words of a general entity is not less than 50, date1 does not satisfy this condition, and date1 should be deleted from the semantic network. When date1 is deleted, node A, node B and node C cannot be directly or indirectly connected to date2, date3 and date4, so node A, node B and node C will also be deleted. Node F is directly connected to date2, and node D and node E can indirectly connect to date2 through node F, so node D, node E and node F can be retained. It should be noted that in addition to this way of optimizing the semantic network, other optimization methods can also be used based on the above idea.

[0100] 504, aggregate the semantic network to obtain a plurality of aggregated communities.

[0101] The words in the semantic network are aggregated to obtain a plurality of aggregated communities. Each initial aggregated community can include a node corresponding to the label of an example word of a general entity. Aggregating the words in the semantic network can better obtain the context semantic environment of each general entity in the text to be processed. The scheme provided by the embodiments of the present application can aggregate the words in the semantic network in various ways, which will be described below.

[0102] In one possible implementation, each word in the semantic network except the example words of the general entities can be traversed. It should be noted that the example words of the general entities in the semantic network have been represented by labels, and the example words in the semantic network in the embodiments of the present application refer to the example words represented by labels. In addition, each word in the semantic network is represented by a node, and traversing the words in the semantic network and traversing the nodes in the semantic network represent the same meaning. After the currently traversed word is added to each aggregated community, the increment of the modularity of each aggregated community will be added to the aggregated community that will make the increment of the modularity of the aggregated community the largest, and then the aggregated community will be updated. The above process is repeatedly executed until each word in the semantic network is added to an aggregated community. It should be noted that traversing each word in the semantic network except the example words of the general entities and traversing each node in the semantic network except the example words of the general entities represent the same meaning, which will not be repeated here.

[0103] In combination with Figure 11 It is assumed that the semantic network constructed by step 503 is as shown in Figure 11 Figure 11 ​The weight on the directed edge is omitted in the semantic network. In the semantic network, the instance words of the general entity are two, i.e. date1 and date2, and there are two initial aggregated communities. It is judged that if the currently traversed word in the semantic network is added to the aggregated community 1 and the aggregated community 2, the increment of the modularity of the aggregated community 1 and the aggregated community 2 is caused. The currently traversed word is added to the aggregated community with the greater increment of the modularity. For example, the currently traversed word is added to the aggregated community 1, and the increment of the modularity of the aggregated community 1 is 2, the currently traversed word is added to the aggregated community 2, and the increment of the modularity of the aggregated community 2 is 8, and then the currently traversed word is added to the aggregated community 2. The formula for calculating the modularity is referred to the formula 1-1, and the difference between the modularity when the currently traversed word is not added to the aggregated community and the modularity when the currently traversed word is added to the aggregated community is obtained, so as to obtain the increment of the modularity of the aggregated community caused by adding the currently traversed word to the aggregated community.

[0104]

[0105] Wherein, Q represents the modularity of the aggregated community, m represents the total number of directed edges in the semantic network. i represents the currently traversed node, and j represents each node in the aggregated community for which the modularity is currently calculated. If the currently traversed node and the node j in the aggregated community for which the modularity is currently calculated are connected through a directed edge, then Aij>=1, and if the currently traversed node and the node j in the aggregated community for which the modularity is currently calculated are not connected, then Aij=0. Wherein, represents the sum of the weights of the directed edges directly connecting the node i, the node i being the first node of the two nodes connected by the directed edges, represents the sum of the weights of all directed edges directly connecting the node i, the node i being the second node of the two nodes connected by the directed edges. It is introduced above that the directed edge is used to represent the order of the words corresponding to the nodes in the text to be processed. For example, if the word represented by the node i appears first and the word represented by the node j appears second in the text to be processed, it is considered that the weight of the directed edge connecting the node i and the node j should be calculated into . Conversely, if the word represented by the node j appears first and the word represented by the node i appears second in the text to be processed, it is considered that the weight of the directed edge connecting the node i and the node j should be calculated into . δ(Ci, Cq) is used to judge whether the node i and the node q are in the same aggregated community, q being any node in the semantic network. If the node i and the node q are in the same aggregated community, then δ(Ci, Cj)=1, otherwise δ(Ci, Cj)=0.

[0106] In one possible implementation, each word in the semantic network except the instance words of the general entities can be traversed. The currently traversed word is added to the aggregated community in which the instance word of the nearest general entity is located. For example, the node corresponding to the currently traversed word is taken as the starting point of a path, and the center nodes of the aggregated communities are taken as the termination points of the path, and the center nodes of the aggregated communities respectively correspond to the instance words of the general entities. A plurality of paths with the least number of nodes passed through in the path are obtained, and a plurality of paths are obtained if there are a plurality of aggregated communities. The currently traversed node is added to the aggregated community in which the termination point of the path with the least number of nodes passed through is located.

[0107] In combination Figure 11 For illustration, it is assumed that the semantic network constructed through step 503 is as shown in FIG. 5B. Figure 11 Figure 11 In the semantic network, the weights on the directed edges are omitted. In the semantic network, the instance words of the general entities are two, i.e., date1 and date2, and there are two initial aggregated communities. The currently traversed node is added to the aggregated community in which the instance word of the nearest general entity is located. For example, the node of "the agreement" can be connected to date1 by passing through the nodes of "both parties" and "sign" with the least number of nodes. The node of "the agreement" can be connected to date2 by passing through the nodes of "both parties", "sign", "date1", "effective date", "come into effect", and "validity period" with the least number of nodes. The node of "the agreement" is closer to the node of date1, and the node of "the agreement" is added to the aggregated community in which date1 is located, i.e., added to the aggregated community 1. It should be noted that if the distance of a node to the nodes of the instance words of a plurality of general entities is the same, the node can be randomly selected and added to any one of the aggregated communities in which the instance words are located. Or if the distance of a node to the nodes of the instance words of a plurality of general entities is the same, the node can also be added to the aggregated community in which the node is located by combining other manners, such as the above-mentioned manner of calculating the increment of the module. For example, if the distance of a node to the nodes of the instance words of a plurality of general entities is the same, the aggregated community in which the termination point of the path with the largest sum of weights of all directed edges included in the path is located can be selected.

[0108] 505, obtaining an identification network of the target entity according to the semantic network.

[0109] ​Through step 504, the instance word of each generic entity corresponds to an aggregated community, and those skilled in the art can determine whether the instance word of each generic entity can correspond to the element of the target entity according to the aggregated community corresponding to the instance word of each generic entity. If the instance word of a certain generic entity cannot correspond to an element of the target entity, it is deleted, which can be specifically reflected in that the aggregated community in which the instance word of the generic entity is located is deleted from the semantic network; if the instance word of a certain generic entity can correspond to an element of the target entity, the instance word of the generic entity and the corresponding element are linked to obtain the recognition network of the target entity. Thus, the recognition network of the target entity is constructed according to the background knowledge network and the semantic network.

[0110] In order to better understand this process, the following will be described in combination with a specific example. Referring to Figure 12 , the target entity is still taken as the example of "contract effective period". The disassembly process of the target entity "contract effective period" has been introduced above, and will not be repeated here. Those skilled in the art can determine which element the instance word in each aggregated community belongs to according to each aggregated community. It is assumed that after judgment, the instance word corresponding to the generic entity in aggregated community 1 corresponds to the element of contract start date, the instance word corresponding to the generic entity in aggregated community 2 corresponds to the element of contract end date, and the instance word corresponding to the generic entity in aggregated community 3 corresponds to the element of contract duration. Then, date1 in aggregated community 1 is connected with the element of contract start date, so that when the target entity is recognized according to the recognition network, if a certain instance word in the text to be processed in the reasoning stage and the semantic environment of aggregated community 1 in which date1 is located are similar, it can be determined that the instance word also corresponds to the element of contract start date according to the connection between date1 in aggregated community 1 and the element of contract start date.

[0111] From Figure 5 As can be known from the corresponding embodiments, the embodiments of the present application do not need a large number of training samples of texts to be processed in which the positions of the target entities are labeled to train a model for recognizing the target entity. The embodiments of the present application can utilize the existing generic entity recognition model by disassembling the target entity. The embodiments of the present application only need to utilize the text including the target entity without labeling as a training sample to obtain a model for recognizing the target entity. The cost of obtaining the model for recognizing the target entity is reduced.

[0112] II. Perform the recognition task of the target entity through the network

[0113] The above describes how to obtain the network for identifying the target entity according to the embodiments of the present application. The following describes how to perform the identification task of the target entity based on the network.

[0114] Referring to Figure 13 A flowchart of a text processing method provided by the embodiments of the present application is shown.

[0115] As Figure 13 shown, the text processing method provided by the embodiments of the present application includes the following steps:

[0116] 1301, obtaining a text to be processed.

[0117] 1302, taking the text to be processed as an input of a general entity identification model to identify general entities from the text to be processed.

[0118] The general entity identification model is determined according to the general entities constituting the target entity.

[0119] In a possible implementation, the general entity identification model can be used to identify the general entities constituting the target entity in the text to be processed.

[0120] In a possible implementation, multiple general entity identification models can be used to identify the general entities constituting the target entity in the text to be processed.

[0121] 1303, performing a word segmentation processing on the text to be processed to obtain a segmented text to be processed.

[0122] The instance words of the same entity in the text to be processed are represented by the same label.

[0123] The text to be processed replaced by the label is subjected to the word segmentation processing to obtain the text to be processed represented by the words (or phrases). It should be noted that although the instance words in the text to be processed are replaced by the labels, the instance words represented by each label can still be found.

[0124] In a possible implementation, the to-be-processed text can be segmented based on a professional dictionary in the field to obtain the to-be-processed text represented by words. It should be noted that the to-be-processed text can be segmented by using any possible implementation to obtain the to-be-processed text represented by words, which is not limited in the embodiments of the present application. For example, the fixed phrases or words in the to-be-processed text can be identified according to the Ngram method, according to the aggregation degree within the phrase and the freedom degree outside the phrase, and then the to-be-processed text can be segmented. In a possible implementation, after the to-be-processed text is segmented, the stop words in the segmented to-be-processed text can be deleted. For example, some words such as mood words and auxiliary words, which have no meaning for describing the context environment of the general entity, are deleted.

[0125] 1304. Constructing at least one candidate path according to the segmented to-be-processed text.

[0126] Each candidate path includes a plurality of words in the segmented to-be-processed text, the plurality of words include at least one label, and each candidate path indicates the order of the plurality of words in the segmented to-be-processed text.

[0127] In order to better utilize the recognition network, a plurality of candidate paths can be constructed according to the segmented to-be-processed text. Each candidate path should include at least one label, that is, at least one instance word of the general entity. For example, assuming that the segmented to-be-processed text is: agreement / framework / both parties / confirmation / date1 / effective date / framework / contract / date2 / formally / sign a contract / valid period / effective / date3 / terminate. The above to-be-processed text can be divided into the following paths. The first path includes: agreement / framework / both parties / confirmation / date1 / effective date; the second path includes framework / contract / date2 / formally / sign; the third path includes contract / valid period / effective / date3 / terminate, and so on. Some words can also be selected every fixed number or non-fixed number to construct a path, for example, the fourth path can also be constructed, including both parties / confirmation / date1 / effective date; the fifth path includes date2 / formally / sign; the sixth path includes date3 / terminate, and so on.

[0128] 1305. Searching the at least one candidate path from the recognition network and obtaining a target path.

[0129] A sufficient number of candidate paths can be constructed through step 1303 to cover as many combinations of words in the context of each general entity in the text to be processed as possible. If all candidate paths of a certain label cannot be found in the recognition network, it is considered that the instance word corresponding to the label is irrelevant to the constituent elements of the target entity, and it can be deleted or not considered in subsequent processing. For example, assume that the candidate paths related to date1 include path1, path2, path3, and path4. If any of the four paths cannot be found in the recognition network, it is considered that the instance word corresponding to date1 is irrelevant to the constituent elements of the target entity, and date1, path1, path2, path3, and path4 can be deleted or not considered in subsequent processing. Take path1 as an example. Assume that path1 includes four words. If any of the four words cannot be found in the recognition network, it is considered that path1 is not included in the recognition network. If multiple candidate paths of a certain label can be found in the recognition network, the path with the maximum total weight is selected as the target path. For example, assume that the candidate paths related to date1 include path1, path2, path3, and path4. If any of the four paths can be found in the recognition network, take path1 as an example. Assume that path1 includes four words. The four words are searched in the recognition network, and the shortest path connecting the four words is obtained. The sum of the weights of the directed edges of the shortest path is the weight of path1. In this way, the weights of path2, path3, and path4 can be obtained, and the path with the maximum weight is selected as the target path.

[0130] 1306、According to the elements of the target entity corresponding to the labels passed through by the target path, the target entity in the text to be processed is obtained.

[0131] The above Figure 5 As introduced in the above embodiments, each label is linked to an element of the target entity, and the elements of the target entity corresponding to the labels passed through by the target path can be obtained. Further, the background knowledge network in the recognition network can be combined to obtain the target entity in the text to be processed. For example, Figure 14As shown, assuming that the target path includes path 1: sign / date1; path 2: valid period / date2 / termination; and path 3: num / termination, according to the identification network, date1 corresponds to the element of the contract start date, date2 corresponds to the element of the contract end date, and num corresponds to the contract duration. The contract start date and the contract end date can constitute the contract validity period, and the instance word corresponding to date1 and the instance word corresponding to date2 can be output. The contract start date and the contract duration can also constitute the contract validity period, and the instance word corresponding to date1 and the instance word corresponding to num can also be output. It should be noted that the target path can have multiple paths, or only one path, that is, the path with the largest weight in all paths is selected as the target path.

[0132] According to the corresponding embodiments, the embodiments of the present application can complete the task of extracting specific domain entities from a large amount of unstructured and unlabeled text, and can greatly reduce the cost and work efficiency of manual extraction. Figure 13 According to the corresponding embodiments, the embodiments of the present application can complete the task of extracting specific domain entities from a large amount of unstructured and unlabeled text, and can greatly reduce the cost and work efficiency of manual extraction.

[0133] The above describes a text processing method and an entity recognition method provided by the present application. It should be understood that the training device and the translation device described above include hardware structures and / or software modules for executing various functions. Those skilled in the art should easily realize that the modules and algorithm steps of the examples described in the embodiments disclosed herein can be realized in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed by hardware or computer software driven hardware depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0134] From the hardware structure, Figure 5 The execution subject in the above method can be realized by one entity device, or by multiple entity devices together, or by a logical functional module in one entity device, and the embodiments of the present application do not make specific limitations. Figure 13 The execution subject in the above method can be realized by one entity device, or by multiple entity devices together, or by a logical functional module in one entity device, and the embodiments of the present application do not make specific limitations.

[0135] Based on the foregoing text processing method and entity recognition method, the text processing device and the entity recognition device provided by the present application are described below. The text processing device is used to execute the steps of the foregoing Figure 5 The entity recognition device is used to execute the steps of the foregoingFigure 13 Steps of the corresponding method.

[0136] For example, the device for text processing or the device for entity recognition can be implemented by a computer device in Figure 15 Figure 15 Fig. 1 shows a hardware structure schematic diagram of a computer device provided by an embodiment of the present application. It includes a communication interface 1501 and a processor 1502, and can also include a memory 1503.

[0137] The communication interface 1501 can use any transceiver-like device for communicating with other devices or communication networks, and in this scheme, the end-side device can communicate with the server using the communication interface 1501, such as uploading a model or downloading a model. In one possible implementation, the communication interface 1501 can use Ethernet, radio access network (RAN), wireless local area networks (WLAN), etc. to communicate with the server.

[0138] The processor 1502 includes but is not limited to one or more of a central processing unit (CPU), a network processor (NP), an application-specific integrated circuit (ASIC), or a programmable logic device (PLD). The above-mentioned PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof. The processor 1502 is responsible for the communication line 1504 and general processing, and can also provide various functions, including timing, peripheral interface, voltage regulation, power management, and other control functions.

[0139] ​The memory 1503 can be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disk storage, a magnetic disk storage or other magnetic storage devices, or any other medium capable of storing desired program code in the form of instructions or data structures and that can be accessed by a computer, but is not limited to. The memory can exist independently of the processor 1502 and be connected to the processor 1502 through the communication line 1504. The memory 1503 can also be integrated with the processor 1502. If the memory 1503 and the processor 1502 are independent devices, the memory 1503 and the processor 1502 are connected, for example, the memory 1503 and the processor 1502 can communicate through the communication line. The communication interface 1501 and the processor 1502 can communicate through the communication line, and the communication interface 1501 can also be directly connected to the processor 1502.

[0140] The communication line 1504 can include any number of interconnected buses and bridges, which link various circuits including one or more processors 1502 represented by the processor 1502 and the memory represented by the memory 1503. The communication line 1504 can also link various other circuits such as peripheral devices, voltage stabilizers, and power management circuits, which are well known in the art, and therefore, further description thereof will not be given.

[0141] In one possible implementation, the computer device is a text processing apparatus including a memory for storing computer readable instructions. It can also include a communication interface and a processor coupled with the memory. The communication interface is configured to obtain a text to be processed. The processor is configured to execute the computer readable instructions in the memory to perform Figure 5 Steps 501 to 505 in the corresponding embodiment.

[0142] In one possible implementation, the computer device is an entity recognition apparatus including a memory for storing computer readable instructions. It can also include a communication interface and a processor coupled with the memory. The communication interface is configured to obtain a text to be processed. The processor is configured to execute the computer readable instructions in the memory to performFigure 13 Step 1301 in the corresponding embodiment. The processor is configured to execute the computer readable instructions in the memory to perform Figure 13 Steps 1302 to 1306 in the corresponding embodiment.

[0143] In the embodiments of the present application, the communication interface can be regarded as a transceiver module 1601 of the computer device, the processor with processing function can be regarded as a processing module 1602 of the computer device, and the memory can be regarded as a storage module (not shown in the figure) of the computer device. Referring to Figure 16 A structural schematic diagram of a computer device provided by the embodiments of the present application.

[0144] The embodiments of the present application do not limit the name of the model, for example, when the computer device is a text processing apparatus, the processing module 1602 can be regarded as a semantic network construction module, an identification network construction module, an updating module, a replacement module and a clustering module. In a possible implementation, the text processing apparatus can include: an acquisition module, configured to acquire a to-be-processed text. A semantic network construction module, configured to construct a semantic network according to the to-be-processed text acquired by the acquisition module, the semantic network including nodes and directed edges, the nodes being configured to represent words in the to-be-processed text, and the directed edges being configured to connect adjacent words in the to-be-processed text, and weights of the directed edges being configured to indicate probabilities of adjacent appearance of the adjacent words in the to-be-processed text. An identification network construction module, configured to link nodes representing target elements in the to-be-processed text in the semantic network constructed by the semantic network construction module with the target elements in a background knowledge network, to acquire a network for identifying target entities, the words representing the target elements being acquired by inputting the to-be-processed text into a general entity recognition model, and the background knowledge network being configured to indicate the target entities represented by the target elements.

[0145] In a possible implementation, the semantic network construction module is specifically configured to: traverse the to-be-processed text.

[0146] According to the currently traversed to-be-processed text, a reference path is constructed, the reference path including nodes representing words in the currently traversed to-be-processed text and directed edges connecting adjacent words in the currently traversed to-be-processed text. The plurality of reference paths are processed according to a first rule to construct the semantic network, the first rule including processing the nodes representing the same words in the plurality of reference paths and processing the same directed edges in the plurality of reference paths, and the weight of the processed directed edge is positively correlated with the number of the same directed edges in the plurality of reference paths.

[0147] In a possible implementation, the method further includes an updating module configured to perform updating processing on the semantic network according to a second rule to obtain an updated semantic network, the second rule including deleting low-weight directed edges in the semantic network and nodes not connected by the directed edges, the low-weight directed edge being a directed edge in the semantic network with a weight lower than a preset threshold.

[0148] In a possible implementation, the method further includes a replacing module configured to replace the node representing the word of the target element in the updated semantic network with a preset node. The identifying network construction module is specifically configured to link the preset node with the target element in the background knowledge network.

[0149] In a possible implementation, the method further includes a clustering module configured to perform multiple times of clustering processing on the nodes included in the semantic network with each preset node as an initial clustering center until each node in the semantic network is added to a clustering community with each preset node as a clustering center, where any one of the multiple times of clustering processing includes: obtaining an increment of modularity of each clustering community obtained after last clustering processing if a to-be-processed node is added to the clustering community, the to-be-processed node being any one node in the semantic network except the preset node; and adding the to-be-processed node to a clustering community with the largest increment of modularity to obtain each clustering community obtained after the clustering processing.

[0150] In a possible implementation, the method further includes a receiving module configured to receive an instruction, the instruction being used to indicate a corresponding relationship between the node representing the word of the target element in the semantic network and the target element in the background knowledge network. The identifying network construction module is specifically configured to link the node representing the word of the target element in the semantic network with the target element in the background knowledge network according to the instruction.

[0151] In a possible implementation, the target element belongs to at least one of the following general entities: time, personal name, place name, organization name, date and time, or proper noun.

[0152] When the computer device is an entity recognition apparatus, the processing module 1601 can be regarded as a constructing module or a recognizing module. In a possible implementation, the entity recognition apparatus can include: an obtaining module, configured to obtain a to-be-processed text. A constructing module, configured to construct at least one candidate path according to the to-be-processed text, wherein each candidate path includes nodes and directed edges, the nodes of the candidate path are used to represent words in the to-be-processed text, and the directed edges of the candidate path are used to connect adjacent words in the to-be-processed text, and each candidate path includes at least one node representing an instance word of a general entity, the instance word representing the general entity being obtained by inputting the to-be-processed text into a general entity recognition model. A recognizing module, configured to obtain a target entity in the to-be-processed text according to a node representing an instance word of a general entity included in a target path in the candidate path constructed by the constructing module and a target element linked to the node representing the instance word of the general entity indicated by a recognition network, the target path being a candidate path with the largest weight in the candidate paths, the weight of the target path being determined according to weights of directed edges included in the target path, the weight of the directed edge included in the target path being determined according to weights of directed edges of the recognition network, the recognition network being obtained by linking nodes representing words of the target element in a semantic network to the target element in a background knowledge network, the nodes representing the words of the target element being regarded as the nodes representing the instance word of the general entity, the semantic network being obtained according to training text, the semantic network including nodes and directed edges, the nodes of the recognition network being used to represent words in the training text, the directed edges of the recognition network being used to connect adjacent words in the training text, and the weight of the directed edge of the recognition network being used to indicate a probability that the adjacent words appear adjacent to each other in the training text, and the background knowledge network being used to indicate a target entity represented by the target element.

[0153] It should be understood that the above merely provides one example of the embodiments of the present application, and the apparatus for processing text / entity recognition apparatus can have more or fewer components than those shown, can combine two or more components, or can have a different configuration of components.

[0154] In the above embodiments, the apparatus for processing text / entity recognition apparatus can be implemented by software, hardware, firmware, or any combination thereof, in whole or in part. When implemented by software, the apparatus for processing text / entity recognition apparatus can be implemented in the form of a computer program product, in whole or in part.

[0155] The apparatus for processing text / entity recognition apparatus provided by the embodiments of the present application can be a chip. The chip includes a processing unit and a communication unit. The processing unit can be, for example, a processor. The communication unit can be, for example, an input / output interface, a pin, or a circuit, etc. When the apparatus for processing text is a chip, the processing unit can execute computer-executable instructions stored in the storage unit to enable the chip to perform the above-mentioned processing. Figure 5The embodiment described above is a method for text processing. In another possible implementation, the entity recognition device is a chip, so that the chip performs the above-mentioned Figure 13 The embodiment described above is a method for entity recognition. Optionally, the storage unit is a storage unit in the chip, such as a register, a cache, etc., and the storage unit can also be a storage unit outside the chip in the wireless access device, such as a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM), etc.

[0156] Specifically, the aforementioned processing unit or processor can be a central processing unit (CPU), a neural-network processing unit (NPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), or a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0157] In addition, it should be noted that the apparatus embodiments described above are merely schematic, and the units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, i.e., they can be located in one place or distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiment. In addition, the connection relationship between the modules in the apparatus embodiment provided in the present application indicates that there is a communication connection between them, which can be implemented as one or more communication buses or signal lines.

[0158] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0159] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0160] This application also provides a computer-readable storage medium storing a program for text processing, which, when run on a computer, causes the computer to perform the aforementioned actions. Figure 5 The steps in the method described in the illustrated embodiment.

[0161] This application also provides a computer-readable storage medium storing an entity identification program that, when run on a computer, causes the computer to perform the aforementioned actions. Figure 13 The steps in the method described in the illustrated embodiment.

[0162] This application also provides a digital processing chip. This digital processing chip integrates circuitry for implementing the aforementioned processor or processor functions, and one or more interfaces. When the digital processing chip integrates a memory, it can complete the method steps of any one or more of the foregoing embodiments. When the digital processing chip does not integrate a memory, it can be connected to an external memory via a communication interface. The digital processing chip implements the actions performed by the text processing device / entity recognition device in the foregoing embodiments based on the program code stored in the external memory.

[0163] The embodiments of the present application further provide a computer program product including one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another, for example, the computer instructions can be transferred from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a server, data center, etc. integrated with one or more available media sets. The available media can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk (SSD)), etc.

[0164] Those of ordinary skill in the art can understand that all or part of the steps in the above-mentioned embodiments of the various methods can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer-readable storage medium, which can include ROM, RAM, magnetic or optical disks, etc.

[0165] The above describes the text processing method, entity recognition method and related device provided by the embodiments of the present application in detail. The principles and implementation manners of the present application are described by applying specific examples in this paper. The above embodiment descriptions are only used to help understand the method of the present application and its core idea; at the same time, for those skilled in the art, according to the idea of the present application, the specific implementation manner and application range will be changed; in summary, the content of the specification should not be understood as a limitation of the present application.

[0166] The terms "first", "second", and the like in the description and in the claims of the present application and above drawings are used for distinguishing between similar objects and not necessarily for describing a specific sequential or chronological order. It is to be understood that the terms so used are interchangeable under appropriate circumstances such that the embodiments of the application described herein are, for example, capable of orderly or inverse order, depending upon the circumstances. The term "and / or" in the present application is merely used to represent an association between associated objects, and it is possible that three relationships exist, for example, A and / or B can mean that A exists alone, A and B exist together, and B exists alone. In addition, the character " / " in the present application generally represents an "or" relationship between the associated objects. Furthermore, the terms "comprising" and "having" and any variations thereof are intended to cover a non-exclusive inclusion, for example, a process, method, system, product or device that includes a list of steps or modules as an example is not necessarily limited to those steps or modules that are clearly listed, but can include other steps or modules that are not clearly listed or inherent to such process, method, product or device. The naming or numbering of steps in the present application does not mean that the steps in the method flow must be executed in the time / logical order indicated by the naming or numbering, and the named or numbered flow steps can be executed in a different order according to the technical purpose to be achieved, as long as the same or similar technical effects can be achieved. The division of modules in the present application is a logical division, and in actual application, there can be another division manner, for example, multiple modules can be combined or integrated in another system, or some features can be ignored or not executed, in addition, the coupling or direct coupling or communication connection between the displayed or discussed modules can be through some ports, and the indirect coupling or communication connection between the modules can be electrical or other similar forms, which are not limited in the present application. Furthermore, the modules or sub-modules described as separate components can or can not be physically separated, and can or can not be physical modules, or can be distributed to multiple circuit modules, and part or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of the present application.

Claims

1. A method of text processing, characterized by, The method comprises: acquiring a to-be-processed text; constructing a semantic network according to the to-be-processed text, the semantic network comprising first nodes and first directed edges, the first nodes being used for representing words in the to-be-processed text, the first directed edges being used for connecting first source nodes and first destination nodes, the words represented by the first source nodes and the words represented by the first destination nodes being adjacent words in the to-be-processed text, and the weights of the first directed edges being acquired according to probabilities of adjacent occurrence of the adjacent words in the to-be-processed text; acquiring a network for identifying a target entity according to the semantic network and a background knowledge network, the background knowledge network being used for indicating constituent elements of the target entity, the constituent elements being obtained by conceptually disassembling the target entity, and a corresponding relationship between the constituent elements and target nodes in the semantic network, the words represented by the target nodes being acquired by inputting the to-be-processed text into a general entity recognition model, the general entity recognition model being used for identifying words belonging to the same entity type as the constituent elements.

2. The method of claim 1, wherein, The constructing of the semantic network according to the to-be-processed text comprises: traversing the to-be-processed text; constructing a sub-semantic network according to the to-be-processed text currently traversed, the sub-semantic network comprising second nodes and second directed edges, the second nodes being used for representing words in the to-be-processed text currently traversed, the second directed edges being used for connecting second source nodes and second destination nodes, the words represented by the second source nodes and the words represented by the second destination nodes being adjacent words in the to-be-processed text currently traversed, and the weights of the second directed edges being acquired according to probabilities of adjacent occurrence of the adjacent words in the to-be-processed text currently traversed; merging a plurality of the sub-semantic networks according to a first rule to construct the semantic network, the first rule comprising merging nodes representing the same words in the plurality of the sub-semantic networks and merging the same directed edges in the plurality of the sub-semantic networks, and the weight of the merged directed edge being positively correlated with the number of the same directed edges in the plurality of the sub-semantic networks.

3. The method of claim 1, wherein, The method further comprises: updating the semantic network according to a second rule to acquire an updated semantic network, the second rule comprising deleting low-weight directed edges in the semantic network and the nodes not connected by the directed edges, the low-weight directed edges being the directed edges in the semantic network whose weights are lower than a preset threshold.

4. The method of claim 3, wherein, The method further comprises: replacing the target nodes in the updated semantic network with preset nodes.

5. The method of claim 4, wherein, The method further comprises: performing multiple clustering processes on the nodes included in the semantic network with each preset node as an initial clustering center until each node in the semantic network is added to a clustering community with each preset node as a clustering center, wherein any one of the multiple clustering processes comprises: obtaining an increment of modularity of each of the aggregated communities obtained after a previous aggregation processing if a to-be-processed node is added to the aggregated communities, the to-be-processed node being any one of the nodes in the semantic network except the preset node; adding the to-be-processed node to the aggregated community with the largest increment of modularity to obtain each of the aggregated communities obtained after the current aggregation processing.

6. The method according to any one of claims 1 to 5, characterized in that, The method further comprises: receiving an instruction, the instruction being used to indicate constituent elements of the target entity and a correspondence between the constituent elements and target nodes in a semantic network; obtaining the background knowledge network according to the instruction.

7. The method according to any one of claims 1 to 5, characterized in that, The constituent elements belong to at least one of general entities of time, person name, place name, organization name, date and time, or proper noun.

8. A method of entity recognition, the method comprising: Comprise: obtaining to-be-processed text; constructing at least one candidate path according to the to-be-processed text, each of the candidate paths comprising nodes and directed edges, the nodes of the candidate paths being used to represent words in the to-be-processed text, the directed edges of the candidate paths being used to connect nodes representing adjacent words in the to-be-processed text, each of the candidate paths comprising at least one target node, the word represented by the target node being obtained by inputting the to-be-processed text into a general entity recognition model, the general entity recognition model being used to identify words belonging to the same entity type as constituent elements of a target entity, the constituent elements being obtained by concept disintegration of the target entity; obtaining the target entity in the to-be-processed text according to target nodes included in a target path in the candidate paths and a recognition network, the target path being a candidate path with the largest weight in the candidate paths, the weight of the target path being determined according to weights of directed edges included in the target path, the weights of the directed edges included in the target path being obtained according to the recognition network, the recognition network being obtained according to a semantic network and a background knowledge network, the semantic network being obtained according to training text, the semantic network comprising nodes and directed edges, the nodes of the semantic network being used to represent words in the training text, the directed edges of the semantic network being used to connect nodes representing adjacent words in the training text, the weights of the directed edges of the semantic network being used to indicate probabilities of the adjacent words appearing adjacent to each other in the training text, the background knowledge network being used to indicate constituent elements of the target entity and a correspondence between the constituent elements and target nodes in the semantic network.

9. The method of claim 8, wherein, The semantic network is obtained by merging a plurality of sub semantic networks according to a first rule, the first rule comprising merging nodes representing the same word in the plurality of sub semantic networks, and merging target directed edges in the plurality of sub semantic networks, the weight of the target directed edge being positively correlated with the number of the target directed edges in the plurality of sub semantic networks, the target directed edge being a directed edge connecting nodes representing the same adjacent word in the plurality of sub semantic networks, the sub semantic network being constructed according to a currently traversed training text.

10. The method of claim 8, wherein, The recognition network is obtained according to the updated semantic network and a background knowledge network, the updated semantic network being obtained by updating the semantic network according to a second rule, the second rule comprising deleting low weight directed edges in the semantic network and nodes not connected by the directed edges, the low weight directed edge being a directed edge in the semantic network with a weight lower than a preset threshold.

11. The method of claim 10, wherein, The target node in the updated semantic network is replaced by a preset node.

12. The method of claim 11, wherein, The semantic network comprises a plurality of aggregated communities, each of the preset nodes being an initial clustering center, the nodes included in the semantic network being clustered a plurality of times until each node in the semantic network is added to a clustering community with each of the preset nodes as a clustering center, wherein any one of the plurality of clustering processes comprises: obtaining an increment of modularity of each aggregated community obtained after a last aggregation process if a to-be-processed node is added to the aggregated community, the to-be-processed node being any one of the nodes in the semantic network except the preset nodes; adding the to-be-processed node to the aggregated community with the largest increment of modularity to obtain each aggregated community obtained after the current aggregation process.

13. The method according to any one of claims 8 to 12, characterized in that, The background knowledge network is obtained according to received instructions, the instructions being used to indicate constituent elements of the target entity and a correspondence between the constituent elements and target nodes in the semantic network.

14. The method according to any one of claims 8 to 12, characterized in that, The constituent elements belong to at least one of general entities of time, name, place name, organization name, date and time, or proper noun.

15. An apparatus for text processing, characterized by Comprise: a obtaining module, configured to obtain a to-be-processed text; a semantic network construction module, configured to construct a semantic network according to the to-be-processed text, the semantic network comprising first nodes and first directed edges, the first nodes being used to represent words in the to-be-processed text, the first directed edges being used to connect first source nodes and first destination nodes, the words represented by the first source nodes and the words represented by the first destination nodes being adjacent words in the to-be-processed text, the weight of the first directed edge being obtained according to the probability of adjacent occurrence of the adjacent words in the to-be-processed text; The identification network construction module is configured to acquire a network for identifying a target entity according to the semantic network and a background knowledge network, the background knowledge network is configured to indicate constituent elements of the target entity and a corresponding relationship between the constituent elements and target nodes in the semantic network, the target nodes represent words obtained by inputting the text to be processed into a general entity identification model, the general entity identification model is configured to identify words belonging to the same entity type as the constituent elements, and the constituent elements are obtained by conceptually disassembling the target entity.

16. The apparatus of claim 15, wherein, The semantic network construction module is specifically configured to: traverse the text to be processed; construct a sub semantic network according to the text to be processed currently traversed, the sub semantic network includes second nodes and second directed edges, the second nodes are configured to represent words in the text to be processed currently traversed, the second directed edges are configured to connect second source nodes and second destination nodes, words represented by the second source nodes and words represented by the second destination nodes are adjacent words in the text to be processed currently traversed, and a weight of the second directed edge is acquired according to a probability of adjacent occurrence of the adjacent words in the text to be processed currently traversed; merge a plurality of the sub semantic networks according to a first rule to construct the semantic network, the first rule includes merging nodes representing the same words in the plurality of the sub semantic networks and merging the same directed edges in the plurality of the sub semantic networks, and the weight of the merged directed edge is positively correlated with a number of the same directed edges in the plurality of the sub semantic networks.

17. The apparatus of claim 15, wherein, The updating module is further configured to update the semantic network according to a second rule to acquire an updated semantic network, the second rule includes deleting low-weight directed edges in the semantic network and nodes not connected by the directed edges, and the low-weight directed edges are the directed edges with weights lower than a preset threshold in the semantic network. The replacing module is further configured to replace the target nodes in the updated semantic network with preset nodes.

18. The apparatus of claim 17, wherein, The clustering module is further configured to: perform a plurality of clustering processes on the nodes included in the semantic network with each preset node as an initial clustering center until each node in the semantic network is added to a clustering community with each preset node as a clustering center, wherein any one of the plurality of clustering processes includes:

19. The apparatus of claim 18, wherein, acquiring an increment of a modularity of each clustering community acquired after a last clustering process if a to-be-processed node is added to the clustering community, the to-be-processed node being any one of the nodes in the semantic network except the preset nodes; adding the to-be-processed node to the clustering community with the largest increment of the modularity to acquire each clustering community acquired after the current clustering process. The receiving module is further configured to: ​ 20. The apparatus of any one of claims 15 to 19, wherein, ​ The receiving module is configured to receive an instruction, the instruction being used to indicate a component element of the target entity and a corresponding relationship between the component element and a target node in a semantic network. The identifying network construction module is further configured to acquire the background knowledge network according to the instruction.

21. The apparatus of any one of claims 15 to 19, wherein, The component element belongs to at least one of a general entity of time, a person name, a place name, an organization name, a date and time, or a proper noun.

22. An apparatus for entity recognition, the apparatus comprising: The method comprises the following steps: The acquiring module is configured to acquire a to-be-processed text. The construction module is configured to construct at least one candidate path according to the to-be-processed text, each candidate path comprising a node and a directed edge, the node of the candidate path being used to represent a word in the to-be-processed text, the directed edge of the candidate path being used to connect nodes representing adjacent words in the to-be-processed text, each candidate path comprising at least one target node, the word represented by the target node being acquired by inputting the to-be-processed text into a general entity recognition model, the general entity recognition model being used to identify a word belonging to a same entity type as a component element of a target entity, the component element being obtained by conceptually disassembling the target entity. The identifying module is configured to acquire the target entity in the to-be-processed text according to the target nodes included in a target path in the candidate paths and an identifying network, the target path being a candidate path with the largest weight in the candidate paths, the weight of the target path being determined according to the weights of the directed edges included in the target path, the weights of the directed edges included in the target path being acquired according to the identifying network, the identifying network being acquired according to a semantic network and a background knowledge network, the semantic network being acquired according to training text, the semantic network comprising nodes and directed edges, the nodes of the semantic network being used to represent words in the training text, the directed edges of the semantic network being used to connect nodes representing adjacent words in the training text, the weights of the directed edges of the semantic network being used to indicate probabilities of the adjacent words appearing adjacently in the training text, the background knowledge network being used to indicate a component element of the target entity and a corresponding relationship between the component element and a target node in the semantic network.

23. An apparatus for text processing, the apparatus comprising: The method comprises the following steps: The memory is configured to store computer readable instructions; The processor coupled to the memory is configured to execute the computer readable instructions in the memory to perform the method described in any one of claims 1 to 7.

24. An apparatus for entity recognition, the apparatus comprising: The memory is configured to store computer readable instructions; The processor coupled to the memory is configured to execute the computer readable instructions in the memory to perform the method described in any one of claims 8 to 14. The instructions, when executed on a computer device, cause the computer device to perform the method described in any one of claims 1 to 7.

25. A computer-readable storage medium, characterized in that, The instructions, when executed on a computer device, cause the computer device to perform the method described in any one of claims 8 to 14.

26. A computer-readable storage medium, characterized in that, ​ 27. A chip, characterized by The chip is coupled with a memory for executing a program stored in the memory to perform the method of any one of claims 1 to 7.

28. A chip, characterized by The chip is coupled with a memory for executing a program stored in the memory to perform the method of any one of claims 8 to 14.

Images