Text recognition method and device, electronic equipment and storage medium

Abstract

This application provides a method, apparatus, electronic device, and computer-readable storage medium for character recognition in text; it relates to natural language processing technology in the field of artificial intelligence; the method includes: extracting multiple candidate character words from text, obtaining at least one matching parameter corresponding to each candidate character word; selecting at least one candidate character word from the multiple candidate character words as a first candidate character entity based on the at least one matching parameter corresponding to each candidate character word; fusing the question and the text corresponding to the text to obtain fused text; performing entity recognition processing on the fused text to obtain at least one second candidate character entity; and performing character classification processing based on the at least one first candidate character entity and the at least one second candidate character entity to obtain the character in the text. This application enables accurate and efficient character recognition from text.

Description

Technical Field

[0001] This application relates to artificial intelligence technology, and more particularly to a method, apparatus, electronic device, and computer-readable storage medium for character recognition in text. Background Technology

[0002] Artificial intelligence (AI) is the theory, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. With the research and advancement of AI technology, it has been studied and applied in multiple fields.

[0003] Taking character recognition in text (such as novels) as an example, the names of characters in text (hereinafter referred to as character names or characters) usually include various titles and appellations, which are different from general personal names. When related technologies use general personal name recognition models to identify characters in text, the special nature of character types leads to incomplete character recognition, that is, the accuracy of the identified characters is low, consuming unnecessary computing resources.

[0004] It is evident that there is currently no effective solution in terms of how to accurately and efficiently identify roles from text. Summary of the Invention

[0005] This application provides a method, apparatus, electronic device, and computer-readable storage medium for identifying roles in text, which can accurately and efficiently identify roles from text.

[0006] The technical solution of this application embodiment is implemented as follows:

[0007] This application provides a method for role recognition in text, including:

[0008] Extract multiple candidate words for roles from the text, and obtain at least one matching parameter corresponding to each candidate word for a role;

[0009] Based on at least one matching parameter corresponding to each of the said role candidate words, at least one role candidate word is selected from the plurality of role candidate words as the first candidate role entity;

[0010] The question corresponding to the text is merged with the text to obtain the merged text;

[0011] The fused text is subjected to entity recognition processing to obtain at least one second candidate role entity;

[0012] Role classification is performed based on at least one first candidate role entity and at least one second candidate role entity to obtain the roles in the text.

[0013] In the above scheme, the extraction of entity features and text features from the entity statement pair includes:

[0014] Multiple word vectors are extracted from the entity sentence pairs, and the average value among the multiple word vectors is determined as the entity feature;

[0015] The entity statement pair is encoded in the direction from the start position to the end position to obtain the forward encoding vector;

[0016] The entity statement pair is encoded in the direction from the end position to the start position to obtain the backward encoding vector;

[0017] The forward encoding vector and the backward encoding vector are fused to obtain the text features.

[0018] In the above scheme, the extraction of multiple role candidate words from the text includes:

[0019] The text is obtained, and the following preprocessing is performed on the text: the text is divided into multiple sentences according to the symbol list, and the symbols in each sentence are filtered out;

[0020] Extract multiple role candidate words from each sentence of the preprocessed text.

[0021] This application provides a text-based role recognition device, including:

[0022] The first entity recognition module is used to extract multiple role candidate words from the text and obtain at least one matching parameter corresponding to each role candidate word;

[0023] The first entity recognition module is further configured to select at least one role candidate word from the plurality of role candidate words as a first candidate role entity based on at least one matching parameter corresponding to each role candidate word;

[0024] The second entity recognition module is used to fuse the question corresponding to the text and the text to obtain fused text;

[0025] The second entity recognition module is further configured to perform entity recognition processing on the fused text to obtain at least one second candidate role entity;

[0026] The classification module is used to perform role classification processing based on at least one first candidate role entity and at least one second candidate role entity to obtain the roles in the text.

[0027] In the above scheme, the matching parameters include word frequency, cohesion, and degrees of freedom. The first entity recognition module is further configured to perform the following processing for each candidate word: determine the word frequency of the candidate word in the text; divide the candidate word into multiple morphemes and determine the probability of occurrence of each morpheme in the text and the probability of occurrence of the candidate word in the text, wherein the types of morphemes include characters and words; determine the cohesion of the candidate word based on the probability of occurrence of each morpheme in the text and the probability of occurrence of the candidate word in the text; determine the left information entropy and right information entropy of the candidate word, and determine the degrees of freedom of the candidate word based on the left information entropy and right information entropy of the candidate word.

[0028] In the above scheme, the first entity recognition module is further configured to determine multiple left adjacent characters and multiple right adjacent characters of the role candidate word in the text; determine the sub-information entropy corresponding to each left adjacent character, and determine the sub-information entropy corresponding to each right adjacent character; determine the left information entropy by taking the negative number of the sum of the sub-information entropies corresponding to each left adjacent character, and determine the right information entropy by taking the negative number of the sum of the sub-information entropies corresponding to each right adjacent character; when the left information entropy is greater than the right information entropy, the right information entropy is determined as the degree of freedom, and when the left information entropy is not greater than the right information entropy, the left information entropy is determined as the degree of freedom.

[0029] In the above scheme, the first entity recognition module is further configured to perform the following processing for each left adjacent word: determine the ratio between the number of times the left adjacent word appears in the text and the number of times all adjacent words of the role candidate word appear in the text as a first ratio; perform logarithmic operation on the first ratio, and determine the product between the logarithmic operation result and the first ratio as the sub-information entropy corresponding to the left adjacent word;

[0030] In the above scheme, the first entity recognition module is further configured to perform the following processing for each right adjacent word: determine the ratio between the number of times the right adjacent word appears in the text and the number of times all adjacent words of the role candidate word appear in the text as a second ratio; perform logarithmic operation on the second ratio, and determine the sub-information entropy corresponding to the right adjacent word as the product between the logarithmic operation result and the second ratio.

[0031] In the above scheme, the first entity recognition module is further configured to perform the following processing for each morpheme: determine the probability of the morpheme appearing in the text as the ratio between the number of times the morpheme appears in the text and the number of times all role candidate words appear in the text; determine the probability of the role candidate word appearing in the text as the ratio between the number of times the role candidate word appears in the text and the number of times all role candidate words appear in the text.

[0032] In the above scheme, the first entity recognition module is further configured to multiply the occurrence probability of each morpheme in the text to obtain a product result; determine the ratio between the occurrence probability of the role candidate word in the text and the product result as a third ratio; perform logarithmic operation on the third ratio, and determine the logarithmic operation result as the cohesion of the role candidate word.

[0033] In the above scheme, the first entity recognition module is further configured to select, from the plurality of role candidate words, a role candidate word that satisfies at least one of the following conditions as the first candidate role entity: the word frequency of the role candidate word in the text exceeds a word frequency threshold, the cohesion of the role candidate word exceeds a cohesion threshold, and the degree of freedom of the role candidate word exceeds a degree of freedom threshold.

[0034] In the above scheme, the second entity recognition module is further configured to perform feature extraction processing on the fused text to obtain a feature sequence; perform mapping processing on the feature sequence to obtain at least one location set; and perform the following processing on each location set: combine the text corresponding to the start position in the location set, the text between the start and end positions in the location set, and the text corresponding to the end position in the location set, and determine the combination result as the second candidate role entity.

[0035] In the above scheme, the second entity recognition module is further configured to divide the feature sequence into multiple sub-features, wherein the multiple sub-features correspond one-to-one with multiple characters in the text; map each sub-feature to a start probability belonging to a start position and an end probability belonging to an end position; select at least one sub-feature with a start probability greater than a start probability threshold as a start sub-feature, and select at least one sub-feature with an end probability greater than an end probability threshold as an end sub-feature; construct at least one candidate start-end feature set based on the selected at least one start sub-feature and at least one end sub-feature, wherein the candidate start-end feature set includes a start sub-feature and an end sub-feature; determine a target start-end feature set from the at least one candidate start-end feature set; determine the character corresponding to the start sub-feature in the target start-end feature set as the start position in the position set, and determine the character corresponding to the end sub-feature in the target start-end feature set as the end position in the position set.

[0036] In the above scheme, the second entity recognition module is further configured to perform the following processing for each candidate start and end feature set: fuse the start sub-feature and end sub-feature in the candidate start and end feature set to obtain a first fused feature, and map the first fused feature to the probability of belonging to the same entity; in the at least one candidate start and end feature set, determine the candidate start and end feature set whose probability of belonging to the same entity is greater than the entity probability threshold as the target start and end feature set.

[0037] In the above scheme, the second entity recognition module is further configured to perform the following processing for each second candidate role entity: dividing the second candidate role entity into multiple morphemes, and determining the occurrence probability of each morpheme in the text and the occurrence probability of the second candidate role entity in the text, wherein the types of the morphemes include characters and words; determining the cohesion of the second candidate role entity based on the occurrence probability of each morpheme in the text and the occurrence probability of the second candidate role entity in the text; determining the left information entropy and right information entropy of the second candidate role entity, and determining the degrees of freedom of the second candidate role entity based on the left information entropy and right information entropy of the second candidate role entity; filtering second candidate role entities that satisfy at least one of the following conditions among the multiple second candidate role entities: the word frequency of the second candidate role entity in the text does not exceed a word frequency threshold, the cohesion of the second candidate role entity does not exceed a cohesion threshold, and the degrees of freedom of the second candidate role entity does not exceed a degrees of freedom threshold.

[0038] In the above scheme, the classification module is further configured to filter out duplicate candidate role entities among the at least one first candidate role entity and the at least one second candidate role entity; and to perform the following processing on each candidate role entity obtained after filtering: determine the statement in which the candidate role entity is located, and combine the candidate role entity and the statement to obtain an entity statement pair; extract entity features and text features from the entity statement pair, and fuse the entity features and the text features to obtain a second fused feature; map the second fused feature to the probability of belonging to a role entity; and when the probability of belonging to a role entity is greater than a role probability threshold, and the word frequency of the candidate role entity in the text is greater than a role word frequency threshold, determine that the candidate role entity is a role in the text.

[0039] In the above scheme, the classification module is further configured to extract multiple word vectors from the entity sentence pair, and determine the mean among the multiple word vectors as the entity feature; encode the entity sentence pair in the direction from the start position to the end position to obtain a forward encoding vector; encode the entity sentence pair in the direction from the end position to the start position to obtain a backward encoding vector; and fuse the forward encoding vector and the backward encoding vector to obtain the text feature.

[0040] In the above scheme, the first entity recognition module is further configured to acquire the text and perform the following preprocessing on the text: divide the text into multiple sentences according to the symbol list and filter out the symbols in each sentence; extract multiple role candidate words from each sentence of the preprocessed text.

[0041] This application provides an electronic device, including:

[0042] Memory is used to store executable instructions for a computer;

[0043] The processor, when executing computer-executable instructions stored in the memory, implements the character recognition method in text provided in the embodiments of this application.

[0044] This application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the role recognition method in text provided in this application.

[0045] This application provides a computer program product, which includes computer-executable instructions for implementing the role recognition method in text provided in this application when executed by a processor.

[0046] The embodiments of this application have the following beneficial effects:

[0047] By extracting candidate role entities from text using two different methods and determining the roles in the text based on the extracted candidate role entities, we can not only ensure the diversity and comprehensiveness of the identified roles, but also ensure the integrity of the identified roles, thereby improving the efficiency and accuracy of role identification from text. Attached Figure Description

[0048] Figure 1 This is a schematic diagram of the architecture of the role recognition system 100 in the text provided in the embodiments of this application;

[0049] Figure 2 This is a schematic diagram of the structure of the server 200 provided in the embodiments of this application;

[0050] Figure 3 This is a flowchart illustrating the role recognition method in text provided in the embodiments of this application;

[0051] Figure 4 This is a flowchart illustrating the role recognition method in text provided in the embodiments of this application;

[0052] Figure 5 This is a flowchart illustrating the role recognition method in text provided in the embodiments of this application;

[0053] Figure 6 This is a schematic diagram of the structure of the machine learning model provided in the embodiments of this application;

[0054] Figure 7 This is a schematic diagram of the role recognition framework provided in the embodiments of this application. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0056] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0057] In the following description, the terms "first / second" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first / second" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0058] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0059] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0060] 1) Natural Language Processing (NLP) is an important field within computer science and artificial intelligence. It studies the theories and methods that enable effective communication between humans and computers using natural language. NLP is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field involves natural language—the language people use in daily life—and thus it has a close relationship with linguistic research. NLP techniques typically include text processing, semantic understanding, machine translation, question answering, and knowledge graphs.

[0061] 2) Term Frequency (TF) refers to the number of times a given word appears in a text.

[0062] 3) Cohesion, used to measure whether a word collocation is reasonable. For example, in “Funny Movie Review”, “movie” is a more reasonable collocation than “laughing movie”. Therefore, the cohesion of “movie” is higher than that of “laughing movie”.

[0063] 4) Degrees of freedom, used to measure the richness of the left and right neighboring characters of a word. The information entropy of a word can be used to represent its degrees of freedom. The higher the degrees of freedom, the richer the left and right neighboring characters of the word.

[0064] Character names in text often include various titles and appellations, which differ from general personal names. Related technologies primarily rely on general named entity recognition models to identify characters in text, such as sequence labeling-based models. These models use Long Short-Term Memory (LSTM), Transformer models, and Bidirectional Encoder Representation from Transformers (BERT) models as their basic framework, and overlay Conditional Random Fields (CRF) to identify entities and their types based on the labeling results.

[0065] During implementation, the applicant discovered the following technical problems with the relevant technology:

[0066] General-purpose named entity recognition models are mainly designed for paragraph and sentence-level text input, and have low compatibility with complete text (such as novels). When using general-purpose named entity recognition models to identify characters in text, the specificity of character types can lead to incomplete character identification.

[0067] To address the aforementioned technical problems, this application provides a method for identifying roles in text, which can accurately and efficiently identify roles from text. The following describes exemplary applications of the text-based role identification method provided in this application. This method can be implemented by various electronic devices. For example, it can be implemented by a terminal alone, where the terminal uses its own computing power to identify roles from text; or it can be implemented collaboratively by a terminal and a server, where the terminal uses the server's computing power to identify roles from text.

[0068] Next, we will describe an embodiment of this application using an example of implementation by a server and a terminal working together. See [link to relevant documentation]. Figure 1 , Figure 1 This is a schematic diagram of the architecture of the text role recognition system 100 provided in the embodiments of this application. The text role recognition system 100 includes: server 200, network 300, and terminal 400, which will be described separately.

[0069] Server 200 is the backend server for client 410. It is used to respond to text reading requests from client 410, and to obtain the corresponding text based on the text identifier (such as novel name, novel number, article name, etc.) in the text reading request. The text can be a novel, news article, article, etc. It is also used to perform character recognition on the text to obtain the characters in the text. It is also used to send the corresponding text and the characters in the text to client 410 for display.

[0070] Network 300 is used as a medium for communication between server 200 and terminal 400, and can be a wide area network, a local area network, or a combination of both.

[0071] Terminal 400 runs client 410, which is a client with text reading capabilities, such as a news client, browser, or novel client. Client 410 responds to the user's text reading operation by sending a text reading request to server 200; it also receives text and characters from the text sent by server 200 and displays them on the human-computer interaction interface. For example, the characters in the text can be displayed as a character list in the text synopsis on the human-computer interaction interface, thereby accelerating the user's understanding of the novel.

[0072] As an example, this can be used to display a character relationship graph. Server 200 performs character recognition on the text, obtains multiple characters in the text, determines the relationships between the multiple characters, and automatically constructs a character relationship graph of the text, which is then sent to client 410. Client 410 displays the character relationship graph in the human-computer interaction interface, thereby providing a reference for users to read and analyze the text.

[0073] As another example, this can be used in text recommendation applications. Server 200 performs role recognition on the text, obtains multiple roles in the text, and determines the character attributes (such as personality, skills, etc.) of each role; Server 200 obtains user information (such as user profile) sent by client 410, determines the role that matches the user information based on the character attributes, and recommends text containing that role to the user, thereby improving the efficiency of text recommendation.

[0074] The embodiments of this application can be implemented with the help of cloud technology, which refers to a hosting technology that unifies a series of resources such as hardware, software, and networks within a wide area network or local area network to realize the computation, storage, processing, and sharing of data.

[0075] Cloud technology is a general term encompassing network technology, information technology, integration technology, management platform technology, and application technology based on the cloud computing business model. It can form resource pools, allowing for on-demand use with flexibility and convenience. Cloud computing technology will become a crucial support. The backend services of cloud computing systems require substantial computing and storage resources.

[0076] As an example, server 200 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. Terminal 400 can be various types of user terminals such as smartphones, tablets, in-vehicle terminals, smart wearable devices, laptops, and desktop computers. Terminal 400 and server 200 can be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment.

[0077] The following will explain... Figure 1 The structure of server 200 in [the document / reference]. See [the document / reference]. Figure 2 , Figure 2 This is a schematic diagram of the structure of the server 200 provided in the embodiments of this application. Figure 2 The server 200 shown includes at least one processor 210, memory 240, and at least one network interface 220. The various components of server 200 are coupled together via a bus system 230. It is understood that the bus system 230 is used to implement communication between these components. In addition to a data bus, the bus system 230 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2 The general labeled all buses as Bus System 230.

[0078] Processor 210 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0079] Memory 240 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 240 described in this application embodiment is intended to include any suitable type of memory. Memory 440 may optionally include one or more storage devices physically located away from processor 210.

[0080] In some embodiments, memory 240 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0081] Operating system 241 includes system programs for handling various basic system services and performing hardware-related tasks, such as framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks; network communication module 242 is used to reach other computing devices via one or more (wired or wireless) network interfaces 220, exemplary network interfaces 220 include: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

[0082] In some embodiments, the character recognition device in the text provided in this application can be implemented in software. Figure 2 A character recognition device 243 for text stored in memory 240 is shown. This device can be software in the form of programs or plugins, and includes the following software modules: a first entity recognition module 2431, a second entity recognition module 2432, and a classification module 2433. These modules can be logical functional modules, and therefore can be arbitrarily combined or further divided according to the functions they implement. The functions of each module will be described below.

[0083] Below, by Figure 1 The following is an example of server 200 executing the text role recognition method provided in this application embodiment. See also... Figure 3 , Figure 3 This is a flowchart illustrating the role recognition method in text provided in the embodiments of this application, which will be combined with... Figure 3 The steps shown are explained.

[0084] In step S101, multiple role candidate words are extracted from the text.

[0085] In some embodiments, multiple words in the text are sequentially extracted using multiple sliding windows of different lengths; the relevance of each word is determined, and words with a relevance exceeding a relevance threshold are identified as candidate words for roles.

[0086] As a successor Figure 6 Example, Figure 6 This is a schematic diagram of the structure of the machine learning model provided in the embodiment of this application. The first candidate role entity recognition model 601 sequentially extracts multiple words from the text, determines the relevance of each word, and identifies words with a relevance exceeding the relevance threshold as role candidate words.

[0087] As an example, the first candidate character entity recognition model 601 can be a neural network model, which can include various types, such as convolutional neural network (CNN) models, recurrent neural network (RNN) models, and Chinese language models (N-Gram), etc.

[0088] As an example, relevance is used to measure whether a word is appropriate for a given phrase. For instance, in the phrase "funny movie review," "movie" is a more appropriate phrase than "laughing movie," therefore, "movie" has a higher relevance than "laughing movie." The relevance threshold can be a parameter obtained during the training of a machine learning model, or it can be a value set by the user, client, or server.

[0089] In some embodiments, text is acquired and the following preprocessing is performed on the text: the text is divided into multiple sentences according to a symbol list and the symbols in each sentence are filtered out; multiple role candidate words are extracted from each sentence of the preprocessed text.

[0090] As an example, the symbol list contains several punctuation marks that can divide sentences, such as period, semicolon, comma, question mark, exclamation mark, etc.

[0091] Taking a novel as an example, a novel is a very long text containing various symbols, requiring some preprocessing before the subsequent extraction of candidate words for characters. Figure 7 For example, Figure 7 The new word discovery model in China is Figure 6 An example of the first candidate role entity recognition model 601 in the model. Figure 7 The named entity recognition model in is Figure 6An example of the second candidate character entity recognition model 602 in the model can be supplemented with a text preprocessing module before the new word discovery model and the named entity recognition model. This text preprocessing module includes a symbol processing module and a text segmentation module. The text segmentation module divides the novel text into multiple sentences based on a pre-built symbol list, and the text symbol processing module filters the symbols in the segmented sentences.

[0092] For example, taking the text "The weather is so nice today, Xiaoming wants to ask Xiaohong out to play. But Xiaohong has to do her homework today, so she refuses Xiaoming," and the symbol list contains a period, we first divide the text into two sentences: "The weather is so nice today, Xiaoming wants to ask Xiaohong out to play." and "But Xiaohong has to do her homework today, so she refuses Xiaoming." Then we filter the symbols from these two sentences to get "The weather is so nice today, Xiaoming wants to ask Xiaohong out to play" and "But Xiaohong has to do her homework today, so she refuses Xiaoming."

[0093] This application embodiment, by preprocessing the text, can avoid the interference of symbols in the extraction process of character candidate words. Therefore, it can process not only text input at the paragraph and sentence level, but also complete text input (such as novels), which has strong compatibility and can improve the speed and accuracy of extracting multiple character candidate words from text.

[0094] In step S102, at least one matching parameter corresponding to each role candidate word is obtained.

[0095] As a successor Figure 6 For example, the first candidate role entity recognition model 601 obtains at least one matching parameter corresponding to each role candidate word, wherein the type of matching parameter includes word frequency, cohesion and degree of freedom.

[0096] In some embodiments, see Figure 4 , Figure 4 This is a flowchart illustrating the role recognition method in text provided in the embodiments of this application, based on... Figure 3 Step S102 may include steps S1021 to S1024.

[0097] In step S1021, the word frequency of the role candidate words in the text is determined.

[0098] In some embodiments, word frequency refers to the number of times a given word appears in a text.

[0099] Taking the text "The weather is so nice today, Xiaoming wants to ask Xiaohong to go out and play. But Xiaohong has to do her homework today, so she refuses Xiaoming" as an example, and the candidate words for the roles are "Xiaoming" and "Xiaohong", the word frequency of "Xiaoming" is 2, and the word frequency of "Xiaohong" is 2.

[0100] In step S1022, the role candidate words are divided into multiple morphemes, and the probability of each morpheme appearing in the text and the probability of the role candidate words appearing in the text are determined.

[0101] As an example, the types of morphemes include characters and words. Taking the role candidate word "Xiaoming" as an example, "Xiaoming" can be divided into two morphemes: "Xiao" and "Ming". Taking the role candidate word "football field" as an example, "football field" can be divided into two morphemes: "football" and "field" or "foot" and "field".

[0102] In some embodiments, role candidate words are divided into multiple morphemes, and the following processing is performed on each morpheme: the ratio between the number of times the morpheme appears in the text and the number of times all role candidate words appear in the text is determined as the probability of the morpheme appearing in the text; the ratio between the number of times the role candidate word appears in the text and the number of times all role candidate words appear in the text is determined as the probability of the role candidate word appearing in the text.

[0103] For example, taking the text "The weather is really nice today, and Xiaoming wants to ask Xiaohong to go out and play. But Xiaohong has to do her homework today, so she refuses Xiaoming," and the candidate words for the roles are "Xiaoming" and "Xiaohong," we can divide "Xiaoming" into two morphemes, "xiao" and "ming," and "Xiaohong" into two morphemes, "xiao" and "hong." The morpheme "xiao" appears 4 times in the text, the morpheme "ming" appears 2 times, the morpheme "hong" appears 2 times, the candidate word for the role "Xiaoming" appears 2 times, and the candidate word for the role "Xiaohong" appears 2 times. The total number of times all candidate words for the roles appear in the text is 4. Thus, the probability of the morpheme "xiao" appearing in the text is 4 / 4 = 1, the probability of the morpheme "ming" appearing in the text is 2 / 4 = 0.5, and the probability of the candidate word "Xiaoming" appearing in the text is 2 / 4 = 0.5.

[0104] In step S1023, the cohesion of the role candidate words is determined based on the probability of each morpheme appearing in the text and the probability of the role candidate words appearing in the text.

[0105] In some embodiments, the occurrence probabilities of each morpheme in the text are multiplied to obtain the product result; the ratio between the occurrence probability of the role candidate word in the text and the product result is determined as the third ratio; the third ratio is logarithmically processed, and the logarithmic result is determined as the cohesion of the role candidate word.

[0106] For example, taking the text "The weather is great today. Xiaoming wants to ask Xiaohong out to play. But Xiaohong has to do her homework today, so she refuses Xiaoming" and the candidate role words being "Xiaoming" and "Xiaohong" as an example, the appearance probability of the morpheme "小" in the text is 4 / 4 = 1, the appearance probability of the morpheme "明" in the text is 2 / 4 = 0.5, and the appearance probability of the candidate role word "Xiaoming" in the text is 2 / 4 = 0.5. Thus, the cohesion degree of the candidate role word "Xiaoming" is 0.5 / (1 × 0.5) = 1.

[0107] In some embodiments, a high cohesion degree indicates that the co-occurrence probability of two morphemes is much greater than the product probability of the free combination of the two morphemes, indicating that the combination of these two morphemes is more reasonable. If a candidate role word contains multiple combination patterns, for example, the candidate role word "足球场 (football field)" can be composed of the morphemes "足球 (football)" and "场 (field)", or can be composed of the morphemes "足 (foot)" and "球场 (ball field)", in this way, the minimum value of the cohesion degrees of all combinations of the candidate role word can be taken as the cohesion degree of the candidate role word.

[0108] For example, when the candidate role word "足球场 (football field)" is divided into the morphemes "足球 (football)" and "场 (field)", the cohesion degree of the candidate role word "足球场 (football field)" is 0.7; when the candidate role word "足球场 (football field)" is divided into the morphemes "足 (foot)" and "球场 (ball field)", the cohesion degree of the candidate role word "足球场 (football field)" is 0.6. At this time, the cohesion degree of the candidate role word "足球场 (football field)" can be determined as 0.6.

[0109] In step S1024, determine the left information entropy and the right information entropy of the candidate role word, and determine the freedom degree of the candidate role word according to the left information entropy and the right information entropy of the candidate role word.

[0110] In some embodiments, determine multiple left adjacent characters and multiple right adjacent characters of the candidate role word in the text; determine the sub-information entropy corresponding to each left adjacent character, and determine the sub-information entropy corresponding to each right adjacent character; take the opposite of the sum of the sub-information entropies corresponding to each left adjacent character as the left information entropy, and take the opposite of the sum of the sub-information entropies corresponding to each right adjacent character as the right information entropy; when the left information entropy is greater than the right information entropy, take the right information entropy as the freedom degree, and when the left information entropy is not greater than the right information entropy, take the left information entropy as the freedom degree.

[0111] For example, taking the text "The weather is great today. Xiaoming wants to ask Xiaohong out to play. But Xiaohong has to do her homework today, so she refuses Xiaoming" and the candidate role words being "Xiaoming" and "Xiaohong" as an example, the left adjacent characters of the candidate role word "Xiaoming" in the text include "好 (good)" and "了 (particle)", the right adjacent character of the candidate role word "Xiaoming" in the text includes "想 (want)", the left adjacent characters of the candidate role word "Xiaohong" in the text include "约 (ask)" and "是 (is)", and the right adjacent characters of the candidate role word "Xiaohong" in the text include "出 (out)" and "今 (today)".

[0112] As an example, determining the sub-information entropy corresponding to each left neighbor word may include: performing the following processing for each left neighbor word: determining the ratio between the number of occurrences of the left neighbor word in the text and the number of occurrences of all neighbor words of the role candidate word in the text as a first ratio; performing a logarithmic operation on the first ratio, and determining the product between the logarithmic operation result and the first ratio as the sub-information entropy corresponding to the left neighbor word.

[0113] As an example, determining the sub-information entropy corresponding to each right neighbor may include: performing the following processing for each right neighbor: determining the ratio between the number of times the right neighbor appears in the text and the number of times all neighboring words of the role candidate word appear in the text as a second ratio; performing a logarithmic operation on the second ratio, and determining the product between the logarithmic operation result and the second ratio as the sub-information entropy corresponding to the right neighbor.

[0114] For example, the degree of freedom of a role candidate word is used to measure the richness of the left and right neighboring words of a role candidate word, and can be calculated using formula (1).

[0115] (1)

[0116] in, Indicating candidate words for roles Left information entropy, Indicating candidate words for roles The right information entropy, the left information entropy, and the right information entropy of the role candidate words are calculated as follows:

[0117] (2)

[0118] (3)

[0119] in, Indicating candidate words for roles The left-adjacent character, express The probability among all adjacent words (i.e., (Number of occurrences in the text / Number of occurrences of all adjacent words in the text) y Indicates the right-adjacent word of the candidate word c. express y The probability among all adjacent words (i.e., (Number of occurrences in the text / Number of occurrences of all adjacent words in the text).

[0120] In step S103, at least one character candidate word is selected from multiple character candidate words as the first candidate character entity based on at least one matching parameter corresponding to each character candidate word.

[0121] In some embodiments, among a plurality of candidate role words, a candidate role word that meets at least one of the following conditions is selected as the first candidate role entity: the word frequency of the candidate role word in the text exceeds a word frequency threshold, the cohesion of the candidate role word exceeds a cohesion threshold, and the degree of freedom of the candidate role word exceeds a degree of freedom threshold.

[0122] As a successor Figure 6 For example, the first candidate role entity recognition model 601 selects a role candidate word from multiple role candidate words that meets at least one of the following conditions as the first candidate role entity: the word frequency of the role candidate word in the text exceeds the word frequency threshold, the cohesion of the role candidate word exceeds the cohesion threshold, and the degree of freedom of the role candidate word exceeds the degree of freedom threshold.

[0123] For example, the word frequency threshold, cohesion threshold, and degree of freedom threshold can be parameters obtained during the training of the machine learning model, or they can be values ​​set by the user, client, or server.

[0124] Using only word frequency to filter candidate words for roles can easily lead to some high-frequency but incomplete candidate words being mistaken for first-line candidate roles. Therefore, this embodiment selects candidate words based on their degree of freedom, cohesion, and word frequency, ensuring the completeness of the identified first-line candidate roles and further improving the efficiency and accuracy of role identification from text.

[0125] In step S104, the question and text corresponding to the text are merged to obtain the merged text.

[0126] As a successor Figure 6 For example, the second candidate role entity recognition model 602 fuses the question and text corresponding to the text (e.g., splices them together) to obtain the fused text.

[0127] For example, the second candidate character entity recognition model 602 can be a neural network model, which can include various types, such as convolutional neural network models, recurrent neural network models, and BERT-MRC models.

[0128] For example, first construct the query corresponding to the text, denoted as a word sequence. ,in, Indicates the first in the query One word, The length of the question is indicated. Considering that this embodiment only needs to extract candidate role entities and does not require entity type classification, the constructed question can be "An entity refers to a concrete, objectively existing thing, usually referring to a real, functioning person, organization, or institution, etc." For text, the formal representation is a sequence of characters. ,in, Indicates the first in the text One word, Indicates the text length. The query and the text are concatenated to form the merged text, formally represented as follows: .

[0129] In this embodiment, since the constructed question does not include the entity type, it can not only improve the fusion speed of the question and the text, but also improve the speed of extracting the second candidate role entity.

[0130] In some embodiments, the following preprocessing is performed on the text: the text is divided into multiple sentences according to the symbol list, and the symbols in each sentence are filtered out; the preprocessed text and the corresponding question are merged to obtain the merged text. Here, the process of performing preprocessing on the text is similar to the process of performing preprocessing on the text in step S101, and will not be described again here.

[0131] In step S105, entity recognition processing is performed on the fused text to obtain at least one second candidate role entity.

[0132] As a successor Figure 6 For example, entity recognition processing is performed on the fused text using the second candidate role entity recognition model 602 to obtain at least one second candidate role entity.

[0133] In some embodiments, see Figure 5 , Figure 5 This is a flowchart illustrating the role recognition method in text provided in the embodiments of this application, based on... Figure 3 Step S105 may include steps S1051 to S1053.

[0134] In step S1051, feature extraction processing is performed on the fused text to obtain a feature sequence.

[0135] In some embodiments, the fused text is divided into multiple morphemes, and feature extraction processing is performed on each morpheme to obtain multiple sub-features that correspond one-to-one with the multiple morphemes. The multiple sub-features are combined to obtain a feature sequence.

[0136] As an example, feature extraction is performed on the fused text to obtain a feature sequence, represented as follows: ,in Indicates the first The output vector of each node (i.e., the morpheme mentioned above), Represents the vector dimension.

[0137] In step S1052, the feature sequence is mapped to obtain at least one set of positions.

[0138] Here, the set of positions includes a start position and an end position.

[0139] In some embodiments, the feature sequence is divided into multiple sub-features, wherein each sub-feature corresponds one-to-one with a multiple character in the text; each sub-feature is mapped to a start probability belonging to a start position and an end probability belonging to an end position; at least one sub-feature with a start probability greater than a start probability threshold is selected as a start sub-feature, and at least one sub-feature with an end probability greater than an end probability threshold is selected as an end sub-feature; based on the selected at least one start sub-feature and at least one end sub-feature, at least one candidate start-end feature set is constructed, wherein the candidate start-end feature set includes a start sub-feature and an end sub-feature; a target start-end feature set is determined from the at least one candidate start-end feature set; the character corresponding to the start sub-feature in the target start-end feature set is determined as the start position in the position set, and the character corresponding to the end sub-feature in the target start-end feature set is determined as the end position in the position set.

[0140] As an example, determining the target start-end feature set from at least one candidate start-end feature set may include: performing the following processing for each candidate start-end feature set: fusing the start sub-features and end sub-features in the candidate start-end feature set to obtain a first fused feature, and mapping the first fused feature to the probability of belonging to the same entity; and determining the candidate start-end feature set whose probability of belonging to the same entity is greater than the entity probability threshold from at least one candidate start-end feature set as the target start-end feature set.

[0141] As an example, the starting probability threshold, ending probability threshold, and entity probability threshold can be parameters obtained during the training of the machine learning model, or values ​​set by the user, client, or server.

[0142] For example, two lengths can be used. The tag sequence is used to determine the start and end positions of the second candidate role entity, denoted as follows: , Representing positions respectively Is it the start and end position of an entity? Also, a... matrix ,in, Indicate the start position and end position respectively. Is the string an entity?

[0143] The calculation method is as follows:

[0144] (4)

[0145] (5)

[0146] (6)

[0147] (7)

[0148] in, This represents the probability that each position is the starting position. This represents the probability that each position is the end position. These are model parameters.

[0149] based on , The calculation method is as follows:

[0150] (8)

[0151] (9)

[0152] Finally, through ,Will The string with a value of 1 is output as the second candidate character entity.

[0153] In step S1053, the following processing is performed for each location set: the text corresponding to the start position in the location set, the text between the start and end positions in the location set, and the text corresponding to the end position in the location set are combined, and the combination result is determined as the second candidate role entity.

[0154] Taking the text "Yang A saw Mu BC drinking in a tavern" as an example, the starting position in the position set is "Mu" and the ending position is "C". Thus, it can be determined that "Mu BC" is a second candidate character entity.

[0155] This application embodiment constructs a question that does not contain entity types, and identifies the second candidate role entity based on the text after fusing the question. Compared with identifying roles in text through a general named entity recognition model, it can solve the problem of incomplete role entity recognition caused by the special nature of role types, thereby improving the completeness of the identified second candidate role entities.

[0156] In some embodiments, after step S105, the following processing may be performed for each second candidate role entity: dividing the second candidate role entity into multiple morphemes, and determining the occurrence probability of each morpheme in the text and the occurrence probability of the second candidate role entity in the text; determining the cohesion of the second candidate role entity based on the occurrence probability of each morpheme in the text and the occurrence probability of the second candidate role entity in the text; determining the left information entropy and right information entropy of the second candidate role entity, and determining the degrees of freedom of the second candidate role entity based on the left information entropy and right information entropy of the second candidate role entity; filtering second candidate role entities among multiple second candidate role entities that satisfy at least one of the following conditions: the word frequency of the second candidate role entity in the text does not exceed a word frequency threshold, the cohesion of the second candidate role entity does not exceed a cohesion threshold, and the degrees of freedom of the second candidate role entity does not exceed a degrees of freedom threshold.

[0157] As an example, the process of filtering the second candidate role entity based on the matching parameters and the process of selecting the first candidate role entity based on the matching parameters in step S103 will not be described again here.

[0158] This application embodiment takes into account the problem that the extracted second candidate role entity may be incomplete. By using the matching parameters obtained in step S102 to filter the second candidate role entity, the efficiency and accuracy of identifying roles from text can be further improved.

[0159] In step S106, role classification is performed based on at least one first candidate role entity and at least one second candidate role entity to obtain the roles in the text.

[0160] As a successor Figure 6 For example, the role classification model 603 performs role classification processing based on at least one first candidate role entity and at least one second candidate role entity to obtain the roles in the text.

[0161] For example, the role classification model 603 can be a neural network model, which can include various types, such as convolutional neural network models, recurrent neural network models, and bidirectional long short-term attention networks (BiLSTM, Bi-directional Long Short-Term Memory).

[0162] In some embodiments, duplicate candidate role entities are filtered out from at least one first candidate role entity and at least one second candidate role entity (e.g., by taking the union). For each candidate role entity obtained after filtering, the following processing is performed: the statement in which the candidate role entity is located is determined, and the candidate role entity and the statement are combined to obtain an entity-statement pair; entity features and text features are extracted from the entity-statement pair, and the entity features and text features are fused (e.g., by concatenation) to obtain a second fused feature; the second fused feature is mapped to the probability of belonging to a role entity; when the probability of belonging to a role entity is greater than a role probability threshold, and the word frequency of the candidate role entity in the text is greater than a role word frequency threshold, the candidate role entity is determined to be a role in the text.

[0163] As an example, extracting entity features and text features from entity sentence pairs may include: extracting multiple word vectors from the entity sentence pairs and determining the average of the multiple word vectors as the entity feature; encoding the entity sentence pairs in the direction from the start position to the end position to obtain a forward encoding vector; encoding the entity sentence pairs in the direction from the end position to the start position to obtain a backward encoding vector; and fusing the forward encoding vector and the backward encoding vector to obtain the text feature.

[0164] As a successor Figure 7 For example, for each candidate character entity obtained after filtering, the following processing is performed: Using the candidate character entity and the text, all <entity, text> pairs are constructed, where the entity represents the candidate character entity and the text represents the sentence containing the candidate character entity. Then, a pre-trained novel character entity classification model is used to obtain the category of the candidate character entity. The novel character entity classification model takes <entity, text> pairs as input, and its formal representation is as follows: , where M represents the candidate character entity and C represents the novel sentence in which M is located.

[0165] For example, a novel character entity classification model extracts the entity features and text features for each input, which can be formally represented as follows: The feature extraction process specifically includes: first, using pre-trained word vectors to obtain the vector matrix of the input text, denoted as... The entity feature extraction process is as follows: The entity features are obtained by averaging all word vectors of the entity. Text features can be extracted using BiLSTM, and the specific calculation method is as follows:

[0166] (10)

[0167] (11)

[0168] in, It is the length of the entity.

[0169] For example, combining entity features and text features as classification features. (i.e., the second fusion feature mentioned above), and uses the Softmax layer to determine whether the entity is a fictional character. The classification result is represented as follows:

[0170] (12)

[0171] (13)

[0172] in, This represents the probability that a candidate character is a fictional character. For model parameters, This refers to the category of the entity predicted by the model (including whether the candidate character entity is a fictional character or not). In this way, the characters in the text can be obtained.

[0173] This application embodiment identifies roles among candidate role entities through a separate role classification model, which can further ensure the completeness of the identified roles, thereby improving the efficiency and accuracy of role identification from text.

[0174] The following uses a novel as an example to illustrate the character recognition method in the text provided in this application.

[0175] Novels typically contain multiple distinct characters with interconnected relationships. Character recognition technology can be used to extract different characters from a novel and display them as a character list in the novel's synopsis, accelerating user comprehension. It can also be used to determine the relationships between multiple characters after extraction, automatically constructing a character relationship graph for user reading and text analysis. Furthermore, it can be used to determine the attributes of multiple characters and, based on these attributes, identify characters that match the user's profile and recommend novels containing those characters, thus improving recommendation efficiency. Finally, it can be used to establish a mapping between novels and characters, so that when users search for characters using search engines, novels with a mapping relationship to the searched characters can be directly displayed, improving search hit rate.

[0176] This application embodiment mainly includes a candidate character entity recognition module (including a new word discovery model and a named entity recognition model) and a novel character entity classification model. Specifically, for the main text of the novel, the new word discovery model (i.e., the first candidate character entity recognition model 601 mentioned above) and the named entity recognition model (i.e., the second candidate character entity recognition model 602 mentioned above) are used to extract candidate character entities, and the novel character entity classification model (i.e., the character classification model 603 mentioned above) is used to extract novel character entities from the candidate character entities.

[0177] This application's embodiments simultaneously employ a new word discovery model and a named entity recognition model to address the issues of incomplete and incomplete entity extraction in related technologies. Furthermore, the use of a separate novel character entity classification model effectively improves the recall and accuracy of novel characters.

[0178] See Figure 7 , Figure 7 This is a schematic diagram of the structure of the character recognition framework provided in this application embodiment. The character recognition framework includes a text preprocessing module, an entity recognition module, a novel character entity classification model, and an entity frequency statistics module, which will be described below.

[0179] (a) Text preprocessing module

[0180] The novel's main text is an extremely long document containing various symbols, requiring preprocessing before subsequent recognition. The text preprocessing module includes a symbol processing module and a text segmentation module. The text segmentation module divides the novel's main text into multiple sentences based on a pre-built symbol list. The symbol processing module cleans the symbols in the novel's main text; since new words discovered by the new word discovery model should not contain various symbols, it is necessary to filter out all symbols from the text.

[0181] (ii) Entity Recognition Module

[0182] The novel's text contains a large number of entities of various types, such as characters, organizations, and locations. To ensure both the quantity (extracting as many characters as possible from the novel) and the quality (ensuring the highest possible accuracy in character extraction) of extracted characters, all candidate character entities are first extracted from the novel's text. Specifically, a new word discovery model and a named entity recognition model are used simultaneously to extract candidate character entities from the novel.

[0183] (1) New word discovery model

[0184] The new word discovery model aims to uncover linguistic features through an unsupervised approach, thereby determining which strings in a novel text might be entities.

[0185] In some embodiments, the given text of a novel consists of a set of sentences. ,in, Indicates the first One sentence. The number of sentences is represented. First, all candidate words for roles are obtained using a Chinese language model (N-Gram), and the word frequency of all candidate words is counted. N-Gram refers to using a sliding window to obtain all strings of a specific length. In this embodiment, all candidate words for roles are formally represented as follows: ,in, This indicates candidate words for roles and their corresponding word frequencies. Indicates the number of candidate words for the role.

[0186] As an example, using only word frequency to filter candidate words for roles can easily lead to some frequently occurring but incomplete candidate words being mistaken for candidate role entities. Therefore, embodiments of this application simultaneously calculate the degrees of freedom and cohesion of candidate words for roles.

[0187] In some embodiments, cohesion is used to measure the appropriateness of a word pairing for a character; for example, in a list of "funny movies," "movies" is a more appropriate pairing than "laughing movies." The formula for calculating cohesion is as follows:

[0188] (14)

[0189] in, express The probability of appearing in the text (i.e., (Number of occurrences in the text / Number of occurrences of all candidate character terms in the text); express The probability of appearing in the text (i.e., (Number of occurrences in the text / Number of occurrences of all candidate character terms in the text); express The probability of appearing in the text (i.e., (Number of occurrences in the text / Number of occurrences of all candidate words in the text).

[0190] A high cohesion level indicates that the probability of two words co-occurring is much greater than the product probability of two words freely combined, suggesting that the two words are more reasonably paired. If a word contains multiple collocations, such as "football field" which can be composed of "football" + "field" or "foot" + "field," then the cohesion level of the entire word can be taken as the value with the lowest cohesion level among all combinations. Formalized as:

[0191] (15)

[0192] in, Indicating candidate words for roles The left part of the combination, Each represents a candidate word for a role. The right part of the combination.

[0193] In some embodiments, degrees of freedom are used to measure the richness of a word's left and right neighboring characters. Information entropy can be used to represent degrees of freedom, and the calculation method is as follows:

[0194] (16)

[0195] in, Indicating candidate words for roles Left information entropy, Indicating candidate words for roles The right information entropy is calculated as follows:

[0196] (17)

[0197] (18)

[0198] in, Indicating candidate words for roles The left-adjacent character, express The probability among all adjacent words (i.e., (Number of occurrences in the text / Number of occurrences of all adjacent words in the text) y Indicates the right-adjacent word of the candidate word c. express y The probability among all adjacent words (i.e., (Number of occurrences in the text / Number of occurrences of all adjacent words in the text).

[0199] In some embodiments, candidate words for roles are filtered by pre-set word frequency thresholds, cohesion thresholds, and degrees of freedom thresholds. For example, candidate words for roles whose word frequency does not exceed the word frequency threshold, whose cohesion does not exceed the cohesion threshold, and whose degrees of freedom do not exceed the degrees of freedom threshold are filtered out, thereby allowing candidate role entities to be selected from the candidate words for roles.

[0200] In some embodiments, the new word discovery model is not limited to the specific model structure described above, but may also be other neural network models, which will not be elaborated here.

[0201] (2) Named Entity Recognition Model

[0202] In some embodiments, the named entity recognition model differs from traditional sequence labeling-based methods. This application uses a named entity recognition model based on Machine Reading Comprehension (MRC). Specifically, the BERT-MRC model can be used. For the input text, a question is constructed, the question and the text are concatenated as input, the BERT model is used to extract features, and based on the extracted features, it is directly determined whether a string is an entity.

[0203] As an example, first construct a question query, denoted as a word sequence. ,in, Indicates the first in the query One word, The length of the question is indicated. Considering that this embodiment only needs to extract candidate role entities and does not need to classify entity types, the constructed question can be "an entity refers to a concrete thing that objectively exists, usually referring to a real person, organization, or institution that actually exists and is in operation."

[0204] As an example, a given input text can be formally represented as a sequence of characters. ,in, Indicates the first in the text One word, Indicates the text length. The question and text are concatenated as input to the BERT model, formally represented as follows: Features are extracted using the BERT model, and the features are represented as follows: ,in Indicates the first The output vector of each node Represents the vector dimension.

[0205] As an example, based on the output of the BERT model, two lengths can be used. The start and end positions of an entity are determined by the tag sequence, denoted as follows: , Representing positions respectively Is it the start and end position of an entity? Also, a... matrix , Indicate the start position and end position respectively. Is the string an entity?

[0206] The calculation method is as follows:

[0207] (19)

[0208] (20)

[0209] (twenty one)

[0210] (twenty two)

[0211] in, This represents the probability that each position is the starting position. This represents the probability that each position is the end position. These are model parameters.

[0212] based on , The calculation method is as follows:

[0213] (twenty three)

[0214] (twenty four)

[0215] Finally, through ,Will The string with a value of 1 is output as a candidate role entity.

[0216] This application embodiment takes into account the problem that the candidate role entities extracted by the named entity recognition model are incomplete. The word frequency threshold, cohesion threshold, and degree of freedom threshold calculated in the new word discovery module can be used to filter the candidate role entities generated by the named entity recognition module. For example, candidate role entities whose word frequency does not exceed the word frequency threshold, whose cohesion does not exceed the cohesion threshold, and whose degree of freedom does not exceed the degree of freedom threshold can be filtered.

[0217] In some embodiments, the named entity recognition model is not limited to the specific model structure described above, but may also be other neural network models, which will not be elaborated here.

[0218] (III) Novel Character Entity Classification Model

[0219] In some embodiments, all <entity, text> pairs are constructed using candidate character entities and the main text of the novel, where the entity represents a candidate character entity and the text represents the sentence in the novel containing the candidate character entity. A pre-trained novel character entity classification model is then used to obtain the category of the candidate character entity. The novel character entity classification model takes <entity, text> pairs as input and is formally represented as follows: , where M represents the candidate character entity and C represents the novel sentence in which M is located.

[0220] As an example, the novel character entity classification model extracts the entity features and text features of each input separately, which can be formally represented as follows: The feature extraction process specifically includes: first, using pre-trained word vectors to obtain the vector matrix of the input text, denoted as... The entity feature extraction process is as follows: The entity features are obtained by averaging all word vectors of the entity; text features are extracted using BiLSTM, and the specific calculation method is as follows:

[0221] (25)

[0222] (26)

[0223] in, It is the length of the entity.

[0224] As an example, entity features and text features are combined as classification features. (i.e., the second fusion feature mentioned above), and uses the Softmax layer to determine whether the entity is a fictional character. The classification result is represented as follows:

[0225] (27)

[0226] (28)

[0227] in, This represents the probability that a candidate character is a fictional character. For model parameters, It is the category of the entity predicted by the model (including whether the candidate entity is a fictional character or not).

[0228] In some embodiments, the novel character entity classification model is not limited to the specific model structure described above, but may also be other neural network models, which will not be elaborated here.

[0229] (iv) Entity Frequency Statistics Module

[0230] In some embodiments, based on the entity classification results described above, all role entities can be filtered out and their frequencies counted. Formal representation is as follows: ,in These represent the character entity and its corresponding frequency. By filtering through a pre-set frequency threshold, entities that meet the requirements are selected as novel characters.

[0231] This application embodiment breaks down the traditional named entity recognition (NAME) model's process of extracting character entities into two parts: candidate character entity extraction and novel character entity classification. This embodiment can be applied to novel character extraction tasks during the construction of a novel-specific knowledge graph, as well as intent recognition applications. In the candidate character entity extraction part, a new word discovery model is used in conjunction with a named entity recognition model based on a reading comprehension framework, improving the completeness and recall rate of the extracted candidate character entities. Furthermore, a separate novel character entity classification model is used, improving the accuracy of the extracted novel characters. Testing showed that the accuracy of the extracted novel characters reached 96.7%, and the novel characters achieved 70% coverage of novel-related queries in search engines.

[0232] The following is combined Figure 2 The implementation of the role recognition device in the text provided in this application embodiment is an exemplary structure of a software module.

[0233] In some embodiments, such as Figure 2 As shown, the software modules in the role recognition device 243 stored in the text in the memory 240 may include: a first entity recognition module 2431, used to extract multiple role candidate words from the text and obtain at least one matching parameter corresponding to each role candidate word; the first entity recognition module 2431 is also used to select at least one role candidate word from the multiple role candidate words as a first candidate role entity based on at least one matching parameter corresponding to each role candidate word; a second entity recognition module 2432, used to fuse the question and text corresponding to the text to obtain fused text; the second entity recognition module 2432 is also used to perform entity recognition processing on the fused text to obtain at least one second candidate role entity; and a classification module 2433, used to perform role classification processing based on at least one first candidate role entity and at least one second candidate role entity to obtain the roles in the text.

[0234] In the above scheme, the matching parameters include word frequency, cohesion, and degrees of freedom. The first entity recognition module 2431 is also used to perform the following processing for each role candidate word: determine the word frequency of the role candidate word in the text; divide the role candidate word into multiple morphemes and determine the probability of occurrence of each morpheme in the text, as well as the probability of occurrence of the role candidate word in the text, wherein the types of morphemes include characters and words; determine the cohesion of the role candidate word based on the probability of occurrence of each morpheme in the text and the probability of occurrence of the role candidate word in the text; determine the left information entropy and right information entropy of the role candidate word, and determine the degrees of freedom of the role candidate word based on the left information entropy and right information entropy of the role candidate word.

[0235] In the above scheme, the first entity recognition module 2431 is further used to determine multiple left adjacent characters and multiple right adjacent characters of the role candidate word in the text; determine the sub-information entropy corresponding to each left adjacent character, and determine the sub-information entropy corresponding to each right adjacent character; determine the left information entropy by summing the sub-information entropies corresponding to each left adjacent character, and determine the right information entropy by summing the sub-information entropies corresponding to each right adjacent character; when the left information entropy is greater than the right information entropy, the right information entropy is determined as the degree of freedom, and when the left information entropy is not greater than the right information entropy, the left information entropy is determined as the degree of freedom.

[0236] In the above scheme, the first entity recognition module 2431 is also used to perform the following processing for each left neighbor word: determine the ratio between the number of occurrences of the left neighbor word in the text and the number of occurrences of all neighbor words of the role candidate word in the text as a first ratio; perform logarithmic operation on the first ratio, and determine the product between the logarithmic operation result and the first ratio as the sub-information entropy corresponding to the left neighbor word;

[0237] In the above scheme, the first entity recognition module 2431 is also used to perform the following processing for each right neighbor: determine the ratio between the number of times the right neighbor appears in the text and the number of times all neighboring words of the role candidate word appear in the text as a second ratio; perform logarithmic operation on the second ratio, and determine the product between the logarithmic operation result and the second ratio as the sub-information entropy corresponding to the right neighbor.

[0238] In the above scheme, the first entity recognition module 2431 is further configured to perform the following processing for each morpheme: determine the probability of the morpheme appearing in the text as the ratio between the number of times the morpheme appears in the text and the number of times all role candidate words appear in the text; determine the probability of the role candidate words appearing in the text as the ratio between the number of times the role candidate words appear in the text and the number of times all role candidate words appear in the text.

[0239] In the above scheme, the first entity recognition module 2431 is also used to multiply the occurrence probability of each morpheme in the text to obtain the product result; determine the ratio between the occurrence probability of the role candidate word in the text and the product result as the third ratio; perform logarithmic operation on the third ratio, and determine the logarithmic operation result as the cohesion of the role candidate word.

[0240] In the above scheme, the first entity recognition module 2431 is also used to select a role candidate word that meets at least one of the following conditions as the first candidate role entity from multiple role candidate words: the word frequency of the role candidate word in the text exceeds the word frequency threshold, the cohesion of the role candidate word exceeds the cohesion threshold, and the degree of freedom of the role candidate word exceeds the degree of freedom threshold.

[0241] In the above scheme, the second entity recognition module 2432 is also used to perform feature extraction processing on the fused text to obtain a feature sequence; perform mapping processing on the feature sequence to obtain at least one location set; and perform the following processing for each location set: combine the text corresponding to the start position in the location set, the text between the start and end positions in the location set, and the text corresponding to the end position in the location set, and determine the combination result as the second candidate role entity.

[0242] In the above scheme, the second entity recognition module 2432 is further configured to divide the feature sequence into multiple sub-features, wherein the multiple sub-features correspond one-to-one with multiple characters in the text; map each sub-feature to a start probability belonging to the start position and an end probability belonging to the end position; select at least one sub-feature with a start probability greater than a start probability threshold as a start sub-feature, and select at least one sub-feature with an end probability greater than an end probability threshold as an end sub-feature; construct at least one candidate start-end feature set based on the selected at least one start sub-feature and at least one end sub-feature, wherein the candidate start-end feature set includes one start sub-feature and one end sub-feature; determine the target start-end feature set from the at least one candidate start-end feature set; determine the character corresponding to the start sub-feature in the target start-end feature set as the start position in the position set, and determine the character corresponding to the end sub-feature in the target start-end feature set as the end position in the position set.

[0243] In the above scheme, the second entity recognition module 2432 is further configured to perform the following processing for each candidate start and end feature set: fuse the start sub-feature and end sub-feature in the candidate start and end feature set to obtain a first fused feature, and map the first fused feature to the probability of belonging to the same entity; in at least one candidate start and end feature set, determine the candidate start and end feature set whose probability of belonging to the same entity is greater than the entity probability threshold as the target start and end feature set.

[0244] In the above scheme, the second entity recognition module 2432 is further configured to perform the following processing for each second candidate role entity: divide the second candidate role entity into multiple morphemes, and determine the occurrence probability of each morpheme in the text and the occurrence probability of the second candidate role entity in the text, wherein the types of morphemes include characters and words; determine the cohesion of the second candidate role entity based on the occurrence probability of each morpheme in the text and the occurrence probability of the second candidate role entity in the text; determine the left information entropy and right information entropy of the second candidate role entity, and determine the degrees of freedom of the second candidate role entity based on the left information entropy and right information entropy of the second candidate role entity; filter the second candidate role entities that satisfy at least one of the following conditions among the multiple second candidate role entities: the word frequency of the second candidate role entity in the text does not exceed the word frequency threshold, the cohesion of the second candidate role entity does not exceed the cohesion threshold, and the degrees of freedom of the second candidate role entity does not exceed the degrees of freedom threshold.

[0245] In the above scheme, the classification module 2433 is further used to filter out duplicate candidate role entities among at least one first candidate role entity and at least one second candidate role entity; for each candidate role entity obtained after filtering, the following processing is performed: determine the sentence in which the candidate role entity is located, and combine the candidate role entity and the sentence to obtain an entity-sentence pair; extract entity features and text features from the entity-sentence pair, and fuse the entity features and text features to obtain a second fused feature; map the second fused feature to the probability of belonging to a role entity; when the probability of belonging to a role entity is greater than the role probability threshold, and the word frequency of the candidate role entity in the text is greater than the role word frequency threshold, the candidate role entity is determined to be a role in the text.

[0246] In the above scheme, the classification module 2433 is also used to extract multiple word vectors from the entity sentence pair and determine the mean of the multiple word vectors as the entity feature; encode the entity sentence pair in the direction from the start position to the end position to obtain the forward encoding vector; encode the entity sentence pair in the direction from the end position to the start position to obtain the backward encoding vector; and fuse the forward encoding vector and the backward encoding vector to obtain the text feature.

[0247] In the above scheme, the first entity recognition module 2431 is also used to acquire text and perform the following preprocessing on the text: divide the text into multiple sentences according to the symbol list and filter out the symbols in each sentence; extract multiple role candidate words from each sentence of the preprocessed text.

[0248] In some embodiments, the logic of the text role recognition method provided in this application can be implemented in a smart contract. Different nodes determine the roles in the text by calling their respective smart contracts, and the final role is determined by taking the intersection. This application embodiment can further improve the accuracy of role recognition from text through collaborative processing between multiple nodes.

[0249] This application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the role recognition method described above in this application.

[0250] This application provides a computer-readable storage medium storing computer-executable instructions. When these computer-executable instructions are executed by a processor, they cause the processor to execute the role recognition method in the text provided in this application, for example... Figure 3 , Figure 4 and Figure 5 The text shown illustrates a method for character recognition, using various computing devices including smart terminals and servers.

[0251] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0252] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0253] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hypertext Markup Language document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0254] As an example, computer-executable instructions can be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0255] In summary, the embodiments of this application extract candidate role entities from text using two different methods and determine the roles in the text based on the extracted candidate role entities. This ensures the diversity and comprehensiveness of the candidate role entities identified from the text, thereby improving the efficiency and accuracy of role identification from text. Furthermore, determining the candidate role entities in the text based on matching parameters or fused text separately ensures the completeness of the identified candidate role entities. This further improves the efficiency and accuracy of role identification from text.

[0256] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A method for role recognition in text, characterized in that, The method includes: Extract multiple candidate words for roles from the text, and obtain at least one matching parameter corresponding to each candidate word for a role; Based on at least one matching parameter corresponding to each of the said role candidate words, at least one role candidate word is selected from the plurality of role candidate words as a first candidate role entity, wherein the first candidate role entity is obtained by an unsupervised first candidate role entity recognition model; The text is merged with the corresponding question (excluding entity types) to obtain the merged text. The fused text is subjected to entity recognition processing to obtain at least one second candidate role entity, wherein the at least one second candidate role entity is obtained through a second candidate role entity recognition model based on machine reading comprehension; Among the at least one first candidate role entity and the at least one second candidate role entity, duplicate candidate role entities are filtered out; For each candidate role entity obtained after filtering, perform the following processing: The statement containing the candidate role entity is combined with the candidate role entity to obtain an entity statement pair; the entity features and text features extracted from the entity statement pair are fused to obtain a second fused feature, and the second fused feature is mapped to the probability of belonging to a role entity; When the probability of a candidate character entity being a role entity is greater than a role probability threshold, and the word frequency of the candidate character entity in the text is greater than a role word frequency threshold, the candidate character entity is determined to be a role in the text.

2. The method according to claim 1, characterized in that, The types of matching parameters include word frequency, cohesion, and degrees of freedom; The step of obtaining at least one matching parameter corresponding to each of the candidate words for the role includes: For each of the aforementioned role candidate words, the following processing is performed: Determine the word frequency of the candidate words for the role in the text; The candidate words for the role are divided into multiple morphemes, and the probability of each morpheme appearing in the text and the probability of the candidate words for the role appearing in the text are determined. The types of the morphemes include characters and words. The cohesion of the role candidate words is determined based on the occurrence probability of each morpheme in the text and the occurrence probability of the role candidate words in the text. The left and right information entropies of the candidate words for roles are determined, and the degrees of freedom of the candidate words for roles are determined based on the left and right information entropies of the candidate words for roles.

3. The method according to claim 2, characterized in that, The process of determining the left and right information entropies of the candidate character words, and determining the degrees of freedom of the candidate character words based on their left and right information entropies, includes: Identify multiple left-adjacent characters and multiple right-adjacent characters of the candidate character word in the text; Determine the sub-information entropy corresponding to each of the left adjacent words, and determine the sub-information entropy corresponding to each of the right adjacent words; The negative number of the sum of the sub-information entropies corresponding to each of the left adjacent words is determined as the left information entropy, and the negative number of the sum of the sub-information entropies corresponding to each of the right adjacent words is determined as the right information entropy; When the left information entropy is greater than the right information entropy, the right information entropy is determined as the degree of freedom; when the left information entropy is not greater than the right information entropy, the left information entropy is determined as the degree of freedom.

4. The method according to claim 3, characterized in that, Determining the sub-information entropy corresponding to each of the left adjacent words includes: For each of the left adjacent words, the following processing is performed: The ratio between the number of times the left adjacent word appears in the text and the number of times all adjacent words of the role candidate word appear in the text is determined as the first ratio. Perform a logarithmic operation on the first ratio, and determine the sub-information entropy corresponding to the left adjacent word as the product of the logarithmic operation result and the first ratio. Determining the sub-information entropy corresponding to each of the right adjacent words includes: For each of the right adjacent words, the following processing is performed: The ratio between the number of times the right adjacent word appears in the text and the number of times all adjacent words of the role candidate word appear in the text is determined as the second ratio. Perform a logarithmic operation on the second ratio, and determine the sub-information entropy corresponding to the right adjacent word by multiplying the logarithmic operation result and the second ratio.

5. The method according to claim 2, characterized in that, Determining the probability of occurrence of each morpheme in the text and the probability of occurrence of the role candidate word in the text includes: For each morpheme, the following processing is performed: the ratio between the number of times the morpheme appears in the text and the number of times all role candidate words appear in the text is determined as the probability of the morpheme appearing in the text; The ratio of the number of times the candidate word for a character appears in the text to the total number of times all candidate words for a character appear in the text is determined as the probability of the candidate word for a character appearing in the text.

6. The method according to claim 2, characterized in that, The step of determining the cohesion of the role candidate words based on the occurrence probability of each morpheme in the text and the occurrence probability of the role candidate words in the text includes: The probability of each morpheme appearing in the text is multiplied to obtain the product result. The ratio between the probability of the occurrence of the candidate word in the text and the product result is determined as the third ratio. The third ratio is subjected to logarithmic operation, and the result of the logarithmic operation is determined as the solidification degree of the candidate words for the role.

7. The method according to claim 2, characterized in that, The step of selecting at least one character candidate word from the plurality of character candidate words as the first candidate character entity based on at least one matching parameter corresponding to each of the character candidate words includes: Among the plurality of candidate character terms, a candidate character term that satisfies at least one of the following conditions is selected as the first candidate character entity: The frequency of the candidate words in the text exceeds the frequency threshold, the cohesion of the candidate words exceeds the cohesion threshold, and the degree of freedom of the candidate words exceeds the degree of freedom threshold.

8. The method according to claim 1, characterized in that, The entity recognition processing of the fused text to obtain at least one second candidate role entity includes: The fused text is subjected to feature extraction processing to obtain a feature sequence; The feature sequence is mapped to obtain at least one set of locations; For each of the aforementioned set of locations, the following processing is performed: The text corresponding to the starting position in the location set, the text between the starting and ending positions in the location set, and the text corresponding to the ending position in the location set are combined, and the combination result is determined as the second candidate character entity.

9. The method according to claim 8, characterized in that, The mapping process of the feature sequence to obtain at least one set of positions includes: The feature sequence is divided into multiple sub-features, wherein each of the multiple sub-features corresponds one-to-one with a multiple character in the text; Each sub-feature is mapped to the starting probability of the starting position and the ending probability of the ending position; At least one sub-feature with an initial probability greater than an initial probability threshold is selected as an initial sub-feature, and at least one sub-feature with an ending probability greater than an ending probability threshold is selected as an ending sub-feature. Based on at least one selected start sub-feature and at least one selected end sub-feature, at least one candidate start and end feature set is constructed, wherein the candidate start and end feature set includes a start sub-feature and an end sub-feature; Determine the target start-end feature set from the at least one candidate start-end feature set; The text corresponding to the starting sub-feature in the target start-end feature set is determined as the starting position in the position set, and the text corresponding to the ending sub-feature in the target start-end feature set is determined as the ending position in the position set.

10. The method according to claim 9, characterized in that, Determining the target start-end feature set from the at least one candidate start-end feature set includes: For each of the candidate start and end feature sets, the following processing is performed: the start sub-features and end sub-features in the candidate start and end feature sets are fused to obtain a first fused feature, and the first fused feature is mapped to the probability of belonging to the same entity; In the at least one set of candidate start and end features, the set of candidate start and end features whose probability of belonging to the same entity is greater than the entity probability threshold is determined as the target start and end feature set.

11. The method according to claim 1, characterized in that, After performing entity recognition processing on the fused text to obtain at least one second candidate role entity, the method further includes: For each of the second candidate role entities, perform the following processing: The second candidate role entity is divided into multiple morphemes, and the occurrence probability of each morpheme in the text and the occurrence probability of the second candidate role entity in the text are determined, wherein the types of the morphemes include characters and words; The cohesion degree of the second candidate character entity is determined based on the occurrence probability of each morpheme in the text and the occurrence probability of the second candidate character entity in the text. Determine the left and right information entropy of the second candidate role entity, and determine the degrees of freedom of the second candidate role entity based on the left and right information entropy of the second candidate role entity; Among the plurality of second candidate role entities, filter the second candidate role entities that satisfy at least one of the following conditions: The word frequency of the second candidate character entity in the text does not exceed the word frequency threshold, the cohesion of the second candidate character entity does not exceed the cohesion threshold, and the degree of freedom of the second candidate character entity does not exceed the degree of freedom threshold.

12. A character recognition device in text, characterized in that, The device includes: The first entity recognition module is used to extract multiple role candidate words from the text and obtain at least one matching parameter corresponding to each role candidate word; The first entity recognition module is further configured to select at least one role candidate word from the plurality of role candidate words as a first candidate role entity based on at least one matching parameter corresponding to each role candidate word, wherein the first candidate role entity is obtained by an unsupervised first candidate role entity recognition model; The second entity recognition module is used to fuse the text with the question that does not include entity types, and the text to obtain fused text; The second entity recognition module is further configured to perform entity recognition processing on the fused text to obtain at least one second candidate role entity, wherein the at least one second candidate role entity is obtained through a second candidate role entity recognition model based on machine reading comprehension; A classification module is used to filter out duplicate candidate role entities from the at least one first candidate role entity and the at least one second candidate role entity; for each candidate role entity obtained after filtering, the following processing is performed: combining the statement containing the candidate role entity with the candidate role entity to obtain an entity statement pair; fusing the entity features and text features extracted from the entity statement pair to obtain a second fused feature, and mapping the second fused feature to the probability of belonging to a role entity; when the probability of belonging to a role entity is greater than a role probability threshold, and the word frequency of the candidate role entity in the text is greater than a role word frequency threshold, the candidate role entity is determined to be a role in the text.

13. The apparatus as claimed in claim 12, characterized in that, The matching parameters include word frequency, cohesion, and degrees of freedom; the first entity recognition module is further configured to perform the following processing for each of the role candidate words: The word frequency of the candidate words for the role is determined in the text; the candidate words for the role are divided into multiple morphemes, and the occurrence probability of each morpheme and the occurrence probability of the candidate words for the role are determined in the text, wherein the types of the morphemes include characters and words; the cohesion of the candidate words for the role is determined based on the occurrence probability of each morpheme and the occurrence probability of the candidate words for the role; the left information entropy and right information entropy of the candidate words for the role are determined, and the degrees of freedom of the candidate words for the role are determined based on the left information entropy and right information entropy of the candidate words for the role.

14. The apparatus as claimed in claim 13, characterized in that, The first entity recognition module is further configured to: The process involves identifying multiple left and right adjacent characters of the candidate word in the text; determining the sub-information entropy corresponding to each left adjacent character and the sub-information entropy corresponding to each right adjacent character; determining the left information entropy by summing the sub-information entropies corresponding to each left adjacent character and the right information entropy by summing the sub-information entropies corresponding to each right adjacent character; determining the right information entropy as the degree of freedom when the left information entropy is greater than the right information entropy, and determining the left information entropy as the degree of freedom when the left information entropy is not greater than the right information entropy.

15. An electronic device, characterized in that, include: Memory is used to store executable instructions for a computer; A processor, when executing computer-executable instructions stored in the memory, implements the character recognition method in text as described in any one of claims 1 to 11.

16. A computer-readable storage medium, characterized in that, The device stores computer-executable instructions, which, when executed, are used to implement the character recognition method in text as described in any one of claims 1 to 11.

17. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the role recognition method in the text as described in any one of claims 1 to 11.

Images