A method for extracting character attributes based on web page texts

Abstract

The application discloses a kind of based on webpage body character attribute extraction method, comprising the following steps: S1, the relevant webpage of target character is obtained, the body text containing character attribute information is extracted from webpage, and text tokenization is carried out to body text;S2, according to character attribute, select the word attribute containing classification information;S3, build feature: use named entity, dependency relationship as additional word token mark, and build the state feature based on static word table, simultaneously using the context relationship of dependency syntax tree to build transition feature;S4, state feature and transition feature are used as the constraint of maximum entropy algorithm, and classification model is trained.The application obtains the body of webpage of target character of interest in complex network environment, realizes the method for extracting character feature in complex network environment by reasonably selecting and constructing feature using maximum entropy algorithm.

Description

TECHNICAL FIELD

[0001] The present application relates to character attribute extraction, in particular, a character attribute extraction method based on web text. BACKGROUND

[0002] With the popularity and rapid development of the Internet, the increasing maturity of Internet technology and the increasing diversification of people's needs make the Internet service show a professional subdivision trend and attract a large number of Internet users. Various Internet services also carry a large amount of user data. Such data is often fragmented, and according to different network platforms, the data format also exists differences, but a considerable part of the information exists in the form of text in the network.

[0003] The problem of character attribute extraction in web text is to search for character-related documents in the network and extract character attributes from the documents. Existing research on character attribute extraction based on web text has formed several methods, but still has some limitations. On the one hand, when searching for possible information according to the character attributes of the target, the existing method usually only extracts the attributes of the character according to a single information such as the part of speech and the position of the word, without fully utilizing the advanced features and some context features in the web text, resulting in the loss of character information. On the other hand, in the selection of text processing models, discriminative models such as conditional random fields are usually used, and the information of the model on the whole sample is rarely considered.

[0004] The goal of character attribute extraction based on web text is to obtain attribute information of a specific entity from the web page, and the character attributes in the web page are mostly in the form of text. Under the premise of extracting the text, the problem can be converted into extracting the attribute information of a specific entity from the text data, such as the name, nationality, and position of the entity. Through the attribute extraction of a specific character entity, the entity attributes of multiple data sources are obtained, and the construction of the character portrait can be completed by processing these attribute information. The commonly used attribute extraction models today are mainly divided into two types: sequence labeling-based method and HTML web-based method.

[0005] 1. Sequence labeling-based attribute extraction method

[0006] For text data, the person attribute problem can be converted into a sequence labeling problem. The text data can be regarded as a linear sequence, and the word units in the text data correspond to the elements in the sequence. The attribute information of each word unit corresponds to the sequence label and attribute value, and finally a sequence labeling model is used for training and extraction. By regarding attribute extraction as a sequence labeling problem, BIO, BIOE, etc. Labeling methods can be used. Labeling requires a certain amount of manual cost. In the context of person attribute extraction, the structured information box of Baidu Encyclopedia and other encyclopedia entries can be used for labeling, which can reduce the cost of manual labeling. At the same time, the Bootstrap method can be used to discover more potential attribute values from the seeds during labeling.

[0007] The commonly used models for sequence labeling include CRF models, neural network models such as BiGRU+CRF models, etc. In recent years, pre-trained language models have been widely used in natural language processing. In attribute extraction based on sequence labeling, pre-trained language models such as bert can still be used for feature extraction, and the classic model framework is bert+BiLSTM+CRF model. Due to the diversity of attribute values in content and form, for descriptive attributes with a large number of words, ideal results cannot be achieved. For sequence labeling, a large amount of manual cost is often consumed, reducing the scalability of the model and failing to solve the multi-attribute value problem. For sequence labeling attribute selection, existing methods generally only consider the part of speech, prefix, and other primary features of the word unit from the lexical perspective, and pay less attention to some obvious high-level features. Considering the context features of the word unit directly from the syntactic perspective does not fully reflect the semantic information of the word unit.

[0008] 2. Attribute extraction method based on HTML web page

[0009] Each web page in the network is based on HTML specification. From the code structure, the content of the web page is presented in the DOM tree. The basic element of the DOM (Document Object Model) is "node", and the structure of the web page is composed of hierarchical nodes. In the DOM model, the entire web document is a document (Document) node, each document node has a root (Root) node, and the root node contains a large number of nodes with different functions, and the nodes have parent-child, sibling, etc. hierarchical relationship.

[0010] The entity-property name-property value relationship naturally exists in the DOM tree, and the character attribute often exists in the property value of a specific node. In the research, the CNN and the LSTM are often used to encode the content of the node at the letter level and the word level, and the XPath encoding, the position encoding, the leaf type encoding and the semantic similarity encoding are spliced into the node encoding, and the attribute is combined with the source text encoding. First, a binary classifier is used to judge whether the attribute value exists, and then the attribute value is extracted, and the node encoding is used to predict the attribute name through a multi-classifier.

[0011] The general idea of the existing character attribute extraction method based on web text can be summarized as four basic steps: sequence labeling of web text at the text level to generate text sequences; selecting word attribute according to the character attribute; using lexical and syntactic level features to construct a model feature function on the basis of the word attribute; training the model to complete the prediction. For the selection of word attribute of character attribute extraction, the existing method generally considers the primary features of the word itself and the position features of the word order, and the advanced features and relative position features with strong representation ability are rarely used, and the feature function constructed loses part of the representation ability; for the selection of the prediction model, the existing method generally uses a discriminative model such as conditional random field, and rarely considers the distribution of the character attribute as a whole, and the generalization ability is weak, which performs poorly in the engineering environment. SUMMARY

[0012] The present application aims at overcoming the shortcomings of the prior art, and provides a method for extracting character features in a complex network environment by reasonably selecting and constructing features and using a maximum entropy algorithm to obtain the text of a web page of a target character of interest.

[0013] The present application aims at overcoming the shortcomings of the prior art, and provides a method for extracting character features in a complex network environment by reasonably selecting and constructing features and using a maximum entropy algorithm to obtain the text of a web page of a target character of interest.

[0014] S1, obtaining the related web page of the target character, extracting the text of the web page containing the character attribute information, and performing text segmentation on the text of the web page;

[0015] S2, selecting the word attribute containing classification information according to the character attribute;

[0016] S3, constructing features: using named entities and dependency relationships as additional word annotations, and constructing state features based on a static word table, and using the context relationship of the dependency syntax tree to construct transition features;

[0017] S4, using the state features and the transition features as constraints of the maximum entropy algorithm to train a classification model.

[0018] The relevant web pages in the step S1 include online encyclopedias, personal websites, institutional homepages and social platform personal accounts. The word is taken as the granularity of word segmentation.

[0019] Further, in the step S2, the text after the word segmentation processing is a word sequence, and the minimum semantic unit of the word segmentation is a word in the sequence; in the character attribute extraction task, the character attribute information is contained in the attribute of the word and the relationship of the context of the word; the word attributes used include the following:

[0020] 1) The attribute value of the "text" attribute is the original text of the word; the original text of the word is retained as the word attribute; in the subsequent feature construction, the high-frequency words in the character attributes are matched against the static word table;

[0021] 2) The "lemma" attribute is the prototype of the word; the "lemma" attribute is used as the word attribute to eliminate ambiguity and as an auxiliary attribute of the "text" attribute for annotation;

[0022] 3) The "shape" attribute represents the case format of the word;

[0023] 4) The "pos" attribute represents the part of speech of the word;

[0024] 5) "index" refers to the index value of the word in the text sequence;

[0025] 6) "prefix" and "suffix" represent the prefix and suffix of the word, respectively.

[0026] The state features in the step S3 include:

[0027] 1) The "word_list" feature is a judgment of whether the "text" and "lemma" attributes of the current word are contained in the static word table;

[0028] 2) "prefix_list" and "suffix_list" are the static word tables of the prefix and suffix, respectively; the static prefix and suffix tables are used as feature functions to select the prefix and suffix;

[0029] 3) "ent" represents the named entity category of the word; the named entity attribute is used as a feature function;

[0030] The transition features include:

[0031] 4) "dep" represents the dependency relationship of the word; this attribute reflects the dependency features of the word in syntax and semantics; the dependency syntax relationship is represented by a dependency relationship tree, and the conditional dependency relationship is used as a feature function.

[0032] Further, the step S4, the maximum entropy algorithm constructs a classification model is a conditional probability distribution P(Y|X), wherein, X is a sample, Y is the character attribute category; Will build the feature function f(x, y) as the distribution P(Y|X) constraint condition, to find the distribution that satisfies these constraint conditions, the entropy of the distribution P(Y|X) is maximum; Wherein, the feature function f(x, y) is used to describe the relationship between the word element x and the category label y, defined as:

[0033]

[0034] The feature function reflects the relationship between the word element and the context that the word element satisfies;

[0035] In order to find the gain degree of different feature functions for classification, the consistency of the feature function on the real distribution and the empirical distribution is considered; For the i-th feature function f i , the expected value about the real expectation is represented by :

[0036]

[0037] P(x, y) represents the real distribution of (x, y);

[0038] The i-th feature function f i , the fuzzy expected value E P (f i ) about the conditional distribution and the empirical distribution is represented as follows:

[0039]

[0040] P(x) refers to the real distribution of x; P(y|x) refers to the distribution of the output y of the model under the condition that x is input;

[0041] In the scene of character attribute extraction, the real expectation of the feature function E P (f i ) represents the expected value of the feature function on the real distribution and the empirical distribution respectively, and it is assumed that the two expectations are equal; Then the character attribute extraction model must satisfy all the feature equations

[0042] In order to minimize the entropy of the model P(Y|X), first, the conditional entropy that satisfies the equation constraint is calculated, which is defined as follows:

[0043]

[0044] The constraint condition of the function is:

[0045]

[0046]

[0047] M represents the number of characteristic functions; H(P) represents the negative value of the entropy of the person attribute extraction model satisfying all characteristic functions; H(P) is a convex function, according to the theory of convex optimization, using the Lagrange function to convert it into an unconstrained optimization function to solve H(P), and the Lagrange multiplier λ corresponding to each constraint condition in the Lagrange function is obtained i ; from the perspective of semantics, λ i reflects the gain degree of the constructed different characteristic functions f i (x, y) for classification, the higher the gain of the characteristic function for classification, the greater the λ i value; thus, the distribution P(y|x) is obtained in the following form about a set of weight values λ:

[0048]

[0049] wherein, z λ (x) is a normalization factor, which ensures that the entire model is a legal probability distribution, and the definition is as follows:

[0050]

[0051] The IIS algorithm is used to complete the optimization of model learning; by minimizing the model H(P), the distribution function P λ (y|x) of the model is obtained, that is, the construction of the model in the person attribute extraction scene is completed.

[0052] The present application proposes a person attribute extraction method based on maximum entropy, in a complex network environment, the text of the web page of the target person of interest is obtained, by reasonably selecting and constructing features, the method for extracting person features in a complex network environment is realized using the maximum entropy algorithm, the following beneficial effects can be achieved:

[0053] 1、In addition to selecting the primary attributes of the text as the label, the present application also uses high-level features such as NER and DEP, and uses the position of the word element in the syntactic dependency tree to obtain context information, which improves the classification performance of the model and enables it to complete training on a small-scale sample;

[0054] 2、The present application uses a maximum entropy model as a prediction model, uses a feature function based on morphology, syntax and context as a model constraint condition, calculates the maximum likelihood of the model to make a prediction, the maximum entropy model reflects more distribution information of the data itself, has wider universality, and the classification effect is obviously improved. BRIEF DESCRIPTION OF DRAWINGS

[0055] Figure 1 This is a flowchart of the character attribute extraction method of the present invention;

[0056] Figure 2 This is a schematic diagram illustrating the dependencies of the present invention. Detailed Implementation

[0057] The technical solution of the present invention will be further illustrated below with reference to the accompanying drawings.

[0058] like Figure 1 As shown, the present invention provides a method for extracting character attributes based on webpage content, comprising the following steps:

[0059] S1. Obtain relevant web pages of the target person, extract the main text containing the person's attribute information from the web pages, and perform text segmentation on the main text; relevant web pages include online encyclopedias, personal websites, institutional homepages, and personal accounts on social media platforms.

[0060] The corpus used in this invention is the main body of web pages in text format. It is also necessary to exclude the influence of other irrelevant content on the selection of the main body, therefore, we need to locate the main body of the web page as accurately as possible. For extracting the main body of the web page, this invention selects web pages that may contain a large amount of information about people, including online encyclopedias, personal websites, institutional homepages, and personal accounts on social media platforms. Online web pages follow HTML specifications at the code level, so the structure of the web page can be roughly regarded as a DOM tree. This invention comprehensively considers attributes such as node position and node type to locate the main body of the web page and stores it in text format, thus completing the extraction of the main body of the web page.

[0061] In the task of extracting character attributes, serialized text can better represent the semantic content of the text and is more convenient for subsequent model processing. Therefore, we need to serialize the web page text to be extracted, which is to perform text segmentation.

[0062] The granularity of word segmentation affects the expression of character attribute information in a text sequence. If phrases are used as lexical units, information is easily lost during the merging process; if individual words are used as lexical units, the information of attributes in phrases and contexts may be ignored. In character attribute extraction tasks, the smallest unit of character attributes can be a single word, such as "position: teacher" or "nationality: England". If word segmentation is performed at the granularity of noun merging or phrases, attribute information within and between words will be lost. Considering the annotation attributes of lexical units, part-of-speech, conditional dependencies, named entities, etc., are all based on words as the basic unit. The relationship between words is also a basic unit reflecting character attributes. Therefore, this study chooses to use words as the granularity of word segmentation, which is reasonable in the context of character attribute extraction.

[0063] S2, according to the character attribute, select the word attribute containing classification information; the text after the word segmentation processing is a word sequence, and the minimum semantic unit of the word segmentation is the word of the sequence; in the character attribute extraction task, the character attribute information is contained in the attribute of the word itself and the relationship of the word context, and the selection of the word attribute has a great influence on the character attribute extraction effect. For example, for the position attribute of the character, the word of the position generally has a suffix such as 'or' and 'er' at the end, and when the sequence is generated, the suffix is taken as the basic attribute of the word, which will bring gain to the word classification. It is very important to select the appropriate word attribute, and the word attribute used by the present application includes the following kinds:

[0064] 1) The attribute value of the 'text' attribute is the original text of the word; in the web text, many words themselves refer to character attributes, such as 'President' and 'Professor' words only have the meaning of position-related interpretation, and such words in the text are most likely to represent the 'position' attribute of the character. Therefore, in order to capture these special words, we need to keep the original text of the word as the word attribute. In the subsequent feature construction, the high-frequency words in the character attribute are matched with the static word table.

[0065] 2) The 'lemma' attribute is the prototype of the word, and the computer cannot directly identify the grammatical variations of the same word, such as the computer cannot identify 'President' and 'president' as the same word, but instead treats them as different words. If the ambiguity in the grammatical form cannot be properly identified, it may reduce the ability of the model to capture attribute information. Therefore, the 'lemma' attribute is used as the word attribute to eliminate ambiguity and as an auxiliary attribute of the 'text' attribute.

[0066] 3) The'shape' attribute represents the case format of the word; in the grammatical rules of English, the capitalization of letters is generally used to identify proper nouns and abbreviations. In particular, considering the scope of the character attribute, the position, name, organization, and country character attribute words generally have the format of the first letter capitalized and the abbreviation, such as 'Chairman', 'Senator', 'BBC', and 'USA'. Therefore, the'shape' attribute is used to capture this kind of information.

[0067] 4) "pos" (part of speech) attribute represents the part of speech of the token; in the scenario of extracting the attributes of the person, we are looking for the position, nationality, name and other attributes of the person, and the tokens of these attributes are generally in the form of nouns, and the "pos" label can well distinguish these features. In addition, prepositions, copulas and other words in the context often appear with certain rules with the tokens of the attributes of the person, such as the prepositions "in" and "of" that usually appear after the tokens of the "position" attribute to indicate that the position of the person belongs to an organization. Therefore, the present application uses the "pos" attribute to capture the above two types of attribute information.

[0068] 5) "index" refers to the index value of the token in the text sequence; in the text sequence, the state of the current token itself often cannot fully reflect the information of the attributes of the person, such as the token "American" which can be interpreted as a noun "American" or as an adjective to indicate that "something is American". At this time, the context needs to be used to assist in obtaining the exact information expressed by the current token. "index" represents the sequential number of the current token in the text sequence, which is used as a token attribute to locate the context of the token, so as to find the relationship between the previous and subsequent tokens in the subsequent work and capture the context features.

[0069] 6) "prefix" and "suffix" represent the prefix and suffix of the token, respectively. In the composition rules of English words, a word is usually composed of one or more parts of the prefix, the root, and the suffix, and the meaning of the word is defined and derived from the meanings of the prefix, the root, and the suffix, that is, the prefix and the suffix also contain the semantic information of the word. In the scenario of extracting the attributes of the person, the meanings of some prefixes and suffixes have a strong correlation with the corresponding attributes of the person, such as the suffixes "ist", "er" and "or" which have the meaning of "a person engaged in something" in English, which is directly related to the "position" attribute. Therefore, the present application labels the prefix and the suffix as token attributes.

[0070] The token features used by the present application are shown in Table 1.

[0071] Table 1

[0072] Feature name Feature description Explanation text Wordpiece The text itself of a wordpiece lemma Lemma The lemma of a wordpiece shape Shape The case shape of a wordpiece pos Part-of-speech The part-of-speech of a wordpiece index Wordpiece index The order of a wordpiece in the original sentence prefix Prefix The prefix of a wordpiece suffix Suffix The suffix of a wordpiece

[0073] S3, constructing features: before training the model for extracting the attributes of the person, the present application needs to construct some feature functions according to the characteristics of the attributes of the person, which are used to describe some experience characteristics that are likely to be true or expected to be true, and can also be regarded as a classification judgment of the token in one dimension. The present application uses named entities and dependency relations as additional token labels, and constructs state features based on a static word table, and uses the context relationship of the dependency syntax tree to construct transition features; the state features include:

[0074] 1)“word_list” feature is the judgment of whether the “text” and “lemma” attributes of the current word element are contained in the static word list; in the text, many words are used to describe the attributes of the characters, and if the “text” or “lemma” attribute value of the word element matches the word, the possibility of the word element being the attribute of the character is very high. Therefore, we need to establish a word list of such words for different character attributes to match the word elements with the words in the list. Therefore, according to different character attribute features, the present application constructs a related character attribute static word list by statistical method.

[0075] 2)“prefix_list” and “suffix_list” are respectively the static word lists of prefixes and suffixes; the construction idea of such word lists is consistent with that of the “word_list” word list. In addition, if the selection of the prefix / suffix word element attribute is not limited, the too many attribute values that can be taken by the word element attribute will consume resources and increase the complexity of the model. Therefore, we use a static prefix / suffix list as a feature function to select the prefix / suffix.

[0076] 3)“ent” represents the named entity category of the word element; it contains entities with specific meanings such as names, organization names, and place names. In the task of character attribute extraction, word elements with specific named entity labels often reflect character attribute information in themselves and context. For example, in the sentence “Assistant manager of R&D department”, if the named entity label of “R&D department” is “organization”, then “assistant manager” appearing before it is very likely to have the “position” attribute. The word element with a specific named entity can usually be used as an important feature for locating the target attribute of the character, such as the word element with the attribute value of the organization name may contain the school and work unit information of the character, and the context of the organization name may contain the position information. Therefore, the named entity attribute is used as a feature function.

[0077] Transition features include:

[0078] 4) “dep” represents the dependency relationship of the word element, which reflects the dependency features of the word element in syntax and semantics; the dependency syntax relationship is represented by a dependency relationship tree, such as Figure 2In the sentence "The car is red", the word "car" is the nominal subject of the word "red", and the word "is" is the dependency of other words, i.e. the virtual root (ROOT). The dependency syntax analysis reflects the character attribute information of word units at the syntactic level across the direct position between word units. In the character attribute extraction task, the character attribute word unit usually appears at a specific position in the syntax tree, such as for the "position" attribute word unit, it can be a nominal object (nobj), attribute word (attr) component, but it cannot appear as a predicate (ROOT) in the dependency syntax tree. In addition, the character attribute information usually exists in a specific position in the dependency syntax tree, and the relative position of the dependency relationship can reflect the context relationship of the word unit from another angle. Therefore, the conditional dependency relationship is used as a feature function in the present application.

[0079] All the features used in the present application are shown in Table 2.

[0080] Table 2

[0081] Feature name Feature description Explanation prefix_list Static prefix list Build a common prefix set for the target attribute suffix_list Static suffix list Build a common suffix set for the target attribute word_list Static word list Build a common word set for the target attribute ent Named entity A person name, organization name, place name, and all other entities identified by a name dep Dependency relation The dependency relation between words in a dependency parse tree

[0082] Unlike the HMM model which can only use one adjacent position transition feature and the conditional random field which only uses the word unit order based position transition feature, the present application rejudges the context relationship based on the dependency syntax tree, and uses the parent node and the child node as the basis for the transition between word units, which can well explore the relationship between the current word unit and the word unit in the syntax. Unlike the context relationship based on the absolute position, the relative position relationship of the syntax dependency relationship can cross multiple word units to find the relationship between the current word unit and its parent (child) node, and improve the classification performance of the character attribute extraction model. The transition features between word units used in the present application are shown in Table 3.

[0083] Table 3

[0084] Feature name Feature description Explanation parent Parent dependency word The parent node (of another wordpiece) of a wordpiece in a syntactic dependency graph child Child dependency word The child node (of another wordpiece) of a wordpiece in a syntactic dependency graph

[0085] S4, using the state feature and the transition feature as the constraint of the maximum entropy algorithm to train the classification model; the essence of extracting the character attribute is to find the word unit indicating a specific character attribute, which requires classifying each word unit. In the present application, given a sample X, the purpose of the character attribute extraction task is to judge whether the sample X belongs to a certain character attribute category Y through the calculation of the model. Therefore, the character attribute extraction task can be regarded as a binary classification problem.

[0086] To find the model of the binary classification problem, we find the distribution with the maximum information entropy among all the distributions satisfying the constraints of the priori to be the distribution of the classification model. According to the maximum entropy principle: among all the possible probability distributions, the distribution with the maximum entropy is the best distribution. This distribution is the optimal algorithm model we are looking for. Specifically, the classification model constructed by the maximum entropy algorithm is a conditional probability distribution P(Y|X), where X is the sample and Y is the category of the character attribute; the feature function f(x, y) constructed is taken as the constraint condition of the distribution P(Y|X) to find the distribution P(Y|X) with the maximum entropy among the distributions satisfying these constraint conditions; wherein the feature function f(x, y) is used to describe the relationship between the word x and the pre-labeled category label y, and is defined as:

[0087]

[0088] The feature function reflects the relationship of the word in the word itself and the context; from the semantic point of view, different feature functions can reflect the character attribute information to different degrees through the attributes of the word and the transformation relationship between the words; from the perspective of the whole model, different feature functions have different degrees of gain for classification. Therefore, it is reasonable to take the constructed feature function as the constraint condition of the maximum entropy model to calculate.

[0089] In order to find the gain degree of different feature functions for classification, the consistency of the feature function on the true distribution and the empirical distribution is considered; the i-th feature function f i of the word x is denoted as:

[0090]

[0091] denotes the true distribution of (x, y);

[0092] The i-th feature function f i The fuzzy expected value E P (f i ) of the conditional distribution and the empirical distribution is as follows:

[0093]

[0094] denotes the true distribution of x; P(y|x) denotes the distribution of the output y of the model under the condition that x is input;

[0095] In the scene of character attribute extraction, the true expectation of the feature function and the model expectation E P (f i ​respectively, represent the expected value of the characteristic function on the real distribution and the empirical distribution, in order to make the character attribute extraction model close to its real distribution, we can assume that the two expectations are equal, so each feature corresponds to an equality constraint. Then the character attribute extraction model must satisfy all the characteristic equation constraints

[0096] In order to minimize the entropy of the model P(Y|X), first calculate the conditional entropy that satisfies the equality constraint , which is defined as follows:

[0097]

[0098] The constraint condition of the function is:

[0099]

[0100]

[0101] M represents the number of characteristic functions; H(P) represents the negative value of the entropy of the character attribute extraction model satisfying all characteristic functions; H(P) is a convex function, according to the theory of convex optimization, it is converted into an unconstrained optimization function by using the Lagrange function to solve H(P), and each constraint condition in the Lagrange function corresponds to the Lagrange multiplier λ i ; From the semantic point of view, λ i reflects the gain degree of the constructed different characteristic function f i (x, y) for classification, the higher the gain of the characteristic function for classification, the greater the value of λ i ; Thus, the distribution P(y|x) is expressed as follows with respect to a set of weights λ:

[0102]

[0103] Where, Z λ (x) is a normalization factor, which ensures that the entire model is a legal probability distribution, and its definition is:

[0104]

[0105] The IIS (improved iterative scaling) algorithm is used to complete the optimization of model learning; by minimizing the model H(P), the distribution function P λ (y|x) of the model is obtained, that is, the construction of the model in the character attribute extraction scene is completed, and the classification model is obtained.

[0106] The maximum entropy model considers the overall distribution of the model on the sample, and selects the model with the maximum information entropy among all models satisfying the constraint conditions, which has high accuracy as a classical classification model. The selection of constraint conditions is also very flexible, which is consistent with the characteristics of using word units and context attributes as features in the task of extracting character attributes.

[0107] Those skilled in the art will appreciate that the embodiments described herein are presented for the purpose of understanding the principles of the present application and should be understood as not limiting the scope of protection of the present application to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations according to the technical inspiration disclosed in the present application without departing from the essence of the present application, and these modifications and combinations are still within the scope of protection of the present application.

Claims

1. A method for extracting character attributes based on web page texts, characterized in that, The method comprises the following steps: S1, obtaining a related webpage of a target person, extracting a text containing attribute information of the person from the webpage, and performing text segmentation on the text; S2, selecting a word attribute containing classification information according to the attribute of the person; S3, constructing a feature: using a named entity and a dependency relationship as an additional word attribute, and constructing a state feature based on a static word table, and using a context relationship of a dependency syntax tree to construct a transition feature; S4, take the state feature and the transition feature as the constraint of the maximum entropy algorithm to train the classification model; the classification model constructed by the maximum entropy algorithm is a conditional probability distribution wherein, is a sample, is a person attribute category; take the constructed feature function as the constraint condition of the distribution to find the distribution with the maximum entropy in the distribution satisfying the constraint condition ; wherein, the feature function is used to describe the relationship between the word element x and the category label y, and is defined as: ； A feature function reflects a relationship met by a word in the word itself and a context; To find the gain degree of different feature functions for classification, the consistency of feature functions on the real distribution and the empirical distribution is considered; for the ith feature function The expected value with respect to the real expectation is denoted as ​ ； representing the true distribution; the ith characteristic function fuzzy expectation values with respect to conditional and empirical distributions has the following representation: ； refers to the true distribution of x; refers to the distribution of y output by the model given input of x; In the scenario of attribute extraction, the true expectation of a feature function and the model expectation represent the expectation of the feature function on the real distribution and the empirical distribution, respectively. Assuming that the two expectations are equal, the attribute extraction model must satisfy all the constraints of the feature equations ; To minimize the entropy of the model the conditional entropy that satisfies the equality constraints is first computed, which is defined as follows: ； The constraint condition of the function is: ； ； M represents the number of characteristic functions; This represents the negative value of the entropy of the character attribute extraction model that satisfies all feature functions; Since it is a convex function, according to the theory of convex optimization, the Lagrangian function is used to transform it into an unconstrained optimization function for solution. This yields the Lagrange multipliers corresponding to each constraint in the Lagrange function. From a semantic perspective, This reflects the different characteristic functions constructed. Regarding the gain for classification, the higher the gain of the feature function for classification, its... The larger the value, the better; thus, the distribution is obtained. Regarding a set of weights The following is the representation: ； where, is a normalizing factor that ensures the entire model is a valid probability distribution, defined as: ； Optimization of model learning is completed using the IIS algorithm; by minimizing the model , the distribution function of the model is obtained , that is, the construction of the model in the scene of extracting the attributes of the person is completed.

2. The method according to claim 1, wherein, The related webpage in the step S1 comprises an online encyclopedia, a personal website, an institutional homepage and a social platform personal account.

3. The method of claim 1, wherein the method further comprises: In the step S1, a word is taken as a granularity of text segmentation.

4. The method of claim 1, wherein the method further comprises: In the step S2, the text after text segmentation is a word sequence, and a minimum semantic unit of text segmentation is a word of the sequence; in a person attribute extraction task, attribute information of a person is contained in an attribute of a word itself and a relationship of a context of the word; the word attribute used comprises the following: 1) an attribute value of a "text" attribute is an original text of a word; the original text of the word is taken as the word attribute; in subsequent feature construction, a static word table is matched to a high-frequency word appearing in a person attribute; 2) a "lemma" attribute is a prototype of a word; the "lemma" attribute is used as the word attribute to eliminate ambiguity and is taken as an auxiliary attribute of the "text" attribute; 3) a "shape" attribute represents a case format of a word; 4) a "pos" attribute represents a part of speech of a word; 5) an "index" is an index value of a word in a text sequence; 6) a "prefix" and a "suffix" represent a prefix and a suffix of a word, respectively.

5. The method of claim 1, wherein the method further comprises: The state feature in the step S3 comprises: 1) a "word_list" feature is a judgment of whether a "text" and a "lemma" attribute of a current word are contained in a static word table; 2) a "prefix_list" and a "suffix_list" are a static prefix table and a static suffix table, respectively; the static prefix and suffix tables are used as a feature function to select a prefix and a suffix; 3) an "ent" represents a named entity category of a word; the named entity attribute is used as the feature function; The transition feature comprises: 4) a "dep" represents a dependency relationship of a word; the attribute reflects a dependency feature of a word in syntax and semantics; a dependency syntax relationship is represented by a dependency relationship tree; a conditional dependency relationship is used as the feature function.

Images