A character attribute extraction method based on big data

Abstract

The application discloses a kind of based on big data's character attribute extraction method, it is related to information extraction technical field. Including the following steps: S1, obtain associated character information text;S2, associated character attribute extraction identification;S3, associated character attribute analysis processing;S4, associated character behavior association;S5, the supplement of associated character attribute data. Through the supplement of attribute set by character attribute extraction model can provide more comprehensive, timely attribute information, meet the individualized requirements of different application scenarios and user needs, promote deep understanding and analysis of character, and improve decision support and business effect, by using word vector technology to obtain the synonym of attribute, similar word can improve the coverage, accuracy and flexibility of attribute extraction model, reduce data sparsity, and support cross-language attribute extraction. This makes the model better adapt to different scenarios and needs, provide more comprehensive, accurate character attribute information.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of information extraction, and particularly relates to a character attribute extraction method based on big data. BACKGROUND

[0002] With the advent of the era of big data, in the face of information explosion and information overload, the low efficiency of information processing and other challenges, with the help of natural language processing technology, the attributes of the characters are extracted and mined, which can effectively alleviate the above problems, and the research and application of the technology can meet the needs of personalized services and data-driven decision-making. There are a large amount of text data in the Internet, including news reports, encyclopedia web pages, forum posts, blog articles, micro-blog articles and the like. Many of these data are about character information. In order to construct an open source character knowledge base and a character knowledge graph, character attribute extraction needs to be performed from the text data. By continuously acquiring data of different media and different sources and performing automatic character attribute extraction, the character attribute information in the text is identified, classified, de-duplicated, disambiguated, fused, and the knowledge base content is constantly enriched. In the aspect of information extraction, it helps to understand the characteristics and relationships of the characters, and better performs information retrieval, knowledge management and decision support. In the aspect of social network analysis, it helps to analyze the social relationships, influence and crowd characteristics between characters. This has important significance for social network analysis, user behavior prediction and recommendation system and the like. In addition, attribute extraction can accurately understand the interests, preferences and needs of users, and can be applied to personalized recommendation and advertisement targeting. Attribute extraction can also help to analyze the emotional tendencies, attitudes and opinions expressed by characters in text data, and can be applied to sentiment analysis and public opinion monitoring. In the application of network security and fraud detection, it can help to detect malicious activities, network attacks and fraud behaviors. In summary, the research of character attribute extraction enables us to obtain more rich information from text data, and deeply understand the characteristics, relationships and behaviors of individuals and groups, which helps to improve efficiency, improve user experience and increase business value. The premise of character attribute extraction includes: obtaining pure text segments through preprocessing, and the text does not contain web page HTML, XML or JSON tags; identifying the subject characters from the text through named entity recognition technology or entity linking technology. There are still some deficiencies in the prior art: including the accuracy problem that a large amount of training data must exist; and the error brought by the tool, the more feature engineering, the more error, the error propagation and accumulation in the whole task, thereby affecting the accuracy of subsequent relationship extraction. SUMMARY

[0003] The present application provides a character attribute extraction method based on big data to solve the problems in the background art.

[0004] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is: a method for extracting personal attributes based on big data, comprising the following steps:

[0005] S1. Obtain the text of related person information;

[0006] S2. Extraction and recognition of related personal attributes;

[0007] S3. Analysis and processing of related character attributes;

[0008] S4. Related Person Behaviors;

[0009] S5, Supplementing related character attribute data.

[0010] A further improvement to the technical solution of the present invention is that step S1 specifically includes:

[0011] S1.1 Obtain information about the social media where related individuals appear through open-source information;

[0012] S1.2. Obtain primary and secondary social media by the frequency of appearance of related personal information;

[0013] S1.3 Obtain information on relevant individuals from major and minor social media platforms and establish a target database.

[0014] A further improvement to the technical solution of the present invention is that step S2 specifically includes:

[0015] S2.1 List all character attributes according to the character attribute system of the target knowledge base, and make necessary additions to form a character attribute seed list. Each item in the list includes a standard attribute name, attribute ID and attribute alias. The attribute alias can be empty or one or more.

[0016] S2.2 Based on word vector technology, the attributes are expanded. Through the Word2Vec library, similar terms are found for each attribute and its aliases and added to the attribute list to form an expanded list of character attributes.

[0017] S2.3. Assign text numbers to the input text, where the assigned text number is D. i ;

[0018] S2.4, Regarding text D i Sentence segmentation is performed on text D. i Each sentence is assigned a number S j Use the character attribute expansion list to analyze sentence S j Perform attribute name matching; once a match is found, then set S... j The attribute name P that is being matched sThe latter part is an attribute value Vk, and the input data is searched for S j The main character name A a , and outputs a five-tuple <A a , P s , V k , S j , D i >, and the above steps are repeatedly executed until all sentences S j are processed.

[0019] S2.5, repeatedly execute step S2.4 until all text D i processing is completed.

[0020] The further improvement of the technical scheme of the application is that the step S3 specifically comprises:

[0021] S3.1, attribute classification is performed on all output five-tuple attributes in step S2.5, a mapping table of attribute name and standard attribute name and attribute ID is constructed according to the character attribute expansion list established in step S2.2, the mapping table is used to replace all five-tuples, and a new five-tuple <A a , P t , V k , S j , D i > is formed.

[0022] S3.2, classification and deduplication are performed on the output five-tuple, <A a , P t , V k > is used as a key, and <S j , D i > is used as a value for aggregation, and optionally, if deduplication is required, deduplication is performed according to the <Aa, Pt, Vk> key, and a four-tuple <A a , P t , <V k , <S j , D i >*>*> is output.

[0023] S3.3, the output four-tuple is fused according to the character name, the attributes of the same character name are aggregated together, and <A a , <P t , <V k , <S j , D i >*>*> is output, facilitating subsequent storage and query.

[0024] The further improvement of the technical scheme of the application is that the step S4 specifically comprises:

[0025] S4.1, processing the data summarized in step S3.3 and establishing a timeline for the summarized data, associating the time of appearance of the character attribute with the character attribute data;

[0026] S4.2, processing the character attribute data established in the timeline, weighting the character attributes that repeatedly appear at different times and the character attributes that continuously appear, and reducing the weight of the character attribute data that appears in large quantities at a specific time point or deleting the character attribute data that appears at a specific time point;

[0027] S4.3, predicting the character attribute data that may appear in the future according to the results calculated in step S4.2.

[0028] The further improvement of the technical scheme of the present application is that the step S5 specifically includes:

[0029] S5.1, obtaining the attributes of different characters in the social media in reverse with the character attribute data summarized in step S3.3, and establishing a supplementary database for the obtained data;

[0030] S5.2, matching the data obtained in the supplementary database with the character attribute data summarized in step S3.3, and deleting the data that fails to match from the database;

[0031] S5.3, obtaining highly consistent character attribute data after processing the supplementary database or failing to obtain additional data due to matching failure;

[0032] S5.4, extracting and identifying the character attribute data obtained in step S5.3 and establishing supplementary data of the character attribute, and integrating the supplementary data into step S3.3.

[0033] The further improvement of the technical scheme of the present application is that the step S4 further includes:

[0034] S4.4, establishing a character attribute data graph based on the results in step S4.2, and differentiating the display of different character attribute data according to the weight of different attributes.

[0035] The further improvement of the technical scheme of the present application is that the step S3 further includes:

[0036] S3.4, comparing the summarized character attribute data, comparing the determined character attribute with the character attribute data indicated by the personal pronoun, and removing the character attribute data with low correlation.

[0037] Due to the adoption of the above technical scheme, the present application has the following technical progress compared with the prior art:

[0038] 1. This invention provides a method for extracting personal attributes based on big data. By supplementing the attribute set through the personal attribute extraction model, more comprehensive and timely attribute information can be provided, meeting the personalized requirements of different application scenarios and user needs, promoting in-depth understanding and analysis of personalities, and improving decision support and business effectiveness.

[0039] 2. This invention provides a method for extracting personal attributes based on big data. By using word vector technology to obtain synonyms and similar words for attributes, the coverage, accuracy, and flexibility of the attribute extraction model can be improved, data sparsity can be reduced, and cross-language attribute extraction can be supported. This allows the model to better adapt to different scenarios and needs, providing more comprehensive and accurate personal attribute information.

[0040] 3. This invention provides a method for extracting personality attributes based on big data. By classifying attributes into different categories or levels, the inherent connections and differences between attributes can be revealed, aiding in understanding the essence and meaning of attribute information. Attribute classification also provides a foundation for the correlation and reasoning between attributes. By dividing attributes into different categories, corresponding labeled datasets can be established for model training and optimization. Attached Figure Description

[0041] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0042] Figure 1 This is a schematic diagram of the process of the present invention; Detailed Implementation

[0043] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0044] like Figure 1 As shown, this invention provides a method for extracting personality attributes based on big data, which is operated through the following steps:

[0045] First, obtain text information about related individuals; then, obtain information about the social media platforms where related individuals appear through open-source information; finally, obtain information about major and minor social media platforms based on the frequency of appearance of related individuals; and finally, obtain information about related individuals from major and minor social media platforms and build a target database.

[0046] Subsequently, the associated person attribute extraction and recognition is performed; all person attributes are listed according to the person attribute series of the target knowledge base, and necessary supplements are performed to form a person attribute seed list, each item of the list including a standard attribute name, an attribute ID and an attribute alias, and the attribute alias can be empty, one or more; based on the word vector technology, the attribute is expanded, the Word2Vec library is used to find similar terms for each attribute and its alias, and the attribute list is added to form an expanded person attribute list; the input text is numbered and assigned, wherein the assigned text number is D i ; the text D i is processed by sentence, each sentence in the text D i is assigned a number S j , the attribute name matching is performed on the sentence S j using the expanded person attribute list, once the matching is successful, the part after the matched attribute name P s in S j is taken as the attribute value V k , the subject person name A a of S j is found in the input data, and the quintuple <A a , P s , V k , S j , D i > is output, and the above steps are repeatedly executed until all sentences S j are processed; step S2.4 is repeatedly executed until all texts D i are processed.

[0047] Secondly, the associated person attribute analysis processing is performed; the attribute classification is performed on all output quintuples, the mapping table of the attribute name and the standard attribute name and the attribute I D is constructed according to the established expanded person attribute list, the attribute replacement is performed on all quintuples using the mapping table to form new quintuples <A a , P t , V k , S j , D i >; the output quintuples are classified and de-duplicated, and <A a , P t , V k > is taken as the key, and <S j , D i > is taken as the value for aggregation, optionally, if de-duplication is required, the de-duplication is performed according to the <A a , P t , V k > key, and <A a , P t , <V k , <Sj ,D i >*>*>four-tuple; the four-tuple of the output is fused according to the character name, and the attributes of the same character name are aggregated together to output <A a ,<P t ,<V k ,<S j ,D i >*>*>*, for subsequent storage and query; compare the aggregated character attribute data, compare the determined character attribute data with the character attribute data indicated by the personal pronoun, and remove the character attribute data with low correlation;

[0048] Then the associated character behavior is associated; the aggregated data is processed and a timeline is established for the aggregated data, the time of the appearance of the character attribute is associated with the character attribute data; the established timeline character attribute data is processed, the repeatedly appearing character attribute at different times and the continuously appearing character attribute are weighted processed, and the character attribute data appearing at a specific time point is reduced in weight or deleted; according to the calculated results, the character attribute data that may appear in the future is predicted; the character attribute data graph is established through the appearance results, and the different character attribute data is differentially displayed according to the weight of different attributes;

[0049] Finally, the associated character attribute data is supplemented; the aggregated character attribute data is used to obtain the attributes of different characters in the social media in reverse, and the obtained data is established as a supplement database; the data obtained in the supplement database is matched through the aggregated character attribute data, and the data that fails to match is deleted from the database; after processing the supplement database, the highly consistent character attribute data or the additional data that fails to match is obtained; the obtained character attribute data is extracted and recognized, and the supplement data of the character attribute is established, and the supplement data is integrated and aggregated and stored.

Claims

1. A method for extracting character attributes based on big data, characterized in that: The method comprises the following steps: S1, obtaining associated person information text; S2, associated person attribute extraction and identification; S3, associated person attribute analysis and processing; S4, associated person behavior association; S5, supplement of associated person attribute data; The step S2 specifically comprises: S2.1, according to the person attribute body series of the target knowledge base, all person attributes are listed and necessary supplements are made to form a person attribute seed list, each item in the list includes a standard attribute name, an attribute ID and an attribute alias, and the attribute alias can be empty, one or more; S2.2, based on the word vector technology, the attributes are expanded, the Word2Vec library is used to find similar terms for each attribute and its alias, and the attributes are added to the attribute list to form an expanded person attribute list; S2.3, text numbering assignment is performed on the input text, wherein the assigned text number is D i ; S2.4, Regarding text D i Sentence segmentation is performed on text D. i Each sentence is assigned a number S j Use the character attribute expansion list to analyze sentence S j Perform attribute name matching; once a match is found, then set S... j The attribute name P that is being matched s The latter part serves as the attribute value V. k Search for S in the input data j The main character's name A a Output the quintuple a ,P s V k ,S j D i Repeat the above steps until all sentences S are completed. j Processing complete;​ S2.5, repeat step S2.4 until all texts D i The processing is completed; The step S3 specifically comprises: S3.

1. Classify the attributes of all the output quintuples in step S2.

5. Based on the character attribute expansion list established in step S2.2, construct a mapping table between attribute names, standard attribute names, and attribute IDs. Use the mapping table to replace the attributes of all quintuples to form new quintuples. a ,P t V k ,S j D i >;​ S3.2, classify and deduplicate the output quintuple to <A a , P t , V k > as key, aggregate to <S j , D i > as value, need deduplication, then deduplicate according to <A a , P t , V k > key, output <A a , P t , <V k , <S j , D i > *> *> quadruple; S3.3, the four-tuples are fused according to the person name, and the attributes of the same person name are aggregated together, and the output is <Aa, <Pt, <Vk, <S j ,D i >*>*>*> for subsequent storage and query. 2.The method of claim 1, wherein: The step S1 specifically comprises: S1.1, obtaining the social media where the associated person information appears through open source information; S1.2, obtaining the main social media and the secondary social media through the frequency of the appearance of the associated person information; S1.3, obtaining the associated person information of the main social media and the secondary social media and establishing a target database. 3.The method of claim 1, wherein: The step S4 specifically comprises: S4.1, processing the data summarized in step S3.3 and establishing a timeline for the summarized data, and associating the time of the appearance of the person attribute with the person attribute data; S4.2, processing the person attribute data of the established timeline, weighting the person attributes that repeatedly appear at different times and continuously appear, and reducing the weight of the person attribute data that appears in large quantities at a specific time point or deleting the person attribute data that appears at a specific time point; S4.3, predicting the person attribute data that may appear in the future according to the results calculated in step S4.

2.

4. The method of claim 1, wherein the method is based on big data. The step S5 specifically comprises: S5.1, obtaining the attributes of different persons in the social media through the person attribute data summarized in step S3.3, and establishing a supplementary database for the obtained data; S5.2, matching the data obtained in the supplementary database through the person attribute data summarized in step S3.3, and deleting the data that fails to match from the database; S5.3, obtaining highly consistent person attribute data or additional data that fails to match after processing the supplementary database; S5.4, extracting and identifying the person attribute data obtained in step S5.3 and establishing supplementary data of the person attribute, and integrating the supplementary data into step S3.

3.

5. The method of claim 1, wherein the method is based on big data. The step S4 further comprises: S4.4, establishing a person attribute data graph based on the results in step S4.2, and differentiating the display of different person attribute data according to the weight of different attributes. 6.The method of claim 1, wherein: The step S3 further comprises: S3.4, comparing the summarized person attribute data, comparing the determined person attribute with the person attribute data indicated by the personal pronoun, and removing the person attribute data with low correlation.

Images