Product information extraction method based on company annual report

Abstract

The invention discloses a company annual report-based product information extraction method. The method comprises the following steps of S1, acquiring and preprocessing company annual report data; s2, constructing a document tree according to the company annual report data; establishing a product entity dictionary, traversing the document trees according to the product entity dictionary, and screening the document trees according to a preset rule to obtain a target document tree; key information is extracted from the target document tree, and a product relation data set is constructed according to the key information; and S3, constructing a visual knowledge graph according to the product relation data set. The method is specially used for extracting the long text data of the annual report, so that the data can be accurately and efficiently extracted from the company annual report information.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing, and in particular to a method for extracting product information based on company annual reports. Background Art

[0002] The company's annual report is a formal document provided by the company to shareholders, investors, regulators and other stakeholders every year. It aims to report in detail the company's operating conditions, financial performance, market strategy, risk management and other aspects in the past year.

[0003] Traditional company annual report analysis relies primarily on the expertise of financial analysts and management consultants, who manually review financial statements, income statements, cash flow statements, and notes to assess a company's financial health and future earnings prospects.

[0004] The defects of the existing technology are: on the one hand, the annual report formats of different companies are different, and the content descriptions are also different. In particular, the product information disclosed in the annual report is mostly presented in the form of unstructured long text, which increases the difficulty of content understanding and data extraction. On the other hand, the traditional manual analysis method is not only inefficient, but also prone to errors, and it is difficult to meet the high requirements for information accuracy. Summary of the invention

[0005] In view of the above-mentioned defects of the prior art, the present invention provides a product information extraction method based on company annual reports, which is specifically aimed at data extraction of long text data in annual reports, so as to achieve accurate and efficient data extraction in company annual report information.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is:

[0007] A method for extracting product information based on company annual reports, comprising the following steps:

[0008] S1: Obtain and preprocess company annual report data;

[0009] S2: construct a document tree based on the company annual report data; establish a product entity dictionary, traverse the document tree based on the product entity dictionary, filter the document tree according to preset rules, and obtain a target document tree; extract key information from the target document tree, and construct a product relationship data set based on the key information;

[0010] S3: Construct a visual knowledge graph based on the product relationship dataset.

[0011] The way to obtain the annual reports of listed companies is to download the PDF document set of the annual reports of listed companies in previous years from the corresponding stock exchange according to the stock code of the listed company. The annual reports of non-listed companies can be obtained through the company's internal management channels. Text preprocessing includes parsing the downloaded PDF files, extracting text information, and performing data cleaning, including removing irrelevant characters, page numbers, headers, footers, etc.

[0012] A document tree is a hierarchical data structure used to represent the structure of a document. It organizes the document content into a tree structure by analyzing the title, paragraph, and other structural elements of the document, where each node represents a specific text paragraph or title. The product entity dictionary is a vocabulary that contains all product names and related information in the company's annual report. It extracts and identifies product entities from the annual report text through natural language processing technology, and standardizes and deduplicates them to form a structured vocabulary set.

[0013] Construct a document tree to reflect the hierarchy and association of the annual report content. Collect vocabulary related to the company's products, including product names, models, categories, etc., to form a product entity dictionary, which can be constructed by consulting the company's official website product introduction, industry standard terms, etc., and continuously updated and improved. According to the product entity dictionary, traverse the document tree, filter the document tree according to preset rules, and obtain the target document tree to focus on the part related to product information. In the target document tree, identify the specific information of the product entity, such as product name, specification parameters, production date, etc. By extracting key information, extract the relationship between product entities, such as the correlation between products (such as supporting products, substitute products, etc.), the relationship between products and market performance (such as sales, market share, etc.), etc., which can be determined by analyzing the semantic associations and syntactic structures in the text. The extracted product entities and their relationships are organized into a structured data set for subsequent analysis and application. Knowledge graph construction: Based on the product relationship data set, use the knowledge graph construction tool to generate a visual knowledge graph. The knowledge graph can display multi-dimensional information such as the relationship between products and products and markets, and intuitively present the full picture of the company's product information.

[0014] Preferably, the step S1 comprises: downloading and parsing the company annual report PDF document set to obtain original text data; performing data cleaning on the original text data to obtain cleaned data; locating the cleaned data through directory title keywords to obtain a text list.

[0015] Download the PDF file of the company's annual report from the stock exchange or other reliable sources and convert it into a processable text format. Use PDF parsing tools to extract the text content in the PDF file to generate raw text data. Perform data cleaning on the raw text data to obtain cleaned data. The purpose is to remove noise and irrelevant information in the raw text data and improve data quality and the efficiency of subsequent processing. Such as automatic spelling check and verification, regular expression cleaning, removal of spaces and special characters, etc. Establish a keyword-text data hierarchical index based on the keywords of the directory title. Through the keyword index, you can quickly locate the specific paragraph or chapter containing product information and generate a text list. By downloading and parsing PDF documents, data cleaning, and locating the text list through the directory title keywords, high-quality text data is provided for subsequent information extraction and knowledge graph construction, ensuring the accuracy and availability of the data, and laying the foundation for efficient and accurate extraction of product information.

[0016] Preferably, the step S2 comprises: traversing the text list, adding text type attributes to the company annual report data, and constructing a document tree through the text type attributes.

[0017] The purpose of constructing a document tree is to organize the text content of the company's annual report into a hierarchical tree structure, which is convenient for subsequent information extraction and processing. By traversing the text list, a type attribute is added to each text paragraph or title to distinguish different text contents (such as text, title, etc.). For example: -1: represents the text type, 1-15: represents different title types (such as 1 represents the first-level title, 2 represents the second-level title, etc.). According to the text type attribute, the text content is organized into a hierarchical tree structure for subsequent information extraction and processing. By traversing the text list and adding text type attributes, the text content of the company's annual report is organized into a hierarchical tree structure. This process not only improves the efficiency of text processing, but also provides a structured foundation for subsequent information extraction and knowledge graph construction. Through the document tree, you can quickly locate a specific paragraph or title containing product information to ensure the accuracy and efficiency of information extraction.

[0018] Preferably, the step S2 comprises: performing word segmentation processing on the company annual report data, and marking each word with BIO annotation.

[0019] The purpose of word segmentation of company annual report data is to provide annotated training data for subsequent product entity recognition and relationship extraction. Split the text content of the company annual report into independent words for more fine-grained analysis and processing, using word segmentation tools such as jieba (for Chinese text) or NLTK (for English text). Mark the product entities in the text through the BIO annotation format, and define the annotation rules: B-Entity: indicates the beginning of the entity, I-Entity: indicates the middle part of the entity, O: indicates a non-entity word. Combine the word segmentation results and the corresponding BIO annotations to generate an annotated dataset. Through word segmentation and BIO annotation, the text content of the company annual report can be converted into annotated training data to provide support for subsequent named entity recognition (NER) tasks. This process not only improves the quality of the data, but also lays the foundation for accurate identification of product entities, thereby improving the efficiency and accuracy of the entire information extraction and knowledge graph construction.

[0020] Preferably, step S2 includes: traversing the document tree, calculating the number of each product entity word in the product entity dictionary, listing the titles corresponding to the product entity words whose number is greater than a set threshold in a target title dictionary, and listing the titles corresponding to the product entity words whose number is less than or equal to the set threshold in other title dictionaries; locating key title nodes according to the target title dictionary, deleting irrelevant nodes according to other title dictionaries, and obtaining the target document tree.

[0021] By traversing the document tree, the frequency of occurrence of product entity words in the text under each title is counted to identify titles containing important product information. A counter is initialized for each product entity word to record its number of occurrences in the document. Starting from the root node, each node of the document tree is traversed layer by layer. For each node (title or paragraph), the words in the product entity dictionary are used for matching and the number of occurrences of each product entity word is counted. According to the frequency of occurrence of product entity words, the titles are divided into target titles and other titles for subsequent screening and processing. According to the statistical results, a threshold is set, and the titles corresponding to product entity words with a number of occurrences higher than the threshold are included in the target title dictionary, and those with a number of occurrences lower than the threshold are included in the other title dictionary. Through the target title dictionary and other title dictionaries, nodes containing important product information are screened out, a target document tree is constructed, and irrelevant content is removed. By traversing the document tree, counting the frequency of occurrence of product entity words, classifying titles, locating key nodes, and deleting irrelevant nodes, a target document tree containing important product information can be constructed. This process not only improves the efficiency of information extraction, but also ensures the accuracy and relevance of the extracted information, providing high-quality structured data for subsequent product relationship extraction and knowledge graph construction.

[0022] Preferably, the step S2 comprises: extracting information from the company annual report having a fixed template by identifying and matching the fixed template.

[0023] For company annual reports with fixed templates, information can be extracted efficiently by identifying and matching these fixed templates. Identify fixed templates to determine whether the annual report follows a known fixed template, which usually contains a specific structure and format. By identifying and matching fixed templates, the required information can be efficiently extracted from the company annual report. This method takes advantage of the consistency and regularity of the annual report format, improves the efficiency and accuracy of information extraction, and provides high-quality structured data for subsequent product relationship extraction and knowledge graph construction.

[0024] Preferably, the step S2 comprises: extracting information based on fixed keywords for the company annual report with an irregular template.

[0025] When processing company annual reports with irregular templates, the fixed template matching method cannot be directly applied because the format and structure of the annual reports may vary greatly. Therefore, a strategy based on fixed keywords can be used to extract information. Locate and extract relevant product information by identifying specific keywords in the text. Identify common fixed keywords related to product information in annual reports, which can help locate the location of product information. Traverse the document tree to locate titles and paragraphs containing fixed keywords, which may contain important product information. Through the fixed keyword-based method, product information can be effectively extracted from company annual reports with irregular templates. This method utilizes the semantic information of keywords, traverses the document tree, locates keywords, extracts key information, and forms structured data, which provides high-quality input for subsequent product relationship extraction and knowledge graph construction, not only improving the efficiency of information extraction, but also ensuring the accuracy and relevance of the extracted information.

[0026] Preferably, step S2 comprises: establishing a general relationship extraction rule, and using a regular expression to extract product name relationship triples.

[0027] Establishing general relation extraction rules and using regular expressions to extract product name relation triplets aims to identify and extract relations between products from text to form structured data. Define a set of rules for identifying and extracting product names and their relations in text. Study the structure and common expressions of annual report texts to identify common patterns of product names and their relations. Determine the types of relations that need to be extracted, such as "includes", "applies to", "competitive relations", etc. Write general relation extraction rules based on the identified patterns and relation types. These rules can be based on keywords, sentence structures, or dependencies. Match product names and their relations in text through regular expressions to extract structured triple data. By establishing general relation extraction rules and using regular expressions to extract product name relation triplets, the relations between products can be effectively extracted from company annual reports to form structured data.

[0028] Preferably, the step S2 comprises: constructing a product relationship data set and a product relationship extraction model according to the relationship triples.

[0029] The purpose of constructing a product relationship dataset and a product relationship extraction model based on the extracted relationship triples is to use the extracted structured information for training and optimizing the model to improve the accuracy and efficiency of relationship extraction. The extracted relationship triples are organized into a structured dataset for training and evaluating the model. A model is trained using the constructed product relationship dataset to automatically identify and extract product relationships in text. A model architecture suitable for relationship extraction is selected, such as BERT, LSTM, BiLSTM, etc. The BERT model performs well in natural language processing tasks, especially for text relationship extraction tasks. By constructing a product relationship dataset and a product relationship extraction model, the relationship between products can be automatically extracted from the company's annual report to form structured data, which not only improves the efficiency of information extraction, but also ensures the accuracy and relevance of the extracted information, providing high-quality input for the subsequent knowledge graph construction. By selecting a suitable model architecture, data preprocessing, model training, and evaluation, the performance of the model can be further improved to ensure that it performs consistently on different datasets.

[0030] Preferably, step S3 includes: the structure of the knowledge graph includes nodes and edges; the nodes represent entities, and the edges represent the relationships between the nodes; the importance score of each node is calculated according to the PageRank algorithm; and a visual data model is constructed according to the importance score.

[0031] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0032] 1. Compared with traditional manual analysis, this method uses natural language processing technology to automatically extract information, greatly shortening the analysis time and improving work efficiency. By constructing a document tree, accurately screening the target content, and using advanced entity recognition and relationship extraction technology, it is possible to more accurately extract product information, reduce human errors, and meet high-precision information needs. The resulting visual knowledge graph presents complex product information and relationships in an intuitive way, making it easier for non-professionals to understand the company's product situation and helping shareholders, investors, etc. make better decisions. The extracted product information and constructed knowledge graph can provide data support for internal management decisions, market research, competitor analysis, and other aspects of the company, and has a wide range of application value.

[0033] 2. Based on the typical directory structure and title keywords of the company's annual report, relevant data is extracted from the document in an automated and hierarchical manner to improve data processing efficiency. First, a document tree based on the title structure is established, and the text is classified and marked according to the different title types in the annual report. By automatically traversing these levels, specific content nodes are located. By traversing the document tree hierarchically, the content of each node is extracted and merged to achieve hierarchical processing of the document tree.

[0034] 3. Since there are a lot of long text data in the annual report and the long text format is diverse, it is difficult to understand the text. When processing long text paragraphs in the annual report, first divide the long sentence into multiple clauses according to punctuation marks, and then use the dependency analysis tool to generate a dependency syntax tree to identify the master-slave structure, parallel relationship, etc. of the sentence. Then divide the long sentence into independent short sentences through the rule chain, so that the long text problem can be converted into a short sentence problem, directly extract the key information in each short sentence, reduce errors and improve the accuracy of parsing.

[0035] 4. For product relationships in the annual report, firstly, based on the product entity dictionary, regular analysis and dependency analysis, general rules are constructed to extract product relationships and construct product relationship triple data sets. Through the self-constructed relationship data set, based on the BERT and LSTM models, sentence semantic features and product entity features are integrated for relationship extraction to improve the efficiency and accuracy of relationship extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 is a flow chart of the method of Example 1;

[0037] Figure 2 This is a flow chart of the method of step S2 in Example 1;

[0038] Figure 3 This is a schematic diagram of the implementation steps of step S2.1 of Example 1;

[0039] Figure 4This is a schematic diagram of the implementation steps of short sentence division in step S2.5.2 of Example 1;

[0040] Figure 5 This is a schematic diagram of the implementation steps of rule-based relationship extraction in step S2.5.3 of Example 1;

[0041] Figure 6 This is a schematic diagram of the implementation steps of the relationship extraction model in step S2.6 of Example 1. DETAILED DESCRIPTION

[0042] In order to make the technical means, creative features, objectives and effects of the invention easier to understand, the invention is further described with reference to specific diagrams. However, the invention is not limited to the following implementation cases.

[0043] It should be noted that the structures, proportions, sizes, etc. illustrated in the drawings in this specification are only used to match the contents disclosed in the specification so as to facilitate understanding and reading by persons familiar with this technology. They are not used to limit the conditions under which the present invention can be implemented, and therefore have no substantive technical significance. Any structural modification, change in proportion or adjustment of size, without affecting the effects and purposes that can be achieved by the present invention, should still fall within the scope of the technical contents disclosed by the present invention.

[0044] Embodiment 1:

[0045] like Figure 1 A method for extracting product information based on a company's annual report is shown, comprising the following steps:

[0046] S1: Obtain and preprocess company annual report data;

[0047] S1.1: Download the PDF document set of the listed company's annual reports from the corresponding stock exchange according to the stock code of the listed company.

[0048] The way to obtain the annual reports of listed companies is to download the PDF document set of the annual reports of listed companies in previous years from the corresponding stock exchange according to the stock code of the listed company. The annual reports of non-listed companies can be obtained through the company's internal management channels. Text preprocessing includes parsing the downloaded PDF files, extracting text information, and performing data cleaning, including removing irrelevant characters, page numbers, headers, footers, etc.

[0049] S1.2: Parse the pdf file. Extract the text information to obtain the original text data. Download and parse the company's annual report pdf document set to obtain the original text data; clean the original text data to obtain the cleaned data; locate the cleaned data through the directory title keywords to obtain the text list.

[0050] S1.3: Preprocess the original text data, that is, automatically spell check and verify the extracted data, use regular expressions to clean the original text data, remove irrelevant data such as page numbers, headers and footers, remove unconventional numeric characters, English characters, and punctuation characters; remove spaces and replace ellipsis with periods.

[0051] S1.4: Use keywords to build a keyword-text data hierarchical index for the cleaned text data, and locate the specified range of data based on the directory and title keywords.

[0052] Download the PDF file of the company's annual report from the stock exchange or other reliable sources and convert it into a processable text format. Use PDF parsing tools to extract the text content in the PDF file to generate raw text data. Perform data cleaning on the raw text data to obtain cleaned data. The purpose is to remove noise and irrelevant information in the raw text data and improve data quality and the efficiency of subsequent processing. Such as automatic spelling check and verification, regular expression cleaning, removal of spaces and special characters, etc. Establish a keyword-text data hierarchical index based on the keywords of the directory title. Through the keyword index, you can quickly locate the specific paragraph or chapter containing product information and generate a text list. By downloading and parsing PDF documents, data cleaning, and locating the text list through the directory title keywords, high-quality text data is provided for subsequent information extraction and knowledge graph construction, ensuring the accuracy and availability of the data, and laying the foundation for efficient and accurate extraction of product information.

[0053] S2: Figure 2 As shown, a document tree is constructed based on the company's annual report data; a product entity dictionary is established, and based on the product entity dictionary, the document tree is traversed, and the document tree is screened according to preset rules to obtain the target document tree; key information is extracted from the target document tree, and a product relationship data set is constructed based on the key information;

[0054] A document tree is a hierarchical data structure used to represent the structure of a document. It organizes the document content into a tree structure by analyzing the title, paragraph, and other structural elements of the document, where each node represents a specific text paragraph or title. The product entity dictionary is a vocabulary that contains all product names and related information in the company's annual report. It extracts and identifies product entities from the annual report text through natural language processing technology, and standardizes and deduplicates them to form a structured vocabulary set.

[0055] S2.1: If Figure 3As shown in the figure, a function to obtain the root text is established, and a document tree is constructed based on the preprocessed text list. The text list is traversed, and a type attribute is added to each text, where -1 is the text type, 1-15 are different title types (such as 1 for the first-level title, 2 for the second-level title, etc.), and a document tree is established based on the title type, so as to facilitate the recursive traversal and locate the product information location according to the hierarchy. The purpose of building a document tree is to organize the text content of the company's annual report into a hierarchical tree structure, which is convenient for subsequent information extraction and processing. By traversing the text list, a type attribute is added to each text paragraph or title to distinguish different text contents (such as text, title, etc.). According to the text type attribute, the text content is organized into a hierarchical tree structure, which is convenient for subsequent information extraction and processing. By traversing the text list and adding the text type attribute, the text content of the company's annual report is organized into a hierarchical tree structure. This process not only improves the efficiency of text processing, but also provides a structured basis for subsequent information extraction and knowledge graph construction. Through the document tree, specific paragraphs or titles containing product information can be quickly located to ensure the accuracy and efficiency of information extraction.

[0056] S2.2: Build a product entity dictionary. Generate a product entity dictionary from the annual report corpus by training the BERT model.

[0057] S2.2.1: Construct annotated dataset. Use the word segmentation tool to segment the text data and annotate each word using the BIO annotation format, and add special markers (such as [CLS] and [SEP]) to construct the input. The annotation rules are: B-Entity: indicates the beginning of the entity. I-Entity: indicates the middle part of the entity. O: indicates a non-entity word.

[0058] S2.2.2: Train a BERT-based named entity recognition model.

[0059] The model input sequence X is:

[0060] X=[[CLS],x1,x2,…,xn,[SEP],y1,y2,…,[SEP]]

[0061] The embedding representation of each word is expressed as:

[0062] E(xi)=T(xi)+S(xi)+P(xi)

[0063] Use the BERT model for pre-training and fine-tune it on the annual report annotation dataset to optimize the named entity recognition task. For text with hierarchical structure (such as titles and directories), use position-enhanced embedding:

[0064] Penhanced(xi)=SegmentEmbedding(xi)+TreeLevelEmbedding(xi)

[0065] The purpose of word segmentation processing on the company's annual report data is to provide labeled training data for subsequent product entity recognition and relationship extraction. The text content of the company's annual report is split into independent words for more fine-grained analysis and processing. This embodiment marks the product entities in the text through the BIO annotation format. The word segmentation results and the corresponding BIO annotations are combined to generate an annotated data set. Through word segmentation processing and BIO annotation, the text content of the company's annual report can be converted into labeled training data to provide support for subsequent named entity recognition (NER) tasks, which not only improves the quality of the data, but also lays the foundation for accurate identification of product entities, thereby improving the efficiency and accuracy of the entire information extraction and knowledge graph construction.

[0066] S2.2.3: Entity extraction and post-processing. Segment the annual report text, input the model for prediction and decode the results to identify the product name information. De-duplicate and standardize product entities and build a product entity dictionary.

[0067] S2.2.4: For the product entity dictionary, calculate its TF-IDF value and select keywords with high frequency and strong discrimination:

[0068]

[0069] Where TF(w,t) represents the frequency of occurrence of word w in text t, N is the total number of texts, and n w is the number of texts containing word w.

[0070] S.2.3: Filter and locate the target content based on the document tree title to narrow the target scope. Traverse the document tree, calculate the number of each product entity word in the product entity dictionary, add the titles corresponding to the product entity words whose number is greater than the set threshold to the target title dictionary, and add the titles corresponding to the product entity words whose number is less than or equal to the set threshold to other title dictionaries; locate the key title nodes based on the target title dictionary, delete irrelevant nodes based on other title dictionaries, and obtain the target document tree.

[0071] S2.3.1: Traverse the document tree hierarchically, calculate the frequency of product entity terms in the text under each title according to the product entity dictionary, add titles with a frequency greater than the set threshold to the target title dictionary, and add titles with a frequency less than or equal to the set threshold to other title dictionaries.

[0072] S2.3.2: Locate key title nodes based on the target title dictionary, and delete irrelevant nodes based on other title dictionaries to obtain the target document tree to ensure accurate extraction of product information.

[0073] By traversing the document tree, the frequency of occurrence of product entity words in the text under each title is counted to identify titles containing important product information. A counter is initialized for each product entity word to record its number of occurrences in the document. Starting from the root node, each node of the document tree is traversed layer by layer. For each node (title or paragraph), the words in the product entity dictionary are used for matching and the number of occurrences of each product entity word is counted. According to the frequency of occurrence of product entity words, the titles are divided into target titles and other titles for subsequent screening and processing. According to the statistical results, a threshold is set, and the titles corresponding to product entity words with a number of occurrences higher than the threshold are included in the target title dictionary, and those with a number of occurrences lower than the threshold are included in the other title dictionary. Through the target title dictionary and other title dictionaries, nodes containing important product information are screened out, a target document tree is constructed, and irrelevant content is removed. By traversing the document tree, counting the frequency of occurrence of product entity words, classifying titles, locating key nodes, and deleting irrelevant nodes, a target document tree containing important product information can be constructed. This process not only improves the efficiency of information extraction, but also ensures the accuracy and relevance of the extracted information, providing high-quality structured data for subsequent product relationship extraction and knowledge graph construction.

[0074] S.2.4: Extract information from documents with fixed templates according to fixed formats. In view of the diversity of annual report formats, we use multiple formats to learn and generate dynamic template matching rules, and identify and match the document structure. If it meets a fixed template, we directly extract key information according to the fixed template.

[0075] For company annual reports with fixed templates, information can be extracted efficiently by identifying and matching these fixed templates. Identify fixed templates to determine whether the annual report follows a known fixed template, which usually contains a specific structure and format. By identifying and matching fixed templates, the required information can be efficiently extracted from the company annual report. This method takes advantage of the consistency and regularity of the annual report format, improves the efficiency and accuracy of information extraction, and provides high-quality structured data for subsequent product relationship extraction and knowledge graph construction.

[0076] S2.5: Locate and segment irregular documents based on titles, and extract key information through regular expression analysis and text analysis.

[0077] When processing company annual reports with irregular templates, the fixed template matching method cannot be directly applied because the format and structure of the annual reports may vary greatly. Therefore, a strategy based on fixed keywords can be used to extract information. Locate and extract relevant product information by identifying specific keywords in the text. Identify common fixed keywords related to product information in annual reports, which can help locate the location of product information. Traverse the document tree to locate titles and paragraphs containing fixed keywords, which may contain important product information. Through the fixed keyword-based method, product information can be effectively extracted from company annual reports with irregular templates. This method utilizes the semantic information of keywords, traverses the document tree, locates keywords, extracts key information, and forms structured data, which provides high-quality input for subsequent product relationship extraction and knowledge graph construction, not only improving the efficiency of information extraction, but also ensuring the accuracy and relevance of the extracted information.

[0078] S2.5.1: Products generally contain fixed keywords in titles, such as business, industry, sector, product, sector, etc. Traverse the document tree titles and perform regular matching to extract product components, and add the identified products to the product entity dictionary. At the same time, use the TF-IDF algorithm to count the word frequency of the annual report content, identify potential product words with high word frequency, and perform intersection operations with the product entity dictionary to obtain a high-frequency product entity dictionary.

[0079] S2.5.2: If Figure 4 As shown, for the body paragraph text, the sentence segmentation is first performed. The text paragraphs are divided into hierarchical structures such as general-specific, general-specific-general, parallel, and progressive. The text paragraphs are divided into a set of sentences according to punctuation marks, and the sentences are subjected to dependency syntactic analysis using DDParser to obtain the dependency tree corresponding to the text, and the subject, predicate, and other sentence components of the sentence are identified. Based on semantic analysis, the core information in the sentence is identified. According to the syntax tree and semantic dependency, the sentence is divided into multiple short sentences and the segmented short sentences are reorganized according to the semantic and grammatical coherence to generate the final short sentence list.

[0080] S2.5.3: If Figure 5As shown, for each short sentence, a general relation extraction rule is established, and a regular expression is used to extract product name relation triples. First, according to regular matching and dependency analysis, the product entity dictionary is matched to extract the subject product of the short sentence. According to the sentence structure and dependency relationship, a product relation analysis rule base is established, and the object / predicate relation hierarchy is parsed according to the rule base to extract product relation triples. The purpose of establishing general relation extraction rules and extracting product name relation triples using regular expressions is to identify and extract the relationship between products from the text to form structured data. Define a set of rules for identifying and extracting product names and their relationships in the text. Study the structure and common expressions of the annual report text to identify common patterns of product names and their relationships. Determine the type of relationship that needs to be extracted, such as "include", "apply to", "competitive relationship", etc. According to the identified patterns and relationship types, write general relation extraction rules. These rules can be based on keywords, sentence structures, or dependencies. Match the product names and their relationships in the text through regular expressions to extract structured triple data. By establishing general relationship extraction rules and using regular expressions to extract product name relationship triples, the relationships between products can be effectively extracted from company annual reports to form structured data.

[0081] S2.6: If Figure 6 As shown in the figure, the company's product relationship extraction model is constructed. The relationship extraction is converted into a relationship classification model. The feature representation of the sentence, the head entity feature representation, and the tail entity feature representation are calculated through the BERT model and the LSTM model. The three features are connected and softmax is used to classify the features into different relationships.

[0082] S2.6.1: Construction of company product relationship dataset. The product entities in the named entity recognition model and the product entity relationship triples extracted in steps 2.4 and 2.5 are converted into the following format (S, e s ,e o )→(e s ,r,e o ). Where S is the sentence prototype, e s is the head product entity, e o is the tail product entity, r∈R, r is the relationship type in the predetermined relationship set, R is the relationship set, (e s ,r,e o ) is the product entity relationship triple result.

[0083] S2.6.2: Construction of company-product relationship extraction model. For a given sentence S = {x1, x2, ..., x n}, use the pre-trained word vector matrix to map each word in a sentence to a low-dimensional digital vector, and then input the numerical vector composed of this sentence into the BERT model to obtain B = {b1, b2, ..., bn}, and then input it into the LSTM network to obtain L = {l1,l2,…,l n}, the encoding formula is as follows:

[0084] B=BERT(x1,x2,…,x n )

[0085] L=LSTM(b1,b2,…,b n )

[0086] Then feature fusion is performed.

[0087] H n =W[concat(L+L1+L2)]+b

[0088] Where L is the sentence feature representation, L1 is the head entity feature representation, L2 is the tail entity feature representation, W is the weight, and b is the bias term.

[0089] The deep feature vector output by feature fusion is output through the softmax function classifier to output the product entity relationship, and compared with the relationship extracted by the rule to obtain the accuracy of the result. Calculate the probability P that the entity relationship contained in the sentence belongs to all entity relationship types, and predict the product entity relationship label y':

[0090] P(r|S)=softmax(w n tanh(H)+e l )

[0091] r'=argmaxP(r|S)

[0092] where w n Generate a vector for target attention, n represents the total number of elements in H, e l It's a deviation.

[0093] The purpose of constructing a product relationship dataset and a product relationship extraction model based on the extracted relationship triples is to use the extracted structured information for training and optimizing the model to improve the accuracy and efficiency of relationship extraction. The extracted relationship triples are organized into a structured dataset for training and evaluating the model. A model is trained using the constructed product relationship dataset to automatically identify and extract product relationships in text. A model architecture suitable for relationship extraction is selected, such as BERT, LSTM, BiLSTM, etc. The BERT model performs well in natural language processing tasks, especially for text relationship extraction tasks. By constructing a product relationship dataset and a product relationship extraction model, the relationship between products can be automatically extracted from the company's annual report to form structured data, which not only improves the efficiency of information extraction, but also ensures the accuracy and relevance of the extracted information, providing high-quality input for the subsequent knowledge graph construction. By selecting a suitable model architecture, data preprocessing, model training, and evaluation, the performance of the model can be further improved to ensure that it performs consistently on different datasets.

[0094] S3: Build a visual knowledge graph based on the product relationship dataset.

[0095] S3.1: Construct the extracted structured product data into a knowledge graph. The knowledge graph structure contains nodes and edges. Each node in the knowledge graph represents a company / product entity, and the edge represents the relationship between nodes. Each edge has a direction indicating the direction of the relationship.

[0096] S3.2: Evaluate the relative importance of each node based on the PageRank algorithm. The node score of each node is calculated according to the following formula:

[0097]

[0098] Where PR(P) is the node score of node P; In(P) is the set of nodes pointing to node P; Out(i) is the out-degree of node i; d is the damping coefficient. The algorithm goes through multiple rounds of iterations and gradually updates the node scores of each node until it converges.

[0099] S3.3: Visualize the company's product knowledge graph, improve the retrieval effect of the knowledge graph through node scores, form a multi-level, visual data model based on the company's annual report, and provide structured information support for data analysis.

[0100] Construct a document tree to reflect the hierarchy and association of the annual report content. Collect vocabulary related to the company's products, including product names, models, categories, etc., to form a product entity dictionary, which can be constructed by consulting the company's official website product introduction, industry standard terms, etc., and continuously updated and improved. Based on the product entity dictionary, traverse the document tree, filter the document tree according to preset rules, and obtain the target document tree to focus on the part related to product information. In the target document tree, identify the specific information of the product entity, such as product name, specification parameters, production date, etc. By extracting key information, extract the relationship between product entities, such as the correlation between products (such as supporting products, substitute products, etc.), the relationship between products and market performance (such as sales, market share, etc.), etc., which can be determined by analyzing the semantic associations and syntactic structures in the text. Organize the extracted product entities and their relationships into a structured data set for subsequent analysis and application. Knowledge graph construction: Based on the product relationship data set, use the knowledge graph construction tool to generate a visual knowledge graph. The knowledge graph can display multi-dimensional information such as the relationship between products and products and markets, and intuitively present the full picture of the company's product information.

Claims

1. A method for extracting product information based on company annual reports, characterized in that: The following steps are involved: S1: Obtain and preprocess company annual report data; S2: construct a document tree according to the company annual report data; establish a product entity dictionary, traverse the document tree according to the product entity dictionary, filter the document tree according to preset rules, and obtain a target document tree; Extracting key information from the target document tree, and constructing a product relationship data set based on the key information; S3: Construct a visual knowledge graph based on the product relationship dataset.

2. The method for extracting product information based on company annual reports according to claim 1, characterized in that: The step S1 comprises: downloading and parsing the company's annual report PDF document set to obtain original text data; performing data cleaning on the original text data to obtain cleaned data; locating the cleaned data through directory title keywords to obtain a text list.

3. The method for extracting product information based on company annual reports according to claim 2, characterized in that: The step S2 includes: traversing the text list, adding text type attributes to the company annual report data, and constructing a document tree through the text type attributes.

4. The method for extracting product information based on company annual reports according to claim 2, characterized in that: The step S2 includes: performing word segmentation processing on the company annual report data, and marking each word using BIO annotation.

5. The method for extracting product information based on company annual reports according to claim 4, characterized in that: The step S2 includes: traversing the document tree, calculating the number of each product entity word in the product entity dictionary, listing the titles corresponding to the product entity words whose number is greater than a set threshold into a target title dictionary, and listing the titles corresponding to the product entity words whose number is less than or equal to the set threshold into other title dictionaries; locating key title nodes according to the target title dictionary, deleting irrelevant nodes according to other title dictionaries, and obtaining the target document tree.

6. The method for extracting product information based on company annual reports according to claim 5, characterized in that: The step S2 comprises: extracting information from the company annual report with a fixed template by identifying and matching the fixed template.

7. The method for extracting product information based on company annual reports according to claim 6, characterized in that: The step S2 comprises: extracting information from the company annual report of the irregular template according to fixed keywords.

8. The method for extracting product information based on company annual reports according to claim 7, characterized in that: The step S2 includes: establishing a general relationship extraction rule, and using a regular expression to extract product name relationship triples.

9. The method for extracting product information based on company annual reports according to claim 8, characterized in that: The step S2 comprises: constructing a product relationship data set and a product relationship extraction model according to the relationship triples.

10. The method for extracting product information based on company annual reports according to any one of claims 1 to 9, characterized in that: The step S3 includes: the structure of the knowledge graph includes nodes and edges; the nodes represent entities, and the edges represent the relationships between the nodes; the importance score of each node is calculated according to the PageRank algorithm; and a visual data model is constructed according to the importance score.

Images