Enterprise public information extraction method and equipment for webpage and medium

Abstract

The invention discloses an enterprise public information extraction method and device for a webpage and a medium, and the method comprises the steps: obtaining sample public data of a sample enterprise according to the Web webpage; according to the sample public data, training the identification and extraction model architecture to obtain a general identification and extraction model; obtaining each set enterprise field corpus, updating the universal recognition and extraction model according to the enterprise field corpus, obtaining a recognition and extraction model of each enterprise field, obtaining a recognition and extraction model library, and obtaining a recognition and extraction model of each enterprise field according to the public data of the to-be-analyzed enterprise and a target recognition and extraction model; and obtaining entity data related to the to-be-analyzed enterprise and an association relationship. The recognition and extraction model is customized in a personalized manner in the enterprise field, so that the recognition and extraction accuracy of the enterprise information is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a method, device and medium for extracting public information of enterprises from web pages. Background Art

[0002] In today's information age, the variety and content of public information released by enterprises through the Internet have far exceeded the scope of traditional media, and this information is an important basis for corporate operations and decision-making, market analysis, investment research, etc. Therefore, it is necessary to effectively extract corporate public information.

[0003] However, faced with such a huge amount of information, the traditional manual extraction method not only consumes a lot of manpower and time costs, but also is limited by human factors, making it difficult to ensure the integrity and accuracy of the information, and it is very easy to make omissions or errors. In addition, the traditional recognition and extraction methods also have problems such as insufficient understanding ability and high computational complexity of the model. As a result, the accuracy of corporate public information extraction for web pages is low. Summary of the invention

[0004] The embodiments of the present application provide a method, device and medium for extracting public information of enterprises from web pages, which are used to solve the problem of low accuracy in extracting public information of enterprises.

[0005] The present application embodiment adopts the following technical solutions:

[0006] On the one hand, an embodiment of the present application provides a method for extracting corporate public information from a web page, the method comprising: obtaining sample public data of a sample enterprise based on a Web page; training a recognition extraction model architecture based on the sample public data to obtain a general recognition extraction model; obtaining each set corporate domain corpus, updating the general recognition extraction model based on the corporate domain corpus, obtaining a recognition extraction model for each corporate domain, and obtaining a recognition extraction model library; receiving an extraction request for corporate public information, obtaining the public data of the enterprise to be analyzed for the web page in the extraction request, and retrieving a target recognition extraction model for the domain in which the enterprise to be analyzed is located in the recognition extraction model library; obtaining entity data and association relationships related to the enterprise to be analyzed based on the public data and the target recognition extraction model of the enterprise to be analyzed.

[0007] In one example, the recognition and extraction model architecture is trained according to the sample public data to obtain a general recognition and extraction model, which specifically includes: segmenting each sentence in the sample public data to obtain multiple sample segmentation sequences; removing stop words from each sample segmentation data to obtain multiple target sample segmentation sequences; in each target sample segmentation sequence, counting the occurrence frequency of each word; calculating the distribution value of a single word in the multiple target sample segmentation sequences, and generating the weight of each word according to the occurrence frequency and distribution value of a single word according to the TF-IDF method; converting each word into a high-dimensional vector according to the word embedding technology to obtain sample feature data; the sample feature data includes multiple sample word vector sequences; training the recognition and extraction model architecture according to the sample feature data to obtain a general recognition and extraction model.

[0008] In one example, the recognition extraction model architecture is trained according to the sample feature data to obtain a general recognition extraction model, which specifically includes: labeling each sample vocabulary vector sequence in the sample feature data to obtain labeled sample feature data; dividing the labeled sample feature data into a data set to obtain a training set, a validation set, and a test set; training the ALBERT model according to the training set to obtain an initial recognition extraction model; and tuning the initial recognition extraction model according to the validation set and the test set to obtain a general recognition extraction model.

[0009] In one example, the ALBERT model is trained according to the training set to obtain an initial recognition and extraction model, specifically including: dividing the training set into different batches according to different data sources and crawling times; the data of each batch comes from the same crawling time and the same data source; the training set is iterated in a mini-batch manner, and in each iteration, the data of each batch is input into the ALBERT model, the ALBERT model loss is calculated and back propagation is performed to update the model parameters of the ALBERT model to obtain the initial recognition and extraction model.

[0010] In one example, before training the ALBERT model according to the training set, the method also includes: constructing an embedding layer, wherein the embedding layer is used to map the input vocabulary vector sequence to a low-dimensional vector space; constructing a Transformer encoder architecture, wherein the Transformer encoder architecture includes multiple stacked encoding layers, each encoding layer includes a self-attention mechanism and a feedforward neural network; the Transformer encoder is used to encode the output vector of the embedding layer to generate a context representation vector for each vocabulary; constructing a classifier, wherein the classifier is in the model output layer and is used to predict the entity category of the vocabulary according to the context representation vector of each vocabulary; and constructing the ALBERT model based on the embedding layer, the encoder architecture and the classifier.

[0011] In one example, before obtaining each set enterprise domain corpus, the method also includes: obtaining enterprise public data in the set domain; performing feature representation based on the enterprise public data to obtain enterprise public feature data; labeling each vocabulary vector sequence in the enterprise public feature data to obtain labeled enterprise public data; and inputting the labeled enterprise public data into the corpus of the set domain.

[0012] In one example, after the public data is input into the target recognition and extraction model of the enterprise to be analyzed to obtain entity data and association relationships related to the enterprise to be analyzed, the method further includes: matching the entity data and association relationships in a pre-set early warning rule library to obtain the risk level of the enterprise to be analyzed; when the risk level is greater than the preset level, generating early warning risk notification information of the enterprise to be analyzed, and feeding back the early warning risk notification information to the client.

[0013] In one example, obtaining sample public data of sample enterprises based on web pages specifically includes: determining a web page data source to be crawled; the web page data source includes an official website of the enterprise, a social media platform, an industry media platform, and a financial news website; crawling original sample public data of the sample enterprise from the web page data source; removing HTML tags from the original sample public data in HTML format to obtain sample text public data; cleaning special characters, redundant punctuation, and extra spaces from the sample text public data to obtain the sample public data of the sample enterprise.

[0014] On the other hand, an embodiment of the present application provides a device for extracting corporate public information for web pages, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any one of the methods for extracting corporate public information for web pages described above.

[0015] On the other hand, an embodiment of the present application provides a non-volatile computer storage medium for extracting corporate public information from a web page, storing computer executable instructions, and the computer executable instructions can execute any of the above-mentioned methods for extracting corporate public information from a web page.

[0016] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects:

[0017] The corpus is divided into different enterprise fields, and the enterprise field corpus is pre-set. After building a general recognition model, the general recognition and extraction model is updated according to the enterprise field corpus to obtain the recognition and extraction model for each enterprise field, and information extraction for specialized fields is performed. The recognition and extraction model is personalized for each field, which can effectively improve the efficiency and accuracy of enterprise public information entity recognition tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solution of the present application, some embodiments of the present application will be described in detail below in conjunction with the accompanying drawings, in which:

[0019] Figure 1 A flowchart of a method for extracting public information of an enterprise from a web page provided in an embodiment of the present application;

[0020] Figure 2 A schematic diagram of the structure of a device for extracting corporate public information from web pages provided in an embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0022] Some embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0023] Figure 1 The present invention provides a flowchart of a method for extracting public information of enterprises from web pages. The method can be applied to different business fields, and some input parameters or intermediate results in the process can be manually adjusted to help improve accuracy.

[0024] The analysis method involved in the embodiments of the present application can be implemented by a terminal device or a server, and the present application does not impose any special restrictions on this. For the convenience of understanding and description, the following embodiments are described in detail by taking a server as an example.

[0025] It should be noted that the server may be a single device or a system composed of multiple devices in series, that is, a distributed server, and this application does not make any specific limitation on this.

[0026] In today's information age, the booming development of the Internet has provided enterprises with an unprecedented information publishing and sharing platform. The public information released by enterprises through the Internet is of various types and rich in content, which has far exceeded the scope of traditional media, covering multiple dimensions such as detailed data of financial reports, the latest developments of press releases, user feedback on social media, and detailed product information. This information is not only an important basis for corporate operations and decision-making, but also a valuable resource for external stakeholders such as market analysts, competitor researchers and investors to obtain key intelligence. However, in the face of such a huge amount of information, the traditional manual extraction method seems to be unable to cope with it. It not only consumes a lot of manpower and time costs, but also is limited by human factors. It is difficult to ensure the integrity and accuracy of the information, and it is very easy to omissions or errors. Therefore, exploring a method that can efficiently and accurately automatically extract corporate public information has become a technical problem that the industry urgently needs to solve.

[0027] At present, a variety of methods have been proposed in this field, including rule-based methods, statistical machine learning-based methods, and deep learning-based methods that have emerged in recent years. Among them, deep learning models, especially pre-trained language models (PLMs) such as BERT, have achieved excellent performance in NER tasks. These models can use large-scale unlabeled text for self-supervised learning, thereby mastering rich language knowledge and contextual information, effectively improving the accuracy and generalization ability of named entity recognition. Although significant progress has been made in the field of corporate public information recognition and extraction, the diversity and complexity of corporate public information make the information extraction task more difficult, and it is necessary to explore more efficient and accurate information extraction methods.

[0028] However, traditional NER methods rely on manual feature design and have limited effects. In recent years, models based on deep learning, such as BERT and its variants, have demonstrated powerful performance, but there are still some challenges in Chinese NER tasks, such as insufficient understanding of context and high computational complexity of the model. The purposes of the present invention are mainly reflected in the following aspects: (1) Improving the efficiency of information extraction: extracting corporate public information in an automated manner, significantly reducing the time and cost of manual intervention, and improving the efficiency of information processing. (2) Enhancing the accuracy of information extraction: using natural language processing and machine learning techniques to improve the accuracy of identifying and extracting corporate information and reduce errors and omissions in the information extraction process. (3) Realizing real-time information updates: by regularly capturing and processing corporate public information, real-time monitoring of market dynamics can be achieved, helping companies to adjust strategies and decisions in a timely manner.

[0029] Figure 1 The process in includes the following steps:

[0030] S101: Obtain sample public data of sample enterprises according to the web page.

[0031] In some embodiments of the present application, when crawling sample public data from a web page, data source selection, data collection and data cleaning are performed, and the process is as follows:

[0032] First, determine the web page data source to be crawled. Web page data sources include corporate official websites, social media platforms, industry media platforms, and financial news websites. Then, crawl the original sample public data of the sample company from the web page data source. Then, remove the HTML tags from the original sample public data in HTML format to obtain the sample text public data. Finally, clean the sample text public data of special characters, redundant punctuation, and extra spaces to obtain the sample public data of the sample company.

[0033] It should be noted that accurately and comprehensively selecting appropriate data sources is a key step in ensuring the quality of analysis. This process requires not only the accuracy and timeliness of data, but also the coverage of multi-dimensional information of the enterprise in order to comprehensively evaluate the enterprise's operating conditions, market position and future development potential. The following are several suitable data source options.

[0034] Official website of a company: The official website of a company is an important channel for obtaining the latest announcements, financial reports, product information, company introductions and news, reflecting the company's operating conditions and development trends. By browsing the official website, you can quickly grasp the latest developments of the company, which directly reflects the company's business strategy, market layout and innovation capabilities. In particular, the financial report section records the company's financial status, operating performance and cash flow in detail, which is the key basis for evaluating the company's economic strength and profitability.

[0035] Social media platforms: These platforms gather a large amount of user content. Information on social media is often more immediate and intuitive, and can capture the public's reaction to the company, market trends and potential problems. For example, users' positive or negative comments on products or services can provide direct feedback for companies to improve their products or services; discussions on hot topics in the industry can help companies grasp market hotspots and competitor dynamics.

[0036] Financial news websites and industry media reports: focus on reporting financial news, industry analysis, policy interpretation, etc. They not only provide the latest industry dynamics and policy changes, but also conduct in-depth analysis of market trends, corporate competition patterns and macroeconomic environment, which can help companies gain a deeper understanding of their industry background and predict possible market changes in the future.

[0037] Information released by securities regulators: The annual and quarterly reports released provide financial data of enterprises. This information is highly authoritative and accurate, and is an important basis for assessing the financial health, compliance and potential risks of enterprises. By analyzing this data, investors can more accurately judge the profitability, growth potential and investment risks of enterprises.

[0038] To sum up, each of the above data sources has its own advantages. In actual operation, different data sources should be flexibly selected and integrated according to specific needs and data availability.

[0039] In some embodiments of the present application, after selecting a suitable data source, it is necessary to crawl the required information from the selected data source. This stage is mainly the data collection stage. In order to obtain this data efficiently and legally, the choice of data collection method is particularly important. The most mainstream methods are crawlers and APIs.

[0040] The crawler's workflow usually starts with seed URLs, which are the web addresses that the crawler initially visits. The crawler will send HTTP requests to these URLs and receive the returned HTML content. Then, it will parse the HTML content, extract the required data, and store the data in the database. At the same time, the crawler will also extract new links from the HTML so that the web pages pointed to by these links can be accessed in the subsequent crawling process. When using web crawlers for data collection, you must strictly abide by laws and regulations and the website's terms of use, including checking the website's robots.txt file, which usually specifies which web pages can be accessed by the crawler and which web pages are restricted. In addition, in order to avoid placing too much burden on the target website, the crawler also needs to set a reasonable crawling frequency and respond to anti-crawling measures that the website may take, such as verification code verification, IP address blocking, etc.

[0041] As for API, it allows developers to directly obtain enterprise data by sending standardized requests, which improves the efficiency and accuracy of data collection. At the same time, API usually provides detailed information on data format, access rights, error handling, etc., making the data collection process more controllable and reliable. Although manual collection is time-consuming and costly, it is sometimes an indispensable method, such as when the enterprise does not provide an API interface.

[0042] In summary, data is collected using three methods: crawlers, APIs, and manual collection to ensure the efficiency and accuracy of data collection.

[0043] In some embodiments of the present application, after completing data collection, faced with various data collected from different information sources, which are often mixed with noise, irrelevant information, inconsistent formats and other problems. If these problems are not properly handled, they will seriously affect the accuracy and efficiency of subsequent data analysis, data mining and entity recognition. Therefore, the collected data must be processed. This stage is the data cleaning stage.

[0044] For data captured from web pages, HTML tags not only do not contain useful information, but may also interfere with the subsequent text analysis process. Therefore, these tags are usually removed to keep only the plain text content. This application is implemented by regular expressions or HTML parsing libraries.

[0045] Regarding regular expressions, you can write complex regular expressions to match and delete HTML tags. If there are many nested tags or attributes, you can use an HTML parsing library. Using an HTML parsing library such as BeautifulSoup can remove HTML tags more efficiently and accurately. BeautifulSoup provides an easy-to-use API that allows developers to access HTML documents in a tree structure and easily extract plain text content.

[0046] After removing HTML tags, the text may still contain some unnecessary special characters, redundant punctuation marks, and extra spaces. For special character replacement, string replacement methods or regular expressions are often used to identify and replace these special characters. For punctuation marks, rules are often written to delete or replace redundant punctuation marks. For space compression, the string split and join methods or regular expressions can be used to compress extra spaces to ensure that there is only one space between words in the text.

[0047] In addition, the data may have problems such as inconsistent data formats, missing values, and duplicate records. This application solves these problems in the following ways:

[0048] For data formats that are not uniform, you can use a parsing library (such as a date parsing library) to identify conversions or use functions to unify the format. For missing values, if there are few missing values ​​and they have little impact on the overall data analysis, you can delete them directly. If there are many missing values ​​and they have a significant impact on the data analysis results, you can use methods such as the mean and median to fill in these missing values. For duplicate record issues, you need to pay attention to identifying and deleting duplicate records. To this end, you can calculate a hash value for each record and compare the hash values ​​to identify and delete duplicate records.

[0049] When information comes from multiple file formats, you need to use the corresponding text parsing engine to extract the required text content. For example, for PDF parsing, use PyPDF2 and other libraries to parse PDF files and extract the text content. For Word document parsing, use the python-docx library to parse Word documents and extract the text and table content. For Excel file parsing, use the pandas library to read Excel files and extract the data. For the text content in Excel, you can use the string operation function of pandas for further cleaning and processing.

[0050] In summary, data cleaning is an essential process that requires comprehensive consideration of multiple factors such as data source, format, quality, etc. Through a reasonable cleaning, the quality and availability of data can be significantly improved, laying a solid foundation for subsequent data analysis and entity recognition.

[0051] S102: According to the sample public data, the recognition and extraction model architecture is trained to obtain a general recognition and extraction model.

[0052] In some embodiments of the present application, during the training process, it is necessary to perform feature representation and data labeling on sample public data.

[0053] In order to achieve more effective and efficient recognition and extraction, data needs to be represented by features, a process that involves multiple key steps aimed at extracting informative features from raw text.

[0054] Based on this, first, each sentence in the sample public data is segmented to obtain multiple sample segmentation sequences. Then, stop words are removed from each sample segmentation data to obtain multiple target sample segmentation sequences. Then, in each target sample segmentation sequence, the frequency of occurrence of each word is counted. Then, the distribution value of a single word in multiple target sample segmentation sequences is calculated, and according to the TF-IDF method, the weight of each word is generated according to the frequency of occurrence and distribution value of a single word. Then, each word is converted into a high-dimensional vector according to the word embedding technology to obtain sample feature data. Among them, the sample feature data includes multiple sample word vector sequences. Then, the recognition and extraction model architecture is trained according to the sample feature data to obtain a general recognition and extraction model.

[0055] It should be noted that word segmentation is the first step of feature representation. Word segmentation is the process of dividing a continuous sequence of characters into independent words or phrases. Effective word segmentation can help to understand the text content more accurately and lay a solid foundation for subsequent steps. Commonly used word segmentation tools include Jieba Word Segmentation and IK Analyzer, which can accurately segment text into words based on contextual information and statistical laws.

[0056] Next, removing stop words is another important preprocessing step. Stop words refer to words that appear frequently in the text but contribute little to the actual meaning of the text. Therefore, by constructing a stop word list, these meaningless words can be removed from the text, thereby simplifying the feature representation.

[0057] After removing stop words, it is necessary to count the frequency of occurrence of each word in the text. This step can help understand the distribution of each word in the text and provide a basis for subsequent feature selection. This application uses TF-IDF and Word2Vec feature representation methods.

[0058] The TF-IDF method is a commonly used feature representation method. Specifically, the TF-IDF method calculates the frequency (TF) of a word in each target sample word segmentation sequence and the inverse document frequency (IDF) of the word in multiple target sample word segmentation sequences, and multiplies these two values ​​to obtain the TF-IDF value of the word. The larger the TF-IDF value, the more important the word is in the document, and the greater its role in distinguishing different documents. This method can effectively reduce the weight of common words and increase the weight of rare words, thereby enhancing the effectiveness of feature representation.

[0059] The word embedding technology Word2Vec method converts words into high-dimensional vectors so that semantically similar words are close to each other in the vector space. Word2Vec maps words into a continuous vector space by training a neural network model so that the cosine similarity between vectors can reflect the semantic similarity between words. This method can capture the potential semantic relationship between words, thereby enhancing the richness and effectiveness of feature representation. In subsequent data analysis and model training, these vectors can be used as features to input into machine learning algorithms to improve model performance.

[0060] That is to say, the sample feature data is feature representation data, and data labeling is continued on the sample feature data.

[0061] In some embodiments of the present application, each sample vocabulary vector sequence in the sample feature data is labeled to obtain labeled sample feature data.

[0062] It should be noted that data labeling is a crucial task in the field of machine learning. The core of data labeling is to convert raw data into labeled samples. These labels not only reflect the category of the data, but also contain rich contextual information, thereby guiding the model to learn the inherent laws and characteristics of the data during the training process.

[0063] In the context of enterprise public data analysis, the goal of data annotation is usually to identify and extract valuable information. The accurate identification of this information is crucial for subsequent decision support, market insights, risk assessment and other applications. In the named entity recognition (NER) task, BIO and BIOES are two widely used annotation methods.

[0064] BIO annotation is a concise and effective annotation method. Specifically, the B (Begin) tag is used to mark the starting word of the entity, the I (Inside) tag is used to mark the internal word of the entity, and the O (Outside) tag is used to mark the non-entity word. This annotation method enables the model to easily identify the entity boundaries in the text, thereby achieving better results in the NER task. However, due to the lack of a clear end mark in the BIO annotation method, when there are multiple closely adjacent entities in the text, the model may find it difficult to accurately determine the end position of the entity. In order to solve this problem, this application adopts the BIOES annotation method, which adds E (End) and S (Single) on the basis of the BIO annotation method. The E (End) tag is used to clearly mark the end position of the entity, thereby solving the ambiguity problem of the BIO annotation method when dealing with tight entities. The S (Single) tag is used to represent a single complete entity. This improvement not only improves the accuracy of the annotation, but also makes the model more accurate and efficient in understanding and identifying entities. In the analysis of corporate public data, the use of the BIOES annotation method can effectively improve the accuracy of the NER task, thereby providing more reliable information extraction and decision support for enterprises.

[0065] In some embodiments of the present application, it is also necessary to perform data partitioning on the labeled data, which directly affects the training effect, tuning strategy and accuracy of the final evaluation of the model.

[0066] Based on this, the labeled sample feature data is divided into data sets to obtain training sets, validation sets, and test sets. The ALBERT model is trained according to the training set to obtain the initial recognition and extraction model. The initial recognition and extraction model is tuned according to the validation set and the test set to obtain the general recognition and extraction model.

[0067] It should be noted that in data partitioning, the common partition ratio is about 70-80% as training set, about 10-15% as validation set, and about 10-15% as test set. This partitioning method ensures that the training set has enough size to support the full learning of the model, and provides enough data for the validation set and test set to tune and evaluate the model.

[0068] The training set is the main source of model learning. It contains a large number of samples that cover the main features and categories of the data. Through the training set, the model can learn the distribution of the data, the relationship between features, and the boundaries between categories. Therefore, when dividing the training set, you need to ensure that it contains sufficiently diverse and representative samples.

[0069] The validation set is used for model tuning. During the model training process, the model parameters and structure need to be constantly adjusted to improve its performance. It provides a data set independent of the training set to evaluate the performance of the model on unseen data. By comparing the performance of the model on the validation set, the optimal model parameters and structure can be selected.

[0070] The test set is used for the final evaluation of the model. After the model is tuned, the test set is needed to evaluate the performance of the model. It contains the same or similar data distribution and features as the training set and validation set. The performance on the test set can be used to objectively evaluate the generalization ability and performance stability of the model.

[0071] It should be noted that the ratio of data partitioning is not fixed. In practical applications, it needs to be adjusted according to the size and distribution of the data and the specific needs of the model. In addition, data partitioning also needs to pay attention to the randomness and independence of the data. It is necessary to ensure that each subset is randomly selected and independent of each other. This can avoid mutual influence and interference between data, thereby improving the accuracy and reliability of model training and evaluation.

[0072] It should be noted that the ALBERT model used in this application adopts the Transformer encoder architecture, which provides strong support for processing such complex information extraction tasks. This architecture consists of multiple stacked encoding layers, each of which is embedded with two core sequence components: the self-attention mechanism and the feedforward neural network, so that the model can fully consider the contextual information of the entire input sequence when processing each word. The construction process is as follows:

[0073] Construct an embedding layer, which is used to map the input vocabulary vector sequence to a low-dimensional vector space.

[0074] Construct a Transformer encoder architecture, which includes multiple stacked encoding layers. Each encoding layer contains a self-attention mechanism and a feedforward neural network. The Transformer encoder is used to encode the output vector of the embedding layer to generate a context representation vector for each word.

[0075] Build a classifier, which is at the output layer of the model and is used to predict the entity category of each word based on the context representation vector of each word.

[0076] Construct the ALBERT model based on the embedding layer, encoder architecture, and classifier.

[0077] It should be noted that the self-attention mechanism is an important feature of the model architecture. It enables the model to consider the entire input sequence when processing each word. This design enables the model to capture rich contextual dependencies and thus more accurately understand entity information in the text. At the same time, the feedforward neural network further enhances the model's nonlinear transformation capabilities, making the model more adept at processing complex text features.

[0078] It is also necessary to explain how the Transformer encoder encodes the input text and generates a contextual representation vector for each word. First, the Transformer encoder converts each word in the input text into a corresponding vector representation. These vectors are called word embedding vectors. The dimension of the word embedding vector is usually low. Since the Transformer model does not rely on the order of time steps like a recurrent neural network, a way is needed to encode the position information of the input data. Position encoding is a method of representing position information by adding to word embedding. It allows the model to distinguish between words at different positions. The position encoding vector is usually generated by sine and cosine functions, and each dimension corresponds to a sine or cosine function of different frequencies. In this way, the encoding vectors at different positions have different angles in space, thereby expressing the relative position relationship between them. Since the Transformer has a self-attention mechanism, it allows the model to see and consider the information of the entire input sequence when processing each word. The self-attention mechanism generates the contextual representation of each word by calculating the correlation score between each word and other words in the input sequence. Since the Transformer has a feedforward neural network, it further processes the output of the self-attention layer. Through the combined effect of the self-attention mechanism and the feedforward neural network, the Transformer encoder is able to integrate the semantic information and position information of each word in the input sequence, which enables the model to capture rich contextual dependencies and generate a contextual representation vector for each word.

[0079] It should be noted here that in addition to the above-mentioned architectural advantages, the ALBERT model also effectively reduces the number of model parameters through the strategy of cross-layer parameter sharing. Specifically, the embedding layer is cleverly decomposed into two smaller matrices, first mapping the input to a low-dimensional space E, and then mapping it to the hidden space H. This design not only reduces the number of parameters in the embedding layer from O(V×H) to O(V×E+E×H) (where V is the size of the vocabulary), but also improves the generalization ability of the model, enabling it to perform well when processing text data in different fields.

[0080] It should also be noted here that in the architecture of the ALBERT model, the output layers are those used to obtain the contextual representation vectors of each input word. The generated vectors capture the meaning of the words in different contexts, and they are used as input for further tasks. These vectors are located in the output of the last layer of the Transformer block of the model. To predict the entity category to which each word belongs based on the ALBERT model, a classifier needs to be added. This classifier is usually a fully connected neural network layer, which receives the contextual representation vectors of each word extracted from the last layer of the Transformer block of the ALBERT model as input, and inputs these contextual representation vectors into a fully connected layer. This fully connected layer calculates the score of each entity category based on the contextual representation of each word. Before output, an activation function is used to convert the score into a probability distribution, and a probability distribution of the entity category is output. For each word, the entity category with the highest probability is selected as the prediction result.

[0081] It should also be noted that in the training process, the cross entropy loss function is the most widely used for entity recognition tasks. It adopts the idea of ​​minimizing KL divergence and optimizes the prediction results of the model by reducing the probability of data noise. At the same time, the Adam optimizer has become the first choice in the training process due to its adaptive learning rate characteristics and bias correction mechanism. It can not only accelerate the training process of the model, but also improve the overall optimization efficiency.

[0082] In addition, the process of training the ALBERT model based on the training set is as follows:

[0083] The training set is divided into different batches according to the data source and crawling time; the data of each batch comes from the same crawling time and the same data source. The training set is iterated in a mini-batch manner. In each iteration, the data of each batch is input into the ALBERT model, the ALBERT model loss is calculated and back propagation is performed to update the model parameters of the ALBERT model and obtain the initial recognition and extraction model.

[0084] In order to further improve the performance of the model, some strategies and techniques need to be adopted during the training process. For example, when the performance on the validation set begins to decline, training should be stopped in time to prevent the model from overfitting. In addition, randomly discarding a part of the neuron connections is also an effective way to reduce the complexity of the model and prevent overfitting. At the same time, normalizing each batch of data can not only speed up the model training, but also improve the stability of the model.

[0085] It should also be noted that after the ALBERT model successfully completes its initial pre-training phase, fine-tuning is usually performed to further optimize its performance on specific tasks. First, a smaller learning rate is selected to update the model parameters. This is because an excessively large learning rate may cause drastic changes in the model parameters, thereby destroying the knowledge learned in the pre-training phase. Secondly, data augmentation techniques are used to expand the training data set. These techniques include synonym replacement, random insertion or deletion of words in the text, etc. Through these operations, more diverse training samples can be generated, which helps the model learn richer feature representations and enhances the generalization ability and robustness of the model. In addition, the performance of the model is regularly evaluated on the validation set. The performance of the model is comprehensively measured by calculating evaluation indicators such as accuracy, recall, and F1 value. Based on these evaluation results, the model parameters or fine-tuning strategies can be adjusted in a timely manner to ensure that the performance of the model on specific tasks can continue to improve.

[0086] S103: Acquire each set enterprise domain corpus, update the general recognition and extraction model according to the enterprise domain corpus, obtain the recognition and extraction model of each enterprise domain, and obtain the recognition and extraction model library.

[0087] In some embodiments of the present application, the process of constructing an enterprise domain corpus is as follows:

[0088] First, obtain the public data of enterprises in the set field. For example, finance, manufacturing, technology, etc. Then, perform feature representation based on the public data of enterprises to obtain public feature data of enterprises. Then, annotate each word vector sequence in the public feature data of enterprises to obtain annotated public data of enterprises. Finally, input the annotated public data of enterprises into the corpus of the set field.

[0089] It should be noted that the method of feature representation and data annotation can be found in the description of the relevant content in S102 and will not be described in detail here.

[0090] S104: receiving an extraction request for enterprise public information, obtaining the public data of the enterprise to be analyzed on the web page in the extraction request, and retrieving a target recognition and extraction model of the field in which the enterprise to be analyzed is located in the recognition and extraction model library.

[0091] It should be noted that the client needs to input the name keywords of the enterprise to be analyzed to generate an extraction request.

[0092] S105: Obtain entity data and association relationships related to the enterprise to be analyzed based on the public data and the target identification and extraction model of the enterprise to be analyzed.

[0093] It should be noted that the output entity data includes entity category, entity text and its location information in the original text, etc., and also includes entity confidence information.

[0094] In some embodiments of the present application, after identifying the entity data and related associations of the enterprise to be analyzed, the risk data of the enterprise to be analyzed can be predicted. For example, there is a problem with the supply quality of Company B, the supplier of Company A. If Company A is the enterprise to be analyzed, the entity data includes Company A and Company B, and the association is that the supply relationship between Company A and Company B is risky. Or, the financial statements of Company A are overdue for N days, the entity data includes Company A, the association is that the financial statements are overdue, and the number of overdue days is N days.

[0095] Based on this, firstly, the entity data and the associated relationships are matched in the pre-set warning rule library to obtain the risk level of the enterprise to be analyzed. Then, when the risk level is greater than the preset level, the warning risk notification information of the enterprise to be analyzed is generated and fed back to the client.

[0096] It should be noted that a series of early warning rules are formulated based on historical data and industry experience, such as setting specific financial ratio thresholds, overdue days, debt ratio, market share, partnership, supply relationship and other rules. For example, the financial statements show that its current ratio is lower than the threshold in the early warning rule library, and the overdue days also exceed the set upper limit. Based on the matching of these characteristic data and rules, it is determined that the risk level of the enterprise is high.

[0097] It should be noted that although the embodiments of the present application are based on Figure 1 Steps S101 to S105 are described in sequence, but this does not mean that steps S101 to S105 must be performed in a strict order. Figure 1 The order shown in the figure is to introduce and explain step S101 to step S105 in sequence, in order to facilitate those skilled in the art to understand the technical solution of the embodiment of the present application. In other words, in the embodiment of the present application, the order between step S101 to step S105 can be appropriately adjusted according to actual needs.

[0098] pass Figure 1The method is used to divide the corpus into different enterprise fields and pre-set the enterprise field corpus. After building a general recognition model, the general recognition extraction model is updated according to the enterprise field corpus to obtain the recognition extraction model of each enterprise field, and information extraction is performed in specialized fields. The recognition extraction model is customized for each field, which can greatly improve the ability to understand the context in information recognition and extraction and simplify the high computational complexity of the model. In addition, there is no other information interference in the field, which can effectively improve the efficiency and accuracy of enterprise public information entity recognition tasks.

[0099] In addition, the breadth of information access has increased: by integrating multiple data sources (such as corporate official websites, social media, financial news and reports released by regulators), comprehensive capture of corporate public information is achieved to ensure the timeliness and accuracy of information.

[0100] Improvement of data quality: A systematic data cleaning and preprocessing process is used to remove noise and irrelevant information, improve data quality, and provide a reliable foundation for subsequent analysis and entity identification.

[0101] Enhanced feature representation: Through technologies such as TF-IDF and Word2Vec, the expressive power of text features is enhanced, enabling the model to better understand and process complex text data.

[0102] Improved model performance and efficiency: The lightweight design and self-attention mechanism of the ALBERT model, combined with appropriate loss functions and optimizers, significantly improves the performance and training efficiency of entity recognition. Compared with the traditional BERT model, the ALBERT model has higher parameter efficiency, faster training speed and more stable performance.

[0103] Model fine-tuning is more flexible: Through data enhancement and regular evaluation strategies, the performance of the model on specific tasks is further improved to ensure its robustness and accuracy in practical applications.

[0104] Figure 2 A schematic diagram of a structure of a device for extracting public information of an enterprise from a web page provided in an embodiment of the present application includes:

[0105] at least one processor; and,

[0106] a memory communicatively connected to the at least one processor; wherein,

[0107] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any one of the above-mentioned methods for extracting public enterprise information from a web page.

[0108] Some embodiments of the present application provide a non-volatile computer storage medium for extracting corporate public information from a web page, which stores computer executable instructions, and the computer executable instructions can execute any of the above-mentioned methods for extracting corporate public information from a web page.

[0109] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0110] The devices and media provided in the embodiments of the present application correspond one-to-one to the methods. Therefore, the devices and media also have similar beneficial technical effects as the corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0111] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0112] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0113] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1A function specified in one or more boxes.

[0114] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0115] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0116] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0117] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0118] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0119] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the technical principle of the present application should all fall within the protection scope of the present application.

Claims

1. A method for extracting public information of an enterprise from a web page, characterized in that: The method comprises: According to the web pages, obtain sample public data of sample enterprises; According to the sample public data, the recognition and extraction model architecture is trained to obtain a general recognition and extraction model; Acquire each set enterprise domain corpus, update the general recognition and extraction model according to the enterprise domain corpus, obtain the recognition and extraction model of each enterprise domain, and obtain the recognition and extraction model library; Receiving an extraction request for enterprise public information, obtaining the public data of the enterprise to be analyzed on the web page in the extraction request, and retrieving a target recognition and extraction model of the field in which the enterprise to be analyzed is located in the recognition and extraction model library; According to the public data and the target recognition and extraction model of the enterprise to be analyzed, entity data and association relationships related to the enterprise to be analyzed are obtained.

2. The method according to claim 1, characterized in that: The training of the recognition and extraction model architecture according to the sample public data to obtain a general recognition and extraction model specifically includes: Segmenting each sentence in the sample public data to obtain multiple sample segmentation sequences; Remove stop words from each sample word segmentation data to obtain multiple target sample word segmentation sequences; In each target sample word segmentation sequence, count the frequency of occurrence of each word; Calculate the distribution value of a single word in the multiple target sample word sequences, and generate the weight of each word according to the occurrence frequency and distribution value of the single word according to the TF-IDF method; Convert each word into a high-dimensional vector according to the word embedding technology to obtain sample feature data; the sample feature data includes a plurality of sample word vector sequences; The recognition and extraction model architecture is trained according to the sample feature data to obtain a universal recognition and extraction model.

3. The method according to claim 2, characterized in that The training of the recognition and extraction model framework according to the sample feature data to obtain a universal recognition and extraction model specifically includes: Labeling each sample vocabulary vector sequence in the sample feature data to obtain labeled sample feature data; Divide the labeled sample feature data into a data set to obtain a training set, a validation set, and a test set; The ALBERT model is trained according to the training set to obtain an initial recognition and extraction model; The initial recognition and extraction model is tuned according to the verification set and the test set to obtain a universal recognition and extraction model.

4. The method according to claim 3, characterized in that The training of the ALBERT model according to the training set to obtain an initial recognition and extraction model specifically includes: The training set is divided into different batches according to different data sources and crawling times; the data of each batch comes from the same crawling time and the same data source; The training set is iterated in a mini-batch manner. In each iteration, each batch of data is input into the ALBERT model, the ALBERT model loss is calculated and back propagation is performed to update the model parameters of the ALBERT model to obtain an initial recognition and extraction model.

5. The method according to claim 3, characterized in that: Before training the ALBERT model according to the training set, the method further includes: Construct an embedding layer, which is used to map the input vocabulary vector sequence to a low-dimensional vector space; Constructing a Transformer encoder architecture, wherein the Transformer encoder architecture includes a plurality of stacked encoding layers, each encoding layer including a self-attention mechanism and a feed-forward neural network; the Transformer encoder is used to encode the output vector of the embedding layer to generate a context representation vector for each word; Constructing a classifier, which is in the model output layer and is used to predict the entity category of each word based on the context representation vector of each word; Construct the ALBERT model based on the embedding layer, encoder architecture, and classifier.

6. The method according to claim 1, characterized in that Before obtaining each set enterprise domain corpus, the method further includes: Obtain public data of enterprises in a specified field; Performing feature representation according to the enterprise public data to obtain enterprise public feature data; Labeling each vocabulary vector sequence in the enterprise public feature number to obtain labeled enterprise public data; The annotated enterprise public data is input into the corpus of the set domain.

7. The method according to claim 1, characterized in that After inputting the public data into the target recognition and extraction model of the enterprise to be analyzed to obtain entity data and association relationships related to the enterprise to be analyzed, the method further includes: Matching the entity data and the association relationship in a pre-set early warning rule library to obtain the risk level of the enterprise to be analyzed; When the risk level is greater than a preset level, early warning risk notification information of the enterprise to be analyzed is generated, and the early warning risk notification information is fed back to the client.

8. The method according to claim 1, characterized in that The obtaining of sample public data of sample enterprises according to the web pages specifically includes: Determine the web page data source to be crawled; the web page data source includes corporate official websites, social media platforms, industry media platforms and financial news websites; Crawling original sample public data of sample enterprises from the web data source; Removing HTML tags from the original sample public data in HTML format to obtain sample text public data; The sample text public data is cleaned of special characters, redundant punctuation marks and extra spaces to obtain the sample public data of the sample enterprise.

9. A device for extracting public information of enterprises from web pages, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method for extracting public enterprise information from a web page as described in any one of claims 1 to 8.

10. A non-volatile computer storage medium for extracting public information of enterprises from web pages, storing computer executable instructions, characterized in that: The computer executable instructions can execute the method for extracting corporate public information from a web page as described in any one of claims 1 to 8.

Images