A method and device for extracting web page information based on attribute perception

Abstract

The present invention discloses a method and device for extracting Web page information based on attribute perception, which belongs to the technical field of web page information extraction. The method comprises: crawling the original web page data of a given target website, saving it as the original input of a model, preprocessing the data on the original data, removing useless label inputs in the original data such as scripts, and reducing the noise information of the input model; extracting variable nodes in the web page, constructing the token sequence and xpath sequence of the input model, and recording the position information of the sequence at the same time, adding an attribute perception embedding module on the basis of the pre-trained model, so that the pre-trained model can perceive the pattern information of the attribute, embedding the input information through a classification layer, and the model outputs the final result of the Web page information extraction. The method can improve the accuracy of Web page information extraction, provide rich structured knowledge for downstream tasks, and promote the development of downstream tasks.

Description

Technical Field

[0001] The present invention belongs to the technical field of web page information extraction, and in particular relates to a method and device for extracting web page information based on attribute perception. Background Art

[0002] With the rapid development of the Internet, the information carried by Web pages is very rich. Extracting information from Web pages is a very basic and important task, which can play a role in many practical applications, such as the construction of knowledge bases, question-answering systems, recommendation systems, and retrieval systems. Web pages are different from traditional documents with rich text (such as news). The information in Web pages is composed of markup languages, which have the characteristics of block division and irregular syntax. These characteristics increase the difficulty of information extraction from Web pages and have also attracted widespread attention from academia and industry.

[0003] Current Web page information extraction methods mainly focus on how to represent HTML pages, such as how to use DOM tree features, Xpath information, visual features, etc., and add additional computational overhead, which will become inefficient in large-scale information extraction tasks. Moreover, current Web information extraction work lacks effective use of the model of the attributes to be extracted, and the model cannot perceive the situation of the attributes to be extracted. Summary of the invention

[0004] In view of the above technical problems, the present invention proposes a method and device for extracting Web page information based on attribute perception; it aims to solve the problem that the attributes of the information to be extracted are not effectively utilized during the Web page information extraction process, while reducing the calculation overhead of additional information.

[0005] The first aspect of the present invention discloses a method for extracting web page information based on attribute perception, the method comprising:

[0006] Step S1, obtaining a target web page from which information is to be extracted, crawling the HTML source code of the target web page using a web crawler, and saving it in a JSON file format;

[0007] Step S2: preprocess the HTML source code in the JSON file format to remove noise in the HTML source code, extract token sequences and xpath sequences based on variable nodes, and record the position information of the sequences;

[0008] Step S3: Using the token sequence, xpath sequence, and location information as inputs to the information extraction model, the information extraction model outputs the information extraction result of the target Web page through feature embedding and attribute classification.

[0009] According to the method of the first aspect of the present invention, in step S1: obtain the target Web page from which information is to be extracted, send an HTTP request message to a specific URL and receive a returned HTTP response message, and based on the pre-set attribute information to be crawled, use a web crawler to crawl the HTML source code corresponding to the attribute information in the target Web page from the HTTP response message, and save it in JSON file format.

[0010] According to the method of the first aspect of the present invention, in step S2, the preprocessing includes:

[0011] Remove noise from HTML source code and perform format normalization, the noise includes: tags that do not contain text content, unicode control characters, non-ASCII characters, spaces;

[0012] Build a DOM tree based on the data after denoising and format normalization, iterate through each node in the DOM tree, save the text on the node and the corresponding xpath path, and use the nodes with changed content corresponding to the same xpath as variable nodes;

[0013] Take the xpath path as the key, the set of all nodes corresponding to the xpath as the value, sort the set in descending order using the size of the set as the sorting key, and take the first n nodes as the variable nodes;

[0014] A token sequence and an xpath sequence are constructed based on the variable node, and the position information of the sequence is recorded.

[0015] According to the method of the first aspect of the present invention, the information extraction model includes a pre-trained model, an attribute perception module and a classifier. In step S3:

[0016] Adding the token sequence, the xpath sequence, and the position information as input of the pre-trained model to perform embedded representation to obtain a first embedded representation;

[0017] The token sequence is used as the input of the attribute perception module to extract the pattern of the token sequence, and the token sequence is embedded represented based on the pattern of the token sequence to obtain a second embedded representation;

[0018] The second embedding representation obtained by the attribute perception module and the first embedding representation obtained by the pre-trained model are combined to obtain a total embedding representation, which is used as the input of the classification layer to output the information extraction result of the target Web page;

[0019] Among them, the classification layer adopts the softmax function.

[0020] The second aspect of the present invention discloses a device for extracting web page information based on attribute perception. The device includes a processing unit, and the processing unit is configured to execute:

[0021] Obtain the target web page to be extracted, crawl the HTML source code of the target web page using a web crawler, and save it in JSON file format;

[0022] Preprocess the HTML source code in JSON file format to remove noise in the HTML source code, extract token sequences and xpath sequences based on variable nodes, and record the position information of the sequences;

[0023] The information extraction model takes token sequence, xpath sequence and location information as input, and outputs the information extraction result of the target Web page through feature embedding and attribute classification.

[0024] According to the apparatus of the second aspect of the present invention, the processing unit is specifically configured to execute:

[0025] Get the target web page from which the information is to be extracted, send an HTTP request message to a specific URL and receive the returned HTTP response message. Based on the pre-set attribute information to be crawled, use a web crawler to crawl the HTML source code corresponding to the attribute information in the target web page from the HTTP response message and save it in JSON file format.

[0026] According to the device of the second aspect of the present invention, the preprocessing comprises:

[0027] Remove noise from HTML source code and perform format normalization, the noise includes: tags that do not contain text content, unicode control characters, non-ASCII characters, spaces;

[0028] Build a DOM tree based on the data after denoising and format normalization, iterate through each node in the DOM tree, save the text on the node and the corresponding xpath path, and use the nodes with changed content corresponding to the same xpath as variable nodes;

[0029] Take the xpath path as the key, the set of all nodes corresponding to the xpath as the value, sort the set in descending order using the size of the set as the sorting key, and take the first n nodes as the variable nodes;

[0030] A token sequence and an xpath sequence are constructed based on the variable node, and the position information of the sequence is recorded.

[0031] According to the device of the second aspect of the present invention, the information extraction model includes a pre-trained model, an attribute perception module and a classifier, and the processing unit is specifically configured to execute:

[0032] Adding the token sequence, the xpath sequence, and the position information as input of the pre-trained model to perform embedded representation to obtain a first embedded representation;

[0033] The token sequence is used as the input of the attribute perception module to extract the pattern of the token sequence, and the token sequence is embedded represented based on the pattern of the token sequence to obtain a second embedded representation;

[0034] The second embedding representation obtained by the attribute perception module and the first embedding representation obtained by the pre-trained model are combined to obtain a total embedding representation, which is used as the input of the classification layer to output the information extraction result of the target Web page;

[0035] Among them, the classification layer adopts the softmax function.

[0036] The third aspect of the present invention discloses an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method for extracting web page information based on attribute perception described in the first aspect of the present disclosure is implemented.

[0037] The fourth aspect of the present invention discloses a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for extracting web page information based on attribute perception described in the first aspect of the present disclosure is implemented.

[0038] In summary, the present invention processes and encodes attribute information so that the model can perceive the attribute pattern, thereby enhancing the information extraction ability of the model. First, crawl the Web page from which information is to be extracted, collect the original input data, and then set the attributes to be extracted. By extracting the attribute pattern, the feature is encoded to obtain the embedding of the attribute pattern. Then, the embedding of other information on the Web page is fused with the embedding of the attribute model and passed to the classifier. The classifier is used to classify the input token to extract the corresponding attributes. The beneficial technical effect brought by the present invention is: the method can efficiently and accurately extract the attribute information to be extracted from the Web page, improve the extraction efficiency, and promote the development of downstream tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0040] Figure 1 Schematic diagram of the process of web page information extraction based on attribute perception. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0042] The first aspect of the present invention discloses a method for extracting web page information based on attribute perception, the method comprising:

[0043] Step S1, obtaining a target web page from which information is to be extracted, crawling the HTML source code of the target web page using a web crawler, and saving it in a JSON file format;

[0044] Step S2: preprocess the HTML source code in the JSON file format to remove noise in the HTML source code, extract token sequences and xpath sequences based on variable nodes, and record the position information of the sequences;

[0045] Step S3: Using the token sequence, xpath sequence, and location information as inputs to the information extraction model, the information extraction model outputs the information extraction result of the target Web page through feature embedding and attribute classification.

[0046] According to the method of the first aspect of the present invention, in step S1: obtain the target Web page from which information is to be extracted, send an HTTP request message to a specific URL and receive a returned HTTP response message, and based on the pre-set attribute information to be crawled, use a web crawler to crawl the HTML source code corresponding to the attribute information in the target Web page from the HTTP response message, and save it in JSON file format.

[0047] According to the method of the first aspect of the present invention, in step S2, the preprocessing includes:

[0048] Remove noise from HTML source code and perform format normalization, the noise includes: tags that do not contain text content, unicode control characters, non-ASCII characters, spaces;

[0049] Build a DOM tree based on the data after denoising and format normalization, iterate through each node in the DOM tree, save the text on the node and the corresponding xpath path, and use the nodes with changed content corresponding to the same xpath as variable nodes;

[0050] Take the xpath path as the key, the set of all nodes corresponding to the xpath as the value, sort the set in descending order using the size of the set as the sorting key, and take the first n nodes as the variable nodes;

[0051] A token sequence and an xpath sequence are constructed based on the variable node, and the position information of the sequence is recorded.

[0052] According to the method of the first aspect of the present invention, the information extraction model includes a pre-trained model, an attribute perception module and a classifier. In step S3:

[0053] Adding the token sequence, the xpath sequence, and the position information as input of the pre-trained model to perform embedded representation to obtain a first embedded representation;

[0054] The token sequence is used as the input of the attribute perception module to extract the pattern of the token sequence, and the token sequence is embedded represented based on the pattern of the token sequence to obtain a second embedded representation;

[0055] The second embedding representation obtained by the attribute perception module and the first embedding representation obtained by the pre-trained model are combined to obtain a total embedding representation, which is used as the input of the classification layer to output the information extraction result of the target Web page;

[0056] Among them, the classification layer adopts the softmax function.

[0057] First embodiment

[0058] The goals of the attribute-aware web page information extraction method include: (1) obtaining the HTML source code of the web page to be extracted; (2) preprocessing the obtained data to meet the model input; and (3) the deep learning model can extract the corresponding attribute information based on the input.

[0059] (1) Website HTML data crawling

[0060] Given the URL of the website to be crawled, analyze the website structure information, use the crawler library of the appropriate programming language, send an HTTP request to the target website, obtain the HTTP message information returned by the target website, process the message information, and save the HTML source code as the original data for the model input.

[0061] (2) HTML data preprocessing

[0062] First, remove the useless tag data in the HTML data to reduce the interference of noise information. For the token data and xpath data required for model input, that is, the relevant information of the variable node, use the corresponding tools to extract them from the HTML source code as the input of the model.

[0063] (3) Information extraction based on attribute perception

[0064] The input token, xpath and position information are embedded through the pre-trained model to obtain token embedding, xpath embedding and position embedding. The token sequence is not only embedded through the pre-trained model, but also embedded through the attribute perception module. The attribute perception module extracts the pattern information of the token and embeds it. The extracted pattern information embedding is fused with the hidden layer output by the pre-trained model as the embedding representation of the entire input. A classification layer is used to classify the hidden layer, and the attributes finally extracted by the model are obtained through classification.

[0065] Second embodiment

[0066] like Figure 1 As shown in the figure, first crawl the original web page data of the given target website and save it as the original input of the model. Then preprocess the data on the original data, and reduce the noise information of the input model by removing useless label inputs such as scripts in the original data; then extract the variable nodes in the web page, construct the token sequence and xpath sequence of the input model, and record the position information of the sequence. The model extracts information from the preprocessed data. On the basis of the basic pre-trained model, the model adds an attribute-aware embedding module, which enables the model to perceive the pattern information of sex. The embedding of the input information passes through a classification layer, and the model outputs the final result of the web page information extraction.

[0067] (1) Obtaining website HTML data

[0068] Website HTML data acquisition aims to crawl the HTML data of web pages to be extracted and store it locally for subsequent processing. It mainly includes two steps: web page data crawling and data storage.

[0069] (1-1) Web data crawling

[0070] A web crawler is a program or script that automatically crawls the World Wide Web according to certain rules.

[12] Given a web page that needs to be extracted, an HTTP request message is sent to a specific URL, and the web crawler is used to crawl the HTML source code of the target web page.

[0071] (1-2) Data storage

[0072] JSON (JavaScript Object Notation) is a lightweight data exchange format. It is based on a subset of ECMAScript (the js specification developed by the European Computer Manufacturers Association) and uses a text format that is completely independent of programming languages ​​to store and represent data. The simplicity and clear hierarchy make JSON an ideal data exchange language. It is easy for people to read and write, and it is also easy for machines to parse and generate, and it effectively improves network transmission efficiency. The original HTML data of the web page crawled by the web crawler is saved in the format of a JSON file.

[0073] (2) HTML data preprocessing

[0074] HTML data preprocessing aims to adopt certain preprocessing methods based on the acquired HTML raw data to process the HTML raw data into the data format required for information extraction by the third-step model. It mainly includes two steps: data noise removal and node sequence extraction.

[0075] (2-1) Data noise removal

[0076] There is a lot of noise information in the original HTML data. First, we need to deal with a lot of tag noise. In HTML, tags are used to create an element. The name of an HTML element is the name used in angle brackets, such as Used for paragraphs. The name of the closing tag is preceded by a slash character " ". In empty elements, the end tag is neither required nor allowed. During data preprocessing, only some major text tags such as 、 、 etc., remove tags without text content such as <script>、<style>等。然后是对数据的格式进行规范化，在原始数据中，可能存在很多unicode控制字符、非ASCII字符以及多余的空格等去除不影响原始数据的语义信息，但是不处理会成为影响模型的噪声信息。在数据预处理过程中，将这些噪声信息去除掉，提高数据的质量。

[0077] （2-2）提取节点序列

[0078] 在给定的DOM树中迭代遍历每一个节点，节点的类型可能有两种情况中间节点或者是叶子结点。将每一个节点上的文本及对应的xpath路径保存下来。在同一类Web页面中，页面上有很多节点是固定不变的，同时也有很多变化的节点，相同xpath对应的内容变化的节点叫做变量节点。数据预处理时，会构造一个字典，将同一个xpath路径作为键，xpath对应的所有node节点组成的集合作为值，然后以集合的大小作为排序关键字降序排列，取前n个节点作为变量节点。

[0079] （3）基于属性感知的模型抽取

[0080] 基于属性感知的模型抽取旨在从Web页面的预处理数据中将属性抽取出来，得到模型最终的抽取结构，主要包含输入数据嵌入、属性分类两个步骤。

[0081] （3-1）输入数据嵌入

[0082] 输入数据包括三个部分，token序列、xpath序列和位置信息。这三部分的嵌入相加起来，作为一个整体输入给预训练模型进行表示。本方法在预训练模型的基础上增加了一个属性感知模块。

[0083] 属性感知模块的输入是token序列，通过将token序列的模式提取出来再进行嵌入。提取的方法是使用小写字母l代表所有的小写字母，大写字母U代表所有的大写字母，数字0代表所有的数字，其余特殊字符保持不变，如表1所示。

[0084]

[0085] 提取得到的模式再进行一个嵌入表示，得到待抽取的属性相对应的模式特征。

[0086] （2）属性分类

[0087] 通过属性感知模块得到的隐藏层嵌入和通过预训练模型得到的隐藏层嵌入两者合并到一起，得到输入信息总的一个表示。这个表示作为分类层的输入，分类层采用softmax函数。

[0088] 可见，基于属性感知的Web网页信息抽取方法通过对输入序列中的模式信息进行提取表示，使得模型能够感知到待抽取属性的模式信息。该方法以预训练模型为骨架模型，融合待抽取属性的模式信息，提高Web页面信息抽取的准确率，为下游任务提供丰富的结构化知识，促进下游任务的发展。

[0089] 本发明第二方面公开了一种基于属性感知的Web网页信息抽取装置。所述装置包括处理单元，所述处理单元被配置为执行：

[0090] 获取待抽取信息的目标Web网页，利用网络爬虫爬取目标Web网页的HTML源码，并以JSON文件格式进行保存；

[0091] 对JSON文件格式的HTML源码进行预处理，以去除HTML源码中的噪声，并基于变量节点提取出token序列和xpath序列，同时记录序列的位置信息；

[0092] 以token序列、xpath序列、位置信息作为信息抽取模型的输入，信息抽取模型通过特征嵌入和属性分类来输出目标Web网页的信息抽取结果。

[0093] 根据本发明第二方面的装置，所述处理单元具体被配置为执行：

[0094] 获取待抽取信息的目标Web网页，向特定URL发送HTTP请求报文并接收返回的HTTP响应报文，基于预先设置的要爬取的属性信息，使用网络爬虫从HTTP响应报文中爬取目标Web网页中与属性信息对应的HTML源码，并以JSON文件格式进行保存。

[0095] 根据本发明第二方面的装置，所述预处理包括：

[0096] 去除HTML源码中的噪声并进行格式规范化处理，所述噪声包括：不包含文本内容的标签、unicode控制字符、非ASCII字符、空格；

[0097] 基于经去噪和格式规范化处理后的数据构建DOM树，在DOM树中迭代遍历每一个节点，保存节点上的文本及对应的xpath路径，将相同xpath对应的内容变化的节点作为变量节点；

[0098] 以xpath路径为键，xpath对应的所有节点组成的集合为值，以集合的大小作为排序关键字进行降序排列，取前n个节点作为变量节点；

[0099] 基于所述变量节点构造token序列和xpath序列，同时记录序列的位置信息。

[0100] 根据本发明第二方面的装置，所述信息抽取模型包括预训练模型、属性感知模块和分类器，所述处理单元具体被配置为执行：

[0101] 将token序列、xpath序列、位置信息进行相加，作为所述预训练模型的输入，以进行嵌入式表示，得到第一嵌入表示；

[0102] 将token序列作为属性感知模块的输入，以提取token序列的模式，基于token序列的模式对token序列进行嵌入式表示，得到第二嵌入表示；

[0103] 将通过属性感知模块得到的第二嵌入表示和通过预训练模型得到的第一嵌入表示进行合并，得到总嵌入表示，将其作为分类层的输入，以输出目标Web网页的信息抽取结果；

[0104] 其中，分类层采用softmax函数。

[0105] 本发明第三方面公开了一种电子设备。所述电子设备包括存储器和处理器，所述存储器存储有计算机程序，所述处理器执行所述计算机程序时，实现本公开第一方面所述的一种基于属性感知的Web网页信息抽取方法。

[0106] 本发明第四方面公开了一种计算机可读存储介质。所述计算机可读存储介质上存储有计算机程序，所述计算机程序被处理器执行时，实现本公开第一方面所述的一种基于属性感知的Web网页信息抽取方法。

[0107] 综上，本发明通过对属性信息进行处理、编码嵌入，使得模型感知到属性模式，以此增强模型的信息抽取能力。首先对要提取信息的Web页面进行爬取，收集原始的输入数据，然后设置需要抽取的属性，通过对属性的模式进行提取，将该特征进行编码，得到属性模式的嵌入。之后将Web页面其他信息的嵌入与属性模型的嵌入进行融合，传递给分类器。使用分类器对输入的token进行分类，从而提取出相应的属性。本发明带来的有益技术效果为：利用所述方法能够高效准确地对Web页面中待抽取的属性信息进行抽取，提高抽取效率，促进下游任务的发展。

[0108] 请注意，以上实施例的各技术特征可以进行任意的组合，为使描述简洁，未对上述实施例中的各个技术特征所有可能的组合都进行描述，然而，只要这些技术特征的组合不存在矛盾，都应当认为是本说明书记载的范围。以上所述实施例仅表达了本申请的几种实施方式，其描述较为具体和详细，但并不能因此而理解为对发明专利范围的限制。应当指出的是，对于本领域的普通技术人员来说，在不脱离本申请构思的前提下，还可以做出若干变形和改进，这些都属于本申请的保护范围。因此，本申请专利的保护范围应以所附权利要求为准。< / script>

Claims

1. A method for extracting web page information based on attribute perception, characterized in that: The method comprises: Step S1, obtaining a target web page from which information is to be extracted, crawling the HTML source code of the target web page using a web crawler, and saving it in a JSON file format; Step S2: preprocess the HTML source code in the JSON file format to remove noise in the HTML source code, extract token sequences and xpath sequences based on variable nodes, and record the position information of the sequences; Wherein, in step S2, the preprocessing includes: Remove noise from HTML source code and perform format normalization, the noise includes: tags that do not contain text content, unicode control characters, non-ASCII characters, spaces; Build a DOM tree based on the data after denoising and format normalization, iterate through each node in the DOM tree, save the text on the node and the corresponding xpath path, and use the nodes with changed content corresponding to the same xpath as variable nodes; Take the xpath path as the key, the set of all nodes corresponding to the xpath as the value, sort the set in descending order using the size of the set as the sorting key, and take the first n nodes as the variable nodes; Constructing a token sequence and an xpath sequence based on the variable node, and recording the position information of the sequence; Step S3: using the token sequence, xpath sequence, and location information as inputs to the information extraction model, the information extraction model outputs the information extraction result of the target Web page through feature embedding and attribute classification; The information extraction model includes a pre-trained model, an attribute perception module and a classifier. In step S3: Adding the token sequence, the xpath sequence, and the position information as input of the pre-trained model to perform embedded representation to obtain a first embedded representation; The token sequence is used as the input of the attribute perception module to extract the pattern of the token sequence, and the token sequence is embedded represented based on the pattern of the token sequence to obtain a second embedded representation; The second embedding representation obtained by the attribute perception module and the first embedding representation obtained by the pre-trained model are combined to obtain a total embedding representation, which is used as the input of the classification layer to output the information extraction result of the target Web page; Among them, the classification layer adopts the softmax function.

2. The method for extracting web page information based on attribute perception according to claim 1 is characterized in that: In step S1: obtain the target web page from which information is to be extracted, send an HTTP request message to a specific URL and receive a returned HTTP response message, and based on the pre-set attribute information to be crawled, use a web crawler to crawl the HTML source code corresponding to the attribute information in the target web page from the HTTP response message, and save it in JSON file format.

3. A web page information extraction device based on attribute perception, characterized in that: The apparatus comprises a processing unit configured to perform: Obtain the target web page to be extracted, crawl the HTML source code of the target web page using a web crawler, and save it in JSON file format; Preprocess the HTML source code in JSON file format to remove noise in the HTML source code, extract token sequences and xpath sequences based on variable nodes, and record the position information of the sequences; Wherein, the preprocessing includes: Remove noise from HTML source code and perform format normalization, the noise includes: tags that do not contain text content, unicode control characters, non-ASCII characters, spaces; Build a DOM tree based on the data after denoising and format normalization, iterate through each node in the DOM tree, save the text on the node and the corresponding xpath path, and use the nodes with changed content corresponding to the same xpath as variable nodes; Take the xpath path as the key, the set of all nodes corresponding to the xpath as the value, sort the set in descending order using the size of the set as the sorting key, and take the first n nodes as the variable nodes; Constructing a token sequence and an xpath sequence based on the variable node, and recording the position information of the sequence; The information extraction model takes token sequence, xpath sequence and location information as input, and outputs the information extraction result of the target Web page through feature embedding and attribute classification. The information extraction model includes a pre-trained model, an attribute perception module and a classifier, and the processing unit is specifically configured to execute: Adding the token sequence, the xpath sequence, and the position information as input of the pre-trained model to perform embedded representation to obtain a first embedded representation; The token sequence is used as the input of the attribute perception module to extract the pattern of the token sequence, and the token sequence is embedded represented based on the pattern of the token sequence to obtain a second embedded representation; The second embedding representation obtained by the attribute perception module and the first embedding representation obtained by the pre-trained model are combined to obtain a total embedding representation, which is used as the input of the classification layer to output the information extraction result of the target Web page; Among them, the classification layer adopts the softmax function.

4. The device for extracting web page information based on attribute perception according to claim 3, characterized in that: The processing unit is specifically configured to perform: Get the target web page from which the information is to be extracted, send an HTTP request message to a specific URL and receive the returned HTTP response message. Based on the pre-set attribute information to be crawled, use a web crawler to crawl the HTML source code corresponding to the attribute information in the target web page from the HTTP response message and save it in JSON file format.

5. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, the method for extracting web page information based on attribute perception according to any one of claims 1 to 2 is implemented.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for extracting web page information based on attribute perception described in any one of claims 1 to 2 is implemented.

Images