A webpage electronic archive processing method based on artificial intelligence algorithm

Abstract

The present invention discloses a webpage electronic archive processing method based on an artificial intelligence algorithm, which relates to the field of information collection technology, including: obtaining a URL set U₀ of a target website, the URL set U₀ containing multiple seed URLs, extracting metadata corresponding to each seed URL through a page parsing engine to generate a metadata set; constructing an initial feature vector based on the metadata set; obtaining a metadata set of a new target website, generating a strategy probability for each hyperlink in the new target website using a trained hybrid traversal model, and determining a traversal strategy based on the strategy probability; obtaining webpage tags and aggregation strengths based on path information recorded during the execution of the traversal strategy, and performing archiving priority classification based on the webpage tags and aggregation strengths. The present invention realizes the fine recognition and dynamic adaptation of the complexity of webpage structures and the semantic value of hyperlinks, and can flexibly adjust the traversal path in webpage structures with different levels and different density distributions.

Description

Technical Field

[0001] The present invention relates to the technical field of information collection, and in particular to a webpage electronic archive processing method based on an artificial intelligence algorithm. Background Art

[0002] With the continuous deepening of informatization construction, the number of electronic archival resources (such as announcement documents, statistical yearbooks, policies and regulations, etc.) published on the web by government websites, enterprise and institution portals, and industry platforms is increasing. Web electronic archives have become an important source of information and data assets. In order to realize the automated acquisition and archiving of these archival resources scattered in different sites and pages with different structures, it is necessary to build an efficient web page collection and processing mechanism to ensure the comprehensiveness, timeliness and structured quality of information collection.

[0003] Traditional web page collection methods are mostly based on breadth-first (BFS) or depth-first (DFS) strategies to achieve traversal control. The former is suitable for horizontally crawling page resources on the same level, while the latter is suitable for mining deep resources under nested directories. In scenarios where the specific website structure is simple or the link distribution is highly regular, this type of traversal strategy has a relatively high collection efficiency. However, faced with the actual complex target website structure - such as multiple entrances, multiple levels, interlaced nesting of heterogeneous modules, and different sub-sites deployed on different domain names or paths, a single strategy is often limited in traversal scope, falls into structural islands, or fails to identify high-value links, resulting in incomplete collection results or excessive redundant resources, which seriously restricts the archiving quality and collection efficiency of web electronic archives.

[0004] Some current research has introduced crawler scheduling models based on link analysis or page scoring, attempting to adjust traversal paths through link priority or page scores to improve resource discovery capabilities. However, such methods generally lack the ability to jointly model the complexity of web page DOM structure, document type characteristics, and page semantic relevance, making it difficult to achieve a dynamic balance between structural adaptability and collection value orientation. At the same time, the lack of a policy feedback optimization mechanism leads to the inability to self-adjust policy preferences when facing different types of target websites, resulting in insufficient generalization and robustness of the model.

[0005] In addition, the current web page collection and archiving process generally lacks the perception and analysis of "aggregation structure" and "resource value", and is unable to effectively distinguish between directory aggregation pages and actual archival resource pages, nor can it prioritize archiving or scheduling resources based on the importance of the content. This leads to redundant storage space occupation, delayed processing of high-value resources, or untimely scheduling responses, which restricts the intelligence level and operational efficiency of the web electronic archive management system. Summary of the Invention

[0006] In order to solve the above technical problems, the present invention provides a webpage electronic archive processing method based on artificial intelligence algorithm, comprising:

[0007] S1, target website parsing, obtaining the target website's URL set U0, which contains multiple seed URLs, and extracting metadata corresponding to each seed URL through the page parsing engine to generate a metadata set;

[0008] S2, initial feature vector acquisition, specifically, constructing the initial feature vector based on the metadata set;

[0009] S3, hybrid ergodic model training. Specifically, a neural network based on structural entropy perception adjustment strategy and policy deviation feedback is used as a hybrid ergodic model;

[0010] S4, traversal strategy determination, specifically, obtain the metadata set of the new target website, use the trained hybrid traversal model to generate strategy probabilities for each hyperlink in the new target website, and determine the traversal strategy based on the strategy probabilities.

[0011] S5, archiving priority classification, specifically, according to the path information recorded during the execution of the traversal strategy, the web page label and aggregation strength are obtained, and the archiving priority classification is performed based on the web page label and aggregation strength.

[0012] Furthermore, the metadata includes the page main title text, the page body content and the page hyperlink attribute set.

[0013] Furthermore, the training process of the neural network based on the structural entropy perception adjustment strategy and strategy deviation feedback is as follows:

[0014] S31, build a three-layer feedforward neural network, which includes an input layer, a hidden layer, and an output layer, initialize the weight matrix and bias vector between each layer, and set the activation function and initial learning rate;

[0015] S32, scaling the initial feature vector by a structural entropy-aware adjustment strategy to obtain an adjustment input vector;

[0016] S33, input the regulated input vector into the feedforward neural network for forward propagation, and output the hidden layer output result and strategy probability ;

[0017] S34, calculating and obtaining a loss function based on the preset strategy label and strategy probability;

[0018] S35, calculating and obtaining the strategy deviation according to the number of successful filings corresponding to each round of strategy probability and the preset expected number of filings;

[0019] S36, updates the neural network parameters based on the loss function and the policy deviation. The neural network parameters include the weight matrix, the bias vector, and the learning rate;

[0020] S37, when the loss function decreases less than , the corresponding neural network is deployed as a hybrid ergodic model.

[0021] Furthermore, the adjustment input vector is obtained, which is expressed as:

[0022] ;

[0023] Where, To adjust the input vector, is the structural entropy adjustment factor, is the initial eigenvector, is the structural entropy.

[0024] Furthermore, the strategy tag includes BFS strategy and DFS strategy. When the strategy tag is BFS strategy, the output value is 1, and when the strategy tag is DFS strategy, the output value is 0.

[0025] Calculate the loss function, expressed as:

[0026] ;

[0027] Where, is the loss function, is the total number of samples in the current training batch, For the The output value of the strategy label of the sample, For the The policy probability of the model for the sample, To suppress the overfitting of the model, is the regularization coefficient, is the weight matrix from the input layer to the hidden layer, is the weight matrix from the hidden layer to the output layer, is the weight matrix The sum of the squares of all elements in , is the weight matrix The sum of the squares of all elements in .

[0028] Furthermore, the logic for calculating the acquisition strategy deviation is:

[0029] Record No. Policy probability for round model training The number of successful archiving of electronic files of the hyperlinks reached, An integer greater than 0, which serves as the actual strategy return , get the strategy probability of this round The corresponding expected number of archives as the expected strategy return , the actual strategy returns and expected strategy returns Perform difference calculation to obtain the policy deviation, which is expressed as: .

[0030] Furthermore, the neural network parameters are updated based on the loss function and policy deviation, which can be expressed as:

[0031] ;

[0032] ;

[0033] Where, is the current round learning rate, is the initial learning rate, is the error adjustment coefficient, is the local adjustment rate used to amplify the weight correction of high misjudgment areas, is the bias vector from the hidden layer to the output layer;

[0034] The gradient is calculated by the back-propagation algorithm, and the weight matrix is updated using the Adam optimizer. , ; and the bias vector , ; expressed as:

[0035] ;

[0036] ;

[0037] ;

[0038] ;

[0039] Where, is the bias vector from the input layer to the hidden layer, , is the loss function with respect to the weight matrix and the bias vector The gradient, , is the loss function with respect to the weight matrix and the bias vector gradient.

[0040] Furthermore, the traversal strategy is: output strategy probability When ≥0.5, the BFS traversal strategy is preferred and the strategy probability is output. When <0.5, the DFS traversal strategy is preferred.

[0041] Furthermore, the path information includes a set of traversed web pages and a set of jump edges of the traversed web pages. The logic for obtaining web page labels and aggregation strength is:

[0042] S51, constructing a belonging structure graph G=(X, E), where X is the set of traversed web pages, ,E is the jump edge set of the traversed web pages, ; and ζ are both integers greater than 0;

[0043] S52, based on each traversal web page in the structure diagram , extracting structural attributes, the structural attributes including page in-degree, page out-degree, structural hierarchy, and proportion of directed archives;

[0044] S53, based on the structural attributes, performing web page tag determination on the traversed web pages, wherein the web page tags include directory aggregation pages and electronic archive resources;

[0045] S54: Calculate the corresponding aggregation strength based on the traversed web page determined to be the directory aggregation page, and prioritize the subordinate document pages of the traversed web page based on the aggregation strength.

[0046] Furthermore, define each directory aggregation page as , ∈X, calculate the aggregation strength of the traversed web pages, expressed as:

[0047] ;

[0048] is the aggregation strength of the directory aggregation page, It is the set of all web pages that the target aggregation page directly points to through hyperlinks in the structure diagram. for The web pages within The proportion of directional files, Directory aggregation page The out-degree of the page.

[0049] Compared with the prior art, the present invention has the following beneficial effects:

[0050] By introducing a hybrid traversal model training mechanism, the present invention achieves efficient understanding and intelligent scheduling of complex, multi-entry, and multi-level web pages, improves the crawling coverage and content relevance matching of web page electronic archives, and has the ability to adaptively identify different types of web page structures and link densities, avoiding problems such as crawling redundancy and omission of key resources that occur in traditional traversal methods in complex websites, thereby enhancing the accuracy and robustness of electronic archive collection.

[0051] In addition, through the construction of structural diagrams and the aggregation strength evaluation mechanism, combined with the archiving role judgment strategy, automatic judgment and priority division of the archiving target value are achieved, which makes it possible to accurately identify high-value archival resources and directory aggregation pages, avoid the waste of storage resources on low-value content, and improve the scheduling response efficiency of high-value archival content. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction to the drawings required for use in the embodiments will be given below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0053] Figure 1 A diagram showing the steps of a webpage electronic archive processing method based on an artificial intelligence algorithm provided by an embodiment of the present invention;

[0054] Figure 2 A diagram of the training steps of a neural network for a webpage electronic archive processing method based on an artificial intelligence algorithm provided by an embodiment of the present invention;

[0055] Figure 3 A flowchart of a neural network training method for processing webpage electronic archives based on an artificial intelligence algorithm provided by an embodiment of the present invention;

[0056] Figure 4 A diagram of the archiving priority classification steps of a web page electronic archive processing method based on an artificial intelligence algorithm provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0057] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0058] The following describes the embodiments of the present disclosure through specific examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all of the embodiments. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present disclosure.

[0059] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present disclosure. The illustrations only show components related to the present disclosure and are not drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component can be changed at will, and the component layout may also be more complicated.

[0060] See Figure 1 , a webpage electronic archive processing method based on artificial intelligence algorithm, the method comprising:

[0061] S1, target website analysis

[0062] Obtain the URL set U0 of the target website, which contains multiple seed URLs. Use the page parsing engine to extract metadata corresponding to each seed URL to generate a metadata set.

[0063] It should be noted that: a target website, such as a government platform, corporate archives website, or institutional announcement page, has different main entrances corresponding to different business modules (such as announcements, policies, and statistical yearbooks). Furthermore, multiple "entry paths" may appear due to page divisions, year directories, and nested columns within the target website. Furthermore, each sub-site (such as a city sub-portal) is deployed on a different second-level domain name or URL path.

[0064] Therefore, in order to cover the starting entrance of the target website structure, it is necessary to extract multiple seed URLs to form the URL set U0 to avoid the crawler falling into a local area and incomplete collection range;

[0065] The seed URL set U0 may be generated by manually specifying an entry page or a sitemap file provided by a target website.

[0066] Specifically, the metadata includes the page main title text, the page body content and the page hyperlink attribute set;

[0067] It should be noted that: the main title text of the page is used to express the core semantic information of the page; the main body content of the page is used to express the main content paragraphs of the page, and the content with the largest DOM density and the most stable text block length is preferentially extracted from the main body content of the page; the target page hyperlink attribute set is the attribute set of all hyperlinks in the target page, which is obtained through the HTML structure. The href attribute of the tag is extracted, and the hyperlink points to other page resources or attachment files within the website.

[0068] S2, constructs the initial feature vector based on the metadata set;

[0069] Specifically, construct the initial feature vector, which is expressed as:

[0070] ={ , , , , }

[0071] Where, The first Hyperlinks, is an integer greater than 0; is the link depth, is the link density, Encode the document type, is the metadata semantic matching degree, is the structural entropy;

[0072] Specifically, link depth Obtained by calculating the number of URL path levels in the page hyperlink attribute set, expressed as;

[0073]

[0074] Where, Indicates that the A hyperlink is split into multiple path segments according to the slash / . For example, / gov / policy / file.pdf will be split into 4 segments, represented as ['', 'gov', 'policy', 'file.pdf'];

[0075] () represents the number of path segments after statistical splitting, c is the compensation constant, preferably, the compensation constant c is set to 2;

[0076] It should be noted that the compensation constant c is derived from the empirical statistics of the actual website path depth distribution, and its value range is 1 to 3;

[0077] Specifically, link density By calculating the DOM structure of the page hyperlink attribute set The quotient of the number of tag nodes and the total number of all nodes in the DOM tree is obtained, expressed as:

[0078]

[0079] Where, for The number of nodes of the label, is the total number of nodes in the DOM tree;

[0080] It should be noted that: The tag statistics range is the link nodes that are directly parsed and visible through the standard HTML structure in the current page, excluding virtual links generated by dynamic JavaScript loading. All standard DOM element nodes are counted (including text, div, table, etc.);

[0081] Specifically, the document type encoding Encoding classification is performed based on the suffix type of files such as .pdf, .docx, and .xls in the page hyperlink property set, expressed as:

[0082]

[0083] For example, when the suffix type in the u-th hyperlink is When , the output is 0.9;

[0084] Preferably, in order to adapt to the diverse file types, a complete document type coding dictionary is constructed, as follows:

[0085]

[0086] Specifically, metadata semantic matching Obtained through cosine similarity calculation, expressed as:

[0087]

[0088] in, is the page semantic vector, the page semantic vector It is generated by splicing the main title text of the page and the main content of the page. is the preset keyword vector, keyword vector Constructed through a collection of archival keywords;

[0089] like =1, indicating that the electronic archive content of the webpage captured on the page perfectly matches the preset keyword content; if =0, indicating that the electronic archive content of the webpage captured on the page has nothing to do with the preset keyword content; preferably, set When it is greater than 0.6, the relevance between the electronic archive content of the web page captured on the page and the preset keyword content is strong;

[0090] It should be noted that: vector and vector The vector construction method adopts the existing BERT model, which uses the pre-trained language model bert-base-chinese;

[0091] Specifically, structural entropy By counting the probability of various HTML tags (including but not limited to div, table, and a tags) in the DOM node distribution in the page hyperlink attribute set, the structure entropy formula is used to calculate it, which is expressed as:

[0092]

[0093]

[0094] Where, For the page The proportion of HTML tags of the class, is the total number of tag types in DOM, For the The probability that the class's HTML tag appears in the DOM structure, For the The number of nodes where the class label appears on the page;

[0095] It should be noted that the label types do not include <script> 与 <style> 标签，以避免非可见内容对结构分布的干扰。

[0096] S3，混合遍历模型训练

[0097] 利用步骤S2中构建的初始特征向量，输入至混合遍历模型中进行优化训练，本发明基于策略偏差反馈与结构熵感知调节策略的神经网络作为混合遍历模型，用于在广度优先搜索（BFS）与深度优先搜索（DFS）之间进行动态权衡，在面对结构复杂或链接密度极化的网页结构时，能够更加精准地分配抓取路径，以提高有价值电子档案资源的采集效率，确保混合遍历模型的适应性与鲁棒性；

[0098] 具体地，基于结构熵感知调节策略和策略偏差反馈的神经网络的训练流程为：

[0099] S31，构建三层前馈神经网络，三层前馈神经网络包括输入层，隐藏层和输出层，初始化每层之间的权重矩阵与偏置向量，并设置激活函数与初始学习率；

[0100] 具体的，初始采用Xavier法进行初始化，表示为：

[0101] ；

[0102] ；

[0103] 式中，为正态分布，为输入层的神经元数量，为隐藏层的神经元数量，为输入层到隐藏层的权重矩阵，为隐藏层到输出层的权重矩阵，为输入层到隐藏层的偏置向量，为隐藏层到输出层的偏置向量，和的初始值设定为零；

[0104] 优选的，三层前馈神经网络中，输入层维度为5，隐藏层神经元数优选为64，输出层神经元为1；隐藏层采用ReLU激活函数；输出层采用Sigmoid激活函数。

[0105] S32，通过结构熵感知调节策略对初始特征向量进行缩放，以获取调节输入向量；

[0106] 具体的，获取调节输入向量，表示为：

[0107]

[0108] 式中，为调节输入向量，为结构熵调节因子，为初始特征向量，为结构熵，优选的，∈[0.2, 1.0]；

[0109] 需要说明的是：当结构熵趋近于0时，对初始特征向量进行调节的幅度近似于1（无调节），结构熵越大，调节越显著；

[0110] S33，将调节输入向量输入前馈神经网络进行前向传播，输出隐藏层输出结果和策略概率；

[0111] 具体的，输出隐藏层输出结果和策略概率，表示为：

[0112] ；

[0113] ；

[0114] ；

[0115] ；

[0116] 式中，为输入层到隐藏层的线性变换结果，为隐藏层到输出层的线性变换结果，为隐藏层激活函数，为输出层激活函数，为输入层到隐藏层的权重矩阵，为隐藏层到输出层的权重矩阵，为输入层到隐藏层的偏置向量，为隐藏层到输出层的偏置向量，优选的，策略概率∈（0，1），策略门限设为0.5；

[0117] 需要说明的是：前向传播顺序为输入层→隐藏层→输出层；BFS为广度优先搜索，DFS为深度优先搜索, 广度优先搜索（BFS）用于优先采集与当前页面同层的链接资源，适合获取页面横向分布的档案内容；而深度优先搜索（DFS）则用于优先深入访问当前页面中的下级链接，适合发掘目录型页面中的嵌套档案资源；

[0118] S34，基于预设策略标签和策略概率，计算获取损失函数；

[0119] 具体的，策略标签包括BFS策略和DFS策略，当策略标签为BFS策略时，输出值为1，当策略标签为DFS策略时，输出值为0；

[0120] 计算获取损失函数，表示为：

[0121] ；

[0122] 其中，为损失函数，为当前训练批次中的样本总数，为第个样本的策略标签的输出值，为第条样本的模型的策略概率，为抑制模型过拟合的正则化项，为正则化系数，为输入层到隐藏层的权重矩阵，为隐藏层到输出层的权重矩阵，为权重矩阵内所有元素的平方和，为权重矩阵内所有元素的平方和，优选的，设为10⁻4；

[0123] 需要说明的是：设为10⁻4用于防止模型过拟合；

[0124] S35，根据每轮策略概率对应的归档成功数和预设的期望归档数，计算获取策略偏差；

[0125] 具体的，计算获取策略偏差的逻辑为：

[0126] 记录第轮次训练的策略概率所触达超链接的电子档案的归档成功数，以作为实际策略收益，获取该轮次策略概率对应的期望归档数，以作为期望策略收益，将实际策略收益与期望策略收益进行差值计算，得到策略偏差,表示为：

[0127] ；

[0128] 需要说明的是：统计归档成功数时去除文件名重复项，期望策略收益通过策略概率的历史数据的平均归档数量进行设定。

[0129] S36，基于损失函数和策略偏差更新神经网络参数，神经网络参数包括权重矩阵，偏执向量和学习率；

[0130] 具体的，基于损失函数和策略偏差更新神经网络参数，表示为：

[0131] ；

[0132] ；

[0133] 式中，为当前轮次学习率，为初始学习率，为误差调整系数，为隐藏层到输出层的偏置向量，为用于对高误判区域权重放大修正的局部调整率，优选的，∈[0.001,0.01]，∈[0.0032,0.013]；

[0134] 通过反向传播算法计算梯度，采用Adam优化器更新权重矩阵，；以及偏置向量，；表示为：

[0135] ；

[0136] ；

[0137] ；

[0138] ；

[0139] 式中，为输入层到隐藏层的偏置向量，为隐藏层到输出层的偏置向量，，为损失函数关于权重矩阵和偏置向量的梯度，，为损失函数关于权重矩阵和偏置向量的梯度。

[0140] S37，当损失函数下降幅度连续三轮小于时，部署对应的神经网络作为混合遍历模型。

[0141] S4，网页遍历策略调整

[0142] 获取新的目标网站的元数据集合，利用已训练完成的混合遍历模型新的目标网站内各个超链接对应生成策略概率，基于策略概率确定遍历策略。

[0143] 具体的，遍历策略为：输出策略概率≥0.5时，优先采用BFS的遍历策略，输出策略概率＜0.5时，优先采用DFS的遍历策略；

[0144] S5，归档优先级分类

[0145] 根据遍历策略执行过程中记录的路径信息，获取网页标签和聚合强度，并基于网页标签和聚合强度进行归档优先级分类；

[0146] 具体的，获取网页标签和聚合强度的逻辑为：

[0147] 所述路径信息包括已遍历网页集合和已遍历网页的跳转边集合；

[0148] 需要说明的是：已遍历网页集合由多个已遍历的网页组成；已遍历网页的跳转边集合为已遍历网页的超链接指向的其他网页的集合；

[0149] S51，构建归属结构图G=（X，E），其中，X为已遍历网页集合，,E为已遍历网页的跳转边集合,；和ζ皆为大于0的整数；

[0150] S52，基于结构图中每个遍历网页，提取结构属性，所述结构属性包括页面入度，页面出度，结构层级，指向型档案比重；

[0151] 需要说明的是：页面入度为网页被其他网页超链接指向的次数，页面出度为超链接指向其他网页的数量，结构层级，基于URL中斜杠" / ”计数，指向型档案比重为网页指向PDF、Word 等电子档案格式资源的比例，通过网页中归档文件链接数和全部有效链接数计算获取；

[0152] S53，基于结构属性，对已遍历网页执行网页标签判定，所述网页标签包括目录聚合页和电子档案资源；

[0153] 具体的，网页标签判定逻辑为：

[0154] 当页面出度大于页面入度，且指向型档案比重小于0.3时，判定网页为目录聚合页；当页面出度小于等于2，且指向型档案比重大于0.5时，判定网页为电子档案资源；其余网页则判定不具备聚合或归档意义；

[0155] S54，基于判定为目录聚合页的已遍历网页，计算对应的聚合强度，并基于聚合强度对已遍历网页的下属文档页进行优先级划分；

[0156] 需要说明的是：下属文档页为判定为目录聚合页的已遍历网页内超链接所指向的网页；

[0157] 具体的，定义每个目录聚合页为，∈X，计算获取；

[0158] 为目录聚合页的聚合强度，为目标聚合页在结构图中通过超链接所直接指向的全部网页集合，为内的网页，为指向型档案比重，为目录聚合页的页面出度；

[0159] 若≥0.6，则将目标聚合页及其下属文档页划分为"高价值批注”，自动优先归档和调度；

[0160] 若＜0.25，则将目标聚合页及其下属文档页划分为"低价值批注”，自动延后归档；

[0161] 若0.25≤＜0.6，则通过人工策略进行处理；

[0162] 需要说明的是：0.25≤＜0.6无法自动进行归档优先级判定，需要人工进行参与确定归档方案。

[0163] 在本实施例中， 首先对网页标签被判定为电子档案资源标签的网页中所包含的电子档案执行优先归档与调度操作；对于聚合强度满足≥0.6的目录聚合页，其所链接的下属文档页所包含的电子档案视为次级归档与调度对象，予以高价值电子档案内容补充；当目录聚合页的聚合强度满足小于＜0.25时，其所链接的下属文档页中的电子档案被标记为延后归档与调度；若聚合强度落于区间0.25≤＜0.6，则对应目录聚合结构无法直接归入自动归档策略范围，需由人工策略判定后再行归档处理。

[0164] 以上所述的实施例仅是对本发明的优选实施方式进行描述，并非对本发明的范围进行限定，在不脱离本发明设计精神的前提下，本领域普通技术人员对本发明的技术方案做出的各种变形和改进，均应落入本发明权利要求书确定的保护范围内。< / script>

Claims

1. A webpage electronic archive processing method based on artificial intelligence algorithm, characterized in that: The method comprises: S1, target website parsing: obtain the URL set U0 of the target website, which contains multiple seed URLs, and extract the metadata corresponding to each seed URL through the page parsing engine to generate a metadata set; S2, initial feature vector acquisition: constructing the initial feature vector based on the metadata set; S3, hybrid ergodic model training: a neural network based on structural entropy-aware regulation strategy and policy deviation feedback is used as a hybrid ergodic model; The training process of the neural network based on the structural entropy perception adjustment strategy and strategy deviation feedback is as follows: S31, build a three-layer feedforward neural network, which includes an input layer, a hidden layer, and an output layer, initialize the weight matrix and bias vector between each layer, and set the activation function and initial learning rate; S32, scaling the initial feature vector by a structural entropy-aware adjustment strategy to obtain an adjustment input vector; Get the adjustment input vector, expressed as: ； Where, To adjust the input vector, is the structural entropy adjustment factor, is the initial eigenvector, is the structural entropy; S33, input the regulated input vector into the feedforward neural network for forward propagation, and output the hidden layer output result and strategy probability ; S34, calculating and obtaining a loss function based on the preset strategy label and strategy probability; S35, calculating and obtaining the strategy deviation according to the number of successful filings corresponding to each round of strategy probability and the preset expected number of filings; S36, updates the neural network parameters based on the loss function and the policy deviation. The neural network parameters include the weight matrix, the bias vector, and the learning rate; S37, when the loss function decreases less than When , the corresponding neural network is deployed as a hybrid ergodic model; S4, traversal strategy determination: Obtain a metadata set of a new target website, use the trained hybrid traversal model to generate strategy probabilities for each hyperlink in the new target website, and determine the traversal strategy based on the strategy probabilities; S5, archiving priority classification: according to the path information recorded during the execution of the traversal strategy, the web page label and aggregation strength are obtained, and the archiving priority classification is performed based on the web page label and aggregation strength.

2. A webpage electronic archive processing method based on artificial intelligence algorithm according to claim 1, characterized in that: The metadata includes the page main title text, the page body content and the page hyperlink attribute set.

3. A webpage electronic archive processing method based on artificial intelligence algorithm according to claim 2, characterized in that: The strategy tags include BFS strategy and DFS strategy. When the strategy tag is BFS strategy, the output value is 1, and when the strategy tag is DFS strategy, the output value is 0. Calculate the loss function, expressed as: ； Where, is the loss function, is the total number of samples in the current training batch, For the The output value of the strategy label of the sample, For the The policy probability of the model for the sample, To suppress the overfitting of the model, is the regularization coefficient, is the weight matrix from the input layer to the hidden layer, is the weight matrix from the hidden layer to the output layer, is the weight matrix The sum of the squares of all elements in , is the weight matrix The sum of the squares of all elements in .

4. A webpage electronic archive processing method based on artificial intelligence algorithm according to claim 3, characterized in that: The logic for calculating the acquisition strategy deviation is: Record No. Policy probability for round model training The number of successful archiving of electronic files of the hyperlinks reached, An integer greater than 0, which serves as the actual strategy return , get the strategy probability of this round The corresponding expected number of archives as the expected strategy return , the actual strategy returns and expected strategy returns Perform difference calculation to obtain the policy deviation, which is expressed as: .

5. The method for processing electronic webpage archives based on artificial intelligence algorithm according to claim 4, characterized in that: Update the neural network parameters based on the loss function and policy deviation, expressed as: ； ； Where, is the current round learning rate, is the initial learning rate, is the error adjustment coefficient, is the local adjustment rate used to amplify the weight correction of high misjudgment areas, is the bias vector from the hidden layer to the output layer; The gradient is calculated by the back-propagation algorithm, and the weight matrix is updated using the Adam optimizer. , ; as well as Bias vector , ; expressed as: ； ； ； ； Where, is the bias vector from the input layer to the hidden layer, , is the loss function with respect to the weight matrix and the bias vector The gradient, , is the loss function with respect to the weight matrix and the bias vector gradient.

6. A webpage electronic archive processing method based on artificial intelligence algorithm according to claim 5, characterized in that: The traversal strategy is: output strategy probability When ≥0.5, the BFS traversal strategy is preferred and the strategy probability is output. When <0.5, the DFS traversal strategy is preferred.

7. A webpage electronic archive processing method based on artificial intelligence algorithm according to claim 6, characterized in that: The path information includes a set of traversed web pages and a set of jump edges of the traversed web pages. The logic for obtaining web page labels and aggregation strength is as follows: S51, constructing a belonging structure graph G=(X, E), where X is the set of traversed web pages, , E is the jump edge set of the traversed web pages, ; and ζ are both integers greater than 0; S52, based on each traversal web page in the structure diagram , extracting structural attributes, the structural attributes including page in-degree, page out-degree, structural hierarchy, and proportion of directed archives; S53, based on the structural attributes, performing web page tag determination on the traversed web pages, wherein the web page tags include directory aggregation pages and electronic archive resources; S54: Calculate the corresponding aggregation strength based on the traversed web page determined to be the directory aggregation page, and prioritize the subordinate document pages of the traversed web page based on the aggregation strength.

8. The method for processing electronic webpage archives based on artificial intelligence algorithm according to claim 7, characterized in that: Define each directory aggregation page as , ∈X, calculate the aggregation strength of the traversed web pages, expressed as: ； is the aggregation strength of the directory aggregation page, It is the set of all web pages that the target aggregation page directly points to through hyperlinks in the structure diagram. for The web pages within The proportion of directional files, Directory aggregation page The out-degree of the page.

Images