A Method and Apparatus for Website Key Resource Analysis Based on Page Rendering Contribution

Abstract

This invention discloses a method and apparatus for analyzing key website resources based on page rendering contribution. The method includes: acquiring website resources during user interaction; rendering a page based on the website resources, acquiring website resource behaviors and resource reference chains during the page rendering process; and analyzing key website resources based on the website resources, website resource behaviors, and resource reference chains to obtain resource analysis results. This invention can automatically identify key resources affecting the normal display and functionality of a website from a set of website resources, reducing the manual cost of resource analysis and helping website administrators maintain website security.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method and apparatus for analyzing key website resources based on page rendering contribution. Background Technology

[0002] The web ecosystem is developing rapidly, and the cost of building a website has been greatly reduced. As a result, websites often introduce many external resources to provide their business functions. This makes the normal operation of a website highly dependent on the use of various resources. Once a resource becomes unavailable, it will affect the normal display and functionality of the website.

[0003] Current methods for evaluating critical resources often rely on manual judgment at the business level. These solutions typically maintain a massive fingerprint database to identify which service a specific resource belongs to. For example, given the resource https: / / www.example.com / path_to / jquery.js, the fingerprint database would identify it as belonging to the specific service jQuery based on the resource name jquery.js, and then determine whether this resource is critical to the website based on prior human knowledge. Such methods require significant manpower, and given the ever-changing web services and the constant emergence of new resources, relying on manual methods to analyze a large number of critical website resources is extremely difficult. Summary of the Invention

[0004] To address the aforementioned issues, this invention discloses a method and apparatus for analyzing key website resources based on page rendering contribution. This method can automatically identify key resources affecting the normal display and functionality of a website from its resource set, reducing the human resource costs of resource analysis and helping website administrators maintain website security.

[0005] The technical solution of the present invention includes:

[0006] A method for analyzing key website resources based on page rendering contribution, comprising the following steps:

[0007] Obtain website resources during website interactions with users;

[0008] Render the page based on the website resources, and obtain the website resource behavior and website resource reference chain during the page rendering process;

[0009] Based on the website resources, website resource behaviors, and website resource reference chains, the key resources of the website are analyzed to obtain resource analysis results.

[0010] Furthermore, the website resources include: implicit resources; the method for obtaining the implicit resources includes:

[0011] Parse the HTML document of the website's page, obtain the relative and absolute path expressions of the web page tags, and obtain the first tag when the text in the path expression contains a first specific word;

[0012] Obtain all tags with id, class, and user-defined attributes in the HTML document of the page, and when the tag attribute contains a second specific word, obtain the second tag;

[0013] Clicking the first tab and the second tab will retrieve the implicit resource.

[0014] Furthermore, the step of clicking the first tag and the second tag to obtain the implicit resource includes:

[0015] When a pop-up window appears, wait for the webpage to finish loading to obtain the implicit resource;

[0016] or

[0017] When a new tab appears, wait for the new tab to finish loading, then switch back to the main window and wait for the main window to finish loading to obtain the implicit resource.

[0018] Furthermore, the website resource behaviors include: network request events and DOM tree modification events.

[0019] Furthermore, the step of rendering the page based on the website resources and obtaining the website resource behavior during the page rendering process includes:

[0020] The rendering engine Blink parses HTML documents into a DOM tree;

[0021] Use the CSS interpreter to mark style information for element nodes in the DOM tree;

[0022] If a script is encountered during the page rendering process, a script request is sent to the remote server, and the script is executed through V8, initiating network request events and DOM tree modification events.

[0023] Furthermore, the network request events include: script requests, image requests, iframe requests, XML requests, CSS requests, and video requests.

[0024] Furthermore, the DOM tree modification events include: node creation, node insertion, node deletion, node rendering, attribute addition, attribute deletion, attribute modification, style addition, and style removal.

[0025] Furthermore, based on the website resources, website resource behaviors, and website resource reference chains, the analysis of key website resources yields resource analysis results, including:

[0026] Count the total number of website resource behaviors that occur, and calculate the event basis vector;

[0027] For each website resource, the number of behaviors occurring for each website resource is counted to form a resource feature vector;

[0028] Projecting the resource feature vector onto the event basis vector yields the length of the projection vector;

[0029] Based on the website resource reference chain, the lengths of the projection vectors are superimposed from bottom to top to obtain the resource analysis results.

[0030] A storage medium storing a computer program, wherein the computer program is configured to execute any of the methods described above when run.

[0031] An electronic device is characterized by comprising a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform any of the methods described above.

[0032] Compared with the prior art, the present invention has at least the following advantages:

[0033] 1. This invention has the function of automatically discovering implicit resources in website pages. This function automatically simulates human behavior and clicks on the page. During the clicking process, it captures invisible implicit resources, thus overcoming the defect of existing technology that cannot capture implicit resources.

[0034] 2. This invention integrates the information collection module into the Chromium browser rendering engine, which generates almost no additional overhead for normal page loading.

[0035] 3. This invention can evaluate the importance of website resources independently of human experience, overcoming the shortcomings of existing technologies that cannot automatically evaluate a batch of target website resources. Attached Figure Description

[0036] Figure 1 A flowchart for analyzing key website resources. Detailed Implementation

[0037] The technical solutions in the embodiments of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are merely specific embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0038] The website key resource analysis method of this invention consists of two parts: data collection and data analysis. Since some resources are only observed in the page source code after being triggered by the user, the data collection part first simulates user clicks on login / registration buttons using fuzzy matching to discover implicit resources on the website. Then, this part collects information from the Chromium rendering engines Blink and V8, tracking the page rendering process from initiating a network request to completion. The data analysis part first extracts the resource-to-resource reference relationships during the rendering process, and then extracts key indicators for evaluating resource importance. Resource importance is evaluated by calculating the weight of each key indicator. This part is guided by the principle of resource contribution to webpage rendering, evaluating resource importance by calculating the proportion of resource-rendered webpages to the total rendering process. This method not only considers the rendering ratio of resources but also the transmission of resource importance; that is, the importance of resources in the same resource reference chain is passed down from bottom to top. The total time spent on the data collection and data analysis parts is similar to the website loading time, making it suitable for periodic website monitoring.

[0039] Figure 1 The diagram shows the workflow of this method. This method is based on Selenium for automated website access and combines Chromium to collect rendering process data for resource scoring. The main steps include:

[0040] (I) Automated Implicit Resource Discovery

[0041] The website contains implicit resources that require user interaction; these resources are only observed on the page after a user clicks on them. Therefore, this method first uses fuzzy matching to discover these implicit resources by clicking on important function buttons such as login / registration during the website's loading process. This fuzzy matching method primarily employs the following two rules:

[0042] 1) Text matching: First, the relative and absolute path expressions of the web page tags are parsed from the page's HTML document. If the path expression text contains the words "login" or "register", then Selenium is used to automatically click the position corresponding to that path.

[0043] 2) Attribute matching: First, find all tags with attributes based on the page's HTML document. These attributes include id, class, and all custom attributes. If a tag's attributes contain the words "login" or "register", then use Selenium to automatically click on that tag.

[0044] Clicking the function button will result in one of the following two scenarios, each requiring specific handling:

[0045] 1) If a pop-up window appears: Do nothing and wait for the webpage to finish loading.

[0046] 2) A new tab appears: Wait for the tab to finish loading, switch back to the main window, and wait for the main window to finish loading.

[0047] (II) Collection of Page Rendering Process

[0048] 1) This method is based on the Chromium rendering engine Blink and V8 to collect the page rendering process. Blink first parses the HTML document into a DOM tree, then uses a CSS interpreter to mark style information for the element nodes in the DOM tree. During the rendering process, if any... <script>则会向远程服务端发起脚本请求，并通过V8执行脚本，脚本的执行会发起网络请求事件与DOM树的修改事件。

[0049] (三)资源行为和引用链获取

[0050] 基于页面渲染过程捕获资源的行为。资源行为可分为以下2类：

[0051] 1)DOM修改类事件：该事件包括节点创建、节点插入、节点删除、节点渲染、属性添加、属性删除、属性修改、样式添加、样式移除总计9种具体事件类型；

[0052] 2)网络请求类事件：该事件包括脚本请求、图片请求、iframe请求、XML请求、CSS请求、视频请求总计6种事件类型。

[0053] 资源的引用链基于网络请求与发起者的关系进行构建。

[0054] (四)资源重要性评分机制

[0055] 根据资源行为和资源引用链，本发明设计了包含重要性传递的资源评价方法。

[0056] 首先，本发明统计页面渲染全过程中发生的各类型事件总数，构成事件基向量。

[0057] 其次，对于每个资源，本发明统计该资源触发的各类事件数量，构成资源特征向量然后，本发明将资源特征向量投影于事件基向量，得到投影向量长度。

[0058] 最后，本发明根据资源引用链，将投影向量长度自底向上叠加，形成具备重要性传递的资源重要性评价方法。

[0059] (五)大规模网站关键资源分析系统

[0060] 对于每个网站，都利用该评分机制找出关键资源，因此对于给定范围网站。

[0061] 下面给出两个实施例，以具体说明本发明：

[0062] 实施例1

[0063] 将页面渲染过程采集方法以补丁的形式集成到Chromium的渲染引擎中，通过hooks技术收集渲染过程中的信息并进行本地存储，这种方式对页面访问速度几乎无影响。

[0064] 在网络通常的情况下，对于给定网站列表，应用selenium自动化操作方法集成的Chromium，应用隐式资源模糊匹配方法找出页面中可能存在的隐式资源，同时收集方法在渲染进程中记录各个资源的行为，即可得到每个网站的渲染过程信息。

[0065] 对渲染过程信息进行预处理，基于本方法定义的资源触发事件类型：

[0066] 1)根据DOM修改类事件类型，提取资源与DOM树元素的二元修改关系，对于每一种修改关系，以数量作为记录标准；

[0067] 2)根据网络请求类事件类型，提取资源与资源之间的二元引用关系，对于每一种引用关系，以数量作为记录标准。

[0068] 最后以资源url作为唯一标识符，将资源的父节点、资源触发的事件数量多维向量、资源类型作为资源属性，构成每个网站的资源特征描述的数据结构。

[0069] 对于评分基向量的处理，取当前页面渲染过程中出现的各个事件以及事件数量的总和，为了避免出现渲染过程中未出现某种事件类型，即事件数量等于0的情况，本方法对每个事件的总数进行了加1平滑，在不影响评分准确性的同时防止了除数为0的情况，最后取倒数构成评分基向量。

[0070] 最后进行资源评分，以资源url作为评分对象，取数据结构中资源的事件数量多维向量，将单个资源的事件向量与评分基向量做点积，相当于将资源的事件向量投影到基向量所在的n维直线上，这使得单个网站中所有的资源都在相同的条件下进行评分比较。根据资源评分公式，资源的重要性会传递给资源的父节点，因此最后的资源评分等于该资源的评分与其后继引用资源评分的总和。

[0071] 将得到的评分按照降序进行排序，得到单个网站中关键的资源url。这些url可以被认定为大概率影响网站正常显示、功能可用的关键资源。最后对于列表中的网站，可以得到范围网站的关键资源。

[0072] 实施例2

[0073] 以xueqiu.com为例，首先通过selenium自动化操作加入了渲染过程捕获功能的Chromium，通过模糊匹配的方式匹配登录 / 注册功能按钮，进行点击。点击后页面中出现了登录框弹窗，同时页面源码中出现了新的资源链接和标签集合。

[0074] 渲染过程可以同时捕获到页面中初始的渲染过程信息，以及点击后出现新内容后的渲染过程信息。

[0075] 根据渲染过程信息，可以提取出资源的触发的事件类型，DOM修改类事件可以通过以下：

[0076] "script_id":"25",

[0077] "script_url":"https: / / assets.imedao.com / ugc / js / home.04daf541fb.js",

[0078] "in_execution":false

[0079] 上述代码片为渲染过程中捕捉到的一个资源实体节点，其节点id为25，节点url为https: / / assets.imedao.com / ugc / js / home.04daf541fb.js。

[0080]

[0081] 上述代码片为渲染过程中捕捉到的一个DOM树节点的情况，其节点id为149，其actor_id为25，代表对该DOM树节点修改的动作执行者，其执行者是上述脚本home.04daf541fb.js，此时event_type：AttrModification代表home.04daf541fb.js进行了元素属性修改事件一次，通过此方式可以捕捉到资源的DOM修改类事件。

[0082]

[0083] 上述代码片段为渲染过程中捕捉到的一个网络请求事件，根据actor_id，可以看到该请求的动作发起者为25，即home.04daf541fb.js，此时event_type代表home.04daf541fb.js进行了一次脚本网络请求。

[0084] 以此方式，在本例中可以得到home.04daf541fb.js触发的DOM修改类事件和网络请求类事件，同时构建了一条资源引用链，该链从https: / / assets.imedao.com / ugc / js / home.04daf541fb.js指向https: / / api.geetest.com / gettype.php？gt＝b68f7c51ff1ee91cad5ece5b8a1c9a56&callback＝geetest_1634622688588。

[0085] 终捕获到的事件类型与数量。本方法定义的事件向量与评分基向量格式如下

[0086] "NodeCreation":xx,

[0087] "NodeInsertion":xx,

[0088] "NodeRemoval":xx,

[0089] "NodeAttachLater":xx,

[0090] "AttrAddition":xx,

[0091] "AttrModification":xx,

[0092] "AttrRemoval":xx,

[0093] "AttrStyleTextAddition":xx,

[0094] "AttrStyleRemoval":xx,

[0095] "NetworkScriptRequest":xx,

[0096] "NetworkImageRequest":xx,

[0097] "NetworkIframeRequest":xx,

[0098] "NetworkXMLHTTPRequest":xx,

[0099] "NetworkLinkRequest":xx,

[0100] "NetworkVideoRequest":xx

[0101] 根据上述方式，将网站上的资源集合表示为：

[0102] R＝{r1，r2，…，rm}

[0103] 定义事件类型集合为：

[0104] Event＝{e1，e2，…,en}

[0105] 其中ei表示具体的资源触发的事件类型，对于资源j，定义其事件向量为：

[0106]

[0107] 该向量表示资源触发的事件类型集合中每一种事件的具体数量，定义评分基向量为：

[0108]

[0109] 其中评分基向量E的每一个维度代表对应事件类型在整个页面渲染过程中总数的倒数，对于每个资源，定义其在资源引用链中的直接后继集合为：

[0110] SubResourcei＝{rj|rj∈R，i≠j}

[0111] 最后，一个资源的评分为：

[0112]

[0113] 该评分方式考虑到了资源重要性的传递，即一个资源的评分等于该资源与该资源的引用链后继资源评分的总和。

[0114] 以上所述仅为本发明的较佳实施方式而已，并不用以限制本发明，凡在本发明的精神和原则之内，所作的任何修改、等同替换、改进等，均应包含在本发明的保护范围之内。< / script>

Claims

1. A method for analyzing key website resources based on page rendering contribution, comprising the following steps: Obtaining external resources introduced during website-user interaction; wherein, the external resources include: implicit resources; the method for obtaining the implicit resources includes: Parse the HTML document of the website's page, obtain the relative and absolute path expressions of the web page tags, and obtain the first tag when the text in the path expression contains a first specific word; Obtain all tags with id, class, and user-defined attributes in the HTML document of the page, and when the tag attribute contains a second specific word, obtain the second tag; Clicking the first tag and the second tag retrieves the implicit resource; Page rendering is performed based on the external resources, and website resource behaviors and reference chains are obtained during the page rendering process; wherein, the website resource behaviors include: network request events and DOM tree modification events, and obtaining the website resource behaviors and reference chains during the page rendering process includes: The rendering engine Blink parses HTML documents into a DOM tree; Use the CSS interpreter to mark style information for element nodes in the DOM tree; If a script is encountered during the page rendering process, a script request is sent to the remote server, and the script is executed through V8, initiating network request events and DOM tree modification events; Based on the external resources, website resource behavior, and website resource reference chains, the website's key resources are analyzed to obtain resource analysis results; wherein, the resource analysis results include a score for each external resource.

2. The method as described in claim 1, characterized in that, The step of clicking the first tag and the second tag to obtain the implicit resource includes: When a pop-up window appears, wait for the webpage to finish loading to obtain the implicit resource; or When a new tab appears, wait for the new tab to finish loading, then switch back to the main window and wait for the main window to finish loading to obtain the implicit resource.

3. The method as described in claim 1, characterized in that, The network request events include: script requests, image requests, iframe requests, XML requests, CSS requests, and video requests.

4. The method as described in claim 1, characterized in that, The DOM tree modification events include: node creation, node insertion, node deletion, node rendering, attribute addition, attribute deletion, attribute modification, style addition, and style removal.

5. The method as described in claim 1, characterized in that, The analysis of key website resources based on external resources, website resource behavior, and website resource reference chains yields resource analysis results, including: Count the total number of website resource behaviors that occur, and calculate the event basis vector; For each external resource, the number of resource behaviors occurring on each website is counted to form a resource feature vector. Projecting the resource feature vector onto the event basis vector yields the length of the projection vector; Based on the website resource reference chain, the lengths of the projection vectors are superimposed from bottom to top to obtain the resource analysis results.

6. A storage medium storing a computer program, wherein, The computer program is configured to execute any of the methods described in claims 1-5 at runtime.

7. An electronic device, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the method as described in any one of claims 1-5.

Images