Multi-modal collaborative webpage version update detection method and device, equipment and medium
By employing a multimodal collaborative webpage version update detection method, which utilizes DOM trees and visual semantic feature mapping to perform incremental DOM reconstruction, the accuracy and resource consumption issues of webpage update detection in existing technologies are resolved, achieving automated and stable webpage updates.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HUNAN XIAOSUAN TECH INFORMATION CO LTD
- Filing Date
- 2026-02-09
- Publication Date
- 2026-05-29
Smart Images
Figure CN122116394A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of webpage update detection technology, and in particular to a multimodal collaborative webpage version update detection method, apparatus, equipment, and medium. Background Technology
[0002] Existing webpage update detection technologies mainly fall into three categories: (1) DOM diff-based structural change detection can detect changes in page structure, but it cannot determine whether the function has undergone semantic changes, and it is easily interfered with by noise such as dynamic IDs and random parameters.
[0003] (2) Visual difference detection based on screenshot comparison: Traditional methods rely on pixel differences and are easily affected by noise such as advertisements, animations, and random rendering.
[0004] (3) Lightweight solutions based on content hash or key element monitoring cannot handle complex web pages or identify semantic layer updates.
[0005] In addition, many web pages have dynamic rendering, asynchronous loading, A / B testing, random styles, etc., which makes the structural differences and visual differences often inconsistent. Traditional technologies are prone to false alarms or false alarms, that is, they cannot determine whether the web page has been effectively updated. Summary of the Invention
[0006] In order to at least solve one of the technical problems existing in the prior art, the present invention provides a multimodal collaborative webpage version update detection method, apparatus, device and medium.
[0007] One aspect of the present invention provides a multimodal collaborative webpage version update detection method, comprising: Obtain the rendered HTML and JS data of the target webpage, and determine the DOM tree based on the HTML and JS data; Obtain a screenshot of the target webpage, extract the visual and semantic features of the webpage screenshot; Visual and semantic features are mapped to DOM nodes in the DOM tree to obtain DOM node matching results; Based on the DOM node matching results, the DOM tree differences of the target webpage are determined according to the historical baseline, and dynamic content adjustments are made. Based on the differences in the DOM tree, perform incremental DOM reconstruction on the target webpage and re-render it, and obtain a screenshot of the re-rendered webpage. The version update result of the target webpage is determined by comparing the re-rendered webpage screenshot with the baseline image of the historical baseline.
[0008] According to the multimodal collaborative webpage version update detection method, the rendered HTML and JS data of the target webpage are obtained, and the DOM tree is determined based on the HTML and JS data, including: Web crawlers are used to obtain the rendered HTML and JS data of the target webpage; The rendered HTML data is converted into a DOM tree using a structured parser. The field values of the DOM nodes in the DOM tree are then normalized and serialized in sequence. In addition, the JS data is normalized. Check the benchmark library. If the benchmark library contains a benchmark tree and benchmark JS, then determine the changed DOM nodes, changed JS lines, and difference snapshots based on the serialized DOM tree, normalized JS data, benchmark tree, and benchmark JS. Otherwise, use the serialized DOM tree and normalized JS data as the benchmark tree and benchmark JS.
[0009] According to the multimodal collaborative webpage version update detection method, a screenshot of the target webpage is obtained, and the visual and semantic features of the webpage screenshot are extracted, including: The target webpage is captured at a preset resolution to obtain a webpage screenshot. It is then determined whether a reference image exists in the webpage screenshot. If not, the webpage screenshot is used as the reference image. If it does exist, the visual structure analysis model is used to obtain the coordinate information, element types, and text content of the page based on the webpage screenshot. After semantic feature encoding, the text vector extraction model CLIP is used to extract features to obtain visual and semantic features.
[0010] According to the multimodal collaborative webpage version update detection method, visual features and semantic features are mapped to DOM nodes in a DOM tree to obtain DOM node matching results, including: Unify the coordinates of the screenshots corresponding to the visual features with the coordinates of the DOM tree; The visual region information of the baseline image and the visual region information of the webpage screenshot are obtained. Element change detection is performed through the visual understanding big model to obtain image consistency differences, which include change markers, change types, bounding box coordinates and reliability scores. The visual region information of the baseline image and the visual region information of the webpage screenshot are used to perform semantic similarity matching using the cosine similarity algorithm to obtain semantic similarity, where semantic similarity includes similarity score and bounding box coordinates; Based on image consistency differences and semantic similarity, a DOM node spatial index structure based on R-Tree is used to map DOM nodes and obtain DOM node matching results. The DOM node matching results are used to include the coordinates of bboxes with consistent visual and semantic features, as well as visually different regions.
[0011] According to the multimodal collaborative webpage version update detection method, the method involves determining the DOM tree differences of the target webpage based on the DOM node matching results and the historical baseline, and then dynamically adjusting the content accordingly. The type of change is determined based on the consistency differences in the images. If the type of change is text, the text of the DOM tree in the webpage screenshot is normalized based on the text of the DOM tree in the historical baseline. If the type of change is advertising, the advertising nodes in the DOM tree in the webpage screenshot are normalized based on the advertising nodes of the DOM tree in the historical baseline.
[0012] According to the multimodal collaborative webpage version update detection method, the target webpage undergoes incremental DOM reconstruction based on DOM tree differences and is re-rendered, and a screenshot of the re-rendered webpage is obtained, including: When a local area of the target webpage is dynamically rendered, new structured content is injected into the root node of the DOM subtree based on the visual difference area and the context of dynamic rendering, so that the root node of the DOM subtree is reconstructed according to the updated HTML structure. Identify dynamically rendered front-end runtime component instances and use the component scheduling system to locally re-render the front-end runtime component instances corresponding to the DOM areas with visual differences. Take a screenshot of the target webpage that has been partially re-rendered to obtain a screenshot of the re-rendered webpage.
[0013] According to the multimodal collaborative webpage version update detection method, the version update result of the target webpage is determined by comparing a re-rendered webpage screenshot with a historical baseline image and based on the comparison result. This includes: The re-rendered webpage screenshot and baseline image are used to perform UI change detection through a large visual understanding model. If the UI change detection score exceeds the preset value, the target webpage is updated, and the updated target webpage screenshot and DOM tree are used to replace the baseline screenshot and DOM tree.
[0014] Another aspect of the present invention provides a multimodal collaborative webpage version update detection device, comprising: The first module is used to obtain the rendered HTML and JS data of the target webpage and determine the DOM tree based on the HTML and JS data. The second module is used to obtain a screenshot of the target webpage, extract the visual features and semantic features of the webpage screenshot; The third module is used to map visual and semantic features to DOM nodes in the DOM tree to obtain DOM node matching results. The fourth module is used to determine the DOM tree differences of the target webpage based on the DOM node matching results and the historical baseline, and to make dynamic content adjustments. The fifth module is used to perform incremental DOM reconstruction on the target webpage based on the differences in the DOM tree and re-render it, as well as to obtain a screenshot of the re-rendered webpage. The sixth module is used to compare the re-rendered webpage screenshot with the baseline image of the historical baseline, and determine the version update result of the target webpage based on the comparison result.
[0015] Another aspect of the present invention provides an electronic device, including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method as described above.
[0016] This invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the methods described above.
[0017] The beneficial effects of this invention are as follows: by extracting visual features and semantic differences through semantic / visual extraction, by judging whether the semantics of elements with the same coordinates are the same, if they are different, by mapping the coordinates to DOM nodes, and by performing a series of clean-ups and additions on the latest DOM and then re-rendering the UI page and the baseline page, visual difference analysis is performed, and only the versions with differences are rendered. By incremental DOM reconstruction, the context is maintained and the whole page is avoided, thereby reducing the resources required for page refresh and rendering stability, and realizing the automatic update of the page version. Attached Figure Description
[0018] Figure 1 This is a schematic diagram of the multimodal collaborative webpage version update detection process according to an embodiment of the present invention.
[0019] Figure 2 This is a schematic diagram of the DOM tree construction process of a webpage according to an embodiment of the present invention.
[0020] Figure 3 This is a schematic diagram of the DOM node mapping process according to an embodiment of the present invention.
[0021] Figure 4 This is a schematic diagram of the webpage re-rendering process according to an embodiment of the present invention.
[0022] Figure 5This is a schematic diagram of a multimodal collaborative webpage version update detection device according to an embodiment of the present invention. Detailed Implementation
[0023] The embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings. Throughout the description, the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions. In the following description, suffixes such as "module," "part," or "unit" used to denote elements are used only for the purpose of illustrative purposes and have no specific meaning in themselves. Therefore, "module," "part," or "unit" can be used interchangeably. Terms such as "first," "second," etc., are used only to distinguish technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the sequential relationship of the indicated technical features. In the following description, the consecutive reference numerals for method steps are for ease of review and understanding. Adjusting the implementation order of steps, in conjunction with the overall technical solution of the present invention and the logical relationship between the various steps, will not affect the technical effect achieved by the technical solution of the present invention. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.
[0024] refer to Figure 1 ,in Figure 1 This is a schematic diagram of the multimodal collaborative webpage version update detection process according to an embodiment of the present invention, which includes, but is not limited to, steps S100~S600: S100: Obtain the rendered HTML and JS data of the target webpage, and determine the DOM tree based on the HTML and JS data.
[0025] In some embodiments, reference Figure 2 The diagram shown illustrates the DOM tree construction process of the webpage, including but not limited to steps S110~S130: S110 uses a web crawler to obtain the rendered HTML and JS data of the target webpage; S120: The rendered HTML data is converted into a DOM tree using a structured parser. The field values of the DOM nodes in the DOM tree are then normalized and serialized in sequence. Additionally, the JS data is normalized. S130, check the baseline library. If the baseline library contains a baseline tree and a baseline JS, then determine the DOM nodes to be changed, the number of lines of JS to be changed, and the difference snapshot based on the serialized DOM tree, the normalized JS data, the baseline tree, and the baseline JS. Otherwise, use the serialized DOM tree and the normalized JS data as the baseline tree and the baseline JS.
[0026] In some embodiments, DOM tree acquisition, parsing, cleaning, and structural difference analysis are performed as follows: We use web scraping to obtain the HTML after the JS has been loaded and rendered, as well as all the JS files used in the same session. We then use a structured parser to convert the HTML into a DOM tree and perform attribute and tag-level cleaning on the DOM nodes.
[0027] Normalize dynamic field values in the DOM, including but not limited to: dynamic timestamps, hash parameters, random tokens, and URLs containing session information in the path. DOM tree normalization methods include: replacing regular expression matches with generic placeholders; zeroing out values after XPath-based location; and dictionary rule replacement based on key names.
[0028] Normalize the content of the crawled JS files (remove timestamps, build hashes, delete or mask experiment flags), and calculate the content hash; Convert the tree-like DOM into a sequence or structured object that is easy to diff. Use preorder traversal, and encode node objects including: tagName, id / class, attributes (normalized), and number of child nodes. Determine if the current webpage exists in the baseline library. If not, save it as the baseline DOM tree and baseline JS. If it exists, compare the current state (DOM_tree, js_file) with the historical baseline, obtain the number of newly added / deleted DOM nodes and JS line changes, and record the difference snapshot.
[0029] S200: Obtain a screenshot of the target webpage, extract the visual and semantic features of the webpage screenshot.
[0030] In some embodiments, a screenshot of the target webpage is taken using a preset resolution to obtain a webpage screenshot. It is then determined whether a reference image exists in the webpage screenshot. If not, the webpage screenshot is used as the reference image. If it does exist, the coordinate information, element type, and text content of the page are obtained based on the webpage screenshot using a visual structure analysis model. After semantic feature encoding, features are extracted using a text vector extraction model CLIP to obtain visual features and semantic features.
[0031] In some embodiments, it is used to extract visual features, semantics, and differences from webpage screenshots to provide data support for subsequent visual-to-DOM inverse mapping.
[0032] Visual Extraction: After the webpage has fully loaded, take a full-screen screenshot and unify the image resolution to 1080P. Determine if a baseline image exists in the screenshot; if not, save it as the baseline image. If a baseline image exists, pass the latest image and the baseline image to the visual structure parsing model (Omni Parser) to obtain the coordinate information, element type, and text content of each element on the page.
[0033] Output format: {bbox:[x,y,w,h],text:"Login",type:button} Then, semantic feature encoding is performed on the text, using the CLIP text vector extraction model as the feature extractor. Visual semantic vectors are generated for cross-modal matching between visual regions and DOM nodes. Output format: {bbox:[x,y,w,h],text:"Login",type:button,embedding: <vector>}
[0034] S300 maps visual and semantic features to DOM nodes in the DOM tree to obtain DOM node matching results.
[0035] In some embodiments, reference Figure 3 The diagram illustrating the DOM node mapping process includes, but is not limited to, steps S310 to S340: S310 unifies the coordinates of the screenshot corresponding to the visual features with the coordinates of the DOM tree.
[0036] Understandably, since the visual area (bbox) belongs to the screenshot coordinate system and the bbox extracted from the DOM belongs to the browser layout coordinate system, it is necessary to unify the coordinate system. By obtaining the browser's device resolution and the ratio of the visual area screenshot, the visual area screenshot coordinates are converted into the actual DOM coordinates on the device.
[0037] S320: Obtain visual region information of the baseline image and visual region information of the webpage screenshot, and perform element change detection through the visual understanding big model to obtain image consistency differences, which include change markers, change types, bounding box coordinates and reliability scores.
[0038] In some embodiments, after unifying the coordinates, the visual region information of the baseline image and the visual region information of the latest DOM screenshot are extracted. The elements with the same x (horizontal coordinate of the center point of the bounding box) are cut and passed to the visual understanding big model (qvqmax) to determine whether there is a change in the element. If there is a change, the change flag and change type (dynamic content changes such as price, title, comments, carousel, random ads, number of likes and comments, etc.) are returned, along with the bounding box coordinates and the reliable score Score_img.
[0039] S330, the visual region information of the baseline image and the visual region information of the webpage screenshot are used to perform semantic similarity matching using the cosine similarity algorithm to obtain semantic similarity, where semantic similarity includes similarity score and bounding box coordinates.
[0040] In some embodiments, semantic similarity matching involves extracting the visual region information of the baseline image and the latest DOM screenshot, and extracting the same x-embeddings from the bounding boxes. A cosine similarity algorithm is used to calculate the similarity score Score_e and record the bounding box coordinates.
[0041] S340. Based on the differences in image consistency and semantic similarity, a DOM node spatial index structure based on R-Tree is used to map DOM nodes and obtain DOM node matching results. The DOM node matching results are used to include the coordinates of bboxes with consistent visual and semantic features, as well as visually different regions.
[0042] To efficiently locate the DOM nodes covered by the bounding box, rather than performing a full brute-force search, this invention employs a DOM node spatial indexing structure based on an R-Tree. This includes: traversing the DOM tree and obtaining the geometric information of each node through a CDP; inserting each node as a rectangular object into a container and querying the candidate node set corresponding to visually different regions; and using a multi-feature fusion DOM node matching algorithm, which relies solely on position to generate numerous false matches. Therefore, this invention uses a multi-feature fusion scoring method (semantic embedding + visual similarity). After matching the coordinates, the semantic embedding + visual similarity are compared; if they are the same, the match is successful.
[0043] S400 determines the DOM tree differences of the target webpage based on the DOM node matching results and the historical baseline, and makes dynamic content adjustments.
[0044] In some embodiments, the historical baseline refers to baseline library storage information before the webpage version was updated, including baseline graphs, baseline trees, etc.
[0045] In some embodiments, the variation type is determined based on the image consistency difference. If the variation type is text, the text of the DOM tree of the webpage screenshot is normalized based on the text of the DOM tree of the historical baseline. If the variation type is advertising, the advertising nodes of the DOM tree of the webpage screenshot are normalized based on the advertising nodes of the DOM tree of the historical baseline.
[0046] For example, text normalization includes determining the type of variation in the results of image consistency differences. If it is price, title, comment, or number of likes / comments, the corresponding text of the historical baseline DOM tree is obtained, and the text in the DOM tree to be compared is modified to make the DOM tree to be compared consistent with the baseline DOM tree.
[0047] Ad normalization involves determining whether the ad to be compared is newly added or missing. If it is newly added, the DOM node of the ad to be compared is deleted; if it is missing, the corresponding ad DOM node from the historical baseline DOM tree is obtained and inserted into the ad DOM of the ad to be compared.
[0048] S500 performs incremental DOM reconstruction on the target webpage based on the differences in the DOM tree and re-renders it, and obtains a screenshot of the re-rendered webpage.
[0049] In some embodiments, reference Figure 4 The diagram shown illustrates the webpage re-rendering process, which includes, but is not limited to, steps S510 to S530: S510, When a local area of the target webpage is dynamically rendered, new structured content is injected into the root node of the DOM subtree based on the root node of the DOM subtree of the visual difference area and the context of dynamic rendering, so that the root node of the DOM subtree is reconstructed according to the updated HTML structure. S520 identifies dynamically rendered front-end runtime component instances and uses the component scheduling system to locally re-render the front-end runtime component instances corresponding to the DOM areas with visual differences. S530 takes a screenshot of the target webpage that has been partially re-rendered, and obtains a screenshot of the re-rendered webpage.
[0050] In some embodiments, incremental DOM reconstruction (partial patch / partial JS reloading) reduces rendering time by only partially loading the modified DOM subtree for visually different areas, instead of loading the entire page, and allows the page to maintain the same context, thus improving rendering stability.
[0051] For example, incremental DOM reconstruction includes: after comparing the baseline page with the current page, for the DOM subtrees where differences have been identified, instead of performing a full refresh of the entire webpage, a "script isolation execution mechanism" based on the browser debugging protocol is used to perform local updates on the target subtrees.
[0052] Specifically, when a portion of a webpage's content is dynamically rendered by a script, simply updating its corresponding DOM node content may fail to make the rendering framework (such as Vue or React) aware of the change, resulting in the update not taking effect. To solve this problem, this invention first identifies the root node of the DOM subtree corresponding to the differing region, and then injects new structured content into that node within the browser's script execution context, causing its internal child nodes to be reconstructed based on the updated HTML structure.
[0053] One example of the new structured content is that if there is an advertisement in the baseline DOM tree, but the latest page does not have an advertisement, the advertisement DOM node in the baseline will be extracted and injected into the DOM of the new current page, and then the DOM tree of the current new page will be re-rendered and loaded.
[0054] Subsequently, for regions rendered using front-end component-based frameworks (such as Vue or React), this invention identifies the corresponding front-end runtime component instance (e.g., React's Fiber node or Vue's VNode) for that DOM region and issues a partial re-rendering instruction to the component scheduling system. This allows the corresponding component to re-execute only its own rendering logic without refreshing the entire webpage. Through this combination of partial script execution and component-level rendering scheduling, this invention enables incremental reconstruction of different regions, ensuring that the page changes only within necessary local areas, significantly reducing rendering overhead and minimizing unnecessary page jumps.
[0055] S600 compares the re-rendered webpage screenshot with the baseline image of the historical baseline, and determines the version update result of the target webpage based on the comparison result.
[0056] In some embodiments, the re-rendered webpage screenshot and the baseline image are used to perform UI change detection through a large visual understanding model. If the UI change detection score exceeds a preset value, the target webpage is updated, and the screenshot and DOM tree of the updated target webpage are used to replace the screenshot and DOM tree of the baseline.
[0057] The visual consistency analysis implemented in this invention involves re-capturing the re-rendered page of the DOM to be compared, and then sending the latest comparison image and the baseline image to the visual understanding large model (qvq max) to determine if there are any UI changes. If the score is greater than a set value, it is determined that an update has occurred.
[0058] The embodiments of the present invention also include baseline model self-evolution, which includes replacing the baseline screenshot and DOM tree with the current version screenshot and DOM tree after the previous step determines that an update has been made, so as to achieve unattended automatic detection of webpage updates.
[0059] Figure 5 This is a schematic diagram of a multimodal collaborative webpage version update detection device according to an embodiment of the present invention. The device includes a first module 510, a second module 520, a third module 530, a fourth module 540, a fifth module 550, and a sixth module 560.
[0060] The system comprises six modules: First, it acquires the rendered HTML and JS data of the target webpage and determines the DOM tree based on this data. Second, it acquires a screenshot of the target webpage, extracts its visual and semantic features, and maps these features to the DOM tree to obtain DOM node matching results. Fourth, based on the DOM node matching results and a historical baseline, it identifies DOM tree differences in the target webpage and dynamically adjusts the content. Fifth, it performs incremental DOM reconstruction on the target webpage based on these differences and re-renders it, acquiring a screenshot of the re-rendered webpage. Sixth, it compares the re-rendered screenshot with the historical baseline and determines the version update result of the target webpage based on the comparison.
[0061] For example, with the cooperation of the first, second, third, fourth, fifth, and sixth modules in the device, the embodiment device can implement any of the aforementioned multimodal collaborative webpage version update detection methods, namely: acquiring the rendered HTML and JS data of the target webpage; determining the DOM tree based on the HTML and JS data; acquiring a screenshot of the target webpage; extracting visual and semantic features from the screenshot; mapping the visual and semantic features to the DOM tree to obtain DOM node matching results; determining the DOM tree differences of the target webpage based on the historical baseline and dynamically adjusting the content based on the DOM node matching results; performing incremental DOM reconstruction processing on the target webpage based on the DOM tree differences and re-rendering it; and acquiring a re-rendered webpage screenshot; comparing the re-rendered webpage screenshot with the historical baseline map; and determining the version update result of the target webpage based on the comparison result. The beneficial effects of this invention are as follows: by extracting visual features and semantic differences through semantic / visual extraction, by judging whether the semantics of elements with the same coordinates are the same, if they are different, by mapping the coordinates to DOM nodes, and by performing a series of clean-ups and additions on the latest DOM and then re-rendering the UI page and the baseline page, visual difference analysis is performed, and only the versions with differences are rendered. By incremental DOM reconstruction, the context is maintained and the whole page is avoided, thereby reducing the resources required for page refresh and rendering stability, and achieving the effect of unattended automatic detection of page updates.
[0062] This invention also provides an electronic device, which includes a processor and a memory; The memory stores the program; The processor executes a program to perform the aforementioned multimodal collaborative webpage version update detection method; the electronic device has the function of carrying and running the software system for multimodal collaborative webpage version update detection provided in the embodiments of the present invention, such as a personal computer, minicomputer, mainframe, workstation, network or distributed computing environment, standalone or integrated computer platform, or communicating with charged particle tools or other imaging devices, etc.
[0063] This invention also provides a computer-readable storage medium storing a program that is executed by a processor to implement the multimodal collaborative webpage version update detection method described above.
[0064] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented in the embodiments of this invention. Alternative embodiments are contemplated, in which the order of various operations is changed and sub-operations described as part of a larger operation are executed independently.
[0065] This invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned multimodal collaborative webpage version update detection method.
[0066] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the described functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, considering the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed in the embodiments of the invention, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.
[0067] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0068] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can include, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.
[0069] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.
[0070] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.
[0071] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.
[0072] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.
[0073] The above is a detailed description of the preferred embodiments of the present invention, but the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention, and these equivalent modifications or substitutions are all included within the scope defined by the claims of this application.< / vector>
Claims
1. A multimodal collaborative webpage version update detection method, characterized in that, include: Obtain the rendered HTML and JS data of the target webpage, and determine the DOM tree based on the HTML and JS data; Obtain a screenshot of the target webpage, extract the visual and semantic features of the webpage screenshot; Visual and semantic features are mapped to DOM nodes in the DOM tree to obtain DOM node matching results; Based on the DOM node matching results, the DOM tree differences of the target webpage are determined according to the historical baseline, and dynamic content adjustments are made. Based on the differences in the DOM tree, perform incremental DOM reconstruction on the target webpage and re-render it, and obtain a screenshot of the re-rendered webpage. The version update result of the target webpage is determined by comparing the re-rendered webpage screenshot with the baseline image of the historical baseline.
2. The multimodal collaborative webpage version update detection method according to claim 1, characterized in that, The process of obtaining the rendered HTML and JS data of the target webpage and determining the DOM tree based on the HTML and JS data includes: Web crawlers are used to obtain the rendered HTML and JS data of the target webpage; The rendered HTML data is converted into a DOM tree using a structured parser. The field values of the DOM nodes in the DOM tree are then normalized and serialized in sequence. In addition, the JS data is normalized. Check the benchmark library. If the benchmark library contains a benchmark tree and benchmark JS, then determine the changed DOM nodes, changed JS lines, and difference snapshots based on the serialized DOM tree, normalized JS data, benchmark tree, and benchmark JS. Otherwise, use the serialized DOM tree and normalized JS data as the benchmark tree and benchmark JS.
3. The multimodal collaborative webpage version update detection method according to claim 1, characterized in that, The step of obtaining a screenshot of the target webpage, and extracting visual and semantic features from the screenshot, includes: The target webpage is captured at a preset resolution to obtain a webpage screenshot. It is then determined whether a reference image exists in the webpage screenshot. If not, the webpage screenshot is used as the reference image. If it does exist, the visual structure analysis model is used to obtain the coordinate information, element types, and text content of the page based on the webpage screenshot. After semantic feature encoding, the text vector extraction model CLIP is used to extract features to obtain visual and semantic features.
4. The multimodal collaborative webpage version update detection method according to claim 3, characterized in that, The process of mapping visual and semantic features to DOM nodes in a DOM tree to obtain DOM node matching results includes: Unify the coordinates of the screenshots corresponding to the visual features with the coordinates of the DOM tree; The visual region information of the baseline image and the visual region information of the webpage screenshot are obtained. Element change detection is performed through the visual understanding big model to obtain image consistency differences, which include change markers, change types, bounding box coordinates and reliability scores. The visual region information of the baseline image and the visual region information of the webpage screenshot are used to perform semantic similarity matching using the cosine similarity algorithm to obtain semantic similarity, where semantic similarity includes similarity score and bounding box coordinates; Based on image consistency differences and semantic similarity, a DOM node spatial index structure based on R-Tree is used to map DOM nodes and obtain DOM node matching results. The DOM node matching results are used to include the coordinates of bboxes with consistent visual and semantic features, as well as visually different regions.
5. The multimodal collaborative webpage version update detection method according to claim 4, characterized in that, The step of determining the DOM tree differences of the target webpage based on the DOM node matching results and the historical baseline, and then dynamically adjusting the content, includes: The type of change is determined based on the consistency differences in the images. If the type of change is text, the text of the DOM tree in the webpage screenshot is normalized based on the text of the DOM tree in the historical baseline. If the type of change is advertising, the advertising nodes in the DOM tree in the webpage screenshot are normalized based on the advertising nodes of the DOM tree in the historical baseline.
6. The multimodal collaborative webpage version update detection method according to claim 5, characterized in that, The incremental DOM reconstruction process based on DOM tree differences and subsequent re-rendering of the target webpage, as well as obtaining a screenshot of the re-rendered webpage, includes: When a local area of the target webpage is dynamically rendered, new structured content is injected into the root node of the DOM subtree based on the visual difference area and the context of dynamic rendering, so that the root node of the DOM subtree is reconstructed according to the updated HTML structure. Identify dynamically rendered front-end runtime component instances and use the component scheduling system to locally re-render the front-end runtime component instances corresponding to the DOM areas with visual differences. Take a screenshot of the target webpage that has been partially re-rendered to obtain a screenshot of the re-rendered webpage.
7. The multimodal collaborative webpage version update detection method according to claim 6, characterized in that, The step of comparing the re-rendered webpage screenshot with the historical baseline image and determining the version update result of the target webpage based on the comparison result includes: The re-rendered webpage screenshot and baseline image are used to perform UI change detection through a large visual understanding model. If the UI change detection score exceeds the preset value, the target webpage is updated, and the updated target webpage screenshot and DOM tree are used to replace the baseline screenshot and DOM tree.
8. A multimodal collaborative webpage version update detection device, characterized in that, include: The first module is used to obtain the rendered HTML and JS data of the target webpage and determine the DOM tree based on the HTML and JS data. The second module is used to obtain a screenshot of the target webpage, extract the visual features and semantic features of the webpage screenshot; The third module is used to map visual and semantic features to DOM nodes in the DOM tree to obtain DOM node matching results. The fourth module is used to determine the DOM tree differences of the target webpage based on the DOM node matching results and the historical baseline, and to make dynamic content adjustments. The fifth module is used to perform incremental DOM reconstruction on the target webpage based on the differences in the DOM tree and re-render it, as well as to obtain a screenshot of the re-rendered webpage. The sixth module is used to compare the re-rendered webpage screenshot with the baseline image of the historical baseline, and determine the version update result of the target webpage based on the comparison result.
9. An electronic device, characterized in that, Including the processor and memory; The memory is used to store programs; The processor executes the program to implement the multimodal collaborative webpage version update detection method as described in any one of claims 1-7.
10. A computer-readable storage medium, characterized in that, The storage medium stores a program that is executed by a processor to implement the multimodal collaborative webpage version update detection method as described in any one of claims 1-7.