Webpage layout element identification method based on image analysis and electronic equipment

Abstract

The invention discloses a webpage layout element identification method based on image analysis and electronic equipment. The method comprises the following steps: acquiring a page image of a to-be-identified page; a candidate element area of the page image is determined, and the candidate element area comprises at least one webpage layout element; and using the target image visual identification model to identify the candidate element region to obtain an element identification result, the element identification result at least comprising bounding box coordinates of the webpage layout element, a predicted semantic category and category confidence, and the category confidence is used for representing the probability that the webpage layout element belongs to the predicted semantic category. According to the method and the device, the technical problem of low identification accuracy of webpage layout elements in related technologies is solved.

Description

Technical Field

[0001] This invention relates to the fields of the Internet and image processing, and more specifically, to a method and electronic device for identifying web page layout elements based on image analysis. Background Technology

[0002] In web design and development, accurately identifying and understanding web page layout elements is crucial. This technology primarily involves the cross-application of multiple fields, including computer vision, deep learning, natural language processing (NLP), and web page parsing. To identify page layout elements, one can directly parse the web page source code, such as Hyper Text Markup Language (HTML), Cascading Style Sheets (CSS), and JavaScript (a lightweight, interpreted scripting language). Using an HTML parser, the tags, attributes, and hierarchy information of each element can be extracted. Combined with CSS and JavaScript, this allows for a deeper understanding of the element's style and dynamic behavior. While this method is direct, the parsing process can be affected by complex nesting and dynamically loaded content, leading to technical problems such as recognition failures in cases of dynamic content, complex CSS, or script errors.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This invention provides a web page layout element recognition method and electronic device based on image analysis, so as to at least solve the technical problem of low recognition accuracy of web page layout elements in related technologies.

[0005] According to one aspect of the present invention, a web page layout element recognition method based on image analysis is provided, comprising: acquiring a page image of a page to be recognized; determining candidate element regions of the page image, wherein the candidate element regions contain at least one web page layout element; and using a target image visual recognition model to recognize the candidate element regions to obtain element recognition results, wherein the element recognition results include at least the bounding box coordinates of the web page layout element, the predicted semantic category, and the category confidence, wherein the category confidence is used to characterize the probability that the web page layout element belongs to the predicted semantic category.

[0006] Further, the candidate element region is identified using a target image visual recognition model to obtain element recognition results, including: using multiple base classifiers in the target image visual recognition model to identify the candidate element region and obtain multiple initial element recognition results, wherein the multiple initial element recognition results include at least multiple initial bounding box coordinates, multiple initial predicted semantic categories, and multiple initial category confidences; and filtering the multiple initial element recognition results to obtain the element recognition result.

[0007] Furthermore, the initial element identification results are filtered to obtain element identification results, including: filtering the initial element identification results based on a preset confidence threshold and multiple initial category confidence to obtain multiple intermediate element identification results; merging the multiple intermediate element identification results to obtain the element identification results.

[0008] Furthermore, the element recognition results of multiple intermediate elements are merged to obtain the element recognition result, including: obtaining at least two intermediate element recognition results with the same initial predicted semantic category among the multiple intermediate element recognition results; merging the intermediate element recognition results with the difference between the corresponding initial bounding box coordinates of the at least two intermediate element recognition results being less than a preset threshold to obtain the element recognition result.

[0009] Furthermore, the candidate element region contains multiple web page layout elements, and the target image visual recognition model includes a feature pyramid network module. Multiple base classifiers in the target image visual recognition model are used to identify the candidate element region, resulting in multiple initial element recognition results. This includes: using the feature pyramid network module to extract features from web page layout elements of different scales among the multiple web page layout elements, obtaining multiple scale features; and guiding multiple base classifiers based on the multiple scale features to identify the candidate element region, resulting in multiple initial element recognition results.

[0010] Furthermore, the target image visual recognition model includes a design rule utilization module; based on multiple scale features, it guides multiple base classifiers to identify candidate element regions and obtain multiple initial element recognition results, including: using the design rule utilization module to identify multiple web page layout elements based on built-in element design rules and obtain multiple reference semantic categories; based on multiple scale features and multiple reference semantic categories, it guides multiple base classifiers to identify candidate element regions and obtain multiple initial element recognition results.

[0011] Furthermore, the method also includes: acquiring a training page image set; using an annotation tool to annotate the web page layout elements in the training page image set to obtain an annotated page image set; performing data augmentation operations on the annotated page image set to obtain an augmented page image set; and training the initial image visual recognition model based on the augmented page image set to obtain the target image visual recognition model.

[0012] Furthermore, obtaining the page image of the page to be identified includes: using an automated tool to access the page to be identified, waiting for a preset time, and then performing screen capture processing on the page to be identified to obtain the page image.

[0013] According to another aspect of the present invention, an electronic device is also provided, comprising: a memory storing an executable program; and a processor for running the program, wherein the program executes the web page layout element recognition method based on image analysis according to the present invention during runtime.

[0014] According to another aspect of the present invention, a computer-readable storage medium is also provided, which stores an executable program that executes the web page layout element recognition method based on image analysis according to the present invention when the program is run.

[0015] According to another aspect of the present invention, an electronic device is also provided, comprising: a memory storing an executable program; and a processor for running the program, wherein the program executes the methods of various embodiments of the present invention during runtime.

[0016] According to another aspect of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein, when the executable program is executed, it controls the device where the computer-readable storage medium is located to perform the methods of various embodiments of the present invention.

[0017] According to another aspect of the present invention, a computer program product is also provided, including a computer program that, when executed by a processor, implements the methods of various embodiments of the present invention.

[0018] According to another aspect of the present invention, a computer program product is also provided, including a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods of various embodiments of the present invention.

[0019] According to another aspect of the present invention, a computer program is also provided, which, when executed by a processor, implements the methods of the various embodiments of the present invention.

[0020] In this embodiment of the invention, after acquiring the page image of the page to be identified, candidate element regions of the page image are determined, and a target image visual recognition model is used to identify the candidate element regions to obtain element recognition results. In other words, this embodiment of the invention uses image analysis and deep learning model recognition to accurately identify and analyze candidate element regions in the page image, achieving the goal of accurately identifying and understanding the semantic categories and category confidence of webpage layout elements. This improves the accuracy and robustness of webpage content recognition, thereby solving the technical problem of low accuracy in identifying webpage layout elements in related technologies. Attached Figure Description

[0021] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0022] Figure 1 This is a flowchart of a web page layout element recognition method based on image analysis, according to existing technology;

[0023] Figure 2 This is a schematic diagram of an optional edge image generation method according to an embodiment of the present invention;

[0024] Figure 3 This is a schematic diagram of an optional image analysis-based webpage layout element recognition device according to an embodiment of the present invention. Detailed Implementation

[0025] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0026] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0027] According to an embodiment of the present invention, a web page layout element recognition method based on image analysis is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0028] Figure 1 This is a flowchart of a webpage layout element recognition method based on image analysis according to an embodiment of the present invention, such as... Figure 1 As shown, the method includes the following steps:

[0029] Step S102: Obtain the page image of the page to be identified.

[0030] The page to be identified in the above steps can be a page that requires webpage layout element identification, such as the page currently being visited by the user. The type of page to be identified can be e-commerce, news, portal, backend management system, etc., but is not limited to these. The webpage layout elements here can be user interface (UI) elements on the page to be identified, and specific types can include at least one of the following: header, title, paragraph, button, image, input box, navigation bar, footer, list, and icon, but are not limited to these.

[0031] In an alternative embodiment, the target webpage can be accessed using automated tools (such as Selenium or Puppeteer), and the page can be waited for to fully load (including all asynchronous requests and dynamic rendering). Then, a screenshot of the entire webpage can be captured and saved as an RGB image file (such as a PNG).

[0032] In another alternative embodiment, the user can upload the page image themselves, which can be a screenshot taken manually by the user or an image captured by an imaging device.

[0033] Furthermore, after obtaining the webpage screenshot, adaptive preprocessing is performed on the screenshot to obtain a standardized image. This adaptive preprocessing may include at least one of the following: adaptive size normalization based on image content complexity, multi-scale image enhancement processing, and color space optimization tailored to the characteristics of the webpage image. Adaptive size normalization based on image content complexity can be achieved by scaling the screenshot to a fixed input size required by the model (e.g., 640x640 pixels), maintaining the original aspect ratio during scaling, and padding the edges to avoid image distortion. Simultaneously, pixel values ​​are normalized from [0,255] to [0,1] or standardized to accelerate model convergence and inference.

[0034] Step S104: Determine candidate element regions of the page image, wherein the candidate element regions contain at least one web page layout element.

[0035] The candidate element region in step S104 above can be the region where at least one web page layout element is located in the page image. For example, a candidate element region can be the region where a web page layout element is located. If the page image contains multiple web page layout elements, the candidate element region can be multiple regions.

[0036] In one optional embodiment, after obtaining the page image (standardized image) of the page to be identified, a hybrid edge detection algorithm can be used to extract edge features in the standardized image. The hybrid edge detection algorithm includes at least one of the following: extracting edges using at least two different edge detection algorithms respectively, adaptively calculating the fusion weight of each algorithm based on image features, and performing weighted fusion of the detection results of multiple algorithms.

[0037] Optionally, two or more different edge detection algorithms (such as the Sobel operator, Canny operator, Laplacian operator, etc.) can be selected to process the standardized image separately to obtain their respective edge feature maps. Each algorithm has its specific application scenarios and advantages and disadvantages. For example, the Canny operator performs well in suppressing noise and detecting sharp edges, while the Sobel operator performs better when detecting edges in larger regions.

[0038] Further, the color image is first converted into a grayscale image, which can clearly represent changes based on brightness. Then, a Gaussian filter is used to smooth the grayscale image, weighting the pixel values ​​of local regions to reduce high-frequency noise and make subsequent edge detection more accurate. Typically, changes in image brightness are identified by calculating the image's gradient. Commonly used gradient operators include the Sobel operator, Prewitt operator, or Laplacian operator. These operators calculate the gradients in the horizontal and vertical directions of the image, respectively, thus obtaining the magnitude and direction of the gradient. After calculating the gradient magnitude, Non-Maximum Suppression (NMS) is performed. The purpose of NMS is to remove points with small gradient magnitudes and retain the points with the largest gradients; these points usually correspond to edges. Then, a double thresholding process is performed to distinguish between strong and weak edges. The algorithm sets two thresholds: a low threshold and a high threshold. For example, points above a high threshold are considered strong edge points, points below a low threshold are considered non-edge points, and points between the two thresholds are marked as edge points if they are connected to strong edge points. Finally, a connection algorithm is used to connect the points marked as edges to form continuous edge lines. For example, depth-first search or breadth-first search algorithms can be used.

[0039] To determine the contribution of each algorithm's result to the final fusion, dynamic fusion weights need to be calculated based on specific image features (such as edge strength, orientation, and local details). This process can employ machine learning methods, such as Support Vector Machines (SVMs) and neural networks, to train a model that automatically adjusts the weights based on the features of the input standardized image to maximize the quality of the fusion result.

[0040] Based on the calculated fusion weights, the results of various edge detection algorithms can be weighted and averaged or summed to generate the final edge feature map. In this way, even under complex or harsh image conditions, the accuracy and stability of edge detection can be improved through the complementarity of multiple algorithms.

[0041] After extracting edge features from the standardized image, candidate element regions can be extracted from the edge features using contour analysis techniques. Contour analysis includes at least one of the following: Multilevel Profile Comparison Analysis (MLPCA), geometric feature-based contour filtering, contour approximation, and shape feature extraction.

[0042] Multi-level profile hierarchy analysis is a data analysis and visualization method primarily used to handle complex data structures and multi-dimensional comparisons. For example, in MLPCA, data is typically organized into multiple levels, each representing a different analytical unit or perspective, such as the bottom level (position, size, color, etc.), the middle level (navigation bar, sidebar, content area, bottom information bar, etc.), and the top level (product display area, promotional activity area, user review area, etc.).

[0043] Furthermore, in this process, each layer of analysis extracts the key features of that layer and transforms them into low-dimensional representations, facilitating further analysis and machine learning applications. For example, the first layer might transform the features of each HTML element into a vector, while the second layer might use these vectors to identify combinations of elements with similar layout features, forming higher-level layout components. This involves preliminary contour detection and coarse feature extraction, providing an informational foundation for more refined filtering and approximation.

[0044] After initially acquiring contour information, the system extracts and filters object edges or contours with specific geometric shapes or characteristics from the image. Following edge detection, edge information can be further refined and optimized, removing irrelevant or noise-induced edges to obtain a more accurate description of the object's shape. Contour filtering can be based on various geometric features, including but not limited to: length, area, perimeter, aspect ratio, roundness / ellipticity (the roundness or ellipticity of the contour can be calculated and filtered according to a preset threshold), orientation, smoothness, convexity, and connectivity.

[0045] After contour filtering, objects in the image are further understood and classified by simplifying the contour details (contour approximation) and extracting specific shape features (such as area, perimeter, concavity, etc.) from the simplified contours. Contour approximation reduces the amount of data (reducing the number of contour points), making the calculation of shape features more efficient, and the output of this step provides key quantitative indicators for the final object recognition or classification.

[0046] Furthermore, in step S104 above, candidate element regions of the page image are determined. The candidate element regions include, but are not limited to, extracting multi-dimensional features of the candidate element regions: texture features, including: local texture patterns, gradient features, etc.

[0047] In one alternative embodiment, an image segmentation algorithm (watershed algorithm) can be used to divide the image into multiple regions, thereby determining the candidate element regions of the page image. By adjusting the parameters, the boundaries of each element on the identified page image can be made to achieve the effect of determining the candidate element regions of the page image.

[0048] Step S106: Use the target image visual recognition model to identify the candidate element region and obtain the element recognition result. The element recognition result includes at least the bounding box coordinates of the web page layout element, the predicted semantic category, and the category confidence. The category confidence is used to characterize the probability that the web page layout element belongs to the predicted semantic category.

[0049] The target image visual recognition model in step S106 above can be a pre-trained image visual recognition model that can recognize images and obtain the recognition results of the elements contained in the image. For example, it can be a convolutional neural network, a deep neural network, etc., but it is not limited to these.

[0050] The bounding box coordinates in step S106 above can be the coordinates of the bounding box in the page image. Since the web layout element is located inside the bounding box, the bounding box coordinates can refer to the coordinates of the web layout element in the page image. The bounding box coordinates can be the coordinates of the top left and bottom right corners of the bounding box, or the coordinates of the center of the bounding box, or the length and width (x_center, y_center, width, height) of the bounding box, but are not limited to these.

[0051] The predicted semantic category in step S106 above can be determined by using a target image visual model to identify web page layout elements and determine the semantic category of the web page layout elements. The prediction results include, but are not limited to: Header, Title, Paragraph, Button, Image, Input Field, Navigation Bar, Footer, List, and Icon.

[0052] The category confidence score in step S106 above can be considered the accuracy of identifying webpage layout elements using the target image visual model. A higher probability that a webpage layout element belongs to the predicted semantic category indicates higher accuracy in identifying the webpage layout element using the target image visual model, resulting in a higher category confidence score. Category confidence scores are typically represented by values ​​between 0 and 1; a higher value indicates greater confidence from the model that the element belongs to that category.

[0053] In one optional embodiment, candidate element regions can be input into a target image visual recognition model. The model then classifies and identifies these regions, outputting detection results that include element location, category confidence, and category label. Low-quality predictions are filtered out by setting a confidence threshold, and non-maximum suppression (NMS) further eliminates duplicate detections, retaining the bounding boxes with the highest confidence. Finally, the bounding box coordinates are inverted to the original screenshot resolution, and the recognition results are output in a structured data format.

[0054] Through the steps provided in the above embodiments of this application, after obtaining the page image of the page to be identified, candidate element regions of the page image are determined, and a target image visual recognition model is used to identify the candidate element regions to obtain element recognition results. In other words, the embodiments of this invention employ image analysis and deep learning model recognition methods to accurately identify and analyze candidate element regions in the page image, achieving the goal of accurately identifying and understanding the semantic categories and category confidence of webpage layout elements. This improves the accuracy and robustness of webpage content recognition, thereby solving the technical problem of low accuracy in webpage layout element recognition in related technologies.

[0055] Optionally, a target image visual recognition model is used to identify candidate element regions and obtain element recognition results, including: using multiple base classifiers in the target image visual recognition model to identify candidate element regions and obtain multiple initial element recognition results, wherein the multiple initial element recognition results include at least multiple initial bounding box coordinates, multiple initial predicted semantic categories, and multiple initial category confidences; and filtering the multiple initial element recognition results to obtain the element recognition results.

[0056] Multiple base classifiers can refer to the basic units that make up a more complex model. For example, in some machine learning and deep learning models, especially in ensemble learning methods, multiple base classifiers are used to work together to complete the recognition task.

[0057] In one alternative embodiment, after identifying candidate element regions in the page image, considering that these regions are believed to likely contain key components of the webpage layout, such as titles, paragraph text, images, or buttons, a pre-trained target image visual recognition model can be used to perform deep analysis on these candidate regions. This model integrates multiple base classifiers, each specifically responsible for identifying a certain type of webpage element. During model operation, it provides multiple initial element recognition results for each candidate region, which may include initial bounding box coordinates, initial predicted semantic category, and initial category confidence corresponding to that category. A higher confidence value indicates that the model believes the element is more likely to belong to the predicted category.

[0058] Furthermore, to improve the accuracy of identification, these initial element identification results can be filtered and fused. A confidence threshold is set, meaning only results with a confidence level higher than a certain threshold are retained. Simultaneously, rules are applied to resolve potential conflicts between different base classifiers regarding the same region, such as using a voting mechanism or a weighted average based on confidence level to determine the final predicted category. After filtering, the final element identification result is obtained, containing the exact bounding box coordinates of the webpage layout elements, the predicted semantic category, and their corresponding high-confidence values, thus yielding the element identification result.

[0059] In the above process, by comprehensively utilizing the synergistic effect of multiple base classifiers, the accuracy and efficiency of webpage element recognition are significantly improved. Each base classifier focuses on the recognition of a specific type of element, and together they can meticulously analyze candidate regions, generating comprehensive recognition results including bounding box coordinates, predicted semantic categories, and category confidence. This provides a solid foundation for subsequent applications such as webpage layout analysis, content extraction, or automated testing.

[0060] Optionally, multiple initial element identification results are filtered to obtain element identification results, including: filtering multiple initial element identification results based on a preset confidence threshold and multiple initial category confidence to obtain multiple intermediate element identification results; merging multiple intermediate element identification results to obtain element identification results.

[0061] The aforementioned pre-set reliability threshold can be a standard or boundary set in advance in fields such as data science, machine learning, and statistical analysis before prediction or classification to judge the reliability of the model's prediction results. This threshold is set based on expectations of the model's prediction accuracy and tolerance for false positives (incorrectly predicting true) and false negatives (incorrectly predicting false). For example, in a binary classification task, if the probability output by the model is higher than the pre-set reliability threshold, the sample is classified as positive; if it is lower than the threshold, it is classified as negative. The threshold determines classification accuracy and error control, and is a key adjustment parameter.

[0062] The aforementioned multiple intermediate element identification results can refer to the identification results obtained using multiple base classifiers, which are the results obtained after multiple base classifiers independently predict the same data. Each base classifier may be based on different subsets of data or use different parameter settings, so their outputs may differ. This approach can improve the accuracy and stability of the model because the error of a single classifier can be compensated for by the correct predictions of other classifiers.

[0063] For example, after obtaining multiple initial bounding box coordinates, multiple initial predicted semantic categories, and multiple initial category confidence scores for webpage elements (which might include text boxes, buttons, images, and links), the model outputs a recognition result containing the semantic category of each element and its corresponding confidence score. Next, a confidence threshold, such as 80%, is set to filter these initial recognition results, removing elements with confidence scores below the threshold, resulting in multiple intermediate element recognition results. Then, to further improve recognition accuracy, these intermediate results are merged. For instance, if the model identifies multiple buttons in similar locations and of the same category, they might be merged into one, with the highest confidence score selected as the final recognition result. Alternatively, if the identified element is inconsistent with the webpage's layout and structure information, the element recognition result will be adjusted based on the layout information, ultimately resulting in a filtered and optimized element recognition result.

[0064] In one alternative embodiment, a dynamic confidence adjustment mechanism is used to improve element recognition results: during the recognition process, the confidence threshold can be dynamically adjusted based on changes in the webpage environment, data quality, or the accuracy of historical recognition results.

[0065] In another alternative embodiment, a machine learning model is used for post-processing: after initially screening out multiple intermediate element identification results, these results and their associated image or signal features are input into a pre-trained machine learning model for further analysis and decision-making.

[0066] In the above process, through fine screening, merging and optimization of multiple initial element recognition results and intelligent post-processing, the problems of unstable recognition and low accuracy of traditional methods in complex web page environments are effectively solved, providing a more reliable and efficient solution for subsequent web page element parsing.

[0067] Optionally, merging multiple intermediate element recognition results to obtain an element recognition result includes: acquiring at least two intermediate element recognition results with the same initial predicted semantic category among the multiple intermediate element recognition results; merging intermediate element recognition results whose difference between the corresponding initial bounding box coordinates is less than a preset threshold among the at least two intermediate element recognition results to obtain an element recognition result.

[0068] For example, the algorithm first processes the webpage image using a target image visual recognition model, obtaining a series of intermediate element recognition results. Each result includes the element's initial predicted semantic category and initial bounding box coordinates. Next, the algorithm filters out all intermediate recognition results with the same semantic category, such as all elements predicted as "buttons." Then, the bounding box coordinates of these elements of the same category can be further analyzed. If the difference between two or more bounding boxes is less than a preset threshold, it indicates that they are very close in position on the image and may be different recognition instances of the same element. At this point, the algorithm merges these intermediate element recognition results with similar positions. Based on their bounding box coordinates and confidence scores, it calculates a more accurate comprehensive bounding box and an optimized semantic category, thus obtaining the final element recognition result.

[0069] In one alternative embodiment, non-maximum suppression (NMS) can be used for result merging. NMS is a common post-processing step in object detection used to select the most likely bounding box from overlapping bounding boxes. In this scenario, all intermediate recognition results belonging to the same category are first filtered out, and then the intersection-over-union (IoU) ratio between these bounding boxes is calculated. If the IoU of two predictions exceeds a preset threshold, the two predictions are considered to overlap. The prediction with the highest confidence is retained, while the other is removed. This process is repeated until no overlapping predictions remain.

[0070] The Intersection over Union (IoU) ratio, mentioned above, is a commonly used concept in computer vision tasks such as object detection and image segmentation. It measures the degree of overlap between two bounding boxes. It is a ratio calculated by dividing the area of ​​the intersection of the two bounding boxes by the area of ​​their union. For example, the IoU ranges from 0 to 1; a higher value indicates a greater degree of overlap between the two bounding boxes. When the two boxes completely overlap, IoU = 1; when they do not overlap at all, IoU = 0.

[0071] In the above process, the element recognition results are merged by using the difference between the initial bounding box coordinates, which can reduce false recognition and duplicate recognition, and improve the accuracy and efficiency of recognition.

[0072] Optionally, the candidate element region contains multiple web page layout elements, and the target image visual recognition model includes a feature pyramid network module; multiple base classifiers in the target image visual recognition model are used to identify the candidate element region to obtain multiple initial element recognition results, including: using the feature pyramid network module to extract features from web page layout elements of different scales among the multiple web page layout elements to obtain multiple scale features; based on the multiple scale features, multiple base classifiers are guided to identify the candidate element region to obtain multiple initial element recognition results.

[0073] The Feature Pyramid Network (FPN) module described above can be a design module in convolutional neural networks used for object detection and image segmentation. It effectively improves the model's ability to recognize objects at different scales by constructing multi-scale feature maps from the bottom up and using lateral connections for feature fusion. This structure allows the network to extract information from feature maps at multiple levels; small-scale feature maps are good at capturing details, while large-scale maps excel at understanding the global picture. Through layer-by-layer fusion, the Feature Pyramid Network achieves multi-level information integration from coarse to fine, significantly enhancing the model's multi-scale perception capability.

[0074] For example, an advanced target image visual recognition model is first used, which integrates a Feature Pyramid Network (FPN) module to enhance the recognition of targets at different scales. When analyzing web pages, the web page image is first preprocessed, and then multi-level feature extraction is performed on the image through the FPN. This allows the model to simultaneously capture the detailed features of large-scale elements (such as full-screen background images) and small-scale elements (such as button icons). Next, the model uses these multi-scale features to drive multiple base classifiers to identify and classify candidate element regions in the web page. Each base classifier focuses on processing layout elements of a specific type or scale. By comprehensively analyzing features at different scales, it can more accurately determine whether a candidate region is a title, body text, or advertisement, etc. For example, one base classifier might be trained to recognize text regions, analyzing all scale features generated by the FPN to determine whether there are continuous alphanumeric patterns in the candidate region, while another base classifier might focus on recognizing image regions, paying more attention to features such as color, shape, and texture.

[0075] In one alternative embodiment, an attention mechanism can first be introduced to enhance the model's ability to focus on key regions, thereby improving recognition accuracy. Specifically, an attention module is added after the feature pyramid network module. This module analyzes features at various scales, automatically learns, and assigns an attention weight to each web page layout element. This weight reflects the importance of the element in the image, guiding the base classifier to pay more attention to high-weight regions and reducing misclassification of background or irrelevant elements.

[0076] In another alternative embodiment, deep reinforcement learning strategies can be used to optimize the identification process of candidate element regions. Specifically, an intelligent agent can be embedded in the target image visual recognition model. This agent learns through trial and error, gradually adjusting its exploration and recognition strategies for web page layout elements. Based on feedback from the initial recognition results, the intelligent agent dynamically adjusts the parameters of the feature pyramid network and the focus of the base classifier to maximize the accuracy of element recognition.

[0077] In the above process, by adding a feature pyramid network module, the system can obtain multiple initial element recognition results. Each result is based on a specific classifier and specific scale features. Subsequently, these results are integrated and optimized to produce the final web page layout element recognition and classification.

[0078] Optionally, the target image visual recognition model includes a design rule utilization module; based on multiple scale features, it guides multiple base classifiers to identify candidate element regions and obtain multiple initial element recognition results, including: using the design rule utilization module to identify multiple web page layout elements based on built-in element design rules and obtain multiple reference semantic categories; based on multiple scale features and multiple reference semantic categories, it guides multiple base classifiers to identify candidate element regions and obtain multiple initial element recognition results.

[0079] The aforementioned design rule utilization module can refer to the key role played in the target image visual recognition model in integrating design theories and rules into machine learning algorithms. It is mainly used to improve the model's accuracy in recognizing web page layout elements and its semantic understanding ability.

[0080] For example, in a practical application, consider a complex web page analysis scenario. A target image visual recognition model is used to parse the web page layout and identify different web page elements such as titles, text blocks, images, and buttons. The model first extracts multi-scale features from the target image. These features include not only local details but also an understanding of the global structure. Next, a design rule utilization module is used, which integrates pre-defined web page design rules, such as titles typically being at the top of the page with large fonts, and images often being displayed alongside or centered with text. This further interprets the extracted features, generating a series of possible reference semantic categories. For example, an element region with large fonts located at the top of the page might be initially identified as a "title" category. Subsequently, multiple base classifiers in the model begin to work, each focusing on different tasks; some specialize in recognizing text, while others excel at recognizing images. Guided by the design rules and multi-scale features, these base classifiers carefully examine candidate element regions, combining their shape, size, position, and content features to ultimately output multiple initial element recognition results.

[0081] In one alternative embodiment, a multimodal fusion recognition framework can be constructed. Specifically, this approach combines deep learning-based visual feature extraction with traditional design rule analysis to create complementary advantages. First, visual features of candidate element regions are extracted from different scales using a convolutional neural network. These features include not only local details but also global structural features. Then, a design rule utilization module generates preliminary semantic category judgments based on built-in web page layout element design rules, such as relative positions, size ratios, color schemes, and common layout patterns between elements. This step provides valuable contextual clues for subsequent recognition. Finally, the visual features and reference semantic categories are input into multiple pre-trained base classifiers (such as support vector machines and random forests). Each classifier identifies elements of a specific type or level. Ultimately, a comprehensive element recognition result is obtained through a fusion strategy (such as weighted voting or model stacking), improving the accuracy and robustness of the recognition.

[0082] In the above process, the integration of design rules into the target image visual recognition model significantly enhances the model's analysis and understanding of web page layout elements. Through the comprehensive application and guidance of multi-scale features, it effectively combines design theory with actual visual data, greatly improving the recognition accuracy and efficiency of the base classifier.

[0083] Optionally, the method further includes: acquiring a training page image set; using an annotation tool to annotate the web page layout elements in the training page image set to obtain an annotated page image set; performing data augmentation operations on the annotated page image set to obtain an augmented page image set; and training the initial image visual recognition model based on the augmented page image set to obtain the target image visual recognition model.

[0084] For example, in the specific implementation process, a large number of webpage screenshots were first collected to form a training page image set. These screenshots covered various webpage design and layout styles to ensure the model's generalization ability. Next, using professional annotation tools (such as LabelImg), each webpage layout element in these page image sets, such as titles, paragraphs, images, and lists, was meticulously annotated, generating an annotated page image set. Subsequently, to further improve the model's robustness and adaptability, data augmentation operations were performed on the annotated page image set, including rotation, scaling, and color adjustment, thus generating an enhanced page image set. This operation effectively increased the diversity of the training data. Finally, based on the enhanced page image set, the initial image visual recognition model, such as a convolutional neural network, was deeply trained and optimized. Weights were adjusted through backpropagation, ultimately yielding the target image visual recognition model. This model can accurately identify and classify webpage layout elements, providing strong technical support for understanding and automating webpage content processing.

[0085] In one alternative embodiment, after acquiring the training page image set, transfer learning is used to optimize the initial image visual recognition model. Specifically, a deep learning model can first be pre-trained on a large-scale general image dataset to learn rich image features. Then, the weights of this pre-trained model are used as the weights of the initial model, and fine-tuned using the labeled page image set. This allows the model to perform better in recognizing web page layout elements because the pre-trained model already has the ability to handle complex images; it only needs fine-tuning to adapt to the specific web page layout recognition task, thus obtaining the target image visual recognition model.

[0086] In another alternative embodiment, an attention mechanism is introduced during training. This mechanism helps the model focus on the most important parts of an image while ignoring irrelevant or distracting information. Specifically, bounding boxes can be added around each element in the labeled page image set, and the network structure can be designed to learn these bounding boxes during the training of the target image visual recognition model, thus automatically focusing on key regions during prediction. Furthermore, trainable attention layers, such as self-attention mechanisms, can be used to dynamically adjust the model's attention to different parts of the image. This allows the model to effectively identify and locate webpage layout elements even with complex and varied page layouts, further improving the accuracy and robustness of recognition.

[0087] Through the aforementioned steps, the method significantly improves the accuracy and generalization ability of the image visual recognition model. First, the collected training page image set covers diverse webpage layout elements, strengthening the model's learning foundation. Second, manual annotation ensures the accuracy and reliability of the training data. Third, data augmentation increases the model's adaptability to different image variations, making it more stable and reliable in practical applications. Finally, trained on the augmented dataset, the target image visual recognition model can more accurately and efficiently identify and understand complex page structures, providing strong support for automated webpage analysis and content extraction.

[0088] Optionally, obtaining the page image of the page to be identified includes: using an automated tool to access the page to be identified, waiting for a preset time, and then performing a screen capture on the page to be identified to obtain the page image.

[0089] For example, an automated tool based on Selenium WebDriver can be used to implement the page image acquisition step in the web layout element recognition method. First, WebDriver is launched, and its target is set to the page to be recognized. Next, to ensure the page is fully loaded, WebDriver waits for a preset 5 seconds (this time can be adjusted according to the page's loading speed). During this waiting period, WebDriver monitors the page status to ensure all dynamic content and resources have been loaded. Then, WebDriver's script execution function triggers a screen capture operation on the current page. This step simulates the user's actual screenshot action in the browser, thus obtaining a complete page image. Finally, the captured page image is saved as a PNG file and used as input data for the subsequent layout element recognition process.

[0090] The Selenium WebDriver mentioned above is a tool for automating web application testing. It provides a higher-level Application Programming Interface (API) that can control browser operations, such as opening web pages, clicking buttons, and filling out forms, thereby achieving automated testing.

[0091] In one alternative embodiment, an image of the page to be identified is obtained using the browser's headless mode. This approach leverages the headless functionality provided by modern browsers, allowing the browser to run without any graphical user interface, thus enabling page rendering and screenshotting in server environments or resource-constrained devices.

[0092] In another alternative embodiment, cloud services are utilized to obtain images of the page to be identified. By submitting the page URL to cloud services, these services can render the page using various browsers within their server environments and provide functions such as screenshots and page snapshots. In the above process, automated tools efficiently access the page to be identified and obtain clear and complete page images, providing high-quality raw data for subsequent image recognition, analysis, and processing, greatly improving work efficiency and accuracy.

[0093] A preferred embodiment of this application will be described in detail below. The method includes the following steps:

[0094] Data preparation and model training (offline phase):

[0095] Step 1, Data Collection: Massively crawl and capture screenshots of different types of web pages (such as e-commerce, news, portals, and backend management systems) to form a dataset. The data should cover different resolutions, browsers, and scrolling states.

[0096] Step 2, Data Annotation: Use an annotation tool (such as LabelImg) to annotate the layout elements in each screenshot. Annotation information includes: bounding boxes (precisely outlining each element to be identified with a rectangle), and category labels (assigning a semantic label to each bounding box). The categories defined in this embodiment include: Header, Title, Paragraph, Button, Image, Input Field, Navigation Bar, Footer, List, and Icon.

[0097] Step 3, Data Augmentation: Augment the training data to improve the model's robustness, including: random rotation (small angle), scaling, color jitter (brightness, contrast), adding Gaussian noise, simulating scaling at different resolutions, etc.

[0098] Step 4, Model Training: Select a pre-trained object detection model (such as YOLOv8s) as the base. Input the prepared dataset (training set) into the model and fine-tune the network weights using the backpropagation algorithm. The training objective is to enable the model to minimize the positional error between the predicted bounding box and the ground truth bounding box, as well as the error in class prediction.

[0099] Web page layout element recognition (online inference stage):

[0100] Step 1, Image Acquisition: Use automated tools (such as Selenium or Puppeteer) to access the target webpage and wait for the page to fully load (including all asynchronous requests and dynamic rendering). Capture a screenshot of the entire webpage and save it as an RGB image file (such as PNG).

[0101] Step 2, Image Preprocessing: Scale the screenshot to the fixed input size required by the model (e.g., 640x640 pixels) to achieve size standardization. Maintain the original aspect ratio during scaling and pad the edges to avoid image distortion. Normalization: Normalize pixel values ​​from [0,255] to [0,1] or perform standardization to accelerate model convergence and inference.

[0102] Step 3, Element Detection and Classification: The preprocessed edge image is input into the trained YOLOv8 model. The model performs forward inference and directly outputs a series of detection results. Each result includes: Bounding box coordinates: (x_center, y_center, width, height), representing the position and size of the element in the image. Class confidence: representing the probability that the bounding box belongs to a certain class (e.g., "Button"). Class label: the predicted class name.

[0103] Step 4, Post-processing: Set a confidence threshold (e.g., 0.5) to filter out all predicted boxes with confidence levels below this threshold, thus removing low-quality detection results and achieving confidence filtering. Apply non-maximum suppression to merge bounding boxes that highly overlap in location and are predicted to belong to the same category, retaining only the one with the highest confidence level, thereby solving the problem of repeated detection of the same element. Calculate the absolute coordinates of the bounding boxes on the scaled and padded image back to the original screenshot resolution.

[0104] Step 5, Output the results: Organize the final recognition results into a structured data format, such as JSON.

[0105] Figure 2 This is a schematic diagram of generating an edge image according to an embodiment of this application, such as... Figure 2The process is as follows: Start; Import image; Perform grayscale conversion, Gaussian filtering, and pixel gradient calculation on the image; Then, based on the gradient direction region, obtain region 1, region 2, region 3, and gradient-free region 4; Interpolate the gradient strength of adjacent pixels in the positive and negative gradient directions for regions 1, 2, and 3; the pixel corresponding to gradient-free region 4 is not an edge; Then, determine the maximum gradient strength; If the result is yes, perform a double thresholding of the gradient strength; If the result is no, the pixel is not an edge; If the double thresholding of the gradient strength is less than the lower threshold, the pixel is not an edge; If the double thresholding of the gradient strength is greater than the higher threshold, the pixel is an edge; If the double thresholding of the gradient strength is between the high and low thresholds, perform an isolation test; If the isolation test shows that adjacent pixels are not edges, the corresponding pixel is not an edge; If the isolation test shows that adjacent pixels have edges, the corresponding pixel is an edge; Finally, generate the edge image; End.

[0106] According to another aspect of the present invention, an embodiment of a web page layout element recognition device based on image analysis is also provided. It should be noted that the device can execute the web page layout element recognition method based on image analysis of the above embodiments. The specific implementation method and preferred application scenarios are the same as those of the above embodiments, and will not be repeated here. Figure 3 This is a schematic diagram of a webpage layout element recognition device based on image analysis according to an embodiment of this application, such as... Figure 3 As shown, the device includes the following: an acquisition module 302, a determination module 304, and an identification module 306.

[0107] The acquisition module 302 is used to acquire a page image of the page to be identified; the determination module 304 is used to determine the candidate element region of the page image, wherein the candidate element region contains at least one web page layout element; the recognition module 306 is used to use a target image visual recognition model to recognize the candidate element region and obtain the element recognition result, wherein the element recognition result includes at least the bounding box coordinates of the web page layout element, the predicted semantic category and the category confidence, and the category confidence is used to characterize the probability that the web page layout element belongs to the predicted semantic category.

[0108] The recognition module is further configured to use multiple base classifiers in the target image visual recognition model to recognize candidate element regions and obtain multiple initial element recognition results. The multiple initial element recognition results include at least multiple initial bounding box coordinates, multiple initial predicted semantic categories, and multiple initial category confidences. The multiple initial element recognition results are then filtered to obtain the element recognition result.

[0109] The identification module is also used to filter multiple initial element identification results based on a preset confidence threshold and multiple initial category confidence levels to obtain multiple intermediate element identification results; and to merge the multiple intermediate element identification results to obtain the element identification result.

[0110] The recognition module is also used to obtain at least two intermediate element recognition results with the same initial predicted semantic category from multiple intermediate element recognition results; and to merge intermediate element recognition results whose difference between the initial bounding box coordinates is less than a preset threshold from at least two intermediate element recognition results to obtain element recognition results.

[0111] The recognition module is also used to extract features from web layout elements of different scales in multiple web layout elements using the feature pyramid network module, and obtain multiple scale features; based on the multiple scale features, it guides multiple base classifiers to recognize candidate element regions and obtain multiple initial element recognition results.

[0112] The identification module is also used to identify multiple web page layout elements based on the built-in element design rules, thereby obtaining multiple reference semantic categories; based on multiple scale features and multiple reference semantic categories, it guides multiple base classifiers to identify candidate element regions, thereby obtaining multiple initial element identification results.

[0113] The acquisition module is further used to acquire a training page image set; to annotate the web page layout elements in the training page image set using an annotation tool to obtain an annotated page image set; to perform data augmentation operations on the annotated page image set to obtain an augmented page image set; and to train the initial image visual recognition model based on the augmented page image set to obtain the target image visual recognition model.

[0114] The acquisition module is also used to access the page to be identified using automated tools, wait for a preset time, and then perform screen capture processing on the page to be identified to obtain the page image.

[0115] Embodiments of this application also provide an electronic device, including: a memory storing an executable program; and a processor for running the program, wherein the program executes the methods in various embodiments of the present invention during runtime.

[0116] Embodiments of this application also provide a computer-readable storage medium including a stored executable program, wherein, when the executable program is running, it controls the device where the computer-readable storage medium is located to perform the methods of various embodiments of the present invention.

[0117] Embodiments of this application also provide a computer program product, including a computer program that, when executed by a processor, implements the methods of various embodiments of the present invention.

[0118] Embodiments of this application also provide a computer program product, including a non-volatile computer-readable storage medium for storing a computer program that, when executed by a processor, implements the methods in various embodiments of the present invention.

[0119] Embodiments of this application also provide a computer program that, when executed by a processor, implements the methods described in the various embodiments of the present invention.

[0120] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0121] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0122] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0123] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0124] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0125] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for identifying layout elements of a web page based on image analysis, characterized in that, The method comprises: obtaining a page image of a page to be recognized; determining a candidate element region of the page image, wherein the candidate element region contains at least one web page layout element; using a target image visual recognition model to recognize the candidate element region to obtain an element recognition result, wherein the element recognition result at least includes a bounding box coordinate of the web page layout element, a predicted semantic category, and a category confidence, and the category confidence is used to represent a probability that the web page layout element belongs to the predicted semantic category.

2. The web page layout element identification method of claim 1, wherein, using a target image visual recognition model to recognize the candidate element region to obtain an element recognition result, comprising: using a plurality of base classifiers in the target image visual recognition model to recognize the candidate element region to obtain a plurality of initial element recognition results, wherein the plurality of initial element recognition results at least include a plurality of initial bounding box coordinates, a plurality of initial predicted semantic categories, and a plurality of initial category confidences; screening the plurality of initial element recognition results to obtain the element recognition result.

3. The web page layout element identification method of claim 2, wherein, screening the plurality of initial element recognition results to obtain the element recognition result, comprising: screening the plurality of initial element recognition results based on a preset confidence threshold and the plurality of initial category confidences to obtain a plurality of intermediate element recognition results; merging the plurality of intermediate element recognition results to obtain the element recognition result.

4. The web page layout element identification method of claim 3, wherein, merging the plurality of intermediate element recognition results to obtain the element recognition result, comprising: obtaining at least two intermediate element recognition results corresponding to the same initial predicted semantic category in the plurality of intermediate element recognition results; merging the intermediate element recognition results corresponding to the initial bounding box coordinates with a difference less than a preset threshold to obtain the element recognition result.

5. The web page layout element identification method of claim 3, wherein, The candidate element region contains a plurality of web page layout elements, and the target image visual recognition model comprises a feature pyramid network module; using a plurality of base classifiers in the target image visual recognition model to recognize the candidate element region to obtain a plurality of initial element recognition results, comprising: using the feature pyramid network module to extract features of web page layout elements of different scales in the plurality of web page layout elements to obtain a plurality of scale features; based on the plurality of scale features, guiding the plurality of base classifiers to recognize the candidate element region to obtain the plurality of initial element recognition results.

6. The web page layout element identification method of claim 5, wherein, The target image visual recognition model comprises a design rule utilization module; based on the plurality of scale features, guiding the plurality of base classifiers to recognize the candidate element region to obtain the plurality of initial element recognition results, comprising: using the design rule utilization module to recognize the plurality of web page layout elements based on built-in element design rules to obtain a plurality of reference semantic categories; based on the plurality of scale features and the plurality of reference semantic categories, guiding the plurality of base classifiers to recognize the candidate element region to obtain the plurality of initial element recognition results.

7. The web page layout element identification method of claim 1, wherein, The method further comprises: obtaining a set of training page images; The training page image set is labeled by using a labeling tool to obtain a labeled page image set; The labeled page image set is subjected to a data enhancement operation to obtain an enhanced page image set; The initial image visual recognition model is trained based on the enhanced page image set to obtain the target image visual recognition model.

8. The web page layout element identification method of claim 1, wherein, A page image of a page to be recognized is obtained, including: An automatic tool is used to access the page to be recognized, and after waiting for a preset time, a screenshot processing is performed on the page to be recognized to obtain the page image.

9. An electronic device, comprising: It includes: A memory storing an executable program; A processor configured to execute the program, wherein the program, when executed, performs the image analysis-based web page layout element recognition method of any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored executable program, wherein the executable program, when executed, controls the device where the storage medium is located to perform the image analysis-based web page layout element recognition method of any one of claims 1 to 8.

Images