Application and Web Page Classification and Recognition Methods Based on Multimodal Feedback Verification
By collecting and processing page screenshots, elements, and OCR text, multimodal features are extracted and consistency verification is performed. This solves the credibility problem of multimodal classification models under local dynamic content interference, and achieves more accurate page classification and continuous adaptability.
Patent Information
- Application Number
- CN202611401257.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-10
- Publication Date
- 2026-11-17
AI Technical Summary
Existing application and webpage multimodal classification models are difficult to effectively correct and verify feedback under local dynamic content interference, resulting in insufficient reliability of classification results.
By collecting page screenshots, page elements, and OCR text, a multimodal dataset is formed. Text, image, and spatial structure features are extracted, cross-modal fusion and consistency verification are performed, multimodal conflict pages are identified, candidate regions of the main content are determined for reclassification, and a review task is generated and fed back to the training sample set.
It enhances the comprehensive representation capability of page business information, reduces the interference of local dynamic content on the overall classification results, improves the system's adaptability to content changes, and achieves reliable classification correction and verification feedback.
Smart Images

Figure CN123119480A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of intelligent page classification technology, specifically to an application and webpage classification and recognition method based on multimodal feedback verification. Background Technology
[0002] With the continuous development of mobile internet and web applications, applications and web pages now carry a wide range of business content, including news, e-commerce, lifestyle services, financial services, education, and social entertainment. To support content management, application distribution, risk monitoring, search and recommendation, and operational analysis, it is typically necessary to combine page screenshots, page element structure, and page text information to automatically identify and classify the business categories of applications and web pages.
[0003] For example, Chinese patent CN114492651B discloses a semi-supervised graph node classification method based on personalized webpage ranking. Step 1: For a given graph, the personalized webpage ranking value of each node is calculated using a personalized webpage ranking algorithm, thus obtaining the ranking distribution vector corresponding to each tag. The dependency vector of each node is obtained based on the ranking distribution vector, and the dependency relationship matrix is obtained after regularization. Step 2: By stacking multiple layers of the hidden layers of the graph neural network, the hidden layer matrix output by the graph neural network after stacking multiple hidden layers is obtained. Step 3: The dependency relationship matrix is added to the last hidden layer matrix of the graph neural network to obtain a new global hidden matrix, thus obtaining a new graph neural network. Step 4: The new graph neural network is used for node classification. This invention ultimately uses the newly constructed graph neural network for node classification, achieving better node classification results.
[0004] For example, Chinese patent CN117456217A discloses a webpage classification and detection method, apparatus, electronic device, and storage medium, including: acquiring webpage text data and webpage image data respectively from the webpage; extracting text features from the webpage text data using a text encoding model, and extracting image features from the webpage image data using an image encoding model; performing comparative learning between text features and image features to obtain a first loss function; performing self-supervised learning on text features and image features respectively to obtain a second loss function; obtaining a joint loss function based on the first and second loss functions; determining a pre-trained classification model when the value of the joint loss function is less than a preset threshold; and using the pre-trained classification model to determine the category of the webpage to be detected. Through the above scheme, this invention achieves multi-modal pre-trained webpage detection including images and text, reducing detection costs and improving the robustness and accuracy of the pre-trained model.
[0005] However, existing multimodal classification models for apps and web pages typically fuse OCR text, interface images, and spatial structure information directly, outputting the page classification result based solely on the probability of the highest fused category. When a page contains main business content, advertising aggregation areas, promotional materials, pop-ups, or other local dynamic content simultaneously, text, image, and spatial structure modalities may each support different classification categories. Relying solely on the probability of the highest fused category makes it difficult to determine whether the classification result receives common support from multiple modalities. For pages with high confidence in fused classification but insufficient modal support, existing methods struggle to identify the main content and reclassify pages with multimodal conflicts, resulting in a lack of reliable verification, classification correction, and review feedback for page classification results affected by local content interference.
[0006] Therefore, to address the above issues, there is an urgent need for applications and web page classification and recognition methods based on multimodal feedback verification. Summary of the Invention
[0007] Technical problems to be solved To address the shortcomings of existing technologies, this invention provides an application and webpage classification and recognition method based on multimodal feedback verification. This method solves the problems in existing multimodal classification processes for applications and webpages, such as conflicting classification criteria caused by local dynamic content interference, insufficient reliability of page classification results, and difficulty in effective correction and verification feedback.
[0008] Technical solution To achieve the above objectives, the present invention provides the following technical solution: an application and webpage classification and recognition method based on multimodal feedback verification, comprising the following steps: S1, collecting screenshots, page elements, and OCR text of the page to be classified; cleaning, unifying the types of the page elements and OCR text, and associating the text elements to form a page state multimodal dataset; S2, based on the page state multimodal dataset, extracting text modal features, image modal features, and spatial structure modal features, and generating single-modal classifications based on the text modal features, image modal features, and spatial structure modal features. S3: Perform cross-modal fusion to obtain the initial classification category and the fusion classification confidence; S4: Combine the single-modal classification and the fusion classification confidence to perform consistency verification on the initial classification category, generate classification verification tags, and identify multimodal conflict pages; For multimodal conflict pages, combine page screenshots, page elements, and OCR text to evaluate the coverage of regional content and determine the main content candidate region, and reclassify based on the main content candidate region; S5: Determine the final classification category based on the classification verification tags, generate the classification review task, and feed the review results of the classification review task back to the classification training sample set.
[0009] Furthermore, the specific process of collecting page screenshots, page elements, and OCR text from the pages to be categorized is as follows: For each page to be categorized, a page status number is generated, and the collection timestamp and page source identifier are recorded; under the same page status number, the rendered image of the page to be categorized is obtained using the screenshot interface, and page screenshot records are collected. The page screenshot records include: page status number, image pixel width, image pixel height, page screenshot area, and image pixel matrix; using the page node tree export interface provided by the page runtime environment, the page node tree corresponding to the page screenshot is exported, and page element records are collected for each page node. The record includes: page status number, element number, parent element number, element type, element visibility marker, element left coordinate, element top coordinate, element width, element height, and element bounding box area; and converts the element left coordinate, element top coordinate, element width, and element height into data with the top left corner of the page screenshot as the origin; inputs the image pixel matrix in the page screenshot record into the OCR recognition model to obtain the OCR text record, which includes: page status number, OCR text record number, text content, text box left coordinate, text box top coordinate, text box width, text box height, text box area, and OCR confidence score.
[0010] Further, the specific process of cleaning, unifying the types, and associating text elements for page elements and OCR text to form a multimodal dataset of page states is as follows: When an element is marked as invisible, the corresponding page element record is deleted; the element type is mapped to a preset unified element type, including: text elements, image elements, button elements, input elements, container elements, pop-up elements, and status indicator elements; whitespace characters, line breaks, and control characters are deleted from the text content; when the deleted text content is empty, the corresponding OCR text record is deleted; for the remaining OCR text records, OCR text records with OCR confidence scores lower than the OCR confidence threshold are deleted; when the OCR confidence score is greater than or equal to the OCR confidence threshold, the corresponding OCR text record is marked as a valid OCR text record; for valid OCR text records, based on the element's left coordinate, element's top coordinate, element width, element height, and the text box's left coordinate, ... The coordinates, width, and height of the text box are used to calculate the intersection area between the text box and the bounding boxes of each element. The text coverage rate is obtained by dividing the intersection area of the text box with each element's bounding box by the text box area. For each valid OCR text record, page element records with a text coverage rate greater than or equal to a text coverage rate threshold are selected, and the record with the smallest bounding box area is chosen as the associated page element for the corresponding valid OCR text record. A text element association record is generated, including the page status number, OCR text record number, and element number. If no page element record has a text coverage rate greater than or equal to the text coverage rate threshold, no text element association record is generated for the corresponding valid OCR text record. Page screenshot records, page element records, valid OCR text records, and text element association records under the same page status number are associated to form a page status multimodal dataset.
[0011] Furthermore, based on the page state multimodal dataset, the specific process of extracting text modal features, image modal features, and spatial structure modal features, and generating single-modal classifications respectively, is as follows: For valid OCR text records under the same page state number, the text content is arranged in ascending order according to the priority order of the coordinates on the top of the text box and the left coordinate of the text box to obtain the page text sequence; the page text sequence is input into a text feature extraction model pre-trained based on the Transformer architecture to obtain a text modal feature vector; the text modal feature vector is input into a pre-trained multilayer perceptron text classification model to obtain the text category probability corresponding to each category, and form a text category probability vector; for page screenshot records under the same page state number, the image pixel matrix is input into a convolutional neural network image feature extraction model to obtain an image modal feature vector; the image modal feature vector is input into a pre-trained multilayer perceptron image classification model to obtain the image category probability corresponding to each category, and form an image category probability vector; for each page element under the same page state number... The system records and calculates the ratios of the element's left coordinate to the image pixel width, the element's top coordinate to the image pixel height, the element's width to the image pixel width, the element's height to the image pixel height, and the element's bounding box area to the page screenshot area to obtain the element's positional size features. It also counts the number of records in the text element association records where the element number is the same as the element number of the current page element record to obtain the number of associated texts. Based on the element type, element positional size features, and the number of associated texts, element structural features are generated. When the parent element number in one page element record equals the element number in another page element record, a parent-child relationship is determined between the two page element records. The element structural features and parent-child relationships of each page element record under the same page state number are input into a spatial structural feature extraction model built based on a graph convolutional network to obtain spatial structural modal feature vectors. These spatial structural modal feature vectors are then input into a pre-trained multilayer perceptron spatial structural classification model to obtain the structural category probabilities corresponding to each classification category, forming a structural category probability vector.
[0012] Furthermore, the specific process of cross-modal fusion based on text modal features, image modal features, and spatial structure modal features to obtain the initial classification category and fused classification confidence is as follows: Input the text modal feature vector, image modal feature vector, and spatial structure modal feature vector into the multimodal Transformer fusion model constructed based on the cross-attention mechanism to obtain the multimodal joint representation; input the multimodal joint representation into the pre-trained decision tree classification model to obtain the fused category probability corresponding to each classification category and form the fused category probability vector; take the classification category corresponding to the maximum fused category probability as the initial classification category and take the maximum fused category probability as the fused classification confidence.
[0013] Furthermore, combining unimodal classification and fused classification confidence, the initial classification categories are validated for consistency, classification validation tags are generated, and the specific process for identifying multimodal conflict pages is as follows: For the text category probability vector, image category probability vector, and structure category probability vector under the same page state number, the classification category corresponding to the largest category probability in each category probability vector is taken as the candidate classification category, resulting in text candidate classification category, image candidate classification category, and structure candidate classification category; if the text candidate classification category is the same as the initial classification category, a text support tag of 1 is generated, otherwise 0; if the image candidate classification category is the same as the initial classification category, a support tag of 1 is generated. Image support is assigned a value of 1, otherwise 0. If the candidate structural classification category is the same as the initial classification category, a structural support label of 1 is generated, otherwise 0. The sum of text support labels, image support labels, and structural support labels is calculated to obtain the number of modal supports. Classification verification labels are generated, including: when the fused classification confidence is less than the fused classification confidence threshold, a verification label is generated; when the fused classification confidence is greater than or equal to the fused classification confidence threshold and the number of modal supports is greater than or equal to the support threshold, a classification confidence label is generated; when the fused classification confidence is greater than or equal to the fused classification confidence threshold and the number of modal supports is less than the support threshold, a multimodal conflict label is generated.
[0014] Furthermore, for multimodal conflict pages, the specific process for evaluating the coverage of content in the evaluation area based on page screenshots, page elements, and OCR text, and determining the candidate areas for main content, is as follows: For the page to be classified that generates multimodal conflict markers, obtain the page element records of element types (container elements, image elements, and button elements) under the page status number, calculate the ratio of the bounding box area of each element to the area of the page screenshot, and obtain the area ratio of the region; obtain the text element association records of the corresponding elements and all child elements obtained by recursive traversal, and count the number of characters in the text content of the associated valid OCR text records to obtain the number of valid OCR characters in the region; count the number of valid OCR characters in all valid OCR text records under the same page status number. The total number of valid OCR characters on the page is obtained by counting the number of characters in the text content. The percentage of valid OCR characters in a region is obtained by dividing the number of valid OCR characters in the region by the total number of valid OCR characters on the page. The number of corresponding elements and all child elements obtained by recursion is obtained by counting the number of elements to obtain the number of regional structure nodes. The total number of page element records under the same page status number is obtained by counting the number of page structure nodes. The percentage of regional structure nodes is obtained by dividing the number of regional structure nodes by the total number of page structure nodes. The percentage of regional structure nodes is obtained by calculating the product of the regional area percentage, the regional text percentage, and the regional structure node percentage, and taking the cube root to obtain the regional content coverage value. The page element record with the largest regional content coverage value is selected as the candidate region for the main content.
[0015] Furthermore, the specific process of reclassifying based on the main content candidate region is as follows: Read the elements corresponding to the main content candidate region and all sub-elements obtained through recursive traversal to obtain the main region page element record; filter the valid OCR text records whose element numbers belong to the main region page element record from the text element association records to obtain the valid OCR text records of the main region; set the image pixels outside the bounding box of the corresponding element in the main content candidate region in the image pixel matrix to zero to obtain the main region image pixel matrix; and combine the main region image pixel matrix, the main region page element record, and the valid OCR text of the main region... The records and their corresponding text elements are associated to form a multimodal dataset of the main content region. Based on the multimodal dataset of the main content region, text modality feature vectors, image modality feature vectors, spatial structure modality feature vectors, and multimodal joint representations are extracted to obtain the main content classification category and the main content classification confidence score. When the main content classification confidence score is greater than or equal to the fusion classification confidence score threshold, the main content classification category is determined as the final classification category of the page to be classified. When the main content classification confidence score is less than the fusion classification confidence score threshold, the multimodal conflict marker is updated to a marker to be reviewed.
[0016] Furthermore, the specific process of determining the final classification category based on the classification verification mark and generating the classification review task is as follows: For the page to be classified that has generated a classification confidence mark, the initial classification category is taken as the final classification category of the page to be classified; for the page to be classified that has generated a review mark, a classification review task is generated, which includes: page status number, page source identifier, collection timestamp, page screenshot record, initial classification category, fusion classification confidence, text candidate classification category, image candidate classification category, structure candidate classification category, text support mark, image support mark, structure support mark, modality support quantity, and review mark.
[0017] Furthermore, the specific process of feeding back the review results of the classification review task to the classification training sample set is as follows: the classification review task is output to the review terminal, and the review classification category submitted by the review terminal is received, and the review classification category is used as the final classification category of the page to be classified; for the page to be classified with the final classification category, a classification output record is generated, including: page status number, page source identifier, collection timestamp, initial classification category, final classification category, and final classification source identifier; the final classification source identifier includes: initial classification result, main content classification result, and review classification result; for the classification output record with the final classification source identifier being the review classification result, the page status multimodal dataset is associated with the review classification category to generate review feedback samples; the review feedback samples are added to the classification training sample set, and the model is updated and trained using the review classification category as the classification label.
[0018] Beneficial effects The present invention has the following beneficial effects: (1) This invention uses the association processing of page screenshots, page elements and OCR text to classify text content, visual images and page spatial structure in a unified page state, thereby improving the comprehensive representation capability of page business information.
[0019] (2) By generating single-modal classification results of text, image and spatial structure respectively and verifying consistency with the fusion classification results, this invention can identify classification results with high fusion confidence but lacking multimodal support, thus avoiding direct misclassification output caused by single-modal dominance.
[0020] (3) For multimodal conflict pages, the present invention determines the candidate area of the main content based on the content coverage of the page area, and reclassifies based on the main content area. This can reduce the interference of local dynamic content such as advertisements, pop-ups, and promotional materials on the overall classification result of the page, and make the final classification result closer to the main business category of the page.
[0021] (4) This invention generates a review task for pages with low confidence or unable to complete the credible classification of the main content, and feeds back the review classification category to the classification training sample set, realizing a closed loop between the manual review results and the classification model update, and improving the system's ability to continuously adapt to changes in page content and new page styles.
[0022] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description
[0023] Figure 1 This is a flowchart of an application and webpage classification and recognition method based on multimodal feedback verification. Figure 2 This is a schematic diagram illustrating multimodal representation fusion and initial page classification. Figure 3 A schematic diagram of the multimodal classification feedback process for applications and web pages. Detailed Implementation
[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. As those skilled in the art will understand, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0025] Please see Figures 1-3 This invention provides a technical solution: a method for classifying and recognizing applications and web pages based on multimodal feedback verification, such as... Figure 1 As shown, the process includes the following steps: S1, collecting screenshots, page elements, and OCR text of the page to be classified; cleaning, unifying the types of the page elements and OCR text, and associating the text elements to form a page state multimodal dataset; S2, based on the page state multimodal dataset, extracting text modal features, image modal features, and spatial structure modal features, and generating unimodal classifications respectively; performing cross-modal fusion based on text modal features, image modal features, and spatial structure modal features to obtain the initial classification category and the fusion classification confidence score; S3, combining the unimodal classification and the fusion classification confidence score, performing consistency verification on the initial classification category, generating classification verification labels, and identifying multimodal conflict pages; for multimodal conflict pages, evaluating the coverage of regional content based on the page screenshots, page elements, and OCR text, determining candidate regions of main content, and reclassifying based on the candidate regions of main content; S4, determining the final classification category based on the classification verification labels, generating a classification verification task, and feeding back the verification results of the classification verification task to the classification training sample set.
[0026] Specifically, the process of collecting screenshots, page elements, and OCR text of the page to be categorized is as follows: For the page to be categorized, a page status number is generated, and the collection timestamp and page source identifier are recorded. Under the same page status number, the rendered screen of the page to be categorized is obtained using the screenshot interface, and a page screenshot record is collected. The page screenshot record includes: page status number, image pixel width, image pixel height, page screenshot area, and image pixel matrix. The page status number uniquely identifies the page display state of the page to be categorized at the corresponding collection time; the page source identifier identifies the application or website to which the page to be categorized belongs. The page screenshot interface can be a screen acquisition interface provided by the application runtime environment and the web browser runtime environment, used to obtain the rendered screen of the current page display area. The image pixel width and image pixel height represent the horizontal and vertical pixel counts of the rendered screen, respectively. The page screenshot area is the product of the image pixel width and image pixel height. The image pixel matrix records the pixel values corresponding to each pixel position in the page screenshot. In this embodiment, after acquiring the page screenshot record, the corresponding page node tree is exported within a preset page state retention time, so that the page screenshot record corresponds to the same page display state as the subsequently acquired page element record. When the page jumps, refreshes, switches pop-ups, or the page content is replaced during the acquisition process, the page state number is regenerated and the page screenshot and page node tree acquisition process is re-executed. Using the page node tree export interface provided by the page runtime environment, the page node tree corresponding to the page screenshot is exported. For each page node, page element records are acquired. The page element records include: page state number, element number, parent element number, element type, element visibility marker, element left coordinate, element top coordinate, element width, element height, and element bounding box area. The element left coordinate, element top coordinate, element width, and element height are all converted into data with the top left corner of the page screenshot as the origin. The page runtime environment includes the application runtime environment and the web browser runtime environment. In this embodiment, the page node tree export interface is used to output the structured node data corresponding to the current page. The structured node data includes node identifier, parent node identifier, original element type, visibility state, left coordinate of the bounding box, top coordinate of the bounding box, width of the bounding box, and height of the bounding box. Element numbers are generated based on node identifiers, parent element numbers are generated based on parent node identifiers, and the corresponding original element types are generated. For application pages, the page node tree corresponding to the currently displayed interface is obtained through the interface node tree export interface provided by the application runtime environment; for web pages, the page node tree corresponding to the current browser viewport is obtained through the document object model node export interface provided by the web browser runtime environment; for web pages embedded in the application, the page node tree is obtained through the document object model node export interface provided by the WebView runtime environment.For application pages, the interface coordinates of page nodes are converted to pixel coordinates relative to the top-left corner of the page screenshot; for web pages, the viewport coordinates of page nodes are converted to pixel coordinates relative to the top-left corner of the page screenshot. This unifies the page node tree obtained from different page runtime environments into page element records. The page node tree represents the hierarchical relationship between page elements in the current page. Element ID is used to uniquely identify a page element, and parent element ID is used to identify the parent page element to which the current page element belongs; when a page element has no parent page element, its parent element ID is set to null. The left and top coordinates of the element represent the position of the top-left corner of the page element's bounding box relative to the top-left corner of the page screenshot; the element width and height represent the horizontal and vertical pixel lengths of the page element's bounding box in the pixel coordinate system of the page screenshot; the area of the element's bounding box is the product of the element's width and height. The element visibility flag indicates whether the page element is in a displayable state at the corresponding moment in the current page screenshot. The left coordinate, top coordinate, width, and height of elements in the page element record are all unified according to the pixel coordinate system corresponding to the page screenshot. This allows the page element record to be positionally associated with the image pixel matrix in the page screenshot record. Based on the logical width and height of the page rendering viewport, as well as the pixel width and height of the page screenshot image, the horizontal and vertical coordinate transformation ratios are determined. The page element bounding boxes are transformed to the page screenshot pixel coordinate system according to these ratios, and any parts exceeding the page screenshot boundaries are cropped to obtain the effective element bounding boxes and their areas. The image pixel matrix from the page screenshot record is input into the OCR recognition model to obtain OCR text records. These OCR text records include: page status number, OCR text record number, text content, left coordinate of the text box, top coordinate of the text box, text box width, text box height, text box area, and OCR confidence score. The preferred OCR recognition model is the pre-trained PP-OCR text recognition model, which includes a text detection network and a text recognition network. The text detection network locates text regions from the image pixel matrix and outputs text boxes corresponding to these regions. The text recognition network performs character recognition on the image pixels within each text box, outputting the text content and the recognition probability of each output character. For each text box, the characters with the highest recognition probabilities among the output characters are connected in recognition order to obtain the corresponding text content. The average of the maximum recognition probabilities of each output character in the corresponding text content is determined as the OCR confidence score. The left and top coordinates of the text box are the coordinates of the top-left corner of the text box in the pixel coordinate system of the page screenshot. The width and height of the text box are the pixel lengths of the text box in the pixel coordinate system of the page screenshot. The area of the text box is the product of its width and height.Since the input to the OCR recognition model is the image pixel matrix in the page screenshot record, the left coordinate, top coordinate, width, and height of the text box in the OCR text record are in the same page screenshot pixel coordinate system as the left coordinate, top coordinate, width, and height of the element in the page element record. Therefore, subsequent text element association processing can be performed based on the positional relationship between the text box and the element bounding box. The OCR text record number is used to uniquely identify an OCR recognition result. The left and top coordinates of the text box represent the position of the top-left corner of the recognized text box relative to the top-left corner of the page screenshot. The width and height of the text box represent its horizontal and vertical pixel lengths, respectively. The area of the text box is the product of its width and height. The OCR confidence score represents the reliability of the OCR recognition model's recognition result for the corresponding text content. The higher the OCR confidence score, the higher the probability that the corresponding text content matches the actual text content in the page screenshot. Since both the OCR text record and the page screenshot record are generated based on the same image pixel matrix, the left and top coordinates of the text box and the left and top coordinates of the element in the page element record are in the same page screenshot pixel coordinate system.
[0027] In this implementation plan, page screenshots, page node trees, and OCR text are collected simultaneously under the same page state, and the pixel coordinate system of page elements and text boxes is unified, so that the visual appearance of the page, the hierarchical structure of elements, and the text content can be accurately matched. At the same time, page state changes are recorded based on page state number, collection timestamp, and page source identifier, which can avoid cross-state data mixing caused by page jumps, refreshes, or pop-up switching, and provide a consistent and traceable data foundation for subsequent text element association, multimodal feature extraction, and page classification.
[0028] Specifically, the process of cleaning, unifying the types of, and associating text elements with page elements and OCR text to form a multimodal dataset of page states is as follows: When an element's visibility marker is changed to invisible, the corresponding page element record is deleted; the element type is mapped to a preset unified element type, including: text elements, image elements, button elements, input elements, container elements, pop-up elements, and status indicator elements; the element visibility marker can be determined by the page runtime environment based on the display state of the page element. For page element records that have not been deleted, the corresponding original element types are merged into the preset unified element type according to the original element types in the page node tree. For example, original text elements such as titles, paragraphs, and tags are mapped to text elements; original visual elements such as images and icons are mapped to image elements; original controls that can trigger click operations are mapped to button elements; original controls that can input content, such as input boxes and search boxes, are mapped to input elements; original layout nodes used to organize other page elements are mapped to container elements; and tooltips, dialog boxes, or overlay nodes that cover the main content of the page are mapped to pop-up elements. Page elements used to display page loading status, processing progress, selection status, or message prompt status, and not used for text input or click operations, are mapped to status indicator elements, including loading indicators, progress bars, status labels, and prompt icons. The process removes whitespace characters, line breaks, and control characters from the text content. If the resulting text is empty, the corresponding OCR text record is deleted. For any remaining OCR text records, those with an OCR confidence score lower than the OCR confidence threshold are deleted. Records with an OCR confidence score greater than or equal to the threshold are marked as valid OCR text records. Whitespace characters include spaces, tabs, and consecutive blank lines; control characters include character codes that do not correspond to displayable text content. The OCR confidence threshold is pre-determined based on the OCR recognition accuracy of historical page screenshots and is used to eliminate OCR text records with incomplete text content, large text region positioning errors, or unstable recognition results. Valid OCR text records are used to participate in subsequent page text sequence construction, text element association, and main content region recognition. For valid OCR text records, the intersection area between the text box and the element bounding boxes is calculated based on the element's left coordinate, top coordinate, width, and height, as well as the text box's left coordinate, top coordinate, width, and height. The text coverage rate is obtained by dividing the intersection area of the text box with each element bounding box by the text box area. The intersection area is determined by measuring the overlap between the text box and the element bounding boxes in both the horizontal and vertical directions. When there is no overlap between the text box and the corresponding element bounding box, the corresponding text coverage rate is set to zero. The text coverage rate characterizes the degree to which the text box corresponding to a valid OCR text record is covered by the page element bounding boxes.For each valid OCR text record, page element records with a text coverage rate greater than or equal to a text coverage rate threshold are selected. The record with the smallest bounding box area is chosen as the associated page element for the corresponding valid OCR text record, generating a text element association record. This record includes: page status number, OCR text record number, and element number. The text coverage rate threshold is pre-determined based on the relationship between text boxes and page elements marked in historical page screenshots, ensuring the text element association accuracy reaches a preset lower limit. Selecting the page element record with the smallest bounding box area ensures that, when page elements have multiple nested relationships, the valid OCR text record is associated with the smallest level page element covering that text box, reducing the possibility of incorrectly associating text with a wide range of container elements. If no page element record with a text coverage rate greater than or equal to the text coverage rate threshold exists, no text element association record is generated for the corresponding valid OCR text record. Page screenshot records, page element records, valid OCR text records, and text element association records under the same page status number are associated to form a page status multimodal dataset. In the page state multimodal dataset, all records are associated through page state numbers, ensuring that the image pixels in the page screenshot, the spatial position and hierarchical relationship of page elements, and the text content obtained by OCR recognition all correspond to the same page display state, providing consistent data input for subsequent joint classification of text modality, image modality, and spatial structure modality.
[0029] In this implementation scheme, by performing visibility filtering, type unification, text reliability filtering, and text element association on page elements and OCR text, the text content in the page screenshot can be accurately mapped to page elements and their hierarchical structure. At the same time, by determining the OCR confidence threshold and text coverage threshold based on historical annotation data, the impact of low-quality recognized text and incorrect text attribution on subsequent processing can be reduced. This allows for the construction of a multimodal dataset of page states with consistent correspondence between images, text, and spatial structure, thereby improving the data reliability for subsequent page classification and main content recognition.
[0030] Specifically, based on the page state multimodal dataset, the process of extracting text modal features, image modal features, and spatial structure modal features, and generating single-modal classifications respectively, is as follows: First, a preset set of classification categories is read. This set is determined by a pre-established page business classification configuration table. The page business classification configuration table includes category numbers and category name enable tags, and the number of filtered classification categories is counted to obtain the preset number of classification categories. Each classification category corresponds to a unique category number. The multilayer perceptron text classification model, multilayer perceptron image classification model, multilayer perceptron spatial structure classification model, and decision tree classification model are all trained based on the same page business classification configuration table, and the number of output categories of each classification model is the same as the preset number of classification categories, so that each probability position in the text category probability vector, image category probability vector, structural category probability vector, and fused category probability vector corresponds to the same category number. For valid OCR text records under the same page status number, the text content is arranged in ascending order according to the priority of the text box coordinates and the left coordinates of the text boxes, resulting in a page text sequence. First, valid OCR text records are arranged in ascending order of text box coordinates. When two or more valid OCR text records have the same text box coordinates, they are then arranged in ascending order of text box left coordinates to ensure that the page text sequence corresponds to the top-to-bottom and left-to-right display order of the page text. When both text box coordinates and left coordinates are the same, they are arranged in ascending order of OCR text record number. Each text content in the page text sequence is connected using a preset separator. The page text sequence is then input into a text feature extraction model pre-trained based on the Transformer architecture to obtain text modality feature vectors. The text feature extraction model pre-trained based on the Transformer architecture preferably uses the BERT pre-trained text feature extraction model. The text feature extraction model includes a word embedding layer, a positional encoding layer, and at least one Transformer encoding layer. First, the page text sequence is converted into a word sequence according to a preset word segmentation rule. The word embedding layer is used to convert the word sequence into a word vector sequence, the positional encoding layer is used to represent the sequential position of each word in the page text sequence, and the Transformer encoding layer is used to extract contextual association features between words based on a self-attention mechanism. The word features output by the last Transformer encoding layer are then averaged and pooled to obtain a fixed-dimensional text modality feature vector. The text modality feature vector is input into a pre-trained multilayer perceptron text classification model to obtain the text category probability corresponding to each category, forming a text category probability vector.The multilayer perceptron text classification model comprises an input layer, at least one fully connected hidden layer, and an output layer. The input layer receives text modal feature vectors, the fully connected hidden layer performs a non-linear mapping on the text modal feature vectors, and the output layer has the same output dimension as the number of preset classification categories. The probability of each text category is obtained through a Softmax function. The multilayer perceptron text classification model is pre-trained based on the text modal feature vectors corresponding to historical page text sequences and the historical page classification categories, using the historical page classification categories as training labels. The model parameters are updated based on the cross-entropy loss between the classification prediction results and the training labels. For page screenshot records under the same page status number, the image pixel matrix is input into a convolutional neural network image feature extraction model to obtain image modal feature vectors. The preferred convolutional neural network image feature extraction model is the ResNet-50 residual convolutional neural network image feature extraction model. This model includes at least one convolutional layer, activation layer, pooling layer, and global average pooling layer. The convolutional layer extracts local texture, color distribution, and visual features of interface components from the page screenshot. The pooling layer reduces the feature map size, and the global average pooling layer converts the last layer's feature map into a fixed-dimensional image modality feature vector. The image modality feature vector is input to a pre-trained multilayer perceptron image classification model to obtain the image class probability corresponding to each classification category, forming an image class probability vector. The multilayer perceptron image classification model includes an input layer, at least one fully connected hidden layer, and an output layer. The input layer receives the image modality feature vector, and the fully connected hidden layer performs linear mapping and non-linear activation processing on the image modality feature vector. The output dimension of the output layer is the same as the number of preset classification categories, and the image class probability corresponding to each classification category is obtained through the Softmax function. The multilayer perceptron image classification model is pre-trained under supervision based on the image modal feature vectors corresponding to historical page screenshots and the historical page classification categories. The historical image modal feature vectors are used as the training input of the multilayer perceptron image classification model, and the corresponding historical page classification categories are used as the training labels. The model parameters are updated based on the cross-entropy loss between the image category probabilities output by the multilayer perceptron image classification model and the training labels.For each page element record under the same page status number, the ratios of the element's left coordinate to the image pixel width, the element's top coordinate to the image pixel height, the element's width to the image pixel width, the element's height to the image pixel height, and the element's bounding box area to the page screenshot area are calculated to obtain the element's position and size features. The number of records in the text element association records whose element numbers are the same as the element number of the current page element record is counted to obtain the element's associated text quantity. Based on the element type, element position and size features, and element associated text quantity, element structure features are generated. When the parent element number in one page element record is equal to the element number in another page element record, a parent-child relationship is determined between the two page element records. The element type is converted into element type features using a preset encoding method; the element position and size features and the element associated text quantity together constitute the element's numerical features. The element type features, element position and size features, and element associated text quantity are concatenated in a preset order to obtain the element structure features of the corresponding page element record. Preferably, for each page element record under the same page status number, a one-hot encoding is performed on the unified element type according to a fixed order of text element, image element, button element, input element, container element, pop-up element, and status indicator element to obtain a seven-dimensional element type feature; wherein, the position corresponding to the unified element type of the current page element record is set to 1, and the other positions are set to 0. The number of element-related texts is divided by the number of valid OCR text records under the same page status number to obtain the normalized number of element-related texts. The element type feature, element position size feature, and normalized number of element-related texts are concatenated in a fixed order to obtain the element structure feature of the corresponding page element record. The page element record is used as a graph node, and the parent-child relationship is used as the connection edge between the graph nodes. The element structure features and parent-child relationships of each page element record under the same page status number are input into a spatial structure feature extraction model built based on a graph convolutional network to obtain a spatial structure modal feature vector. Among them, the spatial structure feature extraction model based on graph convolutional networks preferably adopts the Kipf-Welling graph convolutional network spatial structure feature extraction model. The spatial structure feature extraction model includes graph convolutional layers and graph pooling layers. The graph convolutional layers aggregate the element structure features of adjacent page element records based on parent-child relationships to obtain the node representation of each page element record. The graph pooling layers aggregate the node representations under the same page state number to obtain a fixed-dimensional spatial structure modal feature vector. The spatial structure feature extraction model is pre-trained based on a historical page state multimodal dataset. The training input includes the element structure features and parent-child relationships corresponding to the historical page element records, and the training labels are the corresponding historical page classification categories. The model parameters of the graph convolutional network are updated according to the cross-entropy loss between the spatial structure classification prediction results and the training labels.The spatial structure modality feature vector is input into a pre-trained multilayer perceptron spatial structure classification model to obtain the structure class probability corresponding to each classification category, forming a structure class probability vector. The multilayer perceptron spatial structure classification model includes an input layer, at least one fully connected hidden layer, and an output layer. The input layer receives the spatial structure modality feature vector, and the fully connected hidden layer performs linear mapping and non-linear activation processing on the spatial structure modality feature vector. The output dimension of the output layer is the same as the number of preset classification categories, and the softmax function outputs the structure class probability corresponding to each classification category. The multilayer perceptron spatial structure classification model is pre-trained under supervised supervision based on the spatial structure modality feature vector corresponding to historical page states and the historical page classification categories. The historical spatial structure modality feature vector is used as the training input of the multilayer perceptron spatial structure classification model, and the corresponding historical page classification categories are used as training labels. The model parameters are updated based on the cross-entropy loss between the output structure class probabilities and the training labels.
[0031] In this implementation scheme, by unifying the business classification categories of pages and the correspondence between the output categories of each classification model, classification evidence is extracted from page text semantics, page visual content, and page element spatial structure, respectively, and comparable text category probability vectors, image category probability vectors, and structural category probability vectors are generated. Among them, the pre-trained text model and image model can utilize mature semantic and visual representation capabilities, and the graph convolutional network can represent the page element hierarchy and layout relationship. The three types of single-modal classification results can provide mutually independent and consistent classification basis for subsequent multimodal fusion classification and consistency verification, thereby improving the comprehensiveness, stability, and interpretability of application and web page classification.
[0032] Specifically, the process of cross-modal fusion based on text modality features, image modality features, and spatial structure modality features to obtain the initial classification category and fused classification confidence is as follows: The text modality feature vectors, image modality feature vectors, and spatial structure modality feature vectors are input into a multimodal Transformer fusion model constructed based on a cross-attention mechanism to obtain a multimodal joint representation. The multimodal Transformer fusion model constructed based on the cross-attention mechanism includes a feature dimension mapping layer, a cross-attention layer, a feedforward network layer, and a fusion output layer. The feature dimension mapping layer maps the text modality feature vectors, image modality feature vectors, and spatial structure modality feature vectors to the same preset feature dimension, respectively, to obtain text modality tokens, image modality tokens, and spatial structure modality tokens. The cross-attention layer uses each modality token as a query vector, key vector, and numerical vector, respectively, to extract the correlation features between different modalities. The feedforward network layer performs a nonlinear transformation on the correlation features output by the cross-attention layer. The fusion output layer aggregates the transformation results corresponding to each modality to obtain a fixed-dimensional multimodal joint representation. The multimodal Transformer fusion model is pre-trained on a multimodal dataset of historical page states with historical page classification categories. Specifically, the text modality feature vector, image modality feature vector, and spatial structure modality feature vector corresponding to the historical page states are input into the multimodal Transformer fusion model to obtain a joint historical multimodal representation. This joint historical multimodal representation is then input into the training classification layer to obtain the historical fusion classification prediction result. Using the corresponding historical page classification category as the training label, the model parameters of the feature dimension mapping layer, cross-attention layer, feedforward network layer, and fusion output layer are updated based on the cross-entropy loss between the historical fusion classification prediction result and the training label. After the model parameters are trained, the multimodal Transformer fusion model is used to output the multimodal joint representation. The multimodal joint representation is then input into a pre-trained decision tree classification model to obtain the fusion category probability corresponding to each classification category, forming a fusion category probability vector. The CART decision tree classification model is preferably used, including a root node, internal decision nodes, and leaf nodes. Each internal decision node divides the input samples based on the feature dimensions and corresponding splitting thresholds in the multimodal joint representation. Leaf nodes record the number of samples in each category from the historical training samples that fall into the corresponding leaf node. Dividing the number of samples in each category by the total number of training samples in the corresponding leaf node yields the fusion category probability for each category.The decision tree classification model is pre-trained based on historical multimodal joint representations and historical page classification categories corresponding to historical page states. The historical multimodal joint representations are used as training inputs, and the historical page classification categories are used as training labels. The feature dimensions and splitting thresholds of each internal decision node are determined based on the Gini index until preset tree depth, leaf node sample count, or node purity conditions are met, resulting in a pre-trained decision tree classification model. The historical page classification categories are determined from the classification categories in the page business classification configuration table. The classification category corresponding to the maximum fusion category probability is used as the initial classification category, and the maximum fusion category probability is used as the fusion classification confidence score. When there are two or more identical maximum fusion category probabilities in the fusion category probability vector, the classification category with the smallest category number is determined as the initial classification category to ensure the uniqueness of the initial classification category. The fusion classification confidence score characterizes the degree of classification support the decision tree classification model has for the initial classification category.
[0033] like Figure 2 The diagram illustrates multimodal representation fusion and initial page classification. The upper part of the diagram represents the multimodal feature extraction layer, corresponding to the processes of extracting text modal features based on valid OCR text records, extracting image modal features based on page screenshot records, and extracting spatial structure modal features based on page element records and parent-child relationships. The middle part represents the multimodal representation and classification generation layer. The cross-attention multimodal fusion unit performs correlation modeling on the three types of modal features to obtain a joint multimodal representation. The decision tree classifier outputs the fusion category probability corresponding to each classification category based on this joint multimodal representation. The initial classification category and fusion classification confidence on the right represent the page classification result corresponding to the maximum fusion category probability and the probability value of that result, respectively, providing a classification basis for subsequent multimodal consistency verification, main content reclassification, and manual review.
[0034] In this implementation scheme, the cross-attention mechanism is used to jointly model text, image and spatial structure features, which can extract the correlation information between different modalities and output the fusion classification probability of each page business category based on the decision tree classification model. At the same time, the fusion classification confidence is determined by the distribution of leaf node sample categories, so that the initial classification categories and their credibility have a clear data source, providing a reliable classification basis for subsequent multimodal consistency verification, main content reclassification and manual review.
[0035] Specifically, the process of combining unimodal classification and fusion classification confidence to perform consistency checks on the initial classification categories, generate classification verification tags, and identify multimodal conflicting pages is as follows: For the text category probability vector, image category probability vector, and structure category probability vector under the same page state number, the classification category corresponding to the largest category probability in each category probability vector is taken as the candidate classification category, resulting in text candidate classification category, image candidate classification category, and structure candidate classification category. Each probability position in the text category probability vector, image category probability vector, and structure category probability vector corresponds one-to-one with the category number in the page business classification configuration table, thus allowing the three candidate classification categories to be compared with the initial classification categories by the same category number. When there are two or more identical largest category probabilities in any category probability vector, the classification category with the smallest category number is determined as the corresponding candidate classification category to ensure the uniqueness of the candidate classification category determination result. If the candidate text classification category is the same as the initial classification category, a text support label of 1 is generated; otherwise, 0 is generated. Similarly, if the candidate image classification category is the same as the initial classification category, an image support label of 1 is generated; otherwise, 0 is generated. If the candidate structural classification category is the same as the initial classification category, a structural support label of 1 is generated; otherwise, 0 is generated. The sum of the text support label, image support label, and structural support label is calculated to obtain the modality support count. These labels characterize whether the text modality, image modality, and spatial structural modality independently support the initial classification category, respectively. The modality support count ranges from 0 to 3; a higher count indicates more single-modal classification support for the initial classification category. The generation of classification verification markers includes: generating a verification marker when the fusion classification confidence is less than the fusion classification confidence threshold; generating a reliable classification marker when the fusion classification confidence is greater than or equal to the fusion classification confidence threshold and the number of modal supports is greater than or equal to the support threshold; and generating a multimodal conflict marker when the fusion classification confidence is greater than or equal to the fusion classification confidence threshold and the number of modal supports is less than the support threshold. The fusion classification confidence threshold is pre-determined based on confirmed classification results in the historical page state multimodal dataset and is used to filter pages whose fusion classification results lack reliability. The support threshold is preferably set to 2, indicating that the initial classification category receives support from at least two modalities: text, image, and spatial structure. The verification marker, reliable classification marker, and multimodal conflict marker are mutually exclusive; only one type of classification verification marker is generated for each page to be classified.When a "to be reviewed" marker is generated, it indicates that the fusion classification model does not adequately support the initial classification category. When a "classification confidence" marker is generated, it indicates that the initial classification category simultaneously meets the fusion classification confidence requirement and the multimodal support requirement. When a "multimodal conflict" marker is generated, it indicates that the fusion classification confidence requirement is met, but the number of single modalities supporting the initial classification category is insufficient. Subsequently, candidate regions for the main content will be determined and reclassified based on this multimodal conflict marker.
[0036] In this implementation plan, by cross-validating the unimodal classification results of text, image, and spatial structure with the fusion classification results, it is possible to distinguish between pages with insufficient fusion classification confidence that need to be reviewed, credible pages that have obtained multimodal support, and conflicting pages with high fusion confidence but insufficient modal support. This avoids directly outputting classification results based solely on the highest fusion category probability, improves the credibility of page classification conclusions, and provides a clear triggering basis for subsequent main content reclassification and manual review.
[0037] Specifically, for multimodal conflict pages, the process of evaluating the coverage of content areas and determining candidate areas for main content by combining page screenshots, page elements, and OCR text is as follows: For pages to be classified that have generated multimodal conflict markers, obtain page element records with element types of container elements, image elements, and button elements under the page status number. Calculate the ratio of the bounding box area of each element to the area of the page screenshot to obtain the area proportion. Among them, container elements, image elements, and button elements are used as root elements of candidate areas. Container elements, image elements, and button elements usually have relatively independent bounding boxes and can carry multiple sub-page elements, image content, text content, or interactive content, making them suitable as starting elements for candidate page areas. Text elements and input elements usually only carry local text or input content, with a small coverage area, and are not used as root elements of candidate areas. Pop-up elements are used to identify prompts, dialog boxes, or floating nodes that cover the main content of the page and do not participate in the determination of candidate areas for main content. The root element of the candidate area and all its recursively traversed sub-elements together constitute the corresponding candidate page area. The area of an element's bounding box is the product of the width and height of the root element of the current candidate region. The area of a page screenshot is the product of the image's pixel width and pixel height. The region area ratio is used to characterize the spatial coverage of the candidate page region within the page display area. The specific recursive traversal method is as follows: using the element number of the root element of the current candidate region as the parent element number, filter page element records in the page element records where the parent element number is the same as the current element number to obtain the current level child elements; then use the element numbers of each current level child element as parent element numbers to continue filtering until no new child elements exist; element numbers that have already been traversed are not repeated in the traversal. Obtain the text element association records of the corresponding element and all child elements obtained by recursion, and count the number of characters in the text content of the associated valid OCR text records to obtain the number of valid OCR characters in the region; count the number of characters in the text content of all valid OCR text records under the same page status number to obtain the total number of valid OCR characters on the page; divide the number of valid OCR characters in the region by the total number of valid OCR characters on the page to obtain the region text ratio. The character count in the text content refers to the number of all displayable characters, excluding whitespace, line breaks, and control characters that have been removed during the OCR text cleaning process. Since each valid OCR text record is associated with only one page element, the same valid OCR text record is counted only once within the corresponding candidate page area. The area text percentage represents the proportion of valid page text content carried by the candidate page area to the total valid page text content. When the total number of valid OCR characters on a page is zero, the area text percentage is not calculated, and the multimodal conflict marker for the corresponding page to be classified is updated to a pending review marker.Retrieve the corresponding element and all its child elements obtained through recursive traversal, and count the number of elements to obtain the number of region structure nodes; count the number of all page element records under the same page status number to obtain the total number of page structure nodes; divide the number of region structure nodes by the total number of page structure nodes to obtain the region structure node ratio. The number of region structure nodes includes the number of page element records corresponding to the candidate region root element and each child element obtained through recursive traversal. The region structure node ratio is used to characterize the node coverage of the candidate page region in the current page element hierarchy. When the total number of page structure nodes is zero, the region structure node ratio is not calculated, and the multimodal conflict marker of the corresponding page to be classified is updated to a pending review marker. Calculate the product of the region area ratio, region text ratio, and region structure node ratio, and take the cube root to obtain the region content coverage value; select the page element record with the largest region content coverage value as the main content candidate region. The regional content coverage value is used to comprehensively evaluate the page space coverage, text content carrying capacity, and page structure coverage of candidate page regions. It employs a method of multiplying the product of these three proportions and taking the cube root, ensuring that any candidate page region with a low proportion is unlikely to obtain a high regional content coverage value. This avoids identifying only large areas, areas with abundant text, or areas containing only a large number of page elements as candidate regions for main content. When the root element of a candidate region and all its recursively traversed child elements cover all page element records under the same page status number, the root element of that candidate region is not included in the determination of the candidate region for main content, thus preventing the page root container or the entire page wrapper container from being directly identified as a candidate region for main content. When there are two or more identical maximum regional content coverage values, the page element record with the smaller element bounding box area is prioritized as a candidate region for main content. This prioritizes page regions with more compact boundaries and less non-main content when content coverage is the same. When the element bounding box areas are still the same, the page element record with the smaller element number is prioritized as a candidate region for main content, ensuring the uniqueness of the candidate region determination. If there is no candidate region root element that can be compared, the multimodal conflict flag of the corresponding page to be classified will be updated to a flag to be reviewed.
[0038] In this implementation plan, for multimodal conflict pages, candidate areas for main content are comprehensively identified from three aspects: page space coverage, effective text carrying capacity, and element hierarchy coverage. This avoids the bias caused by determining the main area solely based on page area, text quantity, or element quantity. At the same time, by excluding full-page root containers, setting parallel area selection rules, and transferring to a review process when no effective candidate areas are found, the interference of advertisements, pop-ups, promotional areas, or full-page wrapping containers on the identification of main content can be reduced. This provides a data foundation with clear boundaries, concentrated content, and traceability for subsequent reclassification of main content.
[0039] Specifically, the reclassification process based on the main content candidate region is as follows: Read the elements corresponding to the main content candidate region and all child elements obtained through recursive traversal to obtain the main region page element records; where the main region page element records include the page element records corresponding to the main content candidate region and all their child page element records. For parent-child relationships in the main region page element records, only those where both the parent and child elements belong to the main region page element records are retained to avoid page elements outside the main region participating in spatial structure feature extraction through parent-child relationships. Filter the text element association records to obtain valid OCR text records whose element numbers belong to the main region page element records, obtaining the main region valid OCR text records; specifically, filter the text element association records whose element numbers belong to the main region page element records, and read the corresponding valid OCR text records from the valid OCR text records according to the OCR text record numbers in the filtered text element association records, obtaining the main region valid OCR text records. Thus, the main region valid OCR text records maintain the original text element association relationship with the main region page element records. The image pixels outside the bounding box of the candidate element for the main content in the image pixel matrix are set to zero to obtain the main region image pixel matrix. Specifically, the image pixels within the bounding box of the candidate element for the main content are retained, while the image pixels outside the bounding box are set to a preset zero pixel value. For color images, the preset zero pixel value is a pixel value where all color channels are zero. In this embodiment, page screenshots are uniformly converted into a three-channel RGB image pixel matrix, and the pixel values of each color channel are divided by 255 to obtain an image pixel matrix with values ranging from 0 to 1. For grayscale page screenshots, grayscale pixel values are copied to the red, green, and blue channels to obtain a three-channel RGB image pixel matrix. For page screenshots containing an opacity channel, the page screenshots are first composited onto a preset black background according to the opacity channel, and then converted into a three-channel RGB image pixel matrix. Outside the bounding box of the corresponding element in the main content candidate area, the pixel values of the red, green, and blue channels at the corresponding pixel positions are all set to 0, i.e., the default zero pixel value is (0,0,0). Inside the bounding box of the corresponding element in the main content candidate area, the three-channel pixel values of the corresponding pixel positions in the original image pixel matrix are retained. The image pixel matrix of the main area has the same image pixel width, image pixel height, and page screenshot coordinate system as the original image pixel matrix, thus eliminating the need for coordinate transformation of the left and top coordinates of the elements recorded in the main area page elements, as well as the left and top coordinates of the text boxes of the effective OCR text records in the main area.The main content region image pixel matrix, main content region page element records, main content region valid OCR text records, and corresponding text element association records are associated to form a main content region multimodal dataset. The corresponding text element association records are those whose OCR text record numbers belong to valid OCR text records in the main content region and whose element numbers belong to page element records in the main content region. The image pixels, page elements, text content, and text element associations in the main content region multimodal dataset all correspond to the same main content candidate region. Based on the multimodal dataset of the main content region, text modality feature vector extraction, image modality feature vector extraction, spatial structure modality feature vector extraction, and multimodal joint representation extraction are performed to obtain the main content classification category and the main content classification confidence. Specifically, following the processing method of the aforementioned page state multimodal dataset, the effective OCR text records of the main region are sorted and a main region text sequence is constructed. The main region text sequence is input into the text feature extraction model and the multilayer perceptron text classification model to obtain the main region text category probability vector. The main region image pixel matrix is input into the image feature extraction model and the multilayer perceptron image classification model to obtain the main region image category probability vector. The main region page element records and the preserved parent-child relationships are input into the spatial structure feature extraction model and the multilayer perceptron spatial structure classification model to obtain the main region structure category probability vector. The main region text modality feature vector, the main region image modality feature vector, and the main region spatial structure modality feature vector are input into the multimodal Transformer fusion model to obtain the main region multimodal joint representation. The main region multimodal joint representation is input into the decision tree classification model to obtain the main region fusion category probability vector. The classification category corresponding to the highest category probability in the main area fusion category probability vector is determined as the main content classification category, and the highest category probability is determined as the main content classification confidence level. When there are two or more identical highest category probabilities, the classification category with the smallest category number is determined as the main content classification category. When the main content classification confidence level is greater than or equal to the fusion classification confidence level threshold, the main content classification category is determined as the final classification category of the page to be classified; when the main content classification confidence level is less than the fusion classification confidence level threshold, the multimodal conflict marker is updated to a pending review marker. The main content classification confidence level and the fusion classification confidence level use the same threshold judgment standard to ensure that the credibility of the initial classification and the main content reclassification has a consistent judgment basis. After generating the pending review marker, the main content classification category is not directly output; instead, the corresponding page to be classified is transferred to the classification review task generation process.The candidate area for the main content is determined by the page element record with the largest area content coverage value. Its corresponding area has already characterized the main business content of the page in terms of three aspects: page space coverage, effective text carrying capacity, and structural node coverage. Therefore, when the confidence of the main content classification reaches the confidence threshold of the fusion classification, the main content classification category can be directly used as the final classification category.
[0040] In this implementation scheme, by extracting images, text, page elements, and internal parent-child relationships corresponding to the main area from multimodal conflict pages, and re-performing multimodal classification while maintaining a unified coordinate system, it is possible to avoid page elements such as advertisements, pop-ups, and recommended content outside the main area interfering with the classification results through images or structural relationships. At the same time, based on the classification confidence of the main content, it is decided whether to directly output or transfer to review, so that the final classification result is closer to the main business content of the page, improving the accuracy and reliability of page classification.
[0041] Specifically, the process of determining the final classification category based on classification verification tags and generating classification review tasks is as follows: For the page to be classified that has generated a classification confidence tag, the initial classification category is taken as the final classification category of the page to be classified. Here, generating a classification confidence tag indicates that the fusion classification confidence of the corresponding page to be classified reaches the fusion classification confidence threshold, and the number of modal supports obtained by the initial classification category reaches the support threshold. Therefore, the initial classification category is determined as the final classification category so that the final classification result has the combined support of the fusion classification result and multiple single-modal classification results. For the page to be classified that has already generated a multimodal conflict tag and whose main content classification confidence is greater than or equal to the fusion classification confidence threshold, the main content classification category has already been determined as the final classification category during the main content reclassification process, and no further classification review task is generated. For pages to be categorized that have generated tags for review, a categorization review task is generated. This task includes: page status number, page source identifier, data collection timestamp, page screenshot record, initial classification category, fusion classification confidence level, candidate text classification category, candidate image classification category, candidate structure classification category, text support tags, image support tags, structure support tags, number of modal supports, and tags to be reviewed. The task may also include a trigger source identifier, which includes an identifier for insufficient fusion classification confidence and an identifier for insufficient confidence in reclassifying the main content. The categorization review task is used to submit pages to be categorized that have insufficient fusion classification confidence or insufficient confidence in reclassifying the main content to the review terminal. The page screenshot record is used to display the rendered image of the page to be categorized on the review terminal. The initial classification category, fusion classification confidence level, each candidate classification category, each support tag, and the number of modal supports are used to provide reviewers with the classification basis and conflict information for the corresponding page. When the source identifier for the pending review is an insufficient confidence level in the fusion classification, the review terminal displays the initial classification basis for the corresponding page based on the page screenshot record, initial classification category, fusion classification confidence level, each candidate classification category, each supporting tag, and the number of modal supports. When the source identifier for the pending review is an insufficient confidence level in the main content reclassification, the classification review task also includes the root element number of the main content candidate region, the left coordinate of the main content candidate region, the top coordinate of the main content candidate region, the width of the main content candidate region, and the height of the main content candidate region; the review terminal marks the bounding box of the corresponding element of the main content candidate region in the page screenshot record and displays the initial classification basis and the main content reclassification processing result. The pending review trigger source identifier is written during the processing stage of generating pending review tags: when the fusion classification confidence level is less than the fusion classification confidence level threshold, an insufficient confidence level in the fusion classification is written; when the main content classification confidence level is less than the fusion classification confidence level threshold, an insufficient confidence level in the main content reclassification is written.The review terminal provides review category selection options based on the enabled categories in the page service category configuration table, allowing reviewers to determine the review category from a preset category set. Page status numbers are used to associate the category review task, the review category, and the corresponding page status multimodal dataset.
[0042] In this implementation plan, page classification results are processed by sorting based on classification verification marks. Pages that have obtained multimodal support can directly output classification results, while pages with insufficient confidence in fusion classification or insufficient confidence in reclassifying the main content are subject to manual review. At the same time, the review terminal displays the page screen and the classification basis for each modality, and limits the reviewers to select results from a preset set of classification categories. This ensures that the review classification categories are accurately associated with the page status multimodal dataset, improving the standardization and traceability of classification result output and manual review processing.
[0043] Specifically, the process of feeding back the review results of the classification review task to the classification training sample set is as follows: the classification review task is output to the review terminal, and the review classification categories submitted by the review terminal are received. The review classification categories are then used as the final classification categories for the pages to be classified. The review terminal is a computing terminal for reviewers to view the classification review task and submit review classification categories, including desktop computers, laptops, tablets, mobile communication terminals, or review pages deployed in a browser. The review terminal has functions for task reception, page display, classification category selection, and review result submission. The review terminal reads the classification review task through a preset task reception interface and loads the corresponding task information based on the page status number. The review terminal displays the rendered screen of the page to be classified according to the page screenshot record in the classification review task, displays the corresponding review reason according to the review trigger source identifier, and displays the initial classification category, fusion classification confidence, text candidate classification category, image candidate classification category, structure candidate classification category, text support markers, image support markers, structure support markers, and modality support quantity. When a classification review task includes the root element number of the main content candidate area, the left coordinate of the main content candidate area, the top coordinate of the main content candidate area, the width of the main content candidate area, and the height of the main content candidate area, the review terminal marks the bounding box of the corresponding element of the main content candidate area in the page screenshot record. The reviewer selects a review classification category from the enabled classification categories in the page business classification configuration table in the review terminal; the review terminal generates a review result submission record, which includes the page status number, the review classification category, and the review submission timestamp. After receiving the review result submission record, it verifies that the classification review task corresponding to the page status number has a pending review mark, and verifies that the review classification category belongs to the enabled classification categories in the page business classification configuration table; after successful verification, the review classification category is determined as the final classification category of the corresponding page to be classified. After the review terminal submits the review classification category, it associates the review classification category with the page status number in the classification review task to ensure that the review classification category corresponds to the same page status to be classified. The review classification category is the classification category selected by the reviewer from the enabled classification categories in the page business classification configuration table. For pages to be categorized that have a final category, a categorization output record is generated, including: page status number, page source identifier, collection timestamp, initial category, final category, and final category source identifier. The final category source identifier includes: initial categorization result, main content categorization result, and review categorization result. Specifically, when the final category is determined by the initial category, the final category source identifier is determined as the initial categorization result; when the final category is determined by the main content categorization category, the final category source identifier is determined as the main content categorization result; and when the final category is determined by the review categorization category, the final category source identifier is determined as the review categorization result.The classification output records are written to the page classification result database for subsequent page classification result queries, review result tracing, and training sample construction. For classification output records whose final classification source is identified as review classification results, the page state multimodal dataset is associated with the review classification category to generate review feedback samples. The review feedback samples include the page state multimodal dataset and the corresponding review classification category. The page state multimodal dataset includes page screenshot records, page element records, valid OCR text records, and text element association records. The review classification category serves as the classification label for the corresponding review feedback sample. Since the review classification category is confirmed by the reviewers, the review feedback samples are used to supplement classification training data when the fusion classification results are unreliable, the confidence of reclassifying the main content is insufficient, or the page content style changes. The review feedback samples are added to the classification training sample set, and the model is updated and trained using the review classification category as the classification label. Specifically, when the cumulative number of review feedback samples reaches a preset model update sample number threshold, model update training is performed on the classification training sample set to avoid frequent model parameter updates based on a single or small number of review feedback samples. The models include a multilayer perceptron text classification model, a multilayer perceptron image classification model, a spatial structure feature extraction model, a multilayer perceptron spatial structure classification model, a multimodal Transformer fusion model, and a decision tree classification model. During model update training, the classification training sample set is divided into a training subset and a validation subset. The model parameters are updated based on the training subset, and the updated model is used to determine whether it meets the preset classification accuracy requirement based on the validation subset. When the preset classification accuracy requirement is met, the updated model replaces the current classification model.
[0044] like Figure 3 The diagram illustrates the multimodal classification feedback process for applications and web pages. It demonstrates the basic processing flow for page classification. First, the content interface of the app or website is acquired through data collection, and OCR text features, image visual features, and spatial layout features are extracted. Then, feature alignment and fusion of the three modal features are performed, and a multimodal joint representation is formed through cross-attention multimodal fusion. This representation is then input into a decision tree classifier to obtain the initial classification result and the fused classification confidence score. After the classification result is output, a multimodal consistency check is performed, comparing the support of the three unimodal classification results (text, image, and spatial structure) for the initial classification category, generating classification confidence markers, pending verification markers, or multimodal conflict markers. The "verification correction" in the diagram indicates that the classification verification result can be used to correct subsequent classification processing; the confidence result further enters the confidence feedback mechanism to record the final classification result and support subsequent model optimization.
[0045] In this implementation plan, the classification results confirmed by manual review are associated with the corresponding page state multimodal dataset to form traceable review feedback samples. These samples are then used for model update training after the number of samples meets the update conditions. This allows the system to continuously absorb classification experience from low-confidence pages, conflicting pages, and page style change scenarios, thereby improving the accuracy, stability, and adaptability of subsequent application and webpage classification and recognition.
[0046] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.
[0047] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. As those skilled in the art will understand, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.
Claims
1. A method for classifying and recognizing applications and web pages based on multimodal feedback verification, characterized in that: Includes the following steps: S1. Collect page screenshots, page elements, and OCR text of the page to be classified. Clean the page elements and OCR text, unify their types, and associate the text elements to form a multimodal dataset of page status. S2, based on the page state multimodal dataset, extracts text modal features, image modal features and spatial structure modal features, and generates single-modal classifications respectively. Cross-modal fusion is performed based on text modal features, image modal features and spatial structure modal features to obtain the initial classification category and the fused classification confidence. S3 combines unimodal classification and fusion classification confidence to perform consistency verification on the initial classification categories, generate classification verification tags, and identify multimodal conflict pages; for multimodal conflict pages, it combines page screenshots, page elements, and OCR text to evaluate the coverage of regional content, determine the main content candidate areas, and reclassify based on the main content candidate areas; S4 determines the final classification category based on the classification verification mark, generates a classification verification task, and feeds back the verification results of the classification verification task to the classification training sample set.
2. The application and webpage classification and recognition method based on multimodal feedback verification according to claim 1, characterized in that, The specific process of collecting screenshots, page elements, and OCR text of the page to be categorized is as follows: For pages to be categorized, generate a page status number and record the collection timestamp and page source identifier; Under the same page status number, the rendered screen of the page to be categorized is obtained using the screenshot interface, and page screenshot records are collected. The page screenshot records include: page status number, image pixel width, image pixel height, page screenshot area, and image pixel matrix. Using the page node tree export interface provided by the page runtime environment, export the page node tree corresponding to the page screenshot. Collect page element records for each page node. The page element records include: page status number, element number, parent element number, element type, element visibility marker, element left coordinate, element top coordinate, element width, element height, and element bounding box area. Then, convert the element left coordinate, element top coordinate, element width, and element height into data with the top left corner of the page screenshot as the origin. The image pixel matrix from the page screenshot is input into the OCR recognition model to obtain the OCR text record. The OCR text record includes: page status number, OCR text record number, text content, left coordinate of the text box, top coordinate of the text box, text box width, text box height, text box area, and OCR confidence score.
3. The application and webpage classification and recognition method based on multimodal feedback verification according to claim 1, characterized in that, The specific process of cleaning, unifying the types of page elements and OCR text, and associating text elements to form a multimodal dataset of page states is as follows: When an element's visibility flag is changed to invisible, delete the corresponding page element record; map the element type to a preset unified element type, including: text elements, image elements, button elements, input elements, container elements, pop-up elements, and status indicator elements; Remove whitespace characters, newline characters, and control characters from the text content. If the text content after deletion is empty, delete the corresponding OCR text record. For OCR text records that have not been deleted, delete OCR text records with an OCR confidence score less than the OCR confidence score threshold. If the OCR confidence score is greater than or equal to the OCR confidence score threshold, mark the corresponding OCR text record as a valid OCR text record. For valid OCR text records, the intersection area between the text box and the element bounding box is calculated based on the element's left coordinate, top coordinate, width, and height, as well as the text box's left coordinate, top coordinate, width, and height. The text coverage rate is obtained by dividing the intersection area of the text box with each element bounding box by the text box area. For each valid OCR text record, page element records with text coverage greater than or equal to the text coverage threshold are selected, and the one with the smallest element bounding box area is selected as the associated page element of the corresponding valid OCR text record. A text element associated record is generated, which includes: page status number, OCR text record number, and element number. If there are no page element records with text coverage greater than or equal to the text coverage threshold, no text element associated record is generated for the corresponding valid OCR text record. By associating page screenshot records, page element records, valid OCR text records, and text element association records under the same page status number, a multimodal dataset of page status is formed.
4. The application and webpage classification and recognition method based on multimodal feedback verification according to claim 3, characterized in that, The specific process of extracting text modal features, image modal features, and spatial structure modal features from the page state multimodal dataset, and generating unimodal classifications for each, is as follows: For valid OCR text records under the same page status number, the text content is sorted in ascending order according to the priority of the upper coordinate and the left coordinate of the text box to obtain the page text sequence; the page text sequence is input into a text feature extraction model pre-trained based on the Transformer architecture to obtain text modality feature vectors; the text modality feature vectors are input into a pre-trained multilayer perceptron text classification model to obtain the text category probability corresponding to each category and form a text category probability vector; For page screenshot records under the same page status number, the image pixel matrix is input into the convolutional neural network image feature extraction model to obtain the image modality feature vector; the image modality feature vector is input into the pre-trained multilayer perceptron image classification model to obtain the image category probability corresponding to each category, and form the image category probability vector. For each page element record under the same page status number, calculate the ratio of the element's left coordinate to the image pixel width, the ratio of the element's top coordinate to the image pixel height, the ratio of the element's width to the image pixel width, the ratio of the element's height to the image pixel height, and the ratio of the element's bounding box area to the page screenshot area to obtain the element's position and size features. Also, count the number of records in the text element association records where the element number is the same as the element number of the current page element record to obtain the number of associated text elements. Based on the element type, element position and size features, and the number of associated text elements, generate element structure features. When the parent element number in one page element record is equal to the element number in another page element record, a parent-child relationship is determined between the two page element records. The element structure features and parent-child relationships of each page element record under the same page status number are input into the spatial structure feature extraction model built based on graph convolutional network to obtain spatial structure modal feature vectors; the spatial structure modal feature vectors are input into the pre-trained multilayer perceptron spatial structure classification model to obtain the structure category probabilities corresponding to each classification category and form a structure category probability vector.
5. The application and webpage classification and recognition method based on multimodal feedback verification according to claim 1, characterized in that, The specific process of cross-modal fusion based on text modal features, image modal features, and spatial structure modal features to obtain the initial classification category and the fused classification confidence is as follows: Text modality feature vectors, image modality feature vectors, and spatial structure modality feature vectors are input into a multimodal Transformer fusion model constructed based on a cross-attention mechanism to obtain a multimodal joint representation; the multimodal joint representation is then input into a pre-trained decision tree classification model to obtain the fusion category probability corresponding to each category, and a fusion category probability vector is formed. The classification category corresponding to the maximum fusion category probability is used as the initial classification category, and the maximum fusion category probability is used as the fusion classification confidence.
6. The application and webpage classification and recognition method based on multimodal feedback verification according to claim 4, characterized in that, The specific process of combining unimodal classification and fused classification confidence to perform consistency verification on the initial classification categories, generate classification verification tags, and identify multimodal conflict pages is as follows: For the text category probability vector, image category probability vector, and structure category probability vector under the same page status number, the classification category corresponding to the largest category probability in each category probability vector is taken as the candidate classification category, thus obtaining the text candidate classification category, image candidate classification category, and structure candidate classification category. If the candidate text classification category is the same as the initial classification category, the generated text support label is 1; otherwise, it is 0. If the candidate image classification category is the same as the initial classification category, the generated image support label is 1; otherwise, it is 0. If the candidate structure classification category is the same as the initial classification category, the generated structure support label is 1; otherwise, it is 0. Calculate the sum of text support tags, image support tags, and structural support tags to obtain the number of modal supports; Generate classification verification markers, including: generating a marker to be verified when the fusion classification confidence is less than the fusion classification confidence threshold; generating a classification confidence marker when the fusion classification confidence is greater than or equal to the fusion classification confidence threshold and the number of supported modalities is greater than or equal to the support threshold; and generating a multimodal conflict marker when the fusion classification confidence is greater than or equal to the fusion classification confidence threshold and the number of supported modalities is less than the support threshold.
7. The application and webpage classification and recognition method based on multimodal feedback verification according to claim 6, characterized in that, The specific process for evaluating the coverage of content in multimodal conflict pages, combining page screenshots, page elements, and OCR text, and determining candidate areas for main content is as follows: For the page to be classified that generates multimodal conflict markers, obtain the page element records with element types of container element, image element and button element under the page status number, calculate the ratio of the bounding box area of each element to the area of the page screenshot, and obtain the area ratio of the region. Get the text element associated records of the corresponding element and all child elements obtained by recursive traversal, and count the number of characters in the text content of the associated valid OCR text records to get the number of valid OCR characters in the region; count the number of characters in the text content of all valid OCR text records under the same page status number to get the total number of valid OCR characters on the page. Divide the number of valid OCR characters in a region by the total number of valid OCR characters on the page to obtain the region's text percentage. Get the corresponding element and all child elements obtained by recursion, count the number of elements, and get the number of nodes in the region structure. Count the number of all page element records under the same page status number to get the total number of page structure nodes; divide the number of region structure nodes by the total number of page structure nodes to get the proportion of region structure nodes; Calculate the product of the region area ratio, the region text ratio, and the region structure node ratio, and take the cube root to obtain the region content coverage value; record the page element with the largest region content coverage value as the candidate region for main content.
8. The application and webpage classification and recognition method based on multimodal feedback verification according to claim 7, characterized in that, The specific process of reclassifying based on candidate regions of main content is as follows: Read the elements corresponding to the candidate areas of the main content and all the child elements obtained by recursive traversal to obtain the page element records of the main area; filter the valid OCR text records whose element numbers belong to the page element records of the main area in the text element association records to obtain the valid OCR text records of the main area; set the image pixels outside the bounding box of the corresponding elements of the candidate areas of the main content in the image pixel matrix to zero to obtain the image pixel matrix of the main area. The pixel matrix of the main body region image, the page element record of the main body region, the effective OCR text record of the main body region and the corresponding text element association record are associated to form a multimodal dataset of the main body content region. Based on the multimodal dataset of the main body content region, text modality feature vector extraction, image modality feature vector extraction, spatial structure modality feature vector and multimodal joint representation extraction are performed to obtain the main body content classification category and the main body content classification confidence. When the confidence level of the main content classification is greater than or equal to the confidence level threshold of the fusion classification, the main content classification category is determined as the final classification category of the page to be classified. When the confidence level of the main content classification is less than the confidence level threshold of the fusion classification, the multimodal conflict marker will be updated to a marker to be reviewed.
9. The application and webpage classification and recognition method based on multimodal feedback verification according to claim 8, characterized in that, The specific process of determining the final classification category based on the classification verification mark and generating the classification review task is as follows: For pages to be categorized that have generated reliable classification tags, the initial classification category is used as the final classification category for the pages to be categorized. For the pages to be classified that have generated tags to be reviewed, a classification review task is generated. The classification review task includes: page status number, page source identifier, collection timestamp, page screenshot record, initial classification category, fusion classification confidence, text candidate classification category, image candidate classification category, structure candidate classification category, text support tags, image support tags, structure support tags, modality support quantity, and tags to be reviewed.
10. The application and webpage classification and recognition method based on multimodal feedback verification according to claim 9, characterized in that, The specific process of feeding back the review results of the classification review task to the classification training sample set is as follows: Output the classification review task to the review terminal, and receive the review classification category submitted by the review terminal, and use the review classification category as the final classification category of the page to be classified; For pages to be categorized that have a final category, generate a categorization output record, including: page status number, page source identifier, collection timestamp, initial category, final category, and final category source identifier; the final category source identifier includes: initial categorization result, main content categorization result, and review categorization result; For the classification output record whose final classification source is identified as the verification classification result, the page state multimodal dataset is associated with the verification classification category to generate verification feedback samples; the verification feedback samples are added to the classification training sample set, and the model is updated and trained using the verification classification category as the classification label.
Citation Information
Patent Citations
A semi-supervised graph node classification method based on personalized web page ranking
CN114492651B
Webpage classification detection method and device, electronic equipment and storage medium
CN117456217A
