Multimodal-based bad web page detection method, device, equipment and medium

Abstract

The present application relates to a method, apparatus, device, and medium for detecting bad web pages based on multimodality. The method includes: obtaining a web page to be detected, determining text feature information consisting of the web page title, web page text, web page tag, and web page address, image feature information consisting of a web page screenshot, and position feature information consisting of the one-dimensional position and two-dimensional coordinates of the text image, fusing the three features to form a comprehensive vector of the text and image position, entering the multimodal self-attention network, and converting the vector into a two-dimensional vector through hidden layer linear mapping and nonlinear activation for normalization. When the probability of a bad web page in the two-dimensional vector is greater than the probability of a normal web page, the web page to be detected is determined to be a bad web page. The present application can more accurately identify bad content, prevent website designers from taking improper means to improve web page rankings, promote the purification of the Internet environment, and ensure users' network security.

Description

Technical Field

[0001] The present application relates to the field of network security, and in particular to a multimodal-based bad web page detection method, corresponding device, electronic device and computer-readable storage medium. Background Art

[0002] The rapid development of the internet and the increasing diversity of web content have also brought security risks to the online environment. In particular, the spread of inappropriate web pages poses a threat to users' browsing experience and psychological well-being. Traditional methods for detecting inappropriate web pages rely primarily on text keyword filtering and simple image recognition techniques. These methods are ineffective against web pages with complex page structures and multimodal content (text, images, and their layout), and are prone to false positives and missed detections.

[0003] Search engines are a crucial way for users to access the internet. Most users tend to browse the first few pages of search results, leading to an increasing importance of ranking web pages at the top of search results. To achieve higher rankings, increased traffic, and commercial profits, some website designers employ inappropriate methods to boost page rankings. This not only distorts the authenticity and quality of web content but also potentially exposes a large number of web pages carrying security threats, compromising user security.

[0004] To sum up, in order to adapt to the problem that some website designers use improper means to improve web page rankings, which distorts the authenticity and quality of web page content and may also cause web pages carrying security threats to be displayed in large quantities, this application makes corresponding explorations to solve this problem. Summary of the Invention

[0005] The purpose of this application is to solve the above problems and provide a multimodal bad web page detection method, corresponding device, electronic device and computer-readable storage medium.

[0006] In order to meet the various objectives of this application, this application adopts the following technical solutions:

[0007] A multimodal method for detecting bad web pages is proposed to meet one of the objectives of this application, including:

[0008] In response to a bad web page detection instruction, determining a web page address corresponding to a web page to be detected, and obtaining a web page screenshot and web page HTML node information corresponding to the web page to be detected based on the web page address, wherein the web page HTML node information includes two-dimensional coordinates of the web page where the node is located and web page text data, wherein the web page text data includes a web page title, web page text content, and web page tags;

[0009] Extracting text feature information corresponding to the webpage text data based on a text encoder in a pre-trained bad webpage detection model to construct a word segmentation embedding vector, a word segmentation position index vector, and a word segmentation coordinate index vector corresponding to the webpage text data;

[0010] Extracting image feature information corresponding to the webpage screenshot based on an image encoder in a pre-trained bad webpage detection model to determine an image serialization embedding vector, an image block absolute index position vector, and an image coordinate index vector corresponding to the webpage screenshot;

[0011] Combine and concatenate the word segmentation embedding vector with the image serialization embedding vector to determine a word segmentation image block embedding vector, combine and concatenate the word segmentation position index vector with the image block absolute index position vector to determine one-dimensional position information, combine and concatenate the word segmentation coordinate index vector with the image coordinate index vector to determine two-dimensional position information, and determine a comprehensive embedding vector based on the word segmentation image block embedding vector, the one-dimensional position information, and the two-dimensional position information;

[0012] Based on the multimodal self-attention network, the comprehensive embedding vector is multi-level encoded to determine the web page image and text fusion vector, and the web page image and text fusion vector is normalized and converted into a two-dimensional vector through hidden layer linear mapping and nonlinear activation. When the probability that the web page to be detected in the two-dimensional vector is a bad web page is greater than the probability that the web page to be detected is a normal web page, the web page to be detected is determined to be a bad web page to complete the detection of bad web pages.

[0013] Optionally, the step of extracting text feature information corresponding to the webpage text data based on a text encoder in a pre-trained bad webpage detection model to construct a word segmentation embedding vector, a word segmentation position index vector, and a word segmentation coordinate index vector corresponding to the webpage text data includes:

[0014] Using a word segmenter to subdivide the webpage text data into its corresponding semantic units;

[0015] Mapping the semantic units based on a preset vocabulary, and converting the semantic units into their corresponding numerical representations;

[0016] Converting the numerical representation into an embedding vector through an embedding layer in the text encoder, mapping each vocabulary unit to a vector representation in a high-dimensional space, to determine a word segmentation embedding vector corresponding to the webpage text data;

[0017] The one-dimensional index position of each word in the webpage text data and the two-dimensional coordinate position of the coordinate frame where the text is located are respectively embedded through the embedding layer to determine the word position index vector and the word coordinate index vector of each word.

[0018] Optionally, the step of extracting image feature information corresponding to the webpage screenshot based on an image encoder in a pre-trained bad webpage detection model to determine an image serialization embedding vector, an image block absolute index position vector, and an image coordinate index vector corresponding to the webpage screenshot includes:

[0019] Segmenting the webpage screenshot into individual image blocks, processing the individual image blocks through an image linear flattening module, and linearly projecting the image blocks to determine an image serialization embedding vector;

[0020] Obtain the absolute sequential position corresponding to each image block and the coordinate position of the upper left corner of each image block, and embed the absolute sequential position and the coordinate position of the upper left corner of each image block through an embedding layer to determine the image block absolute index position vector and the image coordinate index vector.

[0021] Optionally, the step of performing multi-level encoding on the comprehensive embedding vector based on a multimodal self-attention network to determine the webpage image-text fusion vector includes:

[0022] In each level of encoding, the encoder corresponding to the current level is used to construct a query vector, a key vector and a value vector for each row vector in the currently input comprehensive embedding vector;

[0023] Perform a weighted sum of the value vectors corresponding to all row vectors, where the weight of each value vector is the normalized score of the dot product of its corresponding key vector and the query vector corresponding to other row vectors;

[0024] The weighted summation result is multiplied by the conversion weight matrix to obtain its corresponding intermediate information matrix. The intermediate information matrix is ​​subjected to feature extraction by a multi-layer perceptron to obtain the image-text interaction vector, which is used as the comprehensive embedding vector required by the next encoder, or directly output as the web page image-text fusion vector in the last encoder.

[0025] Optionally, the step of extracting text feature information corresponding to the webpage text data based on a text encoder in a pre-trained bad webpage detection model includes:

[0026] The web page title, web page text content, web page tag and web page address in the web page text data are combined and spliced ​​and then segmented;

[0027] Using a text encoder to extract the text feature information from the web page title, web page text content, web page tags and web page address after word segmentation;

[0028] The text feature information is activated and then output.

[0029] Optionally, before the step of obtaining webpage text data and webpage screenshot corresponding to the webpage to be detected according to the webpage address, the following steps are included:

[0030] In response to the data preprocessing instruction, determining the webpage title, webpage text content, webpage tag and webpage address corresponding to the webpage to be detected;

[0031] The web page title, web page text content, web page tag and web page address are combined to determine the web page text data.

[0032] Optionally, the bad web pages include one or more of a maintenance-stopped web page, a cheating-inducing web page, an invalid element layout web page, a disorganized layout web page, a harmful information web page, and a web page with many advertisements and horizontal insertion of the text that affects reading.

[0033] A multimodal bad webpage detection device provided for another purpose of the present application includes:

[0034] a data acquisition module configured to respond to a bad web page detection instruction, determine a web page address corresponding to a web page to be detected, and obtain a web page screenshot and web page HTML node information corresponding to the web page to be detected based on the web page address, wherein the web page HTML node information includes the two-dimensional coordinates of the web page where the node is located and web page text data, wherein the web page text data includes a web page title, web page text content, and web page tags;

[0035] a text vector determination module configured to extract text feature information corresponding to the webpage text data based on a text encoder in a pre-trained bad webpage detection model to construct a word segmentation embedding vector, a word segmentation position index vector, and a word segmentation coordinate index vector corresponding to the webpage text data;

[0036] an image vector determination module configured to extract image feature information corresponding to the webpage screenshot based on an image encoder in a pre-trained bad webpage detection model to determine an image serialization embedding vector, an image block absolute index position vector, and an image coordinate index vector corresponding to the webpage screenshot;

[0037] a comprehensive vector determination module configured to combine and splice the word segmentation embedding vector with the image serialization embedding vector to determine a word segmentation image block embedding vector, combine and splice the word segmentation position index vector with the image block absolute index position vector to determine one-dimensional position information, combine and splice the word segmentation coordinate index vector with the image coordinate index vector to determine two-dimensional position information, and determine a comprehensive embedding vector based on the word segmentation image block embedding vector, the one-dimensional position information, and the two-dimensional position information;

[0038] The bad web page detection module is configured to perform multi-level encoding on the comprehensive embedding vector based on a multimodal self-attention network to determine a web page image-text fusion vector, and convert the web page image-text fusion vector into a two-dimensional vector through hidden layer linear mapping and nonlinear activation for normalization. When the probability that the web page to be detected in the two-dimensional vector is a bad web page is greater than the probability that the web page to be detected is a normal web page, the web page to be detected is determined to be a bad web page to complete the detection of bad web pages.

[0039] An electronic device provided to meet another purpose of the present application includes a central processing unit and a memory, wherein the central processing unit is used to call and run a computer program stored in the memory to execute the steps of the multimodal-based bad web page detection method described in the present application.

[0040] A computer-readable storage medium is provided to meet another purpose of the present application, which stores a computer program implemented according to the multimodal-based bad web page detection method in the form of computer-readable instructions. When the computer program is called and executed by a computer, the steps included in the corresponding method are executed.

[0041] Compared with the existing technology, this application addresses the problems of some website designers taking improper measures to improve web page rankings, which distorts the authenticity and quality of web page content and may also cause web pages carrying security threats to be displayed in large quantities. This application includes but is not limited to the following beneficial effects:

[0042] First, the multimodal bad web page detection method of the present application can comprehensively understand multimodal content. Traditional methods often process text and images separately, making it difficult to capture the relationship between them. The present application achieves a more comprehensive understanding of web page content by integrating text, images, and their layout information. In particular, when processing web pages with both text and images, it can more accurately identify bad content.

[0043] Secondly, the multimodal method for detecting bad web pages in this application can effectively utilize contextual information. In detecting bad web pages, it is difficult to accurately determine whether the content is bad based solely on keywords or image features. This application utilizes typesetting coordinate information to understand the positional relationship of text on the page, improving understanding of the context, which helps to reduce false positives and missed positives.

[0044] Third, the multimodal bad web page detection method of this application is capable of handling complex page structures. Modern web page designs are relatively complex, and relying solely on text analysis or simple visual feature recognition is not enough to accurately judge web page content. This application can understand complex page layout and design elements, thereby effectively identifying bad content even in complex web page structures.

[0045] Furthermore, the bad webpage detection model of this application is used to comprehensively process the text, images, and layout information in the webpage to achieve a comprehensive understanding and accurate identification of bad webpages. This multimodal information fusion method enables the system to better cope with the complexity and diversity of modern webpage design.

[0046] The model in this application can understand the positional relationship between text and images in web page layout, so as to better use contextual information to identify objectionable content. This context-aware capability helps reduce false positives and false negatives, and improves the accuracy of objectionable web page detection.

[0047] The system implements a continuous feedback and optimization mechanism for detection results, continuously updating model parameters based on actual recognition situations to improve detection effectiveness and adaptability. This dynamic learning and optimization capability makes the system more adaptable and robust, able to cope with the ever-changing network environment and the characteristics of undesirable web pages;

[0048] It has high efficiency and scalability when processing complex web page structures and large-scale data sets. Both model training and inference processes can be completed quickly in a high-performance computing environment, making it suitable for real-time, large-scale bad web page detection tasks.

[0049] Furthermore, this application can greatly improve the quality of web pages, prevent some website designers from taking improper means to improve web page rankings, and avoid distorting the authenticity and quality of web page content and preventing web pages carrying security threats from being displayed in large quantities. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0051] Figure 1 Schematic diagram of the flow of the multimodal bad web page detection method in an embodiment of the present application;

[0052] Figure 2 This is an exemplary framework diagram of the multimodal bad web page detection method in the embodiment of the present application.

[0053] Figure 3 A schematic diagram illustrating the principle of a breadth-first search strategy in an embodiment of the present application;

[0054] Figure 4 This is a schematic diagram of the HTML DOM tree in an embodiment of the present application;

[0055] Figure 5 A schematic diagram of the depth traversal search principle in an embodiment of the present application;

[0056] Figure 6A schematic diagram of generating word segmentation embedding vectors in an embodiment of the present application;

[0057] Figure 7 Schematic diagram of the absolute sequential positions of image blocks and the coordinate position of the upper left corner of each image block in an embodiment of the present application;

[0058] Figure 8 Schematic diagram of the web page original image processing process in an embodiment of the present application;

[0059] Figure 9 Schematic diagram of multimodal text image embedding vector fusion in an embodiment of the present application;

[0060] Figure 10 A schematic diagram of a confusion matrix predicted by the model in an embodiment of the present application;

[0061] Figure 11 This is a principle block diagram of a multimodal bad web page detection device in an embodiment of the present application;

[0062] Figure 12 Schematic diagram of the structure of the computer device in the embodiment of the present application. DETAILED DESCRIPTION

[0063] The following describes in detail embodiments of the present application, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and are not to be construed as limiting the present application.

[0064] It will be understood by those skilled in the art that, unless expressly stated otherwise, the singular forms "a", "an", "said" and "the" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, "connected" or "coupled" as used herein may include wireless connections or wireless couplings. The term "and / or" used herein includes all or any units and all combinations of one or more associated listed items.

[0065] Those skilled in the art will understand that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with their meanings in the context of the prior art and, unless specifically defined as such, will not be interpreted in an idealized or overly formal sense.

[0066] Those skilled in the art will appreciate that the terms "client," "terminal," and "terminal device" as used herein include both devices that are wireless signal receivers, i.e., devices that only have wireless signal receivers without transmission capabilities, and devices that have receiving and transmitting hardware capable of two-way communication over a two-way communication link. Such devices may include: cellular or other communication devices such as personal computers and tablet computers, which have single-line displays, multi-line displays, or cellular or other communication devices without multi-line displays; PCS (Personal Communications Service), which may combine voice, data processing, fax, and / or data communication capabilities; PDA (Personal Digital Assistant), which may include a radio frequency receiver, a pager, Internet / Intranet access, a web browser, a notepad, a calendar, and / or a GPS (Global Positioning System) receiver; and conventional laptop and / or palmtop computers or other devices, which have and / or include a radio frequency receiver. As used herein, "client," "terminal," or "terminal device" may be portable, transportable, installed in a vehicle (air, sea, and / or land), or adapted and / or configured to operate locally and / or in a distributed manner at any other location on Earth and / or in space. The terms "client," "terminal," or "terminal device" may also refer to a communication terminal, an Internet terminal, or a music / video playback terminal, such as a PDA, a mobile internet device (MID), and / or a mobile phone with music / video playback capabilities, as well as a smart TV, a set-top box, or other similar device.

[0067] The hardware referred to by names such as "server", "client", and "service node" in this application is essentially an electronic device with capabilities equivalent to those of a personal computer. It is a hardware device that has the necessary components revealed by the von Neumann principle, such as a central processing unit (including an arithmetic unit and a controller), a memory, an input device, and an output device. Computer programs are stored in its memory, and the central processing unit loads the program stored in the external memory into the internal memory for execution, executes the instructions in the program, and interacts with the input and output devices to complete specific functions.

[0068] It should be noted that the concept of "server" referred to in this application can also be extended to server clusters. Based on the network deployment principles understood by those skilled in the art, the servers described should be logically divided. In physical space, these servers can be independent of each other but callable through interfaces, or integrated into a single physical computer or a computer cluster. Those skilled in the art should understand this flexibility and should not use it to constrain the implementation of the network deployment method of this application.

[0069] Unless expressly specified, one or more technical features of the present application can be deployed on a server for implementation and accessed by a client through a remote call to obtain an online service interface provided by the server, or can be directly deployed and run on a client for implementation.

[0070] Unless expressly specified otherwise, the neural network models referenced or may be referenced in this application may be deployed on a remote server and remotely called on the client, or may be deployed and directly called on a client with sufficient device capabilities. In some embodiments, when it runs on the client, its corresponding intelligence may be obtained through transfer learning to reduce the requirements for the client's hardware operating resources and avoid excessive occupation of the client's hardware operating resources.

[0071] Unless explicitly specified, the various data involved in this application can be stored remotely on a server or on a local terminal device, as long as they are suitable for being called by the technical solution of this application.

[0072] Those skilled in the art should be aware that although the various methods of this application are described based on the same concept and thus exhibit commonality, unless otherwise specified, these methods can be independently executed. Similarly, the various embodiments disclosed in this application are all based on the same inventive concept. Therefore, concepts with the same expression, as well as concepts that are appropriately transformed for convenience despite different expression, should be understood as equivalent.

[0073] Unless expressly stated to be mutually exclusive, the various embodiments disclosed in this application may be cross-combined with the relevant technical features of the various embodiments to flexibly construct new embodiments, as long as such combination does not deviate from the creative spirit of this application and can meet the needs of the prior art or resolve certain deficiencies in the prior art. Those skilled in the art should be aware of such flexibility.

[0074] The rapid development of the internet and the increasing diversity of web content have also brought security risks to the online environment. In particular, the spread of inappropriate web pages poses a threat to users' browsing experience and psychological well-being. Traditional methods for detecting inappropriate web pages rely primarily on text keyword filtering and simple image recognition techniques. These methods are ineffective against web pages with complex page structures and multimodal content (text, images, and their layout), and are prone to false positives and missed detections. Therefore, there is an urgent need for an inappropriate web page detection technology that can comprehensively understand the multimodal content and structure of a web page.

[0075] Search engines are a crucial means for users to access the internet. Most users tend to browse the first few pages of search results. This phenomenon has led to the increasing importance of webpage rankings at the top of search results. To achieve higher rankings, increased traffic, and commercial profits, some website designers employ inappropriate methods to boost page rankings. This not only distorts the authenticity and quality of webpage content but also potentially leads to the widespread display of webpages carrying security threats, compromising user online security.

[0076] In some embodiments, the multimodal bad web page detection method applied for can be implemented based on the bad web page automatic detection system, describing the construction and operation process of the bad web page automatic detection system, aiming to provide an efficient and accurate network content security review solution.

[0077] This automated bad webpage detection system can be deployed on a high-performance computing server equipped with a multi-core CPU, a high-speed GPU accelerator (such as the NVIDIA Tesla V100), and sufficient memory and storage to ensure efficient data processing and model training. A stable Linux distribution, such as Ubuntu 20.04, can be used as the operating system. The Python programming environment and necessary data processing and machine learning libraries, such as PyTorch and Transformers, should be installed to perform model training and inference.

[0078] A designed web crawler automatically accesses target websites and collects web page URLs. For each URL, the crawler captures the webpage's HTML content, image files, CSS style information, and dynamic content generated by JavaScript. The captured webpage content is preprocessed, including text extraction and cleaning, image resizing, and parsing node layout coordinate information. This step converts the raw webpage data into the input format used by the model processing in this patent. Fine-tuning is performed using a subset of annotated webpage datasets to meet the specific requirements of detecting inappropriate webpages. During fine-tuning, model parameters are adjusted to minimize the loss function, thereby improving the model's performance in identifying inappropriate content. The preprocessed webpage data is then input into the fine-tuned model for inference. The model analyzes the webpage's text content, images, and layout information to predict whether each webpage contains inappropriate content. Based on the detection results, webpage samples identified as inappropriate and their labels are fed back into the dataset for further model training and optimization. This process can be repeated regularly to continuously improve the model's accuracy and adaptability.

[0079] Based on the above example scenarios, please refer to Figure 1 In one embodiment, the multimodal bad web page detection method of the present application includes:

[0080] Step S10: responding to the bad web page detection instruction, determining the web page address corresponding to the web page to be detected, and obtaining a web page screenshot and web page HTML node information corresponding to the web page to be detected based on the web page address, wherein the web page HTML node information includes the two-dimensional coordinates of the web page where the node is located and web page text data, wherein the web page text data includes the web page title, web page text content, and web page tags;

[0081] The web page audit management terminal can respond to the bad web page detection instruction, determine the web page address corresponding to the web page to be detected, and obtain the web page screenshot and web page HTML node information corresponding to the web page to be detected based on the web page address. The web page HTML node information includes the two-dimensional coordinates of the web page where the node is located and the web page text data. The web page text data is composed of the web page title, web page text content, and web page tags. The bad web pages include one or any multiple of maintenance-stop web pages, cheating-inducing web pages, invalid element layout web pages, disordered layout web pages, harmful information web pages, and web pages with many advertisements and horizontal insertion of text that affects reading.

[0082] Specifically, the maintenance-stopped web page is a web page that is no longer maintained: such web pages may contain outdated or inaccurate information because they have not been updated for a long time or no one maintains them, so they need to be marked as bad web pages; the cheating-guided web pages involve fraud, guiding or inducing users to enter bad content and are a type of bad web page, which need to be accurately marked to ensure effective filtering by the system; the element layout invalid web page is a web page with invalid element layout, which may cause the page to appear abnormal or difficult to browse normally; the layout disordered web page is a web page with disordered layout that may cause problems with user reading experience as network content is constantly updated, so this type of web page is also a bad web page; the harmful information web page is a web page involving harmful information and should be clearly marked to ensure that the system can effectively identify and filter such harmful information; the web page with many advertisements that are inserted horizontally into the text and affect reading is a web page with too many advertisements and improper insertion method that may disrupt the text content and affect the user's reading experience.

[0083] In some embodiments, see Figure 2 To train a bad web page detection model, web crawlers can first collect information about web page elements, including web page addresses (URLs), HTML, text, images, and coordinates, to provide raw data for subsequent processing. The collected web page data is manually or semi-automatically annotated to clearly mark bad and normal web pages to meet the model's supervised learning requirements. The annotated web page data is preprocessed, including steps such as text cleaning and image resizing, to ensure that the data format meets the model's input requirements. A multimodal data fusion module integrates the processed web page text, images, and their layout information to provide rich context for the model. Supervised training and fine-tuning are then performed on the prepared dataset to optimize model parameters and improve the accuracy of identifying bad web pages. The fine-tuned model is evaluated using an independent test dataset, with performance assessed using metrics such as precision and recall. If performance does not meet expectations, the data annotation or model structure is adjusted based on feedback, and fine-tuning is performed again. If applicable, the model can be officially released for real-world bad web page detection.

[0084] In some embodiments, the bad web page detection system in the web page audit management terminal includes a data keyword query unit, a web crawler unit, a screenshot unit, an HTML acquisition unit, a text information extraction unit, and the like.

[0085] The data keyword query unit is designed to automatically collect a wide range of search query keywords from multiple online resources to generate a comprehensive query keyword set. This method aims to ensure the diversity and breadth of the dataset, providing a foundation for subsequent web data crawling.

[0086] In the web crawler unit, based on the query keyword set, the web crawler unit uses a breadth-first search (BFS) strategy to automatically crawl web page addresses (URLs) to build a preliminary data set. The breadth-first search strategy ensures that the crawling process can evenly cover various types of web pages, avoiding bias towards a specific topic or field. This strategy ensures the comprehensiveness and representativeness of the data, which helps improve the generalization ability of model training and reasoning. Please refer to Figure 3 , the breadth-first search (BFS) strategy: first visit all adjacent nodes,

[0087] Then their adjacent nodes are traversed in order, and the order of visiting nodes is 1, 2, 3, 4, 5, 6, 7, 8, 9. All the crawled web page addresses (URLs) will be systematically stored in a predetermined database for subsequent processing and analysis.

[0088] The screenshot unit is responsible for taking a comprehensive screenshot of each webpage in the dataset, capturing the entire webpage's presentation. This allows us to capture the webpage's visual appearance, colors, and other visual elements. These screenshots are recorded in detail and stored in the dataset, providing rich visual information for further analysis and model training.

[0089] The HTML acquisition unit uses the webpage address (URL) obtained by the web crawler unit to obtain the HTML source code of the webpage. This step helps capture the structured text information of the webpage, including tags, text content, other HTML elements, and their HTML node coordinates. The acquired HTML content is recorded in detail and stored in a dataset, providing a foundation for subsequent text analysis and model training.

[0090] In the text information extraction unit, the HTML content obtained by the HTML acquisition unit is used to perform a text information extraction operation to extract key text information from the web page. Figure 4 HTML can be thought of as a tree structure. The root node of the document tree is the document itself (Document node). Its branches include HTML elements, text nodes, comments, and so on, forming a hierarchical relationship. Nodes can be element nodes, attribute nodes, text nodes, and so on. Each HTML element is a node, including the root node, element node, and attribute node.

[0091] A deep traversal search strategy is used to extract important web page elements from HTML, including but not limited to the page title and page body, other node label information, and node 2D coordinate information. This approach ensures that text information is extracted in an orderly manner according to the deep structure of the web page, maintaining the integrity and accuracy of the text.

[0092] See also Figure 5 The principle of depth traversal search is: first, explore as deep as possible along a branch until the end of the branch is reached, and then backtrack to the previous branch. The node access order in this process is 1, 2, 4, 5, 8, 3, 6, 7, 9.

[0093] The node's label information and two-dimensional coordinates (x, y, h, w). The two-dimensional coordinates of the web page node are visualized as the coordinate box of each node. x and y represent the coordinates of the upper left corner of the node coordinate box, and h and w represent the horizontal and vertical lengths of the node frame.

[0094] In some embodiments, before the step of obtaining webpage text data and webpage screenshot corresponding to the webpage to be detected according to the webpage address, the following steps are included:

[0095] Step S101, in response to the data pre-processing instruction, determining the webpage title, webpage text content, webpage tag and webpage address corresponding to the webpage to be detected;

[0096] Step S103: Merge the webpage title, webpage text content, webpage tag, and webpage address to determine the webpage text data.

[0097] Step S20: extracting text feature information corresponding to the webpage text data based on the text encoder in the pre-trained bad webpage detection model to construct a word segmentation embedding vector, a word segmentation position index vector, and a word segmentation coordinate index vector corresponding to the webpage text data;

[0098] The bad web page detection model is put into use after being pre-trained to convergence and the evaluation indicators meet the standards. The bad web page detection model mainly includes an image encoder, a text encoder, and a multimodal self-attention network, etc. The multimodal self-attention network mainly includes a neural network model based on a multi-head attention mechanism, etc. After the web page review management terminal obtains the web page text data and web page screenshots corresponding to the web page to be detected according to the web page address, the text encoder in the pre-trained bad web page detection model extracts the text feature information corresponding to the web page text data to construct the word segmentation embedding vector, word segmentation position index vector and word segmentation coordinate index vector corresponding to the web page text data;

[0099] The steps of extracting text feature information corresponding to the webpage text data based on a text encoder in a pre-trained bad webpage detection model to construct a word segmentation embedding vector, a word segmentation position index vector, and a word segmentation coordinate index vector corresponding to the webpage text data include:

[0100] Step S201: using a word segmenter to subdivide the webpage text data into its corresponding semantic units;

[0101] See also Figure 6 , Figure 6 The diagram shows a schematic diagram of generating word segmentation embedding vectors. Based on the text encoder in the pre-trained bad web page detection model, the text feature information corresponding to the web page text data is extracted to construct the word segmentation embedding vector, word segmentation position index vector, and word segmentation coordinate index vector corresponding to the web page text data. In the text processing stage, information such as the web page title, web page text content, web page tags, and web page address is integrated and processed. The purpose of this step is to merge text data from different sources. The task of this stage involves weighing and integrating different information sources so that the input text information can reflect the full picture of the original content as much as possible.

[0102] A word segmenter is used to subdivide the web page title, web page text content, web page tags and web page address into their corresponding semantic units. Using natural language processing technology, the word segmenter in the text encoder subdivides complex natural language text into the smallest semantic units, usually words or a Chinese character. This process helps to establish a more refined semantic representation, laying the foundation for in-depth understanding and processing of text content.

[0103] Step S203: mapping the semantic units based on a preset vocabulary, and converting the semantic units into their corresponding numerical representations;

[0104] After word segmentation, each lexical unit is mapped through a predefined vocabulary table and converted into a corresponding numerical representation, usually called input_ids. The purpose of this step is to convert natural language text into a numerical form that computers can understand. This lays the foundation for subsequent embedding vector generation.

[0105] Step S205: converting the numerical representation into an embedding vector through an embedding layer in the text encoder, mapping each vocabulary unit to a vector representation in a high-dimensional space, to determine a word segmentation embedding vector corresponding to the webpage text data;

[0106] The digitized input_ids are converted into word embedding vectors through the embedding layer and labeled as word_embedding. This step maps each word unit to a vector representation in a high-dimensional space to preserve semantic information and provide input for deep learning models. This vectorized representation can better capture the semantic relationships between words and provide richer contextual information for model training.

[0107] Step S207: embed the one-dimensional index position of each word in the webpage text data and the two-dimensional coordinate position of the coordinate frame where the text is located through the embedding layer to determine the word position index vector and the word coordinate index vector of each word.

[0108] Obtaining the position and coordinates of each word segmentation involves obtaining the one-dimensional index position of each word segmentation in the sequence text, marked as word_1D_pos, and obtaining the two-dimensional coordinate position information of the coordinate box in which the text is located, marked as word_2D_pos. This step is intended to provide the model with information about the position of the word segmentation in the sequence and the overall text coordinates.

[0109] The one-dimensional index position of each word segmentation is embedded to obtain a one-dimensional embedding vector of the word segmentation. The one-dimensional embedding vector of the word segmentation is called the word segmentation position index vector, which is recorded as word_1D_pos_embedding. At the same time, the two-dimensional coordinate position of the coordinate box where the text is located is embedded to generate a word segmentation coordinate index vector, which is recorded as word_2D_pos_embedding. Based on the word segmentation position index vector and the word segmentation coordinate index vector of each word segmentation, the position embedding vector corresponding to the web page title, web page text content, web page tag and web page address is determined. This process helps the model better understand the relative relationship between different parts of the text, strengthen the understanding of position information, and provide the model with more accurate context perception.

[0110] The step of extracting text feature information corresponding to the webpage text data based on the text encoder in the pre-trained bad webpage detection model includes:

[0111] Step S2001: combining and concatenating the web page title, web page text content, web page tags, and web page address in the web page text data and then performing word segmentation;

[0112] Step S2003: using a text encoder to extract the text feature information from the web page title, web page text content, web page tags, and web page address after word segmentation;

[0113] Step S2005: Activate the text feature information and output it.

[0114] This embodiment further discloses the encoding process for the web page text data, and recommends the use of a Bert model or a Transformer model to implement text encoding. These models have achieved remarkable success in the field of natural language processing (NLP) and are capable of understanding complex textual relationships and contexts. Through in-depth analysis of the web page text data, these models can be used to detect objectionable content. By making full use of the outstanding achievements in the existing technology, the encoding efficiency is made higher, and the obtained text semantic vector can more accurately represent the deep semantics of the web page title, web page text content, web page tags, and web page address.

[0115] Step S30: extracting image feature information corresponding to the webpage screenshot based on the image encoder in the pre-trained bad webpage detection model to determine the image serialization embedding vector, image block absolute index position vector, and image coordinate index vector corresponding to the webpage screenshot;

[0116] After extracting text feature information corresponding to the webpage text data based on a text encoder in a pre-trained bad webpage detection model to construct a word segmentation embedding vector, a word segmentation position index vector, and a word segmentation coordinate index vector corresponding to the webpage text data, the image encoder in the pre-trained bad webpage detection model extracts image feature information corresponding to the webpage screenshot to determine an image serialization embedding vector, an image block absolute index position vector, and an image coordinate index vector corresponding to the webpage screenshot;

[0117] The steps of extracting image feature information corresponding to the webpage screenshot based on an image encoder in a pre-trained bad webpage detection model to determine an image serialization embedding vector, an image block absolute index position vector, and an image coordinate index vector corresponding to the webpage screenshot include:

[0118] Step S301: Segment the webpage screenshot into individual image blocks, process the individual image blocks through an image linear flattening module, and perform linear projection on the image blocks to determine an image serialization embedding vector;

[0119] Before the webpage screenshot is divided into individual image blocks based on the image encoder, the image adaptive adjustment module receives the input webpage image and adaptively adjusts it to the specified size of width W and height H. In this process, we denote the width as W and the height as H to ensure the uniform specifications of the image. In the image dicing module of the image encoder, the adaptively adjusted image will be divided into image blocks, denoted as Chunk{C × C}. This step aims to split the large image into C× C small blocks; in the image linear flattening module of the image encoder, the small blocks obtained by segmentation are processed by the image linear flattening module. This process involves linearly projecting the image blocks to finally form an image serialization vector, whose length is given by the formula: The calculated image serialization vector is recorded as chunk_embedding to reflect the characteristics of the image block.

[0120] Step S303: Obtain the absolute sequential position corresponding to each image block and the coordinate position of the upper left corner of each image block, and embed the absolute sequential position and the coordinate position of the upper left corner of each image block through the embedding layer to determine the image block absolute index position vector and the image coordinate index vector.

[0121] Specifically, see Figure 7 In the image sequence and coordinate acquisition module of the image encoder, in the processed image blocks, it is necessary to accurately obtain their sequential index position and coordinate information, which includes the absolute sequential position of the image block, marked as chunk_1D_pos, and the coordinate position of the upper left corner of each image block, marked as chunk_2D_pos.

[0122] See also Figure 8 In the position embedding module of the image encoder, in order to more accurately capture the position information of the image block, the absolute sequential position chunk_1D_pos is embedded to generate a one-dimensional position embedding vector of the image block, called the image block absolute index position vector, denoted as chunk_1D_pos_embedding. At the same time, the coordinate position of the upper left corner of each image block, chunk_2D_pos, is embedded to form a two-dimensional position embedding vector of the image block, called the image coordinate index vector, denoted as chunk_2D_pos_embedding. The image position embedding vector is determined based on the image block absolute index position vector and the image coordinate index vector. This step is dedicated to expressing the position information in the form of an embedding vector so that the accurate position of the image block can be more effectively utilized and analyzed in subsequent processing.

[0123] Step S40: combining and splicing the word segmentation embedding vector with the image serialization embedding vector to determine a word segmentation image block embedding vector, combining and splicing the word segmentation position index vector with the image block absolute index position vector to determine one-dimensional position information, combining and splicing the word segmentation coordinate index vector with the image coordinate index vector to determine two-dimensional position information, and determining a comprehensive embedding vector based on the word segmentation image block embedding vector, the one-dimensional position information, and the two-dimensional position information;

[0124] See also Figure 9 , after determining the image serialization embedding vector, the image block absolute index position vector, and the image coordinate index vector corresponding to the webpage screenshot, combining and splicing the word segmentation embedding vector with the image serialization embedding vector to determine the word segmentation image block embedding vector, combining and splicing the word segmentation position index vector with the image block absolute index position vector to determine one-dimensional position information, combining and splicing the word segmentation coordinate index vector with the image coordinate index vector to determine two-dimensional position information, and determining a comprehensive embedding vector based on the word segmentation image block embedding vector, the one-dimensional position information, and the two-dimensional position information;

[0125] Specifically, first, the word segmentation image block embedding vector is determined by organically combining and splicing the image serialization embedding vector chunk_embedding and the word segmentation embedding vector word_embedding. It is named word_and_chunk_embedding. The word segmentation image block embedding vector not only contains text and image information, but also ensures the orderly integration of image and text information in this way, forming a comprehensive embedding representation, which provides a rich information source for the model.

[0126] Furthermore, the word segmentation position index vector word_1D_pos_embedding is combined and spliced ​​with the image block absolute index position vector chunk_1D_pos_embedding to determine the one-dimensional position information of the image and text. The one-dimensional position information of the image and text is recorded as: 1D_pos_embedding, forming a comprehensive position vector containing the absolute order information of text and image elements.

[0127] Furthermore, the word segmentation coordinate index vector word_2D_pos_embedding and the image coordinate index vector chunk_2D_pos_embedding are combined and spliced ​​to determine the two-dimensional position information of the image and text. The two-dimensional position information of the image and text is recorded as: 2D_pos_embedding, which retains the two-dimensional spatial information of text and image elements and enhances the spatial perception ability of the model.

[0128] The word segmentation image chunk embedding vector, word_and_chunk_embedding, is element-by-element added to the one-dimensional position information of the image and text, 1D_pos_embedding, and the two-dimensional position information of the image and text, 2D_pos_embedding, to form the final input embedding vector, also known as the integrated embedding vector, which can be called inputs_embedding. This integrated fusion result provides more informative and rich input features for subsequent models, supporting more accurate and comprehensive data processing.

[0129] The input embedding vector inputs_embedding is fed into a multimodal self-attention network. Its primary goal is to enable the model to consider the entire sequence while processing any part of the input sequence. The scores generated by the self-attention mechanism are used to weight the input features, generating new feature representations that incorporate global contextual information. This process is repeatable, and through multiple layers of attention, each model output incorporates comprehensive information from the entire webpage.

[0130] Step S50: Perform multi-level encoding on the comprehensive embedding vector based on the multimodal self-attention network to determine the web page image-text fusion vector, convert the web page image-text fusion vector into a two-dimensional vector through hidden layer linear mapping and nonlinear activation for normalization, and when the probability that the web page to be detected in the two-dimensional vector is a bad web page is greater than the probability that the web page to be detected is a normal web page, determine the web page to be detected as a bad web page to complete the detection of the bad web page.

[0131] After determining the comprehensive embedding vector based on the word segmentation image block embedding vector, one-dimensional position information and two-dimensional position information, the comprehensive embedding vector is multi-level encoded based on the multimodal self-attention network to determine the web page image and text fusion vector, and the web page image and text fusion vector is normalized and converted into a two-dimensional vector through hidden layer linear mapping and nonlinear activation. When the probability that the web page to be detected in the two-dimensional vector is a bad web page is greater than the probability that the web page to be detected is a normal web page, the web page to be detected is determined to be a bad web page to complete the detection of bad web pages.

[0132] Furthermore, the step of performing multi-level encoding on the comprehensive embedding vector based on a multimodal self-attention network to determine the webpage image-text fusion vector includes:

[0133] Step S501: In each level of encoding, the encoder corresponding to the current level is used to construct a query vector, a key vector, and a value vector for each row vector in the currently input comprehensive embedding vector;

[0134] Step S503: Perform a weighted sum of the value vectors corresponding to all row vectors, where the weight corresponding to each value vector is the normalized score of the dot product of its corresponding key vector and the query vector corresponding to other row vectors;

[0135] Step S505: Multiply the weighted summation result by the conversion weight matrix to obtain its corresponding intermediate information matrix. Perform feature extraction on the intermediate information matrix through a multi-layer perceptron to obtain a graphic-text interaction vector as the comprehensive embedding vector required by the next encoder, or directly output as a web page graphic-text fusion vector in the last encoder.

[0136] First, an attention score is calculated: for each element in the sequence, the multi-head attention mechanism neural network in the multimodal self-attention network of the bad web page detection model calculates its mutual influence with all other elements in the sequence. This is done through three learnable weight matrices, usually called query, key, and value matrices. The feature vector is multiplied by these three matrices to obtain the query vector, key vector, and value vector respectively.

[0137] Generate attention weights: Calculate the dot product for each pair of query and key vectors to obtain the raw attention score, and adjust it through the scaled dot product attention mechanism.

[0138] ,

[0139] Among them, Q is the query vector, K is the key vector, is the vector dimension. Next, the one-dimensional position information of the image and text and the two-dimensional position information of the image and text are added as offset terms. The formula is as follows:

[0140]

[0141] in, and The bias terms representing the one-dimensional position information of the image and text and the two-dimensional position information of the image and text are added to the original attention score to adjust the relative importance of elements when generating attention weights, thereby better capturing global context information.

[0142] Finally, these scores are normalized by applying the Softmax function to generate the final attention weights:

[0143] ,

[0144] The above formula expresses the relative importance of other elements in generating the context-sensitive representation of each element.

[0145] Context-sensitive output synthesis: By performing a weighted summation of the value vector and the corresponding attention weight, an output vector is generated for each element that incorporates information from other parts of the entire input sequence. This step integrates information from the entire input sequence to form a feature representation rich in global context.

[0146] Multi-head attention parallelization: In order to capture different aspects of the input sequence, the self-attention process can be performed in parallel in a multi-head attention mechanism. Each "head" uses a different weight matrix and independently calculates the attention score and output vector, which can be expressed as follows:

[0147] ,

[0148] in, The self-attention weight of a certain attention head, and then concatenate the outputs of all heads.

[0149]

[0150] Here, h is the number of attention heads.

[0151] Finally, the final multi-head attention output is generated through linear transformation:

[0152] ,

[0153] The output of the multi-head attention mechanism neural network is a web page image and text fusion vector.

[0154] After the multi-head attention mechanism neural network in the bad web page detection model outputs the web page image and text fusion vector, the web page image and text fusion vector is normalized and converted into a two-dimensional vector through hidden layer linear mapping and nonlinear activation. The two-dimensional vector is expressed as [PM, PN], where PM represents the probability of a bad web page and PN represents the probability of a normal web page. This mapping operation is intended to convert the network output into a probability space to support more effective binary classification.

[0155] At this time, when the probability PM of the web page to be detected in the two-dimensional vector being a bad web page is greater than the probability PN of the web page to be detected being a normal web page, the web page to be detected is determined to be a bad web page such as a maintenance-stop web page, a cheating-inducing web page, an element layout invalid web page, a web page with disordered layout, a web page with harmful information, and a web page with many advertisements and horizontal insertion of the text that affects reading, so as to complete the detection of bad web pages.

[0156] In some embodiments, when bad web pages such as maintenance-stopped web pages, cheating-inducing web pages, invalid element layout web pages, disordered layout web pages, harmful information web pages, and web pages with many advertisements and horizontal insertion of text that affect reading are detected, the bad web pages are sorted in order from small to large according to the probability of the bad web pages, so as to complete the detection of bad web pages.

[0157] In some embodiments, the bad web page detection model is trained by supervised learning, taking into account the problem of class imbalance in the sample set, and using Focal Loss function to enhance the model's recognition ability for minority classes (ie, bad web pages).

[0158] The Focal Loss formula is as follows:

[0159] ,

[0160] in, Represents the probability that the model predicts that the sample t belongs to a bad web page, represents the weight of sample t, An adjustment factor representing the difficulty of the sample.

[0161] Evaluation indicators are used to evaluate the bad web page detection model, where positive examples represent bad web pages and negative examples represent normal web pages. True Positive (TP): a positive example is correctly predicted as a positive example, True Negative (TN): a negative example is correctly predicted as a negative example, False Positive (FP): a negative example is incorrectly predicted as a positive example, and False Negative (FN): a positive example is incorrectly predicted as a negative example. In order to evaluate the performance of the model, the following evaluation indicators are used:

[0162] Accuracy: The percentage of samples that are correctly predicted among all samples. The formula is as follows:

[0163] ,

[0164] Precision: The probability of a sample being actually positive among all samples predicted to be positive. The formula is as follows:

[0165] ,

[0166] Recall is the probability of being predicted as a positive sample in an actual positive sample. Its formula is as follows:

[0167] ,

[0168] The F1 score (F1 Score), also known as the balanced F score (balanced F Score), is defined as the harmonic mean of precision and recall.

[0169] ,

[0170] See also Figure 10 , the confusion matrix is ​​as follows Figure 10 As shown, 0 represents a bad web page and 1 represents a normal web page. The vertical axis represents whether the real web page is a bad web page, and the horizontal axis represents whether the predicted web page is a bad web page. A total of 334 bad web pages were identified, of which 293 were bad web pages. A total of 1,830 normal pages were identified, of which 1,815 were identified as normal pages. The precision of bad web pages was 95%, the recall rate was 88%, and the F1 score was 98%. The precision of normal web pages was 98%, the recall rate was 99%, and the F1 score was 98%. These results show that the model training and fine-tuning method proposed in this application can effectively improve the recognition accuracy of bad web pages while maintaining a high recognition ability for normal web pages.

[0171] Compared with the existing technology, this application addresses the problems of some website designers taking improper measures to improve web page rankings, which distorts the authenticity and quality of web page content and may also cause web pages carrying security threats to be displayed in large quantities. This application includes but is not limited to the following beneficial effects:

[0172] First, the multimodal bad web page detection method of the present application can comprehensively understand multimodal content. Traditional methods often process text and images separately, making it difficult to capture the relationship between them. The present application achieves a more comprehensive understanding of web page content by integrating text, images, and their layout information. In particular, when processing web pages with both text and images, it can more accurately identify bad content.

[0173] Secondly, the multimodal method for detecting bad web pages in this application can effectively utilize contextual information. In detecting bad web pages, it is difficult to accurately determine whether the content is bad based solely on keywords or image features. This application utilizes typesetting coordinate information to understand the positional relationship of text on the page, improving understanding of the context, which helps to reduce false positives and missed positives.

[0174] Third, the multimodal bad web page detection method of this application is capable of handling complex page structures. Modern web page designs are relatively complex, and relying solely on text analysis or simple visual feature recognition is not enough to accurately judge web page content. This application can understand complex page layout and design elements, thereby effectively identifying bad content even in complex web page structures.

[0175] Furthermore, the bad webpage detection model of this application is used to comprehensively process the text, images, and layout information in the webpage to achieve a comprehensive understanding and accurate identification of bad webpages. This multimodal information fusion method enables the system to better cope with the complexity and diversity of modern webpage design.

[0176] The model in this application can understand the positional relationship between text and images in web page layout, so as to better use contextual information to identify objectionable content. This context-aware capability helps reduce false positives and false negatives, and improves the accuracy of objectionable web page detection.

[0177] The system implements a continuous feedback and optimization mechanism for detection results, continuously updating model parameters based on actual recognition situations to improve detection effectiveness and adaptability. This dynamic learning and optimization capability makes the system more adaptable and robust, able to cope with the ever-changing network environment and the characteristics of undesirable web pages;

[0178] It has high efficiency and scalability when processing complex web page structures and large-scale data sets. Both model training and inference processes can be completed quickly in a high-performance computing environment, making it suitable for real-time, large-scale bad web page detection tasks.

[0179] Furthermore, this application can greatly improve the quality of web pages, prevent some website designers from taking improper means to improve web page rankings, and avoid distorting the authenticity and quality of web page content and preventing web pages carrying security threats from being displayed in large quantities.

[0180] See also Figure 11A multimodal bad web page detection device is provided to meet one of the purposes of this application, including a data acquisition module 1100, a text vector determination module 1200, an image vector determination module 1300, a comprehensive vector determination module 1400 and a bad web page detection module 1500. Among them, the data acquisition module 1100 is configured to respond to the bad web page detection instruction, determine the web page address corresponding to the web page to be detected, and obtain the web page screenshot and web page HTML node information corresponding to the web page to be detected according to the web page address, wherein the web page HTML node information includes the two-dimensional coordinates of the web page where the node is located and the web page text data, and the web page text data consists of the web page title, web page text content, and web page tags; the text vector determination module 1200 is configured to extract the text feature information corresponding to the web page text data based on the text encoder in the pre-trained bad web page detection model to construct the word segmentation embedding vector, word segmentation position index vector and word segmentation coordinate index vector corresponding to the web page text data; the image vector determination module 1300 is configured to extract the image feature information corresponding to the web page screenshot based on the image encoder in the pre-trained bad web page detection model to determine the image serialization embedding vector, image block absolute index position vector and image block absolute index position vector corresponding to the web page screenshot. Coordinate index vector; comprehensive vector determination module 1400, configured to combine and splice the word segmentation embedding vector with the image serialization embedding vector to determine the word segmentation image block embedding vector, combine and splice the word segmentation position index vector with the image block absolute index position vector to determine one-dimensional position information, combine and splice the word segmentation coordinate index vector with the image coordinate index vector to determine two-dimensional position information, and determine a comprehensive embedding vector based on the word segmentation image block embedding vector, the one-dimensional position information and the two-dimensional position information; bad web page detection module 1500, configured to perform multi-level encoding on the comprehensive embedding vector based on a multimodal self-attention network to determine a web page image-text fusion vector, convert the web page image-text fusion vector into a two-dimensional vector through hidden layer linear mapping and nonlinear activation for normalization, and when the probability that the web page to be detected in the two-dimensional vector is a bad web page is greater than the probability that the web page to be detected is a normal web page, the web page to be detected is determined to be a bad web page to complete the detection of bad web pages.

[0181] Based on any embodiment of this application, please refer to Figure 12 Another embodiment of the present application further provides an electronic device, which can be implemented by a computer device, such as Figure 12As shown, a schematic diagram of the internal structure of a computer device. The computer device includes a processor, a computer-readable storage medium, a memory, and a network interface connected via a system bus. Among them, the computer-readable storage medium of the computer device stores an operating system, a database, and computer-readable instructions, and the database may store a control information sequence. When the computer-readable instructions are executed by the processor, the processor can implement a multi-modal bad web page detection method. The processor of the computer device is used to provide computing and control capabilities to support the operation of the entire computer device. The memory of the computer device may store computer-readable instructions. When the computer-readable instructions are executed by the processor, the processor can execute the multi-modal bad web page detection method of the present application. The network interface of the computer device is used to connect and communicate with the terminal. Those skilled in the art will understand that Figure 12 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0182] In this embodiment, the processor is used to execute Figure 11 The memory stores the program code and various data required to execute the modules and submodules. The network interface is used to transmit data between user terminals and servers. The memory in this embodiment stores the program code and data required to execute all modules and submodules in the multimodal bad web page detection device of this application. The server can call the server's program code and data to execute the functions of all submodules.

[0183] The present application also provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the multimodal bad web page detection method described in any embodiment of the present application.

[0184] The present application also provides a computer program product, including a computer program / instruction, which, when executed by one or more processors, implements the steps of the multimodal-based bad web page detection method described in any embodiment of the present application.

[0185] Those skilled in the art will appreciate that all or part of the processes in the above-described embodiments of the present application can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes in the above-described embodiments of the method. The aforementioned storage medium can be a computer-readable storage medium such as a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0186] The above description is only a partial implementation method of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

[0187] In summary, this application can greatly improve the quality of web pages, prevent some website designers from taking improper means to improve web page rankings, and avoid distorting the authenticity and quality of web page content and preventing web pages carrying security threats from being displayed in large quantities.

Claims

1. A multimodal bad web page detection method, characterized in that: include: In response to a bad web page detection instruction, determining a web page address corresponding to a web page to be detected, and obtaining a web page screenshot and web page HTML node information corresponding to the web page to be detected based on the web page address, wherein the web page HTML node information includes two-dimensional coordinates of the web page where the node is located and web page text data, wherein the web page text data includes a web page title, web page text content, and web page tags; Extracting text feature information corresponding to the webpage text data based on a text encoder in a pre-trained bad webpage detection model to construct a word segmentation embedding vector, a word segmentation position index vector, and a word segmentation coordinate index vector corresponding to the webpage text data; Extracting image feature information corresponding to the webpage screenshot based on an image encoder in a pre-trained bad webpage detection model to determine an image serialization embedding vector, an image block absolute index position vector, and an image coordinate index vector corresponding to the webpage screenshot; Combine and concatenate the word segmentation embedding vector with the image serialization embedding vector to determine a word segmentation image block embedding vector, combine and concatenate the word segmentation position index vector with the image block absolute index position vector to determine one-dimensional position information, combine and concatenate the word segmentation coordinate index vector with the image coordinate index vector to determine two-dimensional position information, and determine a comprehensive embedding vector based on the word segmentation image block embedding vector, the one-dimensional position information, and the two-dimensional position information; Based on the multimodal self-attention network, the comprehensive embedding vector is multi-level encoded to determine the web page image and text fusion vector, and the web page image and text fusion vector is normalized and converted into a two-dimensional vector through hidden layer linear mapping and nonlinear activation. When the probability that the web page to be detected in the two-dimensional vector is a bad web page is greater than the probability that the web page to be detected is a normal web page, the web page to be detected is determined to be a bad web page to complete the detection of bad web pages.

2. The multimodal bad web page detection method according to claim 1, characterized in that: The steps of extracting text feature information corresponding to the webpage text data based on a text encoder in a pre-trained bad webpage detection model to construct a word segmentation embedding vector, a word segmentation position index vector, and a word segmentation coordinate index vector corresponding to the webpage text data include: Using a word segmenter to subdivide the webpage text data into its corresponding semantic units; Mapping the semantic units based on a preset vocabulary, and converting the semantic units into their corresponding numerical representations; Converting the numerical representation into an embedding vector through an embedding layer in the text encoder, mapping each vocabulary unit to a vector representation in a high-dimensional space, to determine a word segmentation embedding vector corresponding to the webpage text data; The one-dimensional index position of each word in the webpage text data and the two-dimensional coordinate position of the coordinate frame where the text is located are respectively embedded through the embedding layer to determine the word position index vector and the word coordinate index vector of each word.

3. The multimodal bad web page detection method according to claim 1, characterized in that: The steps of extracting image feature information corresponding to the webpage screenshot based on an image encoder in a pre-trained bad webpage detection model to determine an image serialization embedding vector, an image block absolute index position vector, and an image coordinate index vector corresponding to the webpage screenshot include: Segmenting the webpage screenshot into individual image blocks, processing the individual image blocks through an image linear flattening module, and linearly projecting the image blocks to determine an image serialization embedding vector; Obtain the absolute sequential position corresponding to each image block and the coordinate position of the upper left corner of each image block, and embed the absolute sequential position and the coordinate position of the upper left corner of each image block through an embedding layer to determine the image block absolute index position vector and the image coordinate index vector.

4. The multimodal bad web page detection method according to claim 1, characterized in that: The step of performing multi-level encoding on the comprehensive embedding vector based on a multimodal self-attention network to determine a web page image and text fusion vector includes: In each level of encoding, the encoder corresponding to the current level is used to construct a query vector, a key vector and a value vector for each row vector in the currently input comprehensive embedding vector; Perform a weighted sum of the value vectors corresponding to all row vectors, where the weight of each value vector is the normalized score of the dot product of its corresponding key vector and the query vector corresponding to other row vectors; The weighted summation result is multiplied by the conversion weight matrix to obtain its corresponding intermediate information matrix. The intermediate information matrix is ​​subjected to feature extraction by a multi-layer perceptron to obtain the image-text interaction vector, which is used as the comprehensive embedding vector required by the next encoder, or directly output as the web page image-text fusion vector in the last encoder.

5. The multimodal bad web page detection method according to claim 1, characterized in that: The step of extracting text feature information corresponding to the webpage text data based on the text encoder in the pre-trained bad webpage detection model includes: The web page title, web page text content, web page tag and web page address in the web page text data are combined and spliced ​​and then segmented; Using a text encoder to extract the text feature information from the web page title, web page text content, web page tags and web page address after word segmentation; The text feature information is activated and then output.

6. The multimodal bad web page detection method according to claim 1, characterized in that: Before the step of obtaining webpage text data and webpage screenshot corresponding to the webpage to be detected according to the webpage address, the method includes: In response to the data preprocessing instruction, determining the webpage title, webpage text content, webpage tag and webpage address corresponding to the webpage to be detected; The web page title, web page text content, web page tag and web page address are combined to determine the web page text data.

7. The multimodal bad web page detection method according to any one of claims 1 to 6, characterized in that: The bad web pages include one or more of maintenance-stopped web pages, cheating-inducing web pages, invalid element layout web pages, disordered layout web pages, harmful information web pages, and web pages with many advertisements and horizontal insertion of the main text that affect reading.

8. A multimodal bad web page detection device, characterized in that: include: a data acquisition module configured to respond to a bad web page detection instruction, determine a web page address corresponding to a web page to be detected, and obtain a web page screenshot and web page HTML node information corresponding to the web page to be detected based on the web page address, wherein the web page HTML node information includes the two-dimensional coordinates of the web page where the node is located and web page text data, wherein the web page text data includes a web page title, web page text content, and web page tags; a text vector determination module configured to extract text feature information corresponding to the webpage text data based on a text encoder in a pre-trained bad webpage detection model to construct a word segmentation embedding vector, a word segmentation position index vector, and a word segmentation coordinate index vector corresponding to the webpage text data; an image vector determination module configured to extract image feature information corresponding to the webpage screenshot based on an image encoder in a pre-trained bad webpage detection model to determine an image serialization embedding vector, an image block absolute index position vector, and an image coordinate index vector corresponding to the webpage screenshot; a comprehensive vector determination module configured to combine and splice the word segmentation embedding vector with the image serialization embedding vector to determine a word segmentation image block embedding vector, combine and splice the word segmentation position index vector with the image block absolute index position vector to determine one-dimensional position information, combine and splice the word segmentation coordinate index vector with the image coordinate index vector to determine two-dimensional position information, and determine a comprehensive embedding vector based on the word segmentation image block embedding vector, the one-dimensional position information, and the two-dimensional position information; The bad web page detection module is configured to perform multi-level encoding on the comprehensive embedding vector based on a multimodal self-attention network to determine a web page image-text fusion vector, and convert the web page image-text fusion vector into a two-dimensional vector through hidden layer linear mapping and nonlinear activation for normalization. When the probability that the web page to be detected in the two-dimensional vector is a bad web page is greater than the probability that the web page to be detected is a normal web page, the web page to be detected is determined to be a bad web page to complete the detection of bad web pages.

9. An electronic device comprising a central processing unit and a memory, characterized in that: The central processing unit is configured to call and run a computer program stored in the memory to execute the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that It stores a computer program implemented according to the method described in any one of claims 1 to 7 in the form of computer-readable instructions, and when the computer program is called and executed by a computer, the steps included in the corresponding method are executed.

Images