A Method and System for Predicting Web Page Saliency Based on Generative Adversarial Networks

Abstract

This invention provides a method and system for predicting webpage saliency based on generative adversarial networks. The method includes: acquiring a webpage snapshot and low-level features of the webpage to be predicted, wherein the low-level features are webpage boundary features and text features of the webpage to be predicted; inputting the webpage snapshot and the low-level features into a trained webpage saliency prediction model, and outputting a predicted webpage saliency result for the webpage to be predicted; the trained webpage saliency prediction model is obtained by training a deep convolutional neural network using sample webpage snapshots and the corresponding low-level features of the sample webpages. This invention improves the accuracy and efficiency of webpage saliency prediction.

Description

Technical Field

[0001] This invention relates to the field of Internet technology, and in particular to a method and system for predicting webpage saliency based on generative adversarial networks. Background Technology

[0002] In recent decades, with the rapid development of the internet and the widespread adoption of search engines and social networks, web pages have gradually become one of the most important channels for people to obtain information. According to statistics from Internet Live Stats Online, the total number of websites worldwide has reached nearly 1.7 billion, and the number of internet users has exceeded 4 billion. Statistics show that people spend approximately 23 hours a week browsing web pages, and this number is still growing. This increased time spent browsing web pages not only changes people's lifestyles but also significantly impacts the marketing behavior of businesses, such as advertising placement, webpage layout design, and webpage loading optimization. Therefore, studying how user attention is deployed and guided when browsing web pages has significant scientific research and commercial application value. Webpage saliency is a saliency map that can be used to represent the distribution of end-user attention on a webpage.

[0003] Webpage saliency is significant for both industry and academia. A study by Amazon showed that reducing webpage loading time on client browsers by one second could increase their profits by approximately $1.6 billion. Inspired by this, Kelton et al. first used webpage saliency to calculate the most salient areas on a webpage, then prioritized sending this content to client browsers to reduce loading time and improve user experience. Furthermore, webpage saliency can be used to boost advertising revenue. Chen et al. used webpage saliency to calculate which areas are more likely to attract end-user attention, then used these areas to improve personalized online document, image, and video ad recommendations. Similarly, Xiang et al. used webpage saliency to select the most suitable ads and used it in their video ad system to increase revenue. Web designers can also benefit from webpage saliency by improving typography and layout. For example, knowing which parts of a webpage are most likely to attract user visual attention allows web designers to arrange these important webpage elements more effectively. Moreover, such improvements can also lead to a more user-friendly experience for end-users.

[0004] Current research on predicting webpage saliency maps is relatively limited, and existing methods for predicting webpage saliency suffer from several drawbacks. These include the need for manual feature selection, the requirement to reselect pixels from webpage snapshots as training data, model sensitivity to data, insufficient accuracy in predicting webpage saliency, and poor generalization and scalability. Therefore, there is an urgent need for a webpage saliency prediction method and system based on generative adversarial networks to address these issues. Summary of the Invention

[0005] To address the problems existing in the prior art, embodiments of the present invention provide a method and system for predicting webpage saliency based on generative adversarial networks.

[0006] In a first aspect, embodiments of the present invention provide a webpage saliency prediction method based on generative adversarial networks, comprising:

[0007] Obtain a snapshot of the webpage to be predicted and the underlying features of the webpage, wherein the underlying features of the webpage are the webpage boundary features and text features of the webpage to be predicted;

[0008] The webpage snapshot and the underlying features of the webpage are input into the trained webpage saliency prediction model, and the webpage saliency prediction result of the webpage to be predicted is output. The trained webpage saliency prediction model is obtained by training a deep convolutional neural network with the sample webpage snapshot and the underlying features of the sample webpage corresponding to the sample webpage snapshot.

[0009] Furthermore, the trained webpage saliency prediction model is obtained through the following steps:

[0010] Obtain a training sample set, which includes sample webpage snapshots, and the sample webpage low-level features and sample webpage saliency maps corresponding to the sample webpage snapshots;

[0011] Based on the sample webpage snapshots and the low-level features of the sample webpages, the generator network is trained to obtain the saliency map of the predicted webpages.

[0012] Based on the saliency map of the predicted webpage and the saliency map of the sample webpage, the discriminator network is trained. If the training result meets the preset conditions, the trained webpage saliency prediction model is obtained; if not, the parameters of the generator network and the discriminator network are updated through the backpropagation algorithm so that the updated generator network and discriminator network are trained until the training result meets the preset conditions.

[0013] Furthermore, the generator network is constructed using residual blocks and a self-attention mechanism.

[0014] Furthermore, the adversarial loss function of the discriminator network is:

[0015]

[0016] The adversarial loss function of the generator network is:

[0017] L' GAN (G,D)=-E F,S [D(F,G(F,S))];

[0018] Where F represents the sample webpage snapshot and its corresponding sample webpage underlying features, S represents the real sample webpage saliency map, G represents the generator network, and D represents the discriminator network.

[0019] Furthermore, after training the discriminator network based on the predicted webpage saliency map and the sample webpage saliency map, the method further includes:

[0020] Based on the discriminator network, the saliency maps of the predicted web pages and the sample web pages are judged to be true or false. A reconstruction loss function is then constructed based on the judgment results. The reconstruction loss function is as follows:

[0021]

[0022] Based on the reconstruction loss function and the adversarial loss function, an optimization objective function is constructed to obtain the trained webpage saliency prediction model. The optimization objective function is:

[0023]

[0024] Among them, L GAN (G,D) represents the adversarial loss function, L recons (G) represents the reconstruction loss function, j represents the number of samples, and S j This represents the true webpage saliency graph of the j-th sample. Let denot represent the webpage saliency map generated by the generator network for the j-th sample; λ is used to control the weights of the sample's predicted webpage saliency map and the sample's webpage saliency map.

[0025] Furthermore, obtaining the training sample set includes:

[0026] The low-level features of the sample webpage snapshot are extracted by the edge detector to obtain the sample low-level features corresponding to the sample webpage snapshot, which are used to construct a training sample set. The edge detector includes a Canny filter, a Sobel filter and a Laplace filter.

[0027] Secondly, embodiments of the present invention provide a webpage saliency prediction system based on generative adversarial networks, comprising:

[0028] The acquisition module is used to acquire a webpage snapshot and low-level features of the webpage to be predicted, wherein the low-level features are the webpage boundary features and text features of the webpage to be predicted.

[0029] The prediction module is used to input the webpage snapshot and the underlying features of the webpage into the trained webpage saliency prediction model, and output the webpage saliency prediction result of the webpage to be predicted; the trained webpage saliency prediction model is obtained by training a deep convolutional neural network with the sample webpage snapshot and the underlying features of the sample webpage corresponding to the sample webpage snapshot.

[0030] Furthermore, the system also includes:

[0031] The sample set construction module is used to obtain a training sample set, which includes sample webpage snapshots, and the sample webpage low-level features and sample webpage saliency maps corresponding to the sample webpage snapshots;

[0032] The generator training module is used to train the generator network based on the sample webpage snapshot and the low-level features of the sample webpage to obtain the saliency map of the sample predicted webpage.

[0033] The discriminator training module is used to train the discriminator network based on the saliency map of the sample predicted webpage and the saliency map of the sample webpage. If the training result meets the preset conditions, the trained webpage saliency prediction model is obtained; if not, the parameters of the generator network and the discriminator network are updated through the backpropagation algorithm so that the updated generator network and discriminator network are trained until the training result meets the preset conditions.

[0034] Thirdly, embodiments of the present invention provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method provided in the first aspect.

[0035] Fourthly, embodiments of the present invention provide a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method provided in the first aspect.

[0036] This invention provides a webpage saliency prediction method and system based on generative adversarial networks (GANs). By taking a complete webpage snapshot and its corresponding features as input, a GAN is constructed based on a deep convolutional neural network to output the webpage saliency. This method addresses the shortcomings of existing webpage saliency prediction algorithms that do not consider spatial information between pixels, thereby improving the accuracy and efficiency of webpage saliency prediction. Furthermore, by eliminating the need for selecting training data, overfitting is mitigated to some extent, and the time spent on data selection is saved. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 A flowchart illustrating the webpage saliency prediction method based on generative adversarial networks provided in this embodiment of the invention;

[0039] Figure 2 This invention provides an overall architecture diagram of a webpage saliency graph prediction model for embodiments of the invention.

[0040] Figure 3 A schematic diagram of the residual block structure is provided for embodiments of the present invention;

[0041] Figure 4 A schematic diagram illustrating the underlying features of a webpage as provided in an embodiment of the present invention;

[0042] Figure 5 A flowchart illustrating the self-attention mechanism provided in an embodiment of the present invention;

[0043] Figure 6 A schematic diagram of the structure of a webpage saliency prediction system based on generative adversarial networks provided in an embodiment of the present invention;

[0044] Figure 7 This is a schematic diagram of the electronic device structure provided in an embodiment of the present invention. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0046] With the introduction of Generative Adversarial Networks (GANs) and the development of related technologies in recent years, the performance of many tasks in the field of image generation has been improved. Since GANs are typically implemented using deep neural networks, they can effectively extract latent important features from data, eliminating the need for manual feature extraction. Furthermore, GANs can guide the image generation process based on certain additional conditional information. These advantages enable GANs to generate visually clear and realistic images in an end-to-end manner. Therefore, GANs have been used to complete many image generation tasks in computer vision, such as image super-resolution reconstruction, image style transfer, and image inpainting.

[0047] Since users ultimately see a loaded and displayed webpage, this can be viewed as a highly complex image containing text, images, and bounding boxes. Therefore, predicting the saliency map of a webpage can be considered an image generation process, meaning generative adversarial networks (GANs) can be used to predict the saliency map. However, webpages have many features different from purely natural images. For example, webpages often have a distinct topological layout, resulting in a visually apparent "blocking" characteristic. Therefore, predicting the saliency map cannot be treated merely as an image generation task; unique structural features of webpages must also be considered. This means that in addition to using a snapshot of the webpage as input to the GAN, other structural features of the webpage, such as element location, element size, and element type, should be input as conditional information. To overcome the shortcomings and deficiencies of existing webpage saliency prediction methods and improve the prediction results and efficiency, this invention proposes a webpage saliency prediction algorithm based on generative adversarial networks. By drawing on new methods and ideas from different fields, this algorithm addresses the shortcomings of existing webpage saliency prediction algorithms and effectively improves the prediction performance.

[0048] Figure 1 This is a flowchart illustrating the webpage saliency prediction method based on generative adversarial networks provided in an embodiment of the present invention, as shown below. Figure 1 As shown, this embodiment of the invention provides a webpage saliency prediction method based on generative adversarial networks, including:

[0049] Step 101: Obtain a webpage snapshot and underlying features of the webpage to be predicted. The underlying features are the webpage boundary features and text features of the webpage to be predicted.

[0050] Step 102: Input the webpage snapshot and the webpage underlying features into the trained webpage saliency prediction model, and output the webpage saliency prediction result of the webpage to be predicted; the trained webpage saliency prediction model is obtained by training a deep convolutional neural network with the sample webpage snapshot and the sample webpage underlying features corresponding to the sample webpage snapshot.

[0051] In this embodiment of the invention, a complete snapshot of the webpage to be predicted and the corresponding underlying features of the webpage are input into a trained model. This model is built on a deep convolutional neural network, and the working method of the deep convolutional neural network can take into account the spatial information of the input. Figure 2 The overall architecture diagram of the webpage saliency graph prediction model is provided for embodiments of the present invention, which can be referred to. Figure 2 As shown, the structure of the webpage saliency map prediction model can be roughly divided into three algorithm modules. The upper left section is a generator network module following an encoder-decoder structure. The encoder extracts features from the webpage snapshot, and the decoder generates the corresponding webpage saliency map based on the extracted snapshot features and the low-level features extracted by the feature extraction network module. The lower left section is a feature extraction network module, which performs high-level abstraction on the low-level features of the webpage extracted beforehand using filters, thus obtaining the low-level features of the webpage. The right side is a discriminator network module, used for adversarial training of the entire network to ensure the generation of clear and realistic-looking webpage saliency maps. In this embodiment, the generator network module and the feature extraction network module together constitute the generator part, while the discriminator network module constitutes the discriminator part alone.

[0052] The webpage saliency prediction method based on generative adversarial networks provided in this invention takes a complete webpage snapshot and its corresponding features as input, constructs a generative adversarial network based on a deep convolutional neural network, and outputs the webpage saliency. This method can, to some extent, address the shortcomings of existing webpage saliency prediction algorithms that do not consider spatial information between pixels, thereby improving the accuracy and efficiency of webpage saliency prediction. Furthermore, by eliminating the process of selecting training data, it reduces overfitting to some extent and saves the time consumed by data selection.

[0053] Based on the above embodiments, the trained webpage saliency prediction model is obtained through the following steps:

[0054] Obtain a training sample set, which includes sample webpage snapshots, and the sample webpage low-level features and sample webpage saliency maps corresponding to the sample webpage snapshots;

[0055] Based on the sample webpage snapshots and the low-level features of the sample webpages, the generator network is trained to obtain the saliency map of the predicted webpages.

[0056] Based on the saliency map of the predicted webpage and the saliency map of the sample webpage, the discriminator network is trained. If the training result meets the preset conditions, the trained webpage saliency prediction model is obtained; if not, the parameters of the generator network and the discriminator network are updated through the backpropagation algorithm so that the updated generator network and discriminator network are trained until the training result meets the preset conditions.

[0057] In this embodiment of the invention, the network structure of the generator network module has a significant impact on the performance of the final algorithm. However, the design of the network structure should not be too complex or too simple. If the network structure of the generator network module is too simple, the generator will be unable to generate sufficiently clear and realistic images; conversely, if the network structure of the generator network module is too complex, training the entire generator network module will become difficult, potentially leading to network convergence failure. It should be noted that the generator network module in this embodiment of the invention is part of the generator network, specifically the encoder-decoder structure generator network module described in the above embodiments. To better ensure the performance of the generative adversarial network, the webpage saliency prediction algorithm proposed in this embodiment of the invention designs and implements a network structure with skip connections. Specifically, this embodiment of the invention constructs the generator network module based on residual blocks. Compared to the traditional network structure of stacked convolutional layers and pooling layers, residual blocks have stronger learning capabilities, thus enabling better extraction of image features. Furthermore, residual blocks are easier to train. Figure 3 A schematic diagram of the residual block structure is provided for an embodiment of the present invention. The network structure of the residual block can be referred to. Figure 3 As shown in Table 1, for the generator network module using the residual block network structure, the parameter configurations of each network layer are as follows:

[0058] Table 1

[0059]

[0060]

[0061] The present invention uses a generator network built from residual blocks for training, and the resulting webpage saliency prediction algorithm will have a relatively better prediction effect.

[0062] Furthermore, existing webpage saliency prediction algorithms only fuse webpage features through linear combination, but this method is too simplistic and fails to extract the low-level features of the webpage. To address this issue, this invention proposes a feature extraction network module based on a deep convolutional network (this feature extraction network module and the generator network module together constitute the generator network) to extract the low-level features of the webpage. In this invention, an edge detector is used, which integrates Canny filters, Sobel filters, and Laplacian filters to extract webpage boundary features and text position features. Figure 4 This is a schematic diagram of the low-level features of a webpage provided in an embodiment of the present invention. These low-level features are then input into a feature extraction network module to encode more representative information, and are then fused with the high-level features of the webpage (i.e., webpage snapshots) to obtain the final information used to characterize the structural features of the webpage. Finally, the obtained structural features characterizing the webpage are input as conditional information into a generative adversarial network (GAN), thereby constraining the GAN's generation process. It should be noted that the feature extraction network module has the same network structure as the encoder in the generator network module. Furthermore, to enable the encoder to simultaneously extract the potential mutual information of webpage snapshots and webpage features, parameters are shared between the feature extraction network module and the encoder.

[0063] Furthermore, since a portion of the generator network's gradient updates comes from the discriminator network, the structure of the discriminator network has a decisive impact on the overall network performance. Correspondingly, if the discriminator network structure is too simple, it cannot effectively guide the generator network's training; conversely, if the discriminator structure is too complex, it will easily distinguish between genuine and fake data, causing the generator network to fail to learn further, ultimately leading to the collapse of the entire generative adversarial network's training. Therefore, this embodiment of the invention employs the PatchGAN structure, which is widely used in image-to-image conversion tasks and has demonstrated compelling results. The parameter configurations of each network layer for the discriminator network using the PatchGAN structure are shown in Table 2.

[0064] Table 2

[0065] serial number Network layer Parameter configuration Output vector 1 enter - 512×512×3 2 Downsampling convolutional layer 1 K = 3 × 3 × 64, P = 2, S = 2 256×256×64 3 Downsampling convolutional layer 2 K = 3 × 3 × 128, P = 2, S = 2 128×128×128 4 Downsampling convolutional layer 3 K = 3 × 3 × 256, P = 2, S = 2 64×64×256 5 Downsampling convolutional layer 4 K = 3 × 3 × 512, P = 2, S = 2 32×32×512 6 5 downsampling convolutional layers K = 3 × 3 × 10²⁴, P = 2, S = 1 31×31×1024 7 Downsampling convolutional layer 6 K = 3 × 3 × 512, P = 2, S = 1 30×30×1

[0066] Where P represents the number of elements used to fill the surrounding area of ​​the feature map during the convolution operation.

[0067] Based on the above embodiments, the generator network is constructed using residual blocks and a self-attention mechanism.

[0068] In this embodiment of the invention, based on the generator network constructed from residual blocks provided in the above embodiments, a self-attention mechanism is applied to the generator network. Figure 5 This is a flowchart illustrating the self-attention mechanism provided in an embodiment of the present invention. The calculation process of the self-attention mechanism can be referred to... Figure 5 As shown. It should be noted that the self-attention mechanism was not applied to the discriminator network in this embodiment of the invention because applying it would degrade the algorithm's performance. More specifically, applying the self-attention mechanism to a network layer with a feature map size of 128×128 yielded the best results in the experiments. For the activation function, the generator network used a Rectified Linear Unit (ReLU), and the discriminator network used a Leaky ReLU with a slope of 0.02 to introduce non-linear elements.

[0069] Based on the above embodiments, the adversarial loss function of the discriminator network is:

[0070]

[0071] The adversarial loss function of the generator network is:

[0072] L' GAN (G,D)=-E F,S [D(F,G(F,S))];

[0073] Where F represents the sample webpage snapshot and its corresponding sample webpage underlying features, S represents the real sample webpage saliency map, G represents the generator network, and D represents the discriminator network.

[0074] In this embodiment of the invention, the adversarial loss function includes the adversarial loss function of the discriminator network and the adversarial loss function of the generator network, which are used to constrain the distribution of the generated image to match the data distribution in the target domain.

[0075] Based on the above embodiments, after training the discriminator network according to the predicted webpage saliency map and the sample webpage saliency map, the method further includes:

[0076] Based on the discriminator network, the saliency maps of the predicted web pages and the sample web pages are judged to be true or false. A reconstruction loss function is then constructed based on the judgment results. The reconstruction loss function is as follows:

[0077]

[0078] Based on the reconstruction loss function and the adversarial loss function, an optimization objective function is constructed to obtain the trained webpage saliency prediction model. The optimization objective function is:

[0079]

[0080] Among them, L GAN (G,D) represents the adversarial loss function, L recons (G) represents the reconstruction loss function, j represents the number of samples, and S j This represents the true webpage saliency graph of the j-th sample. Let denot represent the webpage saliency map generated by the generator network for the j-th sample; λ is used to control the weights of the sample's predicted webpage saliency map and the sample's webpage saliency map.

[0081] In this embodiment of the invention, the objective function is optimized to ensure that the predicted webpage saliency map is as close as possible to the actual webpage saliency map.

[0082] In one embodiment of the present invention, the algorithm for the webpage saliency prediction model is described in detail. First, the input is:

[0083]

[0084] Where S represents the saliency map of the actual webpage, and F represents the webpage snapshot and underlying features of the webpage. G represents the saliency map of the web pages predicted by the generator, Θ represents all updatable parameters of the network, and G represents the saliency map of the web pages predicted by the generator. * L represents the objective function to be optimized. GAN (G, D) denotes the adversarial loss function, L recons (G) represents the reconstruction loss function, λ controls the weights of the adversarial loss function and the reconstruction loss function, iters represents the number of training iterations, and ndis represents the number of times the discriminator is updated for each generator update.

[0085] Output: Θ.

[0086] The specific steps of the above process are as follows:

[0087] Step S1: Randomly initialize all parameters using the He initialization method;

[0088] Step S2, for it = 0 to iters do;

[0089] Step S3: Input F into the generator, and the generator generates the corresponding value based on the input.

[0090] Step S4, put The input is fed into the discriminator, and the output value L of the discriminator is obtained. GAN (G,D);

[0091] Step S5, according to LGAN (G,D), S and Get L recons (G), and obtain the optimization objective function G. * ;

[0092] Step S6: Calculate the objective function G using the gradient descent algorithm. * The gradient values ​​for all parameters Θ;

[0093] Step S7: Update the generator parameters using the backpropagation algorithm;

[0094] Step S8, for i = 0 to ndis do;

[0095] Step S9: Update the parameters of the discriminator using the backpropagation algorithm;

[0096] Step S10, return Θ.

[0097] Based on the above embodiments, obtaining the training sample set includes:

[0098] The low-level features of the sample webpage snapshot are extracted by the edge detector to obtain the sample low-level features corresponding to the sample webpage snapshot, which are used to construct a training sample set. The edge detector includes a Canny filter, a Sobel filter and a Laplace filter.

[0099] In another embodiment of the invention, the following description is provided through experiments. To mitigate the overfitting problem that may result from insufficient training data, several data augmentation techniques were employed to expand the dataset, including random left-right flipping and random cropping. For the random cropping operation during the training phase, the training data was first adjusted to a resolution of 572×57, and then a 512×512 region was randomly cropped as the algorithm's input. During the testing phase, to ensure output consistency, no data augmentation techniques were used; the input was directly adjusted to a resolution of 512×512 and fed into the algorithm for prediction.

[0100] Furthermore, during training, all parameters are updated by the Adaptive Moment Estimation (Adam) optimization algorithm based on the gradient values ​​returned from backpropagation, where β1 is set to 0, β2 to 0.9, and epsilon to 1e-08. This embodiment of the invention synchronously updates the generator and discriminator networks according to the Two Time-scale Update Rule (TTUR), with initial learning rates of 0.0001 and 0.0004 for the generator and discriminator networks, respectively. For both networks, the learning rates remain constant during the first half of training, and in the second half, a linear decay method is used to reduce the learning rate to half of its initial value at the end of training. In all experiments, the hyperparameter of the loss function is set to λ = 0.05.

[0101] Furthermore, in all experiments of this invention, the batch size is set to 1 because increasing the batch size does not improve the experimental results within the limits of GPU memory size. The number of iterations required for the algorithm to complete one training iteration on the entire dataset is set to 200; increasing the number of iterations does not significantly improve the experimental performance. Additionally, this invention uses the deep learning framework Tensorflow to implement the proposed webpage saliency prediction algorithm and conduct experimental verification.

[0102] The webpage saliency prediction method based on generative adversarial networks provided in this invention has the significant advantage of not requiring random sampling of pixels labeled as positive or negative examples from webpage snapshots as training data. Furthermore, the algorithm effectively utilizes the spatial information of the webpage to guide saliency prediction. Moreover, this algorithm generates a saliency map for a given webpage in an end-to-end manner without any preprocessing or post-processing, greatly improving the efficiency of the webpage saliency prediction algorithm.

[0103] Figure 6 This is a schematic diagram of the webpage saliency prediction system based on generative adversarial networks provided in an embodiment of the present invention, as shown below. Figure 6As shown, this embodiment of the invention provides a webpage saliency prediction system based on generative adversarial networks, including an acquisition module 601 and a prediction module 602. The acquisition module 601 is used to acquire a webpage snapshot and low-level features of the webpage to be predicted, wherein the low-level features are the webpage boundary features and text features of the webpage to be predicted. The prediction module 602 is used to input the webpage snapshot and the low-level features into a trained webpage saliency prediction model and output the webpage saliency prediction result of the webpage to be predicted. The trained webpage saliency prediction model is obtained by training a deep convolutional neural network using sample webpage snapshots and the corresponding low-level features of the sample webpages.

[0104] The webpage saliency prediction system based on generative adversarial networks provided in this invention takes a complete webpage snapshot and its corresponding features as input, constructs a generative adversarial network based on a deep convolutional neural network, and outputs the webpage saliency. This system can, to some extent, address the shortcomings of existing webpage saliency prediction algorithms that do not consider spatial information between pixels, thereby improving the accuracy and efficiency of webpage saliency prediction. Furthermore, by eliminating the process of selecting training data, it mitigates overfitting to some extent and saves the time consumed by data selection.

[0105] Based on the above embodiments, the system further includes a sample set construction module, a generator training module, and a discriminator training module. The sample set construction module acquires a training sample set, which includes sample webpage snapshots, and the corresponding low-level features and saliency maps of the sample webpages. The generator training module trains the generator network based on the sample webpage snapshots and the low-level features to obtain a sample predicted webpage saliency map. The discriminator training module trains the discriminator network based on the sample predicted webpage saliency map and the sample webpage saliency map. If the training result meets a preset condition, a trained webpage saliency prediction model is obtained; otherwise, the parameters of the generator network and the discriminator network are updated using a backpropagation algorithm, and the updated generator network and discriminator network are trained until the training result meets the preset condition.

[0106] The system provided in this embodiment of the invention is used to execute the above-described method embodiments. For specific processes and details, please refer to the above embodiments, which will not be repeated here.

[0107] Figure 7 This is a schematic diagram of the electronic device structure provided in an embodiment of the present invention, with reference to... Figure 7The electronic device may include a processor 701, a communications interface 702, a memory 703, and a communication bus 704, wherein the processor 701, communications interface 702, and memory 703 communicate with each other via the communication bus 704. The processor 701 can call logical instructions in the memory 703 to execute the following methods: acquiring a webpage snapshot and low-level features of the webpage to be predicted, wherein the low-level features are the webpage boundary features and text features of the webpage to be predicted; inputting the webpage snapshot and the low-level features into a trained webpage saliency prediction model, and outputting a webpage saliency prediction result for the webpage to be predicted; the trained webpage saliency prediction model is obtained by training a deep convolutional neural network using sample webpage snapshots and the corresponding low-level features of the sample webpages.

[0108] Furthermore, the logical instructions in the aforementioned memory 703 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0109] On the other hand, embodiments of the present invention also provide a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the webpage saliency prediction method based on generative adversarial networks provided in the above embodiments. For example, it includes: acquiring a webpage snapshot and low-level features of the webpage to be predicted, wherein the low-level features are webpage boundary features and text features of the webpage to be predicted; inputting the webpage snapshot and the low-level features into a trained webpage saliency prediction model, and outputting a webpage saliency prediction result for the webpage to be predicted; wherein the trained webpage saliency prediction model is obtained by training a deep convolutional neural network using sample webpage snapshots and the low-level features of the sample webpages corresponding to the sample webpage snapshots.

[0110] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0111] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0112] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A webpage saliency prediction method based on generative adversarial networks, characterized in that, include: Obtain a snapshot of the webpage to be predicted and the underlying features of the webpage, wherein the underlying features of the webpage are the webpage boundary features and text features of the webpage to be predicted; The webpage snapshot and the webpage's underlying features are input into a trained webpage saliency prediction model, and the webpage saliency prediction result of the webpage to be predicted is output. The trained webpage saliency prediction model is obtained by training a deep convolutional neural network using sample webpage snapshots and the sample webpage's underlying features corresponding to the sample webpage snapshots. The trained webpage saliency prediction model is obtained through the following steps: Obtain a training sample set, which includes sample webpage snapshots, and the sample webpage low-level features and sample webpage saliency maps corresponding to the sample webpage snapshots; Based on the sample webpage snapshots and the low-level features of the sample webpages, the generator network is trained to obtain the saliency map of the predicted webpages. Based on the saliency map of the predicted webpage and the saliency map of the sample webpage, the discriminator network is trained. If the training result meets the preset conditions, the trained webpage saliency prediction model is obtained; if not, the parameters of the generator network and the discriminator network are updated through the backpropagation algorithm so that the updated generator network and discriminator network are trained until the training result meets the preset conditions. The adversarial loss function of the discriminator network is: The adversarial loss function of the generator network is: L' GAN (G,D)=-E F,S [D(F,G(F,S))]; Where F represents a snapshot of a sample webpage and its corresponding underlying features, S represents the saliency map of the real sample webpage, G represents the generator network, and D represents the discriminator network. After training the discriminator network based on the predicted webpage saliency map and the sample webpage saliency map, the method further includes: Based on the discriminator network, the saliency maps of the predicted web pages and the sample web pages are judged to be true or false. A reconstruction loss function is then constructed based on the judgment results. The reconstruction loss function is as follows: Based on the reconstruction loss function and the adversarial loss function, an optimization objective function is constructed to obtain the trained webpage saliency prediction model. The optimization objective function is: Among them, L GAN (G,D) represents the adversarial loss function, L recons (G) represents the reconstruction loss function, j represents the number of samples, and S j This represents the true webpage saliency graph of the j-th sample. Let denot represent the webpage saliency map generated by the generator network for the j-th sample; λ is used to control the weights of the sample's predicted webpage saliency map and the sample's webpage saliency map.

2. The webpage saliency prediction method based on generative adversarial networks according to claim 1, characterized in that, The generator network is constructed using residual blocks and a self-attention mechanism.

3. The webpage saliency prediction method based on generative adversarial networks according to claim 1, characterized in that, The acquisition of the training sample set includes: The low-level features of the sample webpage snapshot are extracted by the edge detector to obtain the sample low-level features corresponding to the sample webpage snapshot, which are used to construct a training sample set. The edge detector includes a Canny filter, a Sobel filter and a Laplace filter.

4. A webpage saliency prediction system based on generative adversarial networks, characterized in that, include: The acquisition module is used to acquire a webpage snapshot and low-level features of the webpage to be predicted, wherein the low-level features are the webpage boundary features and text features of the webpage to be predicted. The prediction module is used to input the webpage snapshot and the webpage's underlying features into a trained webpage saliency prediction model, and output the webpage saliency prediction result of the webpage to be predicted; the trained webpage saliency prediction model is obtained by training a deep convolutional neural network using the sample webpage snapshot and the sample webpage's underlying features corresponding to the sample webpage snapshot. The system also includes: The sample set construction module is used to obtain a training sample set, which includes sample webpage snapshots, and the sample webpage low-level features and sample webpage saliency maps corresponding to the sample webpage snapshots; The generator training module is used to train the generator network based on the sample webpage snapshot and the low-level features of the sample webpage to obtain the saliency map of the sample predicted webpage. The discriminator training module is used to train the discriminator network based on the saliency map of the sample predicted webpage and the saliency map of the sample webpage. If the training result meets the preset conditions, the trained webpage saliency prediction model is obtained; if not, the parameters of the generator network and the discriminator network are updated through the backpropagation algorithm so that the updated generator network and discriminator network are trained until the training result meets the preset conditions. The adversarial loss function of the discriminator network is: The adversarial loss function of the generator network is: L' GAN (G,D)=-E F,S [D(F,G(F,S))]; Where F represents a snapshot of a sample webpage and its corresponding underlying features, S represents the saliency map of the real sample webpage, G represents the generator network, and D represents the discriminator network. The system is also used for: Based on the discriminator network, the saliency maps of the predicted web pages and the sample web pages are judged to be true or false. A reconstruction loss function is then constructed based on the judgment results. The reconstruction loss function is as follows: Based on the reconstruction loss function and the adversarial loss function, an optimization objective function is constructed to obtain the trained webpage saliency prediction model. The optimization objective function is: Among them, L GAN (G,D) represents the adversarial loss function, L recons (G) represents the reconstruction loss function, j represents the number of samples, and S j This represents the true webpage saliency graph of the j-th sample. Let denot represent the webpage saliency map generated by the generator network for the j-th sample; λ is used to control the weights of the sample's predicted webpage saliency map and the sample's webpage saliency map.

5. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the webpage saliency prediction method based on generative adversarial networks as described in any one of claims 1 to 3.

6. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the webpage saliency prediction method based on generative adversarial networks as described in any one of claims 1 to 3.

Images