Webpage quality evaluation method, neural network training method, device and equipment

Abstract

The present disclosure provides a webpage quality evaluation method, a neural network training method, device and equipment, relates to the technical field of artificial intelligence, specifically relates to the technical field of deep learning, natural language processing and computer network. The neural network comprises a first subnetwork and an evaluation subnetwork. The method comprises the following steps: extracting a first text from a webpage source code of a target webpage, the first text comprising at least one hypertext markup language tag; processing the first text by using the first subnetwork to obtain a first feature of the target webpage; and processing the first feature by using the evaluation subnetwork to obtain an evaluation result of the target webpage.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, specifically to deep learning technology, natural language processing technology, and computer network technology, and particularly to a webpage quality assessment method based on neural networks, a neural network training method, a webpage quality assessment device based on neural networks, a neural network training device, electronic equipment, computer-readable storage media, and computer program products. Background Technology

[0002] With the development of the internet, the number of web pages on the global internet has now exceeded one trillion. Search engines, as the gateway to the internet, theoretically need to index all content on the entire network. However, due to the abundance of low-quality (expired content, unanswered questions, etc.), cheating (malicious scraping, illegal information, etc.), and broken links (content-related broken links, protocol-related broken links), only a portion of the total online resources are truly valuable to users. If search engines were to index all of the nearly one trillion web pages on the internet, it would result in a huge waste of their storage and computing resources. Furthermore, the prevalence of low-value web pages would negatively impact the user experience.

[0003] The methods described in this section are not necessarily methods that had been previously conceived or adopted. Unless otherwise specified, no method described in this section should be assumed to be prior art simply because it is included in this section. Similarly, unless otherwise specified, the issues mentioned in this section should not be considered to be accepted in any prior art. Summary of the Invention

[0004] This disclosure provides a method for webpage quality assessment based on neural networks, a method for training neural networks, a device for webpage quality assessment based on neural networks, a neural network training device, an electronic device, a computer-readable storage medium, and a computer program product.

[0005] According to one aspect of this disclosure, a method for webpage quality assessment based on a neural network is provided. The neural network includes a first subnetwork and an evaluation subnetwork. The method includes: extracting first text from the webpage source code of a target webpage, the first text including at least one Hypertext Markup Language tag; processing the first text using the first subnetwork to obtain a first feature of the target webpage; and processing the first feature using the evaluation subnetwork to obtain an evaluation result for the target webpage.

[0006] According to another aspect of this disclosure, a method for training a neural network is provided. The neural network includes a first sub-network and an evaluation sub-network. The method includes: acquiring a sample webpage and a true evaluation result of the sample webpage; extracting first sample text from the webpage source code of the sample webpage, the first sample text including at least one supersample text markup language tag; processing the first sample text using the first sub-network to obtain first sample features of the sample webpage; processing the first sample features using the evaluation sub-network to obtain a predicted evaluation result of the sample webpage; and adjusting the parameters of the neural network based on the true quality evaluation result and the predicted quality evaluation result to obtain a trained neural network.

[0007] According to one aspect of this disclosure, a webpage quality assessment apparatus based on a neural network is provided. The neural network includes a first subnetwork and an evaluation subnetwork. The apparatus includes: a first extraction unit configured to extract first text from the webpage source code of a target webpage, the first text including at least one Hypertext Markup Language tag; a first processing unit configured to process the first text using the first subnetwork to obtain a first feature of the target webpage; and a second processing unit configured to process the first feature using the evaluation subnetwork to obtain an evaluation result for the target webpage.

[0008] According to one aspect of this disclosure, a training apparatus for a neural network is provided. The neural network includes a first subnetwork and an evaluation subnetwork. The apparatus includes: an acquisition unit configured to acquire a sample webpage and a true evaluation result of the sample webpage; a second extraction unit configured to extract first sample text from the webpage source code of the sample webpage, the first sample text including at least one supersample text markup language tag; a third processing unit configured to process the first sample text using the first subnetwork to obtain first sample features of the sample webpage; a fourth processing unit configured to process the first sample features using the evaluation subnetwork to obtain a predicted evaluation result for the sample webpage; and a parameter tuning unit configured to adjust the parameters of the neural network based on the true quality evaluation result and the predicted quality evaluation result to obtain a trained neural network.

[0009] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the methods described above.

[0010] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause a computer to perform the above-described method.

[0011] According to another aspect of this disclosure, a computer program product is provided, including a computer program, wherein the computer program implements the above-described method when executed by a processor.

[0012] According to one or more embodiments of this disclosure, by extracting corresponding features from text including Hyper Text Markup Language (HTML) tags extracted from the source code of a target webpage, and obtaining an evaluation result of the target webpage based on these features, it is possible to evaluate the quality of a webpage solely based on information about the target webpage itself, eliminating the need for posterior signals. Furthermore, HTML tags contain a great deal of information related to the structure, framework, and layout of the webpage, which can help improve the accuracy of the webpage quality evaluation results.

[0013] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0014] The accompanying drawings exemplify embodiments and form part of the specification, serving together with the textual description to explain exemplary implementations of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of the claims. Throughout the drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0015] Figure 1 A schematic diagram of an exemplary system in which the various methods described herein may be implemented according to embodiments of the present disclosure is shown;

[0016] Figure 2 A flowchart of a webpage quality assessment method according to an exemplary embodiment of the present disclosure is shown;

[0017] Figure 3 A flowchart of a webpage quality assessment method according to an exemplary embodiment of the present disclosure is shown;

[0018] Figure 4 A flowchart illustrating a method for training a neural network according to an exemplary embodiment of the present disclosure is shown;

[0019] Figure 5 A structural block diagram of a webpage quality assessment apparatus according to an exemplary embodiment of the present disclosure is shown;

[0020] Figure 6 A structural block diagram of a training apparatus for a neural network according to exemplary embodiments of the present disclosure is shown; and

[0021] Figure 7A structural block diagram of an exemplary electronic device that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0022] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0023] In this disclosure, unless otherwise stated, the use of terms such as "first," "second," etc., to describe various elements is not intended to limit the positional, temporal, or importance relationships of these elements; such terms are merely used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of that element, while in other cases, based on the context, they may refer to different instances.

[0024] The terminology used in the description of the various examples described in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context clearly indicates otherwise, an element may be one or more unless the number of elements is specifically limited. Furthermore, the term "and / or" as used in this disclosure covers any one of the listed items and all possible combinations thereof.

[0025] In related technologies, existing webpage quality assessment methods typically rely on posterior signals for judgment, that is, based on user feedback. However, enriching webpage user signals is costly, and user behavior can be manipulated through cheating, thus affecting the assessment results.

[0026] To address the aforementioned issues, this disclosure extracts relevant features from the text, including Hyper Text Markup Language (HTML) tags, extracted from the target webpage's source code. Based on these features, an evaluation result for the target webpage is obtained, enabling quality assessment solely based on information about the webpage itself, eliminating the need for posterior signals. Furthermore, HTML tags contain a wealth of information related to the webpage's structure, framework, and layout, which can help improve the accuracy of the webpage quality assessment results.

[0027] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0028] Figure 1A schematic diagram of an exemplary system 100 in which the various methods and apparatus described herein can be implemented according to embodiments of this disclosure is shown. Reference Figure 1 The system 100 includes one or more client devices 101, 102, 103, 104, 105 and 106, a server 120, and one or more communication networks 110 coupling the one or more client devices to the server 120. The client devices 101, 102, 103, 104, 105 and 106 can be configured to execute one or more applications.

[0029] In embodiments of this disclosure, server 120 may run one or more services or software applications that enable the execution of methods according to embodiments of this disclosure.

[0030] In some embodiments, server 120 may also provide other services or software applications that may include non-virtual and virtual environments. In some embodiments, these services may be provided as web-based services or cloud services, such as to users of client devices 101, 102, 103, 104, 105 and / or 106 under a Software as a Service (SaaS) model.

[0031] exist Figure 1 In the configuration shown, server 120 may include one or more components that implement the functions performed by server 120. These components may include software components, hardware components, or combinations thereof that can be executed by one or more processors. Users operating client devices 101, 102, 103, 104, 105, and / or 106 can sequentially interact with server 120 using one or more client applications to utilize the services provided by these components. It should be understood that various different system configurations are possible and may differ from system 100. Therefore, Figure 1 This is an example of a system used to implement the various methods described herein, and is not intended to be limiting.

[0032] Users can use client devices 101, 102, 103, 104, 105, and / or 106 to access the search engine and use search services. The client devices can provide an interface that allows users to interact with the client devices. The client devices can also output information to the user through this interface. Although... Figure 1 Only six client devices are described, but those skilled in the art will understand that this disclosure can support any number of client devices.

[0033] Client devices 101, 102, 103, 104, 105, and / or 106 may include various types of computer devices, such as portable handheld devices, general-purpose computers (such as personal computers and laptops), workstation computers, wearable devices, smart screen devices, self-service terminal devices, service robots, gaming systems, thin clients, various messaging devices, sensors, or other sensing devices. These computer devices can run various types and versions of software applications and operating systems, such as Microsoft Windows, Apple iOS, UNIX-like operating systems, Linux or Linux-like operating systems (such as Google Chrome OS); or include various mobile operating systems, such as Microsoft Windows Mobile OS, iOS, Windows Phone, and Android. Portable handheld devices may include cellular phones, smartphones, tablets, personal digital assistants (PDAs), etc. Wearable devices may include head-mounted displays (such as smart glasses) and other devices. Gaming systems may include various handheld gaming devices, internet-enabled gaming devices, etc. Client devices are capable of executing various applications, such as various internet-related applications, communication applications (such as email applications), short message service (SMS) applications, and can use various communication protocols.

[0034] Network 110 can be any type of network well known to those skilled in the art, and can use any of a variety of available protocols (including but not limited to TCP / IP, SNA, IPX, etc.) to support data communication. By way of example only, one or more networks 110 can be a local area network (LAN), an Ethernet-based network, a token ring network, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., Bluetooth, WIFI), and / or any combination of these and / or other networks.

[0035] Server 120 may include one or more general-purpose computers, special-purpose server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and / or combination. Server 120 may include one or more virtual machines running a virtual operating system, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that can be virtualized to maintain virtual storage devices for servers). In various embodiments, server 120 may run one or more services or software applications that provide the functionality described below.

[0036] The computing unit in server 120 can run one or more operating systems, including any of the aforementioned operating systems and any commercially available server operating system. Server 120 can also run any of a variety of additional server applications and / or middleware applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.

[0037] In some implementations, server 120 may include one or more applications to analyze and merge data feeds and / or event updates received from users of client devices 101, 102, 103, 104, 105, and 106. Server 120 may also include one or more applications to display data feeds and / or real-time events via one or more display devices of client devices 101, 102, 103, 104, 105, and 106.

[0038] In some implementations, server 120 can be a server for a distributed system or a server integrated with blockchain. Server 120 can also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. A cloud server is a host product in the cloud computing service system, designed to address the shortcomings of traditional physical hosts and Virtual Private Server (VPS) services, such as high management difficulty and weak business scalability.

[0039] System 100 may also include one or more databases 130. In some embodiments, these databases may be used to store data and other information. For example, one or more of the databases 130 may be used to store information such as audio files and video files. Databases 130 may reside in various locations. For example, a database used by server 120 may be local to server 120, or it may be located away from server 120 and may communicate with server 120 via a network-based or dedicated connection. Databases 130 may be of different types. In some embodiments, the database used by server 120 may be, for example, a relational database. One or more of these databases may store, update, and retrieve data from and from the databases in response to commands.

[0040] In some embodiments, one or more of the databases 130 may also be used by an application to store application data. The databases used by the application may be of different types, such as key-value stores, object stores, or regular stores supported by a file system.

[0041] Figure 1 The system 100 can be configured and operated in various ways to enable the application of the various methods and apparatus described in this disclosure.

[0042] According to one aspect of this disclosure, a method for webpage quality assessment based on a neural network is provided. The neural network may include a first subnetwork and an evaluation subnetwork. Figure 2 As shown, the webpage quality assessment method includes: step S201, extracting first text from the webpage source code of the target webpage, the first text including at least one hypertext markup language tag; step S202, processing the first text using a first sub-network to obtain a first feature of the target webpage; and step S203, processing the first feature using an evaluation sub-network to obtain an evaluation result for the target webpage.

[0043] Therefore, by extracting features from the text including HyperText Markup Language (HTML) tags extracted from the target webpage's source code, and using these features to obtain the evaluation result of the target webpage, the quality of the webpage can be evaluated based solely on information about the webpage itself, eliminating the need for posterior signals. Furthermore, HTML tags contain a wealth of information related to the webpage's structure, framework, and layout, which can help improve the accuracy of the webpage quality evaluation results.

[0044] Search engines crawl web pages periodically (e.g., daily). For the large number of crawled web pages, search engines can obtain numerous web page features by analyzing the HTML source code, analyzing the Uniform Resource Locator (URL), and / or simulating browser rendering. These features can then be used to determine the quality of the web pages and whether to include them in their index.

[0045] In some embodiments, in step S201, first text including HTML tags can be extracted from the source code of the target webpage. Besides the text actually displayed on the webpage, the source code also includes information related to the webpage's structure, framework, layout, etc., which is presented through HTML tags such as js and CSS in the source code. Furthermore, the source code also includes the URLs of various external links, and this content and information can help evaluate the webpage quality. Therefore, by using first text including HTML tags to extract the first feature of the target webpage, the accuracy of webpage quality evaluation can be improved, thereby enhancing the user experience for search engines.

[0046] In some embodiments, the webpage source code itself can be used directly as the first text, so that the first feature can provide as much information as possible to the neural network related to the target page, thereby maximizing the accuracy of webpage quality assessment.

[0047] In some embodiments, after obtaining the first text, it can be segmented into words, and then a first feature can be obtained based on the segmentation results. It is understood that the first feature can be a text feature representing the textual information of the first text, or a feature representing the quality assessment of the first text / target page; this is not limited here. Correspondingly, the first sub-network can be a fine-tuned pre-trained long-text deep semantic model (e.g., BERT, ERNIE, etc.), which can process the segmentation results to obtain text features that incorporate all the textual information of the first text. It can also include both a large model for extracting textual features and a text quality assessment model for quality assessment based on textual features; this is not limited here. In addition to the networks described above, those skilled in the art can select other existing neural network models or build their own neural network models as needed to achieve the above functions.

[0048] According to some embodiments, the neural network may include a second sub-network. For example... Figure 3 As shown, the webpage quality assessment method may further include: step S303, determining the second text corresponding to the target webpage, the second text including at least one anchor link in the target webpage and at least one title of the target webpage; and step S304, processing the second text using a second sub-network to obtain the second feature of the target webpage. Figure 3 The operations of steps S301, S302, and S310 in the process and Figure 2 The operations of steps S201-S203 are similar and will not be described in detail here. Step S310, processing the first feature using the evaluation subnetwork to obtain the evaluation result of the target webpage, may include: processing the first feature and the second feature using the evaluation subnetwork to obtain the evaluation result of the target webpage.

[0049] Anchor links are navigational links on a webpage, also known as anchor text. Typically, anchor links point to other pages related to the current page; for example, on a portal website, anchor links can point to different sections or other pages within the same section. In some low-quality webpages, the page title or anchor links often contain enticing keywords to entice users to click. Therefore, using the title and / or anchor links as secondary text, and evaluating webpage quality based on secondary features of this secondary text, can improve the accuracy of the evaluation results from the sub-network output.

[0050] According to some embodiments, step S303, determining the second text corresponding to the target webpage, includes concatenating the title of the target webpage and at least one anchor link of the target webpage to obtain the second text. This maximizes the use of the title and anchor links to improve the accuracy of the evaluation results. It is understood that when either the title or anchor link is missing, the second text can be constructed based on only one of them.

[0051] According to some embodiments, the second sub-network can be a Text Convolutional Neural Network (TextCNN). TextCNN is adept at distinguishing different short texts when processing them; therefore, using TextCNN to model titles and anchor links can yield high-quality second features. It is understood that the second feature can be a text feature representing the title and / or anchor link, or a feature representing the quality assessment of the title and / or anchor link, without limitation. Accordingly, the second sub-network can be the aforementioned TextCNN, or it can simultaneously include TextCNN and a text quality assessment model based on text features, without limitation. Besides the networks described above, those skilled in the art can also select other existing neural network models or build their own neural network models to achieve the above functions as needed.

[0052] According to some embodiments, the neural network may further include a third sub-network. The webpage quality assessment method may further include: step S305, determining third text corresponding to the target webpage based on its Uniform Resource Locator (URL); and step S306, processing the third text using the third sub-network to obtain a third feature of the target webpage. Step S310, processing the first feature using an evaluation sub-network to obtain an evaluation result for the target webpage, may include: processing the first and third features using an evaluation sub-network to obtain an evaluation result for the target webpage.

[0053] In some low-quality web pages, the URL itself may already contain misleading words. Therefore, third-party text can be determined based on the web page URL, and third-party features based on the third-party text can be used to evaluate the quality of the web page, thereby improving the accuracy of the evaluation results output by the evaluation subnetwork.

[0054] According to some embodiments, step S305, determining the third text corresponding to the target webpage based on the target webpage's Uniform Resource Locator (URL), includes: segmenting the target webpage's URL to obtain segmentation results; and processing the segmentation results using an n-gram model to obtain the third text. Therefore, by segmenting the webpage's URL and processing the segmentation results using an n-gram model to obtain the third text, the understanding of the URL can be further improved, thereby increasing the accuracy of the final evaluation result. It should be noted that the third text may include multiple text fragments, each fragment consisting of n adjacent fragments from the URL's segmentation results.

[0055] In some embodiments, trigrams (i.e., n=3) can be used to process the segmentation results to further improve the accuracy of the final evaluation results. In some embodiments, the third sub-network can be a long short-term memory network to better understand the relationships between multiple segments in the third text, thereby obtaining higher quality third features.

[0056] According to some embodiments, the webpage quality assessment method may further include: rendering a target webpage, obtaining webpage stability parameters of the target webpage, wherein the webpage stability parameters of the target webpage are determined based on the return codes of at least one external link included in the target webpage; and determining a fourth feature based on the webpage stability parameters of the target webpage. Step S310, processing the first feature using an evaluation subnetwork to obtain an evaluation result for the target webpage, may include: processing the first feature and the fourth feature using an evaluation subnetwork to obtain an evaluation result for the target webpage. Thus, by rendering the target webpage and obtaining the return codes of the external links in the webpage, it is possible to determine whether these external links have been successfully rendered, and based on this information, the webpage stability parameters of the target webpage can be determined as one of the evaluation criteria for webpage quality. In this way, the dimensions of webpage quality assessment are further enriched, and the accuracy of the webpage quality assessment results is improved.

[0057] In some embodiments, backlinks may include images, videos, JavaScript, CSS, etc., from a webpage. If a webpage contains a large number of backlinks that fail to render, it indicates that the webpage is of poor quality.

[0058] According to some embodiments, such as Figure 3As shown, the webpage quality assessment method may further include: step S307, obtaining the site stability parameters of the site where the target webpage is located, wherein the site stability parameters are determined based on the webpage stability parameters of at least one webpage included in the site, and the webpage stability parameters are determined based on the return codes of at least one external link included in the corresponding webpage; and step S308, determining the fifth feature based on the site stability parameters. Step S310, processing the first feature using an evaluation subnetwork to obtain the assessment result for the target webpage may include: processing the first feature and the fifth feature using an evaluation subnetwork to obtain the assessment result for the target webpage. Therefore, by obtaining the site stability of the site where the target website is located for assessment, and using site stability as one of the assessment criteria for webpage quality, the dimensions of webpage quality assessment are further enriched, and the accuracy of the webpage quality assessment results is improved.

[0059] According to some embodiments, such as Figure 3 As shown, the webpage quality assessment method may further include: step S309, obtaining a sixth feature of the target webpage, wherein the sixth feature indicates at least one of the following: link depth, text length, or domain classification. Step S310, processing the first feature using an evaluation subnetwork to obtain an assessment result for the target webpage may include: processing the first feature and the sixth feature using an evaluation subnetwork to obtain an assessment result for the target webpage. Therefore, by obtaining the link depth, text length, domain classification, and / or other basic webpage features, the dimensions of webpage quality assessment can be further enriched, thereby improving the accuracy of the webpage quality assessment results.

[0060] In some embodiments, the first feature, second feature, third feature, fourth feature, fifth feature, and sixth feature, or any combination thereof, can be fused to obtain a fused feature. It is understood that those skilled in the art can fuse these features in various ways, such as concatenation, weighted summation, etc., and this is not limited thereto. In an exemplary embodiment, the first feature, second feature, third feature, fifth feature, and sixth feature can be concatenated to obtain a blended feature.

[0061] In some embodiments, an evaluation sub-model can be used to process the fused features to obtain the final evaluation result. In some embodiments, a point-wise Learning To Rank (LTR) model can be used as the evaluation sub-model to process the fused features to obtain the final evaluation result. It is understood that other network models can also be used as evaluation sub-models, and this is not limited thereto.

[0062] In some embodiments, the evaluation result can be a score in the range of 0-1, denoted as a score. The closer to 1, the higher the quality; the closer to 0, the lower the quality. A threshold w can be preset, and the comparison between the score and w is used to determine whether to include the corresponding target webpage. It is understood that those skilled in the art can design other evaluation result presentation methods (e.g., binary classification results indicating whether a webpage is a low-quality webpage) as needed, and set corresponding judgment criteria to determine whether to include the corresponding target webpage, which is not limited here.

[0063] According to another aspect of this disclosure, a method for training a neural network is provided, the neural network including a first sub-network and an evaluation sub-network. For example... Figure 4 As shown, the training method includes: step S401, obtaining sample web pages and their true evaluation results; step S402, extracting the first sample text from the web page source code of the sample web pages, the first sample text including at least one supersample text markup language tag; step S403, processing the first sample text using a first sub-network to obtain the first sample features of the sample web pages; step S404, processing the first sample features using an evaluation sub-network to obtain the predicted evaluation results for the sample web pages; and step S405, adjusting the parameters of the neural network based on the true quality evaluation results and the predicted quality evaluation results to obtain the trained neural network. It is understandable that... Figure 4 The operations in steps S402-S404 and Figure 2 The operations of steps S201-S203 are similar and will not be described in detail here.

[0064] In some embodiments, the sample webpages and their corresponding real evaluation results are obtained based on annotations. The real evaluation results can instruct annotators to score the corresponding sample webpages. In some embodiments, the sample webpages may be cheating webpages identified by an automated webpage anti-fraud system (or low-quality / high-quality webpages identified by other similar automated systems), in which case the corresponding real evaluation result can characterize the low quality of the webpage. Furthermore, for some higher-quality sites (e.g., government official websites, portals, etc.), webpages can be directly crawled from these sites as high-quality sample webpages. It is understood that those skilled in the art can obtain sample webpages and their corresponding real evaluation results through other means, which are not limited here.

[0065] According to some embodiments, the neural network includes a second sub-network. The training method may further include: determining second sample text corresponding to a sample webpage, the second sample text including at least one anchor link in the sample webpage and at least one title of the sample webpage; and processing the second sample text using the second sub-network to obtain second sample features of the sample webpage. Step S404, processing the first sample features using an evaluation sub-network to obtain an evaluation result for the sample webpage, may include: processing the first sample features and the second sample features using an evaluation sub-network to obtain a predicted evaluation result for the sample webpage.

[0066] According to some embodiments, the neural network further includes a third sub-network. The training method may also include: determining third sample text corresponding to the sample webpage based on the Uniform Resource Locator (URL) of the sample webpage; and processing the third sample text using the third sub-network to obtain third sample features of the sample webpage. Step S404, processing the first sample features using the evaluation sub-network to obtain an evaluation result for the sample webpage, may include: processing the first sample features and the third sample features using the evaluation sub-network to obtain a predicted evaluation result for the sample webpage.

[0067] According to some embodiments, the training method may further include: obtaining webpage stability parameters of the sample webpage, wherein the webpage stability parameters of the sample webpage are determined based on the return codes of at least one external link included in the sample webpage; and determining a fourth sample feature based on the webpage stability parameters of the sample webpage. Step S404, processing the first sample feature using an evaluation subnetwork to obtain an evaluation result for the sample webpage may include: processing the first sample feature and the fourth sample feature using an evaluation subnetwork to obtain a predicted evaluation result for the sample webpage.

[0068] According to some embodiments, the training method may further include: obtaining the site stability parameters of the site where the sample webpage is located, wherein the site stability parameters are determined based on the webpage stability parameters of at least one webpage included in the site, and the webpage stability parameters are determined based on the return codes of at least one external link included in the corresponding webpage; and determining the fifth sample feature based on the site stability parameters. Step S404, processing the first sample feature using an evaluation subnetwork to obtain a prediction evaluation result for the sample webpage may include: processing the first sample feature and the fifth sample feature using an evaluation subnetwork to obtain a prediction evaluation result for the sample webpage.

[0069] According to some embodiments, the training method may further include: obtaining a sixth sample feature of the sample webpage, wherein the sixth sample feature indicates at least one of the link depth, text length, or domain classification of the sample webpage. Step S404: Processing the first sample feature using an evaluation subnetwork to obtain a prediction evaluation result for the sample webpage includes: processing the first sample feature and the sixth sample feature using an evaluation subnetwork to obtain a prediction evaluation result for the sample webpage.

[0070] It is understood that, in step S405, those skilled in the art can adjust the parameters of the neural network and its included subnetworks (e.g., the first subnetwork, the second subnetwork, the third subnetwork, and the evaluation subnetwork) in various ways based on the actual evaluation results and the predicted evaluation results to obtain the trained neural network, without limitation herein.

[0071] According to another aspect of this disclosure, a webpage quality assessment apparatus based on a neural network is provided. The neural network includes a first subnetwork and an evaluation subnetwork. Figure 5 As shown, the apparatus 500 includes: a first extraction unit 510 configured to extract first text from the webpage source code of a target webpage, the first text including at least one Hypertext Markup Language tag; a first processing unit 520 configured to process the first text using a first subnetwork to obtain a first feature of the target webpage; and a second processing unit 530 configured to process the first feature using an evaluation subnetwork to obtain an evaluation result for the target webpage. It is understood that the operations of units 510-530 in the apparatus 500 are similar to... Figure 2 The operations of steps S201-S203 are similar and will not be described in detail here.

[0072] According to another aspect of this disclosure, a training apparatus for a neural network is provided. The neural network includes a first subnetwork and an evaluation subnetwork. Figure 6 As shown, the device 600 includes: an acquisition unit 610 configured to acquire a sample webpage and a true evaluation result of the sample webpage; a second extraction unit 620 configured to extract a first sample text from the webpage source code of the sample webpage, the first sample text including at least one supersample text markup language tag; a third processing unit 630 configured to process the first sample text using a first sub-network to obtain a first sample feature of the sample webpage; a fourth processing unit 640 configured to process the first sample feature using an evaluation sub-network to obtain a predicted evaluation result for the sample webpage; and a parameter tuning unit 650 configured to adjust the parameters of the neural network based on the true quality evaluation result and the predicted quality evaluation result to obtain a trained neural network. It is understood that the operations of units 610-650 in the device 600 are similar to... Figure 4The operations of steps S401-S405 are similar and will not be described in detail here.

[0073] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0074] According to embodiments of this disclosure, an electronic device, a readable storage medium, and a computer program product are also provided.

[0075] refer to Figure 7 The present invention describes a structural block diagram of an electronic device 700 that can serve as a server or client of the present disclosure, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0076] like Figure 7 As shown, device 700 includes a computing unit 701, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 702 or a computer program loaded from storage unit 708 into random access memory (RAM) 703. RAM 703 may also store various programs and data required for the operation of device 700. The computing unit 701, ROM 702, and RAM 703 are interconnected via bus 704. Input / output (I / O) interface 705 is also connected to bus 704.

[0077] Multiple components in device 700 are connected to I / O interface 705, including: input unit 706, output unit 707, storage unit 708, and communication unit 709. Input unit 706 can be any type of device capable of inputting information to device 700. Input unit 706 can receive input numerical or character information and generate key signal inputs related to user settings and / or function control of the electronic device, and may include, but is not limited to, a mouse, keyboard, touchscreen, trackpad, trackball, joystick, microphone, and / or remote control. Output unit 707 can be any type of device capable of presenting information, and may include, but is not limited to, a monitor, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 708 may include, but is not limited to, a hard disk and an optical disk. Communication unit 709 allows device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers, and / or chipsets, such as Bluetooth™ devices, 802.11 devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0078] The computing unit 701 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as web page quality assessment methods and / or neural network training methods. For example, in some embodiments, the web page quality assessment methods and / or neural network training methods can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed on device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the web page quality assessment methods and / or neural network training methods described above can be performed. Alternatively, in other embodiments, the computing unit 701 may be configured by any other suitable means (e.g., by means of firmware) to perform web page quality assessment methods and / or neural network training methods.

[0079] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0080] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0081] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0082] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0083] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0084] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service ecosystem, addressing the shortcomings of traditional physical hosts and VPS (Virtual Private Server, or simply "VPS") services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.

[0085] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0086] While embodiments or examples of this disclosure have been described with reference to the accompanying drawings, it should be understood that the methods, systems, and devices described above are merely exemplary embodiments or examples, and the scope of the invention is not limited by these embodiments or examples, but only by the granted claims and their equivalents. Various elements in the embodiments or examples may be omitted or replaced by their equivalents. Furthermore, the steps may be performed in a different order than that described in this disclosure. Further, various elements in the embodiments or examples may be combined in various ways. Importantly, as the technology evolves, many elements described herein can be replaced by equivalents that appear after this disclosure.

Claims

1. A webpage quality assessment method based on a neural network, wherein the neural network includes a first sub-network and an evaluation sub-network, the method comprising: Extract first text from the source code of the target webpage. The first text includes at least one hypertext markup language tag that represents the structure, frame or layout of the target webpage. The at least one hypertext markup language tag includes tags for rendering JS scripts or cascading style sheets (CSS). The first text is processed using the first sub-network to obtain the first feature of the target webpage; as well as The first feature is processed using the evaluation subnetwork to obtain an evaluation result for the target webpage.

2. The method according to claim 1, wherein, The neural network includes a second sub-network, and the method further includes: Determine second text corresponding to the target webpage, the second text including at least one anchor link in the target webpage and at least one title of the target webpage; and The second text is processed using the second sub-network to obtain the second feature of the target webpage. The evaluation subnetwork is used to process the first feature to obtain an evaluation result for the target webpage, including: The evaluation subnetwork is used to process the first feature and the second feature to obtain the evaluation result of the target webpage.

3. The method according to claim 2, wherein, Determining the second text corresponding to the target webpage includes: The title of the target webpage and at least one anchor link of the target webpage are concatenated to obtain the second text.

4. The method according to claim 2 or 3, wherein, The second sub-network is a text convolutional neural network.

5. The method according to any one of claims 1-3, wherein, The neural network further includes a third sub-network, and the method includes: Based on the Uniform Resource Locator (URL) of the target webpage, determine the third text corresponding to the target webpage; and The third text is processed using the third sub-network to obtain the third feature of the target webpage. The evaluation subnetwork is used to process the first feature to obtain an evaluation result for the target webpage, including: The evaluation subnetwork is used to process the first feature and the third feature to obtain the evaluation result of the target webpage.

6. The method according to claim 5, wherein, Based on the Uniform Resource Locator (URL) of the target webpage, determine the third text corresponding to the target webpage, including: The Uniform Resource Locator (URL) of the target webpage is segmented to obtain the segmentation results; and The word segmentation results are processed using an n-gram model to obtain the third text.

7. The method according to claim 5, wherein, The third sub-network is a Long Short-Term Memory (LSTM) network.

8. The method according to any one of claims 1-3, further comprising: Render the target webpage to obtain a webpage stability parameter for the target webpage, wherein the webpage stability parameter is determined based on the return codes of at least one external link included in the target webpage; and Based on the webpage stability parameters of the target webpage, a fourth feature is determined. The evaluation subnetwork is used to process the first feature to obtain an evaluation result for the target webpage, including: The evaluation subnetwork is used to process the first feature and the fourth feature to obtain the evaluation result of the target webpage.

9. The method according to any one of claims 1-3, further comprising: Obtain the site stability parameters of the site where the target webpage is located, wherein the site stability parameters are determined based on the webpage stability parameters of at least one webpage included in the site, and the webpage stability parameters are determined based on the return codes of at least one external link included in the corresponding webpage; and Based on the site stability parameters, the fifth feature is determined. The evaluation subnetwork is used to process the first feature to obtain an evaluation result for the target webpage, including: The evaluation subnetwork is used to process the first feature and the fifth feature to obtain the evaluation result of the target webpage.

10. The method according to any one of claims 1-3, further comprising: Obtain a sixth feature of the target webpage, wherein the sixth feature indicates at least one of the following: link depth, text length, or domain classification of the target webpage. The evaluation subnetwork is used to process the first feature to obtain an evaluation result for the target webpage, including: The evaluation subnetwork is used to process the first feature and the sixth feature to obtain the evaluation result of the target webpage.

11. A method for training a neural network, said neural network comprising a first sub-network and an evaluation sub-network, said method comprising: Obtain the sample webpage and the actual evaluation results of the sample webpage; Extract first sample text from the webpage source code of the sample webpage. The first sample text includes at least one hypertext markup language tag that characterizes the structure, frame, or layout of the sample webpage. The at least one hypertext markup language tag includes tags for rendering JS scripts or cascading style sheets (CSS). The first sample text is processed using the first sub-network to obtain the first sample features of the sample webpage; The evaluation subnetwork is used to process the features of the first sample to obtain the prediction evaluation result of the sample webpage; as well as Based on the actual evaluation results and the predicted evaluation results, the parameters of the neural network are adjusted to obtain the trained neural network.

12. The method of claim 11, wherein, The neural network includes a second sub-network, and the method further includes: Determine a second sample text corresponding to the sample webpage, the second sample text including at least one anchor link from the sample webpage and at least one title of the sample webpage; and The second sample text is processed using the second sub-network to obtain the second sample features of the sample webpage. The evaluation subnetwork is used to process the features of the first sample to obtain the evaluation result of the sample webpage, including: The evaluation subnetwork is used to process the first sample features and the second sample features to obtain the prediction evaluation result for the sample webpage.

13. The method according to claim 11 or 12, wherein, The neural network further includes a third sub-network, and the method further includes: Based on the Uniform Resource Locator (URL) of the sample webpage, determine the third sample text corresponding to the sample webpage; and The third sample text is processed using the third sub-network to obtain the third sample features of the sample webpage. The evaluation subnetwork is used to process the features of the first sample to obtain the evaluation result of the sample webpage, including: The evaluation subnetwork is used to process the first sample features and the third sample features to obtain the prediction evaluation result for the sample webpage.

14. The method according to claim 11 or 12, further comprising: Obtain the webpage stability parameter of the sample webpage, wherein the webpage stability parameter of the sample webpage is determined based on the return code of each of the at least one external link included in the sample webpage; Based on the webpage stability parameters of the sample webpages, the fourth sample features are determined. The evaluation subnetwork is used to process the features of the first sample to obtain the evaluation result of the sample webpage, including: The evaluation subnetwork is used to process the first sample features and the fourth sample features to obtain the prediction evaluation result for the sample webpage.

15. The method according to claim 11 or 12, further comprising: Obtain the site stability parameters of the site where the sample webpage is located, wherein the site stability parameters are determined based on the webpage stability parameters of at least one webpage included in the site, and the webpage stability parameters are determined based on the return codes of at least one external link included in the corresponding webpage. Based on the site stability parameters, the features of the fifth sample are determined. The evaluation sub-network is used to process the features of the first sample to obtain a prediction evaluation result for the sample webpage, including: The evaluation subnetwork is used to process the first sample features and the fifth sample features to obtain the prediction evaluation result for the sample webpage.

16. The method according to claim 11 or 12, further comprising: Obtain the sixth sample feature of the sample webpage, wherein the sixth sample feature indicates at least one of the following: link depth, text length, or domain classification of the sample webpage. The evaluation sub-network is used to process the features of the first sample to obtain a prediction evaluation result for the sample webpage, including: The evaluation subnetwork is used to process the first sample features and the sixth sample features to obtain the prediction evaluation result for the sample webpage.

17. A webpage quality assessment device based on a neural network, the neural network comprising a first sub-network and an evaluation sub-network, the device comprising: The first extraction unit is configured to extract first text from the webpage source code of the target webpage. The first text includes at least one hypertext markup language tag that represents the structure, frame, or layout of the target webpage. The at least one hypertext markup language tag includes tags for rendering JS scripts or cascading style sheets (CSS). The first processing unit is configured to process the first text using the first sub-network to obtain a first feature of the target webpage; as well as The second processing unit is configured to process the first feature using the evaluation subnetwork to obtain an evaluation result for the target webpage.

18. A training apparatus for a neural network, the neural network comprising a first subnetwork and an evaluation subnetwork, the apparatus comprising: The acquisition unit is configured to acquire a sample webpage and the actual evaluation result of the sample webpage; The second extraction unit is configured to extract first sample text from the webpage source code of the sample webpage. The first sample text includes at least one hypertext markup language tag that characterizes the structure, frame, or layout of the sample webpage. The at least one hypertext markup language tag includes tags for rendering JS scripts or cascading style sheets (CSS). The third processing unit is configured to process the first sample text using the first sub-network to obtain the first sample features of the sample webpage. The fourth processing unit is configured to process the features of the first sample using the evaluation subnetwork to obtain a prediction evaluation result for the sample webpage. as well as The parameter tuning unit is configured to adjust the parameters of the neural network based on the actual evaluation results and the predicted evaluation results to obtain the trained neural network.

19. An electronic device comprising: At least one processor; as well as A memory that is communicatively connected to the at least one processor; in The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-16.

20. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-16.

21. A computer program product comprising a computer program, wherein, When the computer program is executed by a processor, it implements the method of any one of claims 1-16.

Images