Webpage classification method, webpage classification model training method and device

Abstract

The present disclosure provides a webpage classification method and device, electronic equipment and storage medium, relates to the technical field of data processing, in particular to the field of artificial intelligence and intelligent recommendation. The specific implementation scheme is: respectively extracting features of a uniform resource locator (URL) of a webpage to be classified and a hypertext markup language (HTML) corresponding to the URL, obtaining a first feature vector and a second feature vector corresponding to the webpage to be classified; preprocessing the first feature vector and the second feature vector to obtain a target feature vector; and determining the type of the webpage to be classified based on the target feature vector.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of data processing, in particular to a webpage classification method, a webpage classification model training method and device in the field of artificial intelligence and intelligent recommendation. BACKGROUND

[0002] Artificial intelligence (AI) is a comprehensive technology of computer science, which enables machines to have the functions of perception, reasoning and decision-making by studying the design principles and implementation methods of various intelligent machines.

[0003] Webpage classification is one of the important applications in the field of artificial intelligence, which can classify webpages in an information overload environment. SUMMARY

[0004] The present disclosure provides a webpage classification method and device, an electronic device and a storage medium.

[0005] According to a first aspect of the present disclosure, a webpage classification method is provided, comprising:

[0006] respectively extracting features of a uniform resource locator (URL) of a webpage to be classified and a hyper text markup language (HTML) corresponding to the URL, to obtain a first feature vector and a second feature vector corresponding to the webpage to be classified;

[0007] preprocessing the first feature vector and the second feature vector to obtain a target feature vector;

[0008] determining a type of the webpage to be classified based on the target feature vector;

[0009] wherein the first feature vector comprises a feature vector corresponding to the URL of the webpage to be classified; and the second feature vector comprises a feature vector corresponding to the HTML of the webpage to be classified.

[0010] According to a second aspect of the present disclosure, a webpage classification model training method is provided, the webpage classification model comprising a classification sub-model and a word segmentation sub-model, the method comprising:

[0011] respectively extracting features of a uniform resource locator (URL) sample subset included in a first sample set and a hyper text markup language (HTML) sample subset corresponding to the URL sample subset, to obtain a first sample feature vector and a second sample feature vector corresponding to the first sample set;

[0012] perform preprocessing on the first sample feature vector and the second sample feature vector based on the word segmentation sub-model, to obtain a target sample feature vector;

[0013] input the target feature vector into the classification sub-model, and determine a predicted type of the first sample set based on an output of the classification sub-model;

[0014] match the target sample feature vector and a target label feature vector corresponding to the first sample set, and adjust parameters of the word segmentation sub-model based on a matching result; and / or match the predicted type of the first sample set and a label type of the first sample set, and adjust parameters of the classification sub-model based on a matching result.

[0015] According to a third aspect of the present disclosure, a web page classification apparatus is provided, comprising:

[0016] a first extraction unit configured to perform feature extraction on a URL of a web page to be classified and on HTML corresponding to the URL, respectively, to obtain a first feature vector and a second feature vector corresponding to the web page to be classified;

[0017] a first preprocessing unit configured to perform preprocessing on the first feature vector and the second feature vector, to obtain a target feature vector;

[0018] a first determination unit configured to determine a type of the web page to be classified based on the target feature vector;

[0019] The first feature vector comprises a feature vector corresponding to the URL of the web page to be classified; and the second feature vector comprises a feature vector corresponding to the HTML of the web page to be classified.

[0020] According to a fourth aspect of the present disclosure, a training apparatus of a web page classification model is provided, comprising:

[0021] a second extraction unit configured to perform feature extraction on a uniform resource locator URL sample subset included in a first sample set and on a hypertext markup language HTML sample subset corresponding to the URL sample subset, respectively, to obtain a first sample feature vector and a second sample feature vector corresponding to the first sample set;

[0022] a second preprocessing unit configured to perform preprocessing on the first sample feature vector and the second sample feature vector based on a word segmentation sub-model, to obtain a target sample feature vector;

[0023] a second determination unit configured to input the target feature vector into a classification sub-model, and determine a predicted type of the first sample set based on an output of the classification sub-model;

[0024] The matching unit is configured to match the target sample feature vector and a target label feature vector corresponding to the first sample set, adjust parameters of the word segmentation sub-model based on a matching result; and / or match a predicted type of the first sample set and a label type of the first sample set, and adjust parameters of the classification sub-model based on a matching result.

[0025] The fifth aspect of the present disclosure provides an electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the webpage classification method or the training method of the webpage classification model.

[0026] The sixth aspect of the present disclosure provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to perform the webpage classification method or the training method of the webpage classification model.

[0027] The seventh aspect of the present disclosure provides a computer program product comprising computer programs / instructions, which, when executed by a processor, implement the webpage classification method or the training method of the webpage classification model.

[0028] It should be understood that the contents described in this part are not intended to identify key or important features of the embodiments of the present disclosure, nor to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0029] The accompanying drawings are used to better understand the present scheme, and do not constitute a limitation on the present disclosure. Among them:

[0030] Figure 1 is an architecture schematic diagram of a webpage classification system provided by an embodiment of the present disclosure;

[0031] Figure 2 is an optional flow schematic diagram of a webpage classification method provided by an embodiment of the present disclosure;

[0032] Figure 3 is another optional flow schematic diagram of a webpage classification method provided by an embodiment of the present disclosure;

[0033] Figure 4 is an optional schematic diagram of obtaining a first training sample set provided by an embodiment of the present disclosure;

[0034] Figure 5 is an optional schematic diagram of preprocessing the spliced features provided by an embodiment of the present disclosure;

[0035] Figure 6 is another optional flow diagram of a webpage classification method provided by an embodiment of the present disclosure;

[0036] Figure 7 is an optional flow diagram of a webpage classification model training method provided by an embodiment of the present disclosure;

[0037] Figure 8 is an optional structural diagram of a webpage classification device provided by an embodiment of the present disclosure;

[0038] Figure 9 is an optional structural diagram of a webpage classification model training device provided by an embodiment of the present disclosure;

[0039] Figure 10 is a schematic block diagram of an example electronic device that can be used to implement an embodiment of the present disclosure. DETAILED DESCRIPTION

[0040] Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, which include various details of the embodiments of the present disclosure to assist in understanding, which should be considered in a descriptive sense only. Thus, it will be apparent to one of ordinary skill in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Also, for the sake of brevity and clarity, descriptions of well-known functions and constructions are omitted from the following description.

[0041] In related art, an electronic device can classify webpages accessed by the electronic device in the following two ways:

[0042] 1) By issuing a keyword list, matching keywords in the keyword list with titles of webpages accessed by the electronic device, and determining the type of the webpage accessed by the electronic device according to the matching result;

[0043] 2) By issuing a host list to the electronic device, and determining the type of the webpage accessed by the electronic device by matching domain names in the host list with domain names of urls in the webpage accessed by the electronic device.

[0044] However, in the above-mentioned solution 1), although the size of the keyword list is relatively small, the matching accuracy is low, and webpage classification cannot be accurately achieved. In the above-mentioned solution 2), since the number of websites is huge, if the host list needs to cover more webpages, the size of the host list will be very large, affecting the performance of the electronic device. In solution 2), it cannot be guaranteed to cover all webpages, and the classification labels of some webpages not in the host list cannot be obtained.

[0045] Therefore, in view of the defects in the webpage classification method in the related art, the present disclosure provides a webpage classification method, which can overcome some or all of the shortcomings of the prior art.

[0046] Referring to Figure 1 , Figure 1 is a schematic diagram of an architecture of a webpage classification system provided by an embodiment of the present disclosure, to implement a supporting example application, the electronic device 800 includes a webpage classification apparatus 200 (hereinafter referred to as the apparatus 200), and the webpage classification apparatus 200 is connected to a database 500 through a network 300, wherein the network 300 can be a wide area network or a local area network, or a combination of the two, and data transmission is implemented using a wireless or wired link. The apparatus 200 can be software installed in the electronic device 800.

[0047] The webpage classification method provided by the embodiment of the present disclosure can be implemented by an electronic device. For example, the electronic device 800 runs a client 810, which can be a client for webpage classification. The client can collect the URL and HTML of a webpage to be classified, and perform feature extraction through the apparatus 200 to obtain a first feature vector and a second feature vector corresponding to the webpage to be classified; pre-process the first feature vector and the second feature vector to obtain a target feature vector; and determine the type of the webpage to be classified based on the target feature vector.

[0048] Before that, the apparatus 200 can obtain a URL sample subset from the database through the network 300, wherein the URL sample subset includes a certain number of URLs of webpage samples of different types; then the apparatus 200 can control a first electronic device (which can be an electronic device corresponding to the apparatus, or at least one first electronic device set in advance, which is not specifically limited in the present disclosure) to open each URL in the URL sample subset through a real machine crawler controller (which can also be a crawler controller), and the apparatus can start the corresponding application program in the first electronic device, such as a browser application program or a reading application program, to open the URL. After completing the first screen rendering of the webpage corresponding to each URL, the corresponding HTML is stored in the first electronic device, and the crawler controller obtains the HTML sample subset from the first electronic device. The part-of-speech sub-model and the classification sub-model are trained based on the URL sample subset and the HTML sample subset.

[0049] When a webpage needs to be classified, the client 810 acquires the URL of the webpage to be classified input by a first user, wherein the client can also confirm the URL of the webpage to be classified based on the information input by the first user, such as the information input in the search box of an application. Then, the webpage classification apparatus 200 draws the webpage page corresponding to the URL based on the URL of the webpage to be classified, and acquires the HTML corresponding to the URL based on the webpage page, and performs feature extraction on the URL and the HTML. The first feature vector and the second feature vector corresponding to the webpage to be classified are obtained. The first feature vector and the second feature vector are preprocessed to obtain a target feature vector. The type of the webpage to be classified is determined based on the target feature vector.

[0050] In some embodiments, the electronic device 800 or the webpage classification apparatus 200 can implement the webpage classification method provided in the embodiments of the present disclosure by running a computer program. For example, the computer program can be a native program or a software module in an operating system; can be a native application program (APP), that is, a program that needs to be installed in an operating system to run; can also be a mini program, that is, a program that only needs to be downloaded into a browser environment to run; and can also be a mini program that can be embedded into any APP. In summary, the above computer program can be any form of application program, module or plug-in.

[0051] In some embodiments, the electronic device 800 can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart television, a smart watch, etc., but is not limited thereto. The electronic device 800 and the webpage classification apparatus 200 can be directly or indirectly connected through wired or wireless communication, which is not limited in the present disclosure.

[0052] In some embodiments, the webpage classification method provided in the embodiments of the present disclosure can be implemented by an electronic device or a webpage classification apparatus included in the electronic device. The webpage classification method provided in the embodiments of the present disclosure will be described below by taking the webpage classification apparatus as an example. Figure 2 An optional flowchart of the webpage classification method provided in the embodiments of the present disclosure is shown, which will be described according to each step.

[0053] In step S301, feature extraction is performed on the URL of the webpage to be classified and the HTML corresponding to the URL, respectively, to obtain the first feature vector and the second feature vector corresponding to the webpage to be classified.

[0054] In some embodiments, a webpage classification device (hereinafter referred to as the device) can acquire a URL of a webpage to be classified and HTML corresponding to the URL; perform feature extraction on the URL of the webpage to be classified to obtain a first feature vector corresponding to the webpage to be classified; and perform feature extraction on the HTML to obtain a second feature vector corresponding to the webpage to be classified.

[0055] In a specific implementation, the device can be arranged inside an electronic device; the electronic device loads the URL of the webpage to be classified; the URL is used to determine a page corresponding to the webpage to be classified; HTML corresponding to the URL is obtained based on the page corresponding to the webpage to be classified; feature extraction is performed on the URL to obtain the first feature vector corresponding to the URL; and feature extraction is performed on the HTML to obtain the second feature vector corresponding to the HTML.

[0056] In some optional embodiments, the feature extraction performed by the device on the URL can include extracting at least one of a protocol (Protocol), a host name (Hostname), a domain name, a port number (Port), a path (Path), parameters (Parameters), query information (Query), and an information fragment (Fragment) in the URL, and the first feature vector is used to represent the features extracted from the URL. Optionally, the device can remove useless characters and characters corresponding to a server network in the URL; the useless characters can include “.”, “ / ”, “: / / ”, and the like; and the characters corresponding to the server network can include “www” and “com”. For example, when the URL includes www.XXXXX.com, the first feature vector can be determined as “XXXXX”; or when the URL includes www.YYYYY.edu.cn, the first feature vector can be determined as “YYYYYeducn”.

[0057] In some other embodiments, the HTML corresponding to the URL is extracted by extracting tags (such as HTML tags) in the HTML. <title>), at least one keyword and title (as in HTML< / title> <h1>to< / h1> <h6>In at least one of the above embodiments (or any combination of the above embodiments), the second feature vector is used to represent features in the extracted HTML. Wherein, the title can be extracted by <hn>represents, n is an integer greater than 1.

[0058] For example, in the case that the HTML includes the following text, based on the following text, the label of the HTML can be extracted to include "intellectual property", and the title can be extracted to include "patent", and it can be determined that the second feature vector corresponding to the HTML at least represents "intellectual property" and "patent".

[0059]

[0060] In step S302, the first feature vector and the second feature vector are preprocessed to obtain a target feature vector.

[0061] In some embodiments, the apparatus can perform splicing processing on the first feature vector and the second feature vector to obtain the target feature vector; wherein the position of the first feature vector in the target feature vector can be before the second feature vector or after the second feature vector.

[0062] In some optional embodiments, the apparatus can further perform the following operations on the target feature vector: removing first type characters in the target feature vector; truncating first length characters in the target feature vector; performing word segmentation processing on the first length characters to obtain a word segmentation result; and merging at least one of adjacent and same characters in the word segmentation result. The first type characters can include punctuation symbols and / or English. When truncating, if the position of the first feature vector in the target feature vector is before the second feature vector, the feature represented by the first feature vector will not be truncated, and part of the feature represented by the second feature vector will be truncated.

[0063] In some optional embodiments, the first length characters or the target feature vector can be subjected to word segmentation processing based on a word segmentation sub-model to obtain a word segmentation result. The apparatus can further obtain a URL sample subset and an HTML sample subset corresponding to the URL sample subset; and determine a first training sample set based on the URL sample subset and the HTML sample subset.

[0064] In implementation, the apparatus can acquire a certain number of URLs of different types of web page samples to form the URL sample subset; based on the crawler controller, control a first electronic device (which can be the electronic device corresponding to the apparatus, or at least one first electronic device preset, without specific limitation in the present disclosure) to open each URL in the URL sample subset (the apparatus can start the corresponding application program, such as a browser type application program, to open the URL), and after completing the first screen rendering of the web page corresponding to each URL, store the corresponding HTML in the first electronic device, and the crawler controller acquires the HTML sample subset, and determines the first training sample set based on the URL sample subset and the HTML sample subset.

[0065] In some optional embodiments, the apparatus can further parse the URL sample subset respectively, and extract at least one of the protocol, host name, domain name, port number, path, parameter, query information and information segment in each URL in the URL sample subset; and parse the HTML sample subset and extract at least one of the label, at least one keyword and title of the HTML corresponding to the URL. Optionally, the apparatus can further store the features extracted from the URL sample subset and the HTML sample subset.

[0066] In some embodiments, the apparatus can determine the target sample feature vector corresponding to the first training sample set based on the word segmentation sub-model; determine the difference between the target label feature vector corresponding to the first training sample set and the target sample feature vector; and based on the difference between the target label feature vector and the target sample feature vector, adjust the model parameters of the word segmentation sub-model.

[0067] In implementation, the apparatus can split the first training sample set into a first training sample subset, a first test sample subset and a first validation sample subset; before training the word segmentation sub-model, determine the range of the hyperparameters of the word segmentation sub-model; based on the first training sample subset, the first test sample subset and the first validation sample subset, train the word segmentation sub-model, based on the training result, adjust the parameters of the word segmentation sub-model; evaluate the word segmentation sub-model with adjusted parameters to obtain the model evaluation index of the word segmentation sub-model; and based on the model evaluation index of the word segmentation sub-model, adjust the parameters of the word segmentation sub-model. Optionally, the parameters of the word segmentation sub-model can be adjusted using gradient descent until the model evaluation index of the word segmentation sub-model is greater than or equal to a first index. The first index is used to indicate the model evaluation standard, and if the model evaluation index of the word segmentation sub-model is greater than or equal to the first index, it indicates that the word segmentation sub-model does not need to be further trained.

[0068] In step S303, the type of the webpage to be classified is determined based on the target feature vector.

[0069] In some embodiments, the apparatus determines a model of the webpage to be classified based on the target feature vector.

[0070] In implementation, the apparatus can determine the type of the webpage to be classified corresponding to the target feature vector based on the classification sub-model.

[0071] In some optional embodiments, the apparatus can determine a feature vector sample set corresponding to the first training sample set based on the classification sub-model; determine a predicted type corresponding to the feature vector sample set based on the classification sub-model; determine a difference between a labeled type corresponding to the feature vector sample set and the predicted type; and adjust parameters of the classification sub-model based on the difference between the labeled type and the predicted type.

[0072] In implementation, the apparatus can obtain a second training sample set by performing at least one of the following on each training sample in the first training sample set: removing the first type of characters, truncating the first length, performing word segmentation on the first length of characters based on a word segmentation sub-model to obtain a word segmentation result, and merging adjacent and identical characters in the word segmentation result; determine a feature vector corresponding to a feature in each training sample in the second training sample set based on the classification sub-model, to form a feature vector sample set.

[0073] In implementation, the apparatus can further split the second training sample set into a second training sample subset, a second test sample subset, and a second validation sample subset; determine a range of hyperparameters of the classification sub-model before training the classification sub-model; train the classification sub-model based on the second training sample subset, the second test sample subset, and the second validation sample subset, adjust parameters of the classification sub-model based on a training result, evaluate the classification sub-model with adjusted parameters to obtain a model evaluation index of the classification sub-model, and adjust the parameters of the classification sub-model based on the model evaluation index of the classification sub-model. Optionally, the parameters of the classification sub-model can be adjusted using gradient descent until the model evaluation index of the classification sub-model is greater than or equal to a second index. The second index is used to indicate a model evaluation standard, and if the model evaluation index of the classification sub-model is greater than or equal to the second index, it indicates that the classification sub-model does not need to be further trained.

[0074] Thus, by the webpage classification method provided by the embodiment of the present disclosure, the URL of the webpage to be classified and the HTML corresponding to the URL are respectively subjected to feature extraction, to obtain the first feature vector and the second feature vector corresponding to the webpage to be classified; the first feature vector and the second feature vector are preprocessed to obtain a target feature vector; and the type of the webpage to be classified is determined based on the target feature vector. The target feature vector is determined based on the URL and the HTML, and the type of the webpage corresponding to the target feature vector is determined based on artificial intelligence, thereby improving the accuracy of webpage classification. In addition, without loading a large keyword list or host list in the electronic device, the related device can be deployed on the electronic device in a lightweight manner, thereby reducing the performance loss of the electronic device in the webpage classification process.

[0075] Figure 3 Another optional flowchart of the webpage classification method provided by the embodiment of the present disclosure is shown, which will be described according to each step.

[0076] In some optional embodiments, the webpage classification method provided by the present disclosure can be divided into an online part and an offline part; wherein the offline part involves training sample acquisition, model training, feature storage, etc. (steps S401 to S403); and the online part involves identification, feature extraction, feature preprocessing, etc. (steps S404 to S406).

[0077] In step S401, a first training sample set is acquired.

[0078] In some embodiments, the webpage classification device (hereinafter referred to as device) can acquire the first training sample set from the server, or can construct a URL sample subset by acquiring a certain number of URLs of webpages of different types; based on the crawler controller, a first electronic device (which can be an electronic device corresponding to the device, or can be at least one first electronic device pre-set, which is not specifically limited by the present disclosure) opens each URL in the URL sample subset (the device can start the corresponding application program in the first electronic device, such as a browser application program, to open the URL), and after completing the first screen rendering of the webpage corresponding to each URL, the corresponding HTML is stored in the first electronic device, the crawler controller acquires the HTML sample subset, and determines the first training sample set based on the URL sample subset and the HTML sample subset. Wherein, the crawler controller can be arranged in the device, or can be arranged outside the device.

[0079] In some optional embodiments, if the device obtains the first training sample set from the server, the server can obtain a number of URLs of different types of web page samples to form the URL sample subset; based on a crawler controller controlling the first electronic device to open each URL in the URL sample subset, after completing the first screen rendering of the web page corresponding to each URL, the corresponding HTML is stored in the first electronic device, the crawler controller obtains the HTML sample subset, and determines the first training sample set based on the URL sample subset and the HTML sample subset.

[0080] In some optional embodiments, the device can further parse the URL sample subset and extract at least one of the protocol, host name, domain name, port number, path, parameter, query information and information segment in each URL in the URL sample subset; and parse the HTML sample subset and extract at least one of the label, at least one keyword and title of the HTML corresponding to the URL. Optionally, the device can further store the features extracted from the URL sample subset and the HTML sample subset.

[0081] Figure 4 An optional schematic diagram for obtaining the first training sample set is shown.

[0082] As shown in Figure 4 , the crawler controller starts the crawler in response to the obtaining instruction of the device; the first electronic device starts the client, loads each URL in the URL sample subset, and stores the corresponding HTML in the first electronic device after completing the first screen rendering of the web page corresponding to each URL; the crawler controller obtains the HTML sample subset stored in the first electronic device, and / or stores the URL sample subset and the HTML sample subset in the first electronic device; the device extracts the features in the URL sample subset and the HTML sample subset stored in the first electronic device and parses them, and stores the parsed features in the URL sample subset and the HTML sample subset in the electronic device or the device as the first training sample set.

[0083] In some embodiments, after obtaining the first training sample set, the device extracts features in the URL sample subset and the HTML sample subset corresponding to the URL sample subset in the first training sample set, and sequentially splices the features. Optionally, the features corresponding to the URL sample subset precede the features corresponding to the HTML sample subset. The spliced features include features of any URL and features of HTML corresponding to the URL, i.e., there is a corresponding relationship between the URL and the HTML, and the splicing is also based on the corresponding relationship.

[0084] In some optional embodiments, after sequentially splicing the features in the URL sample subset and the HTML sample subset corresponding to the URL sample subset, the device can further pre-process the spliced features.

[0085] Figure 5 An optional schematic diagram of pre-processing the spliced features is shown.

[0086] As shown in Figure 5 the device removes the first type of characters (useless characters and English) in the spliced features, where the useless characters can be punctuation marks. The device truncates characters of a first length in the spliced features after removing the first type of characters. Optionally, the device truncates the features from left to right (i.e., maximally retains the features corresponding to the URL). The device performs word segmentation on the characters of the first length to obtain word segmentation results. Adjacent and identical characters in the word segmentation results are merged.

[0087] By truncating the spliced features after removing the first type of characters, the input of the model (word segmentation sub-model and / or classification sub-model) can be reduced, and the impact of the device on the performance of the electronic device can be reduced. In addition, since the URL features and the HTML features are spliced, adjacent and identical characters will inevitably appear, affecting the accuracy of the results, and the identical characters need to be normalized (merged). Optionally, the adjacent and identical features can be removed by a for loop traversal manner.

[0088] In some optional embodiments, to ensure consistency between the features obtained online and the features used when training the model offline, the target feature vector obtained from the webpage to be classified is of the same type as the features in the first training sample set.

[0089] Step S402: training a word segmentation sub-model.

[0090] In some embodiments, the word segmentation sub-model is used to segment the target feature vector or the features in the first training set after splicing, removing the first type of characters, and truncation. The device can determine the target sample feature vector corresponding to the first training sample set based on the word segmentation sub-model; determine the difference between the target label feature vector corresponding to the first training sample set and the target sample feature vector; adjust the model parameters of the word segmentation sub-model based on the difference between the target label feature vector and the target sample feature vector.

[0091] In specific implementation, the device can split the first training sample set into a first training sample subset, a first test sample subset, and a first validation sample subset; determine the range of the hyperparameters of the word segmentation sub-model before training the word segmentation sub-model; train the word segmentation sub-model based on the first training sample subset, the first test sample subset, and the first validation sample subset, adjust the parameters of the word segmentation sub-model based on the training result; evaluate the word segmentation sub-model with adjusted parameters to obtain the model evaluation index of the word segmentation sub-model; and adjust the parameters of the word segmentation sub-model based on the model evaluation index of the word segmentation sub-model. Optionally, the parameters of the word segmentation sub-model can be adjusted using gradient descent until the model evaluation index of the word segmentation sub-model is greater than or equal to a first index. The first index is used to indicate the model evaluation standard. If the model evaluation index of the word segmentation sub-model is greater than or equal to the first index, it indicates that the word segmentation sub-model does not need further training.

[0092] In some optional embodiments, the word segmentation sub-model can be a Lexical Analysis of Chinese (LAC) model. The LAC model is a joint grammar analysis tool that can realize Chinese word segmentation, part-of-speech tagging, and proper noun (or professional noun) recognition. It has the following advantages:

[0093] 1) Through joint learning of word segmentation, part-of-speech tagging, and proper noun (or professional noun) recognition by a deep learning model, the F1 value of the overall function of the LAC model can exceed 0.91; the F1 value of part-of-speech tagging can exceed 0.94; and the F1 value of proper noun (or professional noun) recognition can exceed 0.85. The F1 value is used to represent the accuracy and recall rate of word segmentation, recognition, or tagging. The higher the F1 value, the higher the accuracy and recall rate of word segmentation, recognition, or tagging.

[0094] 2) It has more compact model parameters, and can optimize the performance of the LAC model in combination with the Paddle prediction library. In the running process, the performance of a single thread of a central processing unit (CPU) can reach 800 QPS (QPS represents query rate per second), and has higher efficiency.

[0095] 3) Stronger customizability; can follow user dictionary to achieve simple and controllable intervention mechanism, accurately match user dictionary, and intervene in the model; and long segment characters can exist in the user dictionary, which can make the model more customizable.

[0096] 4) Convenient to call; the LAC model supports "one-key installation" in electronic devices, and also provides Python, Java and C++ calling interfaces and calling examples to realize fast calling and integration.

[0097] 5) Support for mobile terminals. The LAC model is a customized ultra-lightweight model with a volume of only 2 megabits (Mbit), which is suitable for most single-thread performance 200QPS electronic devices in the market with a price of 1000 yuan, and can meet the application requirements of most mobile terminals.

[0098] Those skilled in the art understand that the LAC model is only an example in the embodiments of the present disclosure, and any model or method capable of realizing word segmentation function can be used in specific implementation, and the present disclosure does not make specific limitations.

[0099] Step S403, training a classification sub-model.

[0100] In some embodiments, the classification sub-model is used to determine the type of a webpage to be classified; the apparatus can determine a feature vector sample set corresponding to the first training sample set based on the classification sub-model; determine a predicted type corresponding to the feature vector sample set based on the classification sub-model; determine the difference between the labeled type corresponding to the feature vector sample set and the predicted type; and adjust the parameters of the classification sub-model based on the difference between the labeled type and the predicted type.

[0101] In specific implementation, the apparatus can obtain a second training sample set by at least one of the following processing on each training sample in the first training sample set: removing the first type of characters, truncating the first length, performing word segmentation processing on the first length of characters based on the word segmentation sub-model to obtain a word segmentation result, and merging adjacent and identical characters in the word segmentation result.

[0102] In some optional embodiments, the classification sub-model can be a FastText model. The FastText model is a fast text classification sub-model, which has the following advantages compared with the classification algorithm based on neural network:

[0103] 1) In the case of maintaining high accuracy, the training speed and test speed are accelerated, and the offline model training time is saved.

[0104] 2) The features input into the FastText model do not need to be pre-trained word vectors corresponding to the features, and the FastText model can realize training of word vectors (i.e., determining a feature vector sample set corresponding to the first training sample set).

[0105] In addition, the FastText model has a small volume and is more suitable for electronic devices and does not affect the performance of the electronic devices.

[0106] Those skilled in the art understand that the FastText model in the embodiments of the present disclosure is only an example, and in specific implementation, any model or method capable of realizing webpage classification can be used, and the present disclosure does not make specific limitations.

[0107] In some embodiments, the trained word segmentation sub-model and the classification sub-model (or a device including the word segmentation sub-model and the classification sub-model) are deployed into an electronic device. When the electronic device loads a URL of a webpage to be classified and determines a page corresponding to the URL, first feature vectors corresponding to the URL and second feature vectors corresponding to HTML are obtained. After the first feature vectors and the second feature vectors are preprocessed, the first feature vectors and the second feature vectors are input into the word segmentation sub-model to obtain target feature vectors (text features including the first feature vectors and the second feature vectors). The target features are input into the classification sub-model to complete prediction of a type of the webpage to be classified, obtain a prediction result, and store the prediction result in a cache of the electronic device or the device. Specific steps can include:

[0108] In step S404, first feature vectors corresponding to the webpage to be classified and second feature vectors are obtained by performing feature extraction on the URL of the webpage to be classified and the HTML corresponding to the URL, respectively.

[0109] In some embodiments, the device can obtain a URL of a webpage to be classified and HTML corresponding to the URL. Feature extraction is performed on the URL of the webpage to be classified to obtain first feature vectors corresponding to the webpage to be classified. Feature extraction is performed on the HTML of the webpage to be classified to obtain second feature vectors corresponding to the webpage to be classified.

[0110] In specific implementation, the device can be arranged inside an electronic device. The electronic device loads a URL of a webpage to be classified. Based on the URL, a page corresponding to the webpage to be classified is determined. Based on the page corresponding to the webpage to be classified, HTML corresponding to the URL is obtained. Feature extraction is performed on the URL to obtain first feature vectors corresponding to the URL. Feature extraction is performed on the HTML to obtain second feature vectors corresponding to the HTML.

[0111] In some optional embodiments, the feature extraction on the URL can include extracting at least one of a protocol, a host name, a domain name, a port number, a path, a parameter, query information, and an information segment in the URL, and the first feature vector is used to represent the extracted features in the URL. Optionally, the device can remove useless characters and server network corresponding characters in the URL; wherein the useless characters can include ".”, " / " and ": / / ", etc.; and the server network corresponding characters can include "www" and "com". For example, in the case of a URL including www.XXXXX.com, the first feature vector can be determined as "XXXXX"; or in the case of a URL including www.YYYYY.edu.cn, the first feature vector can be determined as "YYYYYeducn".

[0112] In some other embodiments, at least one of a label, at least one keyword, and a title of the HTML corresponding to the URL is extracted, and the second feature vector is used to represent the extracted features in the HTML.

[0113] In step S405, the first feature vector and the second feature vector are preprocessed to obtain a target feature vector.

[0114] In some embodiments, the device can perform splicing processing on the first feature vector and the second feature vector to obtain the target feature vector; wherein the position of the first feature vector in the target feature vector can be before the second feature vector or after the second feature vector.

[0115] In some optional embodiments, the device can further perform the following operations on the target feature vector: removing first type characters in the target feature vector; truncating first length characters in the target feature vector; performing word segmentation processing on the first length characters based on a word segmentation sub-model to obtain a word segmentation result; and merging at least one of adjacent and same characters in the word segmentation result. Wherein the first type characters can include punctuation marks or English. And when truncating, if the position of the first feature vector in the target feature vector is before the second feature vector, the feature represented by the first feature vector will not be truncated, and part of the feature represented by the second feature vector will be truncated.

[0116] In step S406, the type of the webpage to be classified is determined based on the target feature vector.

[0117] In some embodiments, the device can determine the type of the webpage to be classified corresponding to the target feature vector based on a classification sub-model, to realize the prediction of the webpage to be classified.

[0118] Thus, by the webpage classification method provided by the embodiment of the present disclosure, the URL of the webpage to be classified and the HTML corresponding to the URL are respectively subjected to feature extraction, to obtain the first feature vector and the second feature vector corresponding to the webpage to be classified; the first feature vector and the second feature vector are preprocessed to obtain a target feature vector; and the type of the webpage to be classified is determined based on the target feature vector. The target feature vector is determined based on the URL and the HTML, and the type of the webpage corresponding to the target feature vector is determined based on artificial intelligence, thereby improving the accuracy of webpage classification. In addition, without loading a large volume of keyword list or host list in the electronic device, the related device can be deployed on the electronic device in a lightweight manner, thereby reducing the performance loss of the electronic device in the webpage classification process.

[0119] Figure 6 Another optional flowchart of the webpage classification method provided by the embodiment of the present disclosure is shown, which will be described according to each step.

[0120] In step S501, a URL sample subset is determined based on webpage samples of different types.

[0121] In some embodiments, the webpage classification device (hereinafter referred to as device) obtains the URLs of a certain number of webpage samples of different types to constitute the URL sample subset.

[0122] In step S502, an HTML sample subset is determined based on the URL sample subset.

[0123] In some embodiments, based on the crawler controller controlling the first electronic device (which can be the electronic device corresponding to the device, or at least one first electronic device set in advance, which is not specifically limited in the present disclosure) to open each URL in the URL sample subset (the device can start the corresponding application program, such as a browser type application program, to open the URL), after the first screen rendering of the webpage corresponding to each URL is completed, the corresponding HTML is stored in the first electronic device, and the crawler controller obtains the HTML sample subset. The crawler controller can be set in the device or outside the device.

[0124] In step S503, the URL sample subset and the HTML sample subset are processed to obtain a first training sample set.

[0125] In some embodiments, the apparatus can parse the subset of URL samples and extract at least one of a protocol, a host name, a domain name, a port number, a path, a parameter, query information, and a fragment in each URL in the subset of URL samples; and parse the subset of HTML samples and extract at least one of a tag, at least one keyword, and a title of the HTML corresponding to the URL. Optionally, the apparatus can further store the features extracted from the subset of URL samples and the subset of HTML samples.

[0126] In some embodiments, the apparatus can further concatenate the features in the subset of URL samples and the subset of HTML samples in sequence; optionally, the features corresponding to the subset of URL samples are before the features corresponding to the subset of HTML samples. The concatenated features include features of any URL and features of the HTML corresponding to the URL, i.e., there is a corresponding relationship between the URL and the HTML, and the concatenation is based on the corresponding relationship.

[0127] In some optional embodiments, after the apparatus concatenates the features in the subset of URL samples and the subset of HTML samples corresponding to the subset of URL samples in sequence, the apparatus can further pre-process the concatenated features to obtain a first training sample set.

[0128] In implementation, the apparatus can remove first type characters (useless characters and English) in the concatenated features, where the useless characters can be punctuation marks; the apparatus can truncate characters of a first length in the concatenated features after removing the first type characters; optionally, the apparatus can truncate the features from left to right (i.e., maximize the preservation of features corresponding to the URL).

[0129] In implementation, the apparatus can further perform word segmentation on the characters of the first length to obtain a word segmentation result; and merge adjacent and same characters in the word segmentation result. Optionally, the apparatus can perform word segmentation on Chinese characters in the characters of the first length.

[0130] The apparatus can perform word segmentation on the characters of the first length based on a word segmentation sub-model to obtain a word segmentation result; training of the word segmentation sub-model can be based on step S402, which will not be repeated here.

[0131] The apparatus can further perform the following steps: Figure 6 The middle feature processing part can be applied to the offline part and the online part. When applied to the offline part, the URL sample subset and the HTML sample subset are processed to obtain the first training sample set, as described in step S503. When applied to the online part, the URL and the HTML of the webpage to be classified are processed to obtain the target feature vector, as described in steps S506-S507.

[0132] In step S504, the features in the first training sample set are labeled and stored.

[0133] In some embodiments, the device can label the type of the webpage corresponding to the first training sample set, and / or store the first training sample set and the type of the webpage corresponding to the first training sample set. The type of the webpage corresponding to the first training sample set is the target label feature vector. The first training sample set is used to train the classification sub-model.

[0134] In this way, the device completes the process of the offline part.

[0135] In step S505, it is determined whether there is a classification result corresponding to the URL of the webpage to be classified.

[0136] In some embodiments, after loading the URL of the webpage to be classified and determining the page corresponding to the webpage to be classified based on the URL, the device can determine whether there is a classification result corresponding to the URL of the webpage to be classified in the electronic device based on the URL of the webpage to be classified. If there is a classification result corresponding to the URL of the webpage to be classified in the electronic device, the classification result is obtained, and the determination of the type of the webpage to be classified is completed. If there is no classification result corresponding to the URL of the webpage to be classified in the electronic device, step S507 is performed.

[0137] In some embodiments, the device can determine whether there is a classification result corresponding to the URL of the webpage to be classified in the electronic device based on the hostname included in the URL of the webpage to be classified.

[0138] In step S506, the URL of the webpage to be classified and the HTML corresponding to the URL are respectively extracted to obtain the first feature vector and the second feature vector corresponding to the webpage to be classified.

[0139] In some embodiments, the device can obtain the URL of the webpage to be classified and the HTML corresponding to the URL, extract the URL of the webpage to be classified to obtain the first feature vector corresponding to the webpage to be classified, and extract the HTML of the webpage to be classified to obtain the second feature vector corresponding to the webpage to be classified. In some embodiments, the device can obtain the URL of the webpage to be classified and the HTML corresponding to the URL, extract the URL of the webpage to be classified to obtain the first feature vector corresponding to the webpage to be classified, and extract the HTML of the webpage to be classified to obtain the second feature vector corresponding to the webpage to be classified.

[0140] In practice, the device can be arranged inside an electronic device; the electronic device loads a URL of a webpage to be classified; the URL is used to determine a page corresponding to the webpage to be classified; HTML corresponding to the URL is obtained based on the page corresponding to the webpage to be classified; feature extraction is performed on the URL to obtain the first feature vector corresponding to the URL; and feature extraction is performed on the HTML to obtain the second feature vector corresponding to the HTML.

[0141] In some optional embodiments, the device can perform feature extraction on the URL, which can include extracting at least one of a protocol, a host name, a domain name, a port number, a path, a parameter, query information, and an information segment in the URL, and the first feature vector is used to represent the extracted features in the URL. Optionally, the device can remove useless characters and server network corresponding characters in the URL; the useless characters can include ".”, " / " and ": / / ”, etc.; and the server network corresponding characters can include "www” and "com”. For example, when the URL includes www.XXXXX.com, the first feature vector can be determined as "XXXXX”; or when the URL includes www.YYYYY.edu.cn, the first feature vector can be determined as "YYYYYeducn”.

[0142] In some other embodiments, the device can perform feature extraction on the URL, which can also include extracting at least one of a label, at least one keyword, and a title of the HTML corresponding to the URL, and the second feature vector is used to represent the extracted features in the HTML.

[0143] In step S507, the first feature vector and the second feature vector are processed to obtain a target feature vector.

[0144] In some embodiments, the device can perform splicing processing on the first feature vector and the second feature vector to obtain the target feature vector; and the position of the first feature vector in the target feature vector can be before the second feature vector or after the second feature vector.

[0145] In some optional embodiments, the apparatus can further perform, on the target feature vector: removing a first type of character in the target feature vector; truncating a first length of characters in the target feature vector; performing word segmentation on the first length of characters based on a word segmentation sub-model to obtain a word segmentation result; and merging at least one of adjacent and same characters in the word segmentation result. The first type of character can include a symbol of a punctuation or English. When truncating, if a position of a first feature vector in the target feature vector is before a second feature vector, a feature represented by the first feature vector is not truncated, and a part of a feature represented by the second feature vector is truncated.

[0146] In step S508, the type of the webpage to be classified is determined based on the target feature vector.

[0147] In some embodiments, the apparatus can determine the type of the webpage to be classified corresponding to the target feature vector based on a classification sub-model, to realize prediction of the webpage to be classified.

[0148] In some optional embodiments, the apparatus can further store a corresponding relationship between a URL and / or HTML of the webpage to be classified and the type of the webpage to be classified in the electronic device; after the URL of the webpage to be classified is recorded again in the electronic device, the type of the webpage to be classified can be determined based on the URL of the webpage to be classified and the corresponding relationship between the URL and / or HTML of the webpage to be classified and the type of the webpage to be classified stored in the electronic device.

[0149] In this way, the webpage classification method provided by the embodiments of the present disclosure can improve the accuracy of webpage classification based on artificial intelligence on the one hand, and can reduce the impact on the terminal performance in the webpage classification process by using a smaller classification sub-model and a word segmentation sub-model on the other hand. In addition, after the type of the webpage is determined, the corresponding relationship between the URL and the type of the webpage is stored in the electronic device, which can improve the efficiency and accuracy of webpage type classification on the one hand, and only store the corresponding relationship between the URL and the type of the webpage corresponding to the operation of the user of the electronic device in the electronic device on the other hand, which is equivalent to providing a lightweight and customizable host list, to facilitate subsequent accurate delivery of content of interest to the user according to the historical record of the user.

[0150] Figure 7 is an optional flowchart of a webpage classification model training method provided by the embodiments of the present disclosure, which will be described according to each step.

[0151] In step S601, feature extraction is respectively performed on a uniform resource locator (URL) sample subset included in the first sample set and a hypertext markup language (HTML) sample subset corresponding to the URL sample subset, to obtain a first sample feature vector and a second sample feature vector corresponding to the first sample set.

[0152] In some embodiments, the training apparatus of the webpage classification model can obtain a URL sample subset and an HTML sample subset corresponding to the URL sample subset; and determine a first training sample set based on the URL sample subset and the HTML sample subset.

[0153] In some embodiments, the training apparatus of the webpage classification model can extract at least one of a sample label, at least one sample keyword and the sample label of an HTML sample corresponding to a URL sample in the URL sample subset, and the second sample feature vector is used to represent sample features in the extracted HTML sample; and the training apparatus of the webpage classification model can further perform feature extraction on the URL sample to obtain the first sample feature.

[0154] In step S602, the first sample feature vector and the second sample feature vector are preprocessed based on the word segmentation sub-model to obtain a target sample feature vector.

[0155] In some embodiments, the training apparatus of the webpage classification model performs splicing processing on the first sample feature vector and the second sample feature vector based on the word segmentation sub-model to obtain the target sample feature vector.

[0156] In some embodiments, the training apparatus of the webpage classification model can perform splicing processing on the first sample feature vector and the second sample feature vector to obtain the target sample feature vector; and the position of the first sample feature vector in the target sample feature vector can be before the second sample feature vector or after the second sample feature vector.

[0157] In some optional embodiments, the training apparatus of the webpage classification model can further perform the following operations on the target sample feature vector: removing a first type of character in the target sample feature vector; cutting a first length of characters in the target sample feature vector; performing word segmentation processing on the first length of characters to obtain a word segmentation result; and merging at least one of adjacent and same characters in the word segmentation result. The first type of character can include a symbol of a punctuation mark and / or English. When cutting, if the position of the first sample feature vector in the target sample feature vector is before the second sample feature vector, the feature represented by the first sample feature vector will not be cut, and part of the feature represented by the second sample feature vector will be cut.

[0158] In some optional embodiments, the first length of characters or the target feature vector can be processed by a word segmentation sub-model to obtain a word segmentation result. The training apparatus of the web page classification model can further obtain a URL sample subset and an HTML sample subset corresponding to the URL sample subset; and determine a first training sample set based on the URL sample subset and the HTML sample subset.

[0159] In implementation, the training apparatus of the web page classification model can obtain a certain number of URLs of web page samples of different types to form the URL sample subset; control a first electronic device (which can be an electronic device corresponding to the apparatus, or at least one first electronic device preset, without specific limitation in the present disclosure) to open each URL in the URL sample subset based on a crawler controller (the apparatus can start an application program, such as a browser application program, in the first electronic device to open the URL), and store the corresponding HTML in the first electronic device after completing the first screen rendering of the web page corresponding to each URL. The crawler controller obtains the HTML sample subset, and determines the first training sample set based on the URL sample subset and the HTML sample subset.

[0160] In some optional embodiments, the training apparatus of the web page classification model can further analyze the URL sample subset respectively, and extract at least one of a protocol, a host name, a domain name, a port number, a path, a parameter, query information, and an information segment in each URL in the URL sample subset; and analyze the HTML sample subset and extract at least one of a label, at least one keyword, and a title of the HTML corresponding to the URL. Optionally, the apparatus can further store the features extracted from the URL sample subset and the HTML sample subset.

[0161] In some embodiments, the training apparatus of the web page classification model can determine a target sample feature vector corresponding to the first training sample set based on the word segmentation sub-model; determine a difference between a target label feature vector corresponding to the first training sample set and the target sample feature vector; and adjust a model parameter of the word segmentation sub-model based on the difference between the target label feature vector and the target sample feature vector.

[0162] In particular implementation, the training apparatus of the webpage classification model can split the first training sample set into a first training sample subset, a first test sample subset and a first validation sample subset; before training the word segmentation sub-model, determine a range of hyperparameters of the word segmentation sub-model; train the word segmentation sub-model based on the first training sample subset, the first test sample subset and the first validation sample subset, adjust the parameters of the word segmentation sub-model based on the training result; evaluate the word segmentation sub-model with the adjusted parameters to obtain a model evaluation index of the word segmentation sub-model; and adjust the parameters of the word segmentation sub-model based on the model evaluation index of the word segmentation sub-model. Optionally, the parameters of the word segmentation sub-model can be adjusted using gradient descent until the model evaluation index of the word segmentation sub-model is greater than or equal to a first index. The first index is used to indicate a model evaluation standard. If the model evaluation index of the word segmentation sub-model is greater than or equal to the first index, it indicates that the word segmentation sub-model does not need further training.

[0163] In step S603, the target feature vector is input into the classification sub-model, and a predicted type of the first sample set is determined based on an output of the classification sub-model.

[0164] In some embodiments, the training apparatus of the webpage classification model inputs the target feature vector into the classification sub-model, and determines a predicted type of the first sample set based on an output of the classification sub-model.

[0165] In some optional embodiments, the training apparatus of the webpage classification model can determine a feature vector sample set corresponding to the first training sample set based on the classification sub-model; determine a predicted type corresponding to the feature vector sample set based on the classification sub-model; determine a difference between a labeled type corresponding to the feature vector sample set and the predicted type; and adjust the parameters of the classification sub-model based on the difference between the labeled type and the predicted type.

[0166] In particular implementation, the training apparatus of the webpage classification model can obtain a second training sample set by performing at least one of the following on each training sample in the first training sample set: removing the first type of characters, truncating the first length, performing word segmentation on the characters of the first length based on a word segmentation sub-model to obtain a word segmentation result, and merging adjacent and identical characters in the word segmentation result; determine a feature vector corresponding to a feature in each training sample in the second training sample set based on the classification sub-model to form a feature vector sample set.

[0167] In specific implementation, the training apparatus of the webpage classification model can further split the second training sample set into a second training sample subset, a second test sample subset and a second validation sample subset; before training the classification sub-model, determine a range of hyperparameters of the classification sub-model; train the classification sub-model based on the second training sample subset, the second test sample subset and the second validation sample subset, adjust the parameters of the classification sub-model based on the training result; evaluate the classification sub-model with the adjusted parameters to obtain a model evaluation index of the classification sub-model; and adjust the parameters of the classification sub-model based on the model evaluation index of the classification sub-model. Optionally, the parameters of the classification sub-model can be adjusted in a gradient descent manner until the model evaluation index of the classification sub-model is greater than or equal to a second index. The second index is used to indicate a model evaluation standard. If the model evaluation index of the classification sub-model is greater than or equal to the second index, it indicates that the classification sub-model does not need to be further trained.

[0168] In step S604, the target sample feature vector is matched with a target label feature vector corresponding to the first sample set, and the parameters of the word segmentation sub-model are adjusted based on the matching result; and / or, the predicted type of the first sample set is matched with the label type of the first sample set, and the parameters of the classification sub-model are adjusted based on the matching result.

[0169] In some embodiments, if the target sample feature vector is the same as the target label feature vector, the training apparatus of the webpage classification model determines not to adjust the parameters of the word segmentation sub-model; or, if the target sample feature vector is different from the target label feature vector, the training apparatus of the webpage classification model adjusts the word segmentation sub-model based on the difference between the target sample feature vector and the target label feature vector.

[0170] In other embodiments, if the predicted type is the same as the label type, the training apparatus of the webpage classification model determines not to adjust the parameters of the classification sub-model; or, if the predicted type is different from the label type, the training apparatus of the webpage classification model adjusts the classification sub-model based on the difference between the predicted type and the label type.

[0171] In this way, the training method of the webpage classification model provided by the embodiments of the present disclosure can effectively train the word segmentation sub-model and the classification sub-model, and provide strong support for subsequent webpage classification.

[0172] In the technical solutions of the present disclosure, the acquisition, storage and application of user personal information comply with relevant laws and regulations and do not violate public order and good customs.

[0173] According to embodiments of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0174] Figure 8 is an optional structure schematic diagram of the web page classification device provided by the embodiments of the present disclosure, which will be described according to each part.

[0175] In some embodiments, the web page classification device 200 comprises a first extraction unit 201, a first preprocessing unit 202 and a first determination unit 203.

[0176] The first extraction unit 201 is configured to perform feature extraction on the URL of the web page to be classified and the HTML corresponding to the URL respectively, to obtain a first feature vector and a second feature vector corresponding to the web page to be classified.

[0177] The first preprocessing unit 202 is configured to preprocess the first feature vector and the second feature vector to obtain a target feature vector.

[0178] The first determination unit 203 is configured to determine the type of the web page to be classified based on the target feature vector.

[0179] The first extraction unit 201 is specifically configured to load the URL of the web page to be classified, confirm the page corresponding to the web page to be classified based on the URL, obtain the HTML corresponding to the URL based on the page corresponding to the web page to be classified, perform feature extraction on the URL to obtain the first feature vector corresponding to the URL, and perform feature extraction on the HTML to obtain the second feature vector corresponding to the HTML.

[0180] The first extraction unit 201 is specifically configured to extract at least one of a label, at least one keyword and a title of the HTML corresponding to the URL, and the second feature vector is used to represent the features in the extracted HTML.

[0181] The first preprocessing unit 202 is specifically configured to perform splicing processing on the first feature vector and the second feature vector to obtain the target feature vector.

[0182] The first preprocessing unit 202 is further configured to implement one of the following:

[0183] Remove the first type of character in the target feature vector;

[0184] Truncate the first length of characters in the target feature vector;

[0185] Perform word segmentation processing on the first length of characters to obtain a word segmentation result;

[0186] Merge adjacent and identical characters in the word segmentation result.

[0187] The first determination unit 203 is further configured to input the target feature vector into a classification sub-model, and determine a type of the webpage to be classified based on an output of the classification sub-model.

[0188] Figure 9 An optional structure schematic diagram of a training device of a webpage classification model provided by the embodiments of the present disclosure will be described according to various parts.

[0189] In some embodiments, the webpage classification model 700 includes a second extraction unit 701, a second preprocessing unit 702, a second determination unit 703, and a matching unit 704.

[0190] The second extraction unit 701 is configured to perform feature extraction on a uniform resource locator (URL) sample subset included in the first sample set and a hypertext markup language (HTML) sample subset corresponding to the URL sample subset, respectively, to obtain a first sample feature vector and a second sample feature vector corresponding to the first sample set.

[0191] The second preprocessing unit 702 is configured to perform preprocessing on the first sample feature vector and the second sample feature vector based on the word segmentation sub-model, to obtain a target sample feature vector.

[0192] The second determination unit 703 is configured to input the target feature vector into the classification sub-model, and determine a predicted type of the first sample set based on an output of the classification sub-model.

[0193] The matching unit 704 is configured to match the target sample feature vector and a target label feature vector corresponding to the first sample set, and adjust parameters of the word segmentation sub-model based on a matching result; and / or match the predicted type of the first sample set and a labeled type of the first sample set, and adjust parameters of the classification sub-model based on a matching result.

[0194] The second determination unit 703 is further configured to obtain a URL sample subset and an HTML sample subset corresponding to the URL sample subset, and determine a first training sample set based on the URL sample subset and the HTML sample subset.

[0195] The matching unit 704 is specifically configured to determine not to adjust the parameters of the word segmentation sub-model if the target sample feature vector is the same as the target label feature vector, or adjust the word segmentation sub-model based on a difference between the target sample feature vector and the target label feature vector if the target sample feature vector is different from the target label feature vector.

[0196] The matching unit 704 is specifically configured to determine not to adjust the parameters of the classification sub-model if the prediction type is the same as the label type, or to adjust the classification sub-model based on a difference between the prediction type and the label type if the prediction type is different from the label type.

[0197] Figure 10 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present disclosure described and / or claimed in this document.

[0198] As shown in Figure 10 The electronic device 800 includes a computing unit 801 that can perform various appropriate actions and processes in accordance with a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. Various programs and data required for the operation of the electronic device 800 can also be stored in the RAM 803. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other through a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0199] Various components in the electronic device 800 are connected to the I / O interface 805, including an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, a speaker, etc.; the storage unit 808, such as a magnetic disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the electronic device 800 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0200] The computing unit 801 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 801 performs various methods and processes described above, such as the webpage classification method or the training method of the webpage classification model. For example, in some embodiments, the webpage classification method or the training method of the webpage classification model can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded onto the RAM 803 and executed by the computing unit 801, one or more steps of the webpage classification method or the training method of the webpage classification model described above can be performed. Alternatively, in other embodiments, the computing unit 801 can be configured to perform the webpage classification method or the training method of the webpage classification model by any other appropriate means, such as by means of firmware.

[0201] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0202] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces a means for implementing the functions / acts specified in the flowcharts and / or block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0203] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0204] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0205] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0206] The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0207] It should be understood that the various forms of flow shown above can be re-ordered, added to, or have steps deleted, using the flow. For example, the steps described in the present disclosure can be performed in parallel, in series, or in a different order, as long as the desired results of the technical solutions of the present disclosure can be achieved, which are not limited herein.

[0208] The specific implementation described above does not constitute a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present disclosure shall be included in the protection scope of the present disclosure.< / hn> < / h6>

Claims

1. A webpage classification method, comprising: deploying a trained segmentation sub-model and a classification sub-model in an electronic device; loading a URL of a webpage to be classified in the electronic device, determining a page corresponding to the URL, and performing feature extraction on a uniform resource locator URL of the webpage to be classified and a hypertext markup language HTML of the page corresponding to the URL respectively to obtain a first feature vector and a second feature vector corresponding to the webpage to be classified; performing preprocessing on the first feature vector and the second feature vector based on the segmentation sub-model to obtain a target feature vector; the segmentation sub-model adopts an LAC model, and the LAC model is an ultra-lightweight model; inputting the target feature vector into the classification sub-model, and determining a type of the webpage to be classified based on an output of the classification sub-model; the classification sub-model adopts a FastText model; storing the type of the webpage to be classified in a cache of the electronic device; wherein the first feature vector comprises a feature vector corresponding to the URL of the webpage to be classified; and the second feature vector comprises a feature vector corresponding to the HTML of the webpage to be classified.

2. The method of claim 1, wherein, The preprocessing on the first feature vector and the second feature vector to obtain the target feature vector comprises: performing splicing processing on the first feature vector and the second feature vector to obtain the target feature vector.

3. The method of claim 2, wherein, The method further comprises one of the following: removing a first type of character in the target feature vector; truncating a first length of characters in the target feature vector; performing segmentation processing on the first length of characters to obtain a segmentation result; merging adjacent and same characters in the segmentation result.

4. The method of claim 1, wherein, The determination of the type of the webpage to be classified based on the target feature vector comprises: inputting the target feature vector as an input of the classification sub-model, and determining the type of the webpage to be classified based on an output of the classification sub-model.

5. A training method of a webpage classification model, the webpage classification model comprising a classification sub-model and a segmentation sub-model, the method comprising: performing feature extraction on a uniform resource locator URL sample subset included in a first sample set and a hypertext markup language HTML sample subset corresponding to the URL sample subset to obtain a first sample feature vector and a second sample feature vector corresponding to the first sample set; performing preprocessing on the first sample feature vector and the second sample feature vector based on the segmentation sub-model to obtain a target sample feature vector; inputting the target sample feature vector into the classification sub-model, and determining a predicted type of the first sample set based on an output of the classification sub-model; matching the target sample feature vector and a target label feature vector corresponding to the first sample set, and adjusting parameters of the segmentation sub-model based on a matching result; and / or, matching the predicted type of the first sample set and a label type of the first sample set, and adjusting parameters of the classification sub-model based on a matching result.

6. The method of claim 5, wherein, Before the feature extraction on the uniform resource locator URL included in the first sample set and the hyper text markup language HTML corresponding to the URL respectively, the method further comprises: obtaining a URL sample subset and an HTML sample subset corresponding to the URL sample subset; determining a first training sample set based on the URL sample subset and the HTML sample subset.

7. The method of claim 5, wherein, The matching of the target sample feature vector and the target label feature vector corresponding to the first sample set and the adjustment of the parameters of the word segmentation sub-model based on the matching result comprise: if the target sample feature vector is the same as the target label feature vector, it is determined that the parameters of the word segmentation sub-model are not adjusted; or, if the target sample feature vector is different from the target label feature vector, the word segmentation sub-model is adjusted based on the difference between the target sample feature vector and the target label feature vector.

8. The method of claim 5, wherein, The matching of the predicted type of the first sample set and the label type of the first sample set and the adjustment of the parameters of the classification sub-model based on the matching result comprise: if the predicted type is the same as the label type, it is determined that the parameters of the classification sub-model are not adjusted; or, if the predicted type is different from the label type, the classification sub-model is adjusted based on the difference between the predicted type and the label type.

9. A web page classification device, comprising: deploying the trained word segmentation sub-model and classification sub-model in an electronic device, loading the URL of the web page to be classified in the electronic device, determining the page corresponding to the URL, a first extraction unit for respectively extracting features from the uniform resource locator URL of the web page to be classified and the hyper text markup language HTML corresponding to the URL, obtaining the first feature vector and the second feature vector corresponding to the web page to be classified; a first preprocessing unit for preprocessing the first feature vector and the second feature vector based on the word segmentation sub-model, obtaining a target feature vector; the word segmentation sub-model adopts an LAC model, and the LAC model is an ultra-lightweight model; a first determination unit for inputting the target feature vector into the classification sub-model and determining the type of the web page to be classified based on the output of the classification sub-model; the classification sub-model adopts a FastText model; storing the type of the web page to be classified in the cache of the electronic device; wherein the first feature vector comprises a feature vector corresponding to the URL of the web page to be classified; and the second feature vector comprises a feature vector corresponding to the HTML of the web page to be classified.

10. A training device of a web page classification model, comprising: a second extraction unit for respectively extracting features from the uniform resource locator URL sample subset included in the first sample set and the hyper text markup language HTML sample subset corresponding to the URL sample subset, obtaining the first sample feature vector and the second sample feature vector corresponding to the first sample set; a second preprocessing unit, configured to preprocess the first sample feature vector and the second sample feature vector based on a word segmentation sub-model to obtain a target sample feature vector; a second determining unit, configured to input the target sample feature vector into a classification sub-model, and determine a predicted type of the first sample set based on an output of the classification sub-model; a matching unit, configured to match the target sample feature vector and a target label feature vector corresponding to the first sample set, and adjust parameters of the word segmentation sub-model based on a matching result; and / or match the predicted type of the first sample set and a label type of the first sample set, and adjust parameters of the classification sub-model based on a matching result.

11. An electronic device comprising: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-4; or, to enable the at least one processor to perform the method of any one of claims 5-8.

12. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, the computer instructions for causing the computer to perform the method of any one of claims 1-4; or, to perform the method of any one of claims 5-8.

13. A computer program product comprising computer programs / instructions which, when executed by a processor, implement the method of any one of claims 1-4; or, implement the method of any one of claims 5-8.

Images