A webpage identification method and device, a computer program product and an electronic device

Abstract

The application provides a webpage identification method and device, a computer program product and an electronic device, and is applied to the technical field of network security. The webpage identification method comprises the following steps: obtaining text content of a webpage to be identified; wherein the text content comprises keywords corresponding to the webpage to be identified and text information corresponding to webpage body in the webpage to be identified; obtaining background knowledge information related to the webpage to be identified according to the keywords; encoding and representing the background knowledge information and the text information to obtain a background knowledge vector and a text vector respectively; and determining a webpage type of the webpage to be identified according to the background knowledge vector and the text vector. Therefore, in the process of identifying the webpage type of the webpage to be identified, the background knowledge vector corresponding to the background knowledge information and the text vector corresponding to the text information can be identified simultaneously, and compared with identification according to only the text information, the accuracy of identification can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of network security, in particular to a webpage identification method and device, a computer program product and an electronic device. BACKGROUND

[0002] With the development of Internet technology, more and more users obtain information through the network. The network provides great convenience for people, but also brings network security threats, which seriously threatens the information security of the country and society. Therefore, in the face of a large number of unknown web pages emerging every day, in order to better protect the online security of users, bad web pages will generally be identified. However, in the prior art, the identification accuracy of the traditional identification method is low. SUMMARY

[0003] The purpose of the embodiments of the present application is to provide a webpage identification method and device, a computer program product and an electronic device, to solve the technical problem of low identification accuracy of webpage identification in the prior art.

[0004] In a first aspect, the embodiments of the present application provide a webpage identification method, comprising: obtaining text content of a webpage to be identified; wherein the text content comprises keywords corresponding to the webpage to be identified and text information corresponding to webpage body in the webpage to be identified; obtaining background knowledge information related to the webpage to be identified according to the keywords; encoding and representing the background knowledge information and the text information to obtain background knowledge vectors and text vectors respectively; determining the webpage type of the webpage to be identified according to the background knowledge vectors and the text vectors.

[0005] In the above scheme, after obtaining the keywords corresponding to the webpage to be identified and the text information corresponding to the webpage body, the background knowledge information related to the webpage to be identified can be obtained according to the keywords, and the background knowledge information and the text information are converted into background knowledge vectors and text vectors respectively. Therefore, in the process of identifying the webpage type of the webpage to be identified, the background knowledge vectors corresponding to the background knowledge information and the text vectors corresponding to the text information can be identified at the same time, compared with only identifying according to the text information, the accuracy of identification can be improved.

[0006] In an optional implementation, the obtaining the background knowledge information related to the webpage to be identified according to the keyword comprises: extracting the background knowledge information corresponding to the keyword from a knowledge base, wherein the knowledge base stores a plurality of keywords and the background knowledge information corresponding to each keyword. In the foregoing solution, a plurality of keywords and the background knowledge information corresponding to each keyword can be stored in the knowledge base in advance, so that when the keyword corresponding to the webpage to be identified is obtained, the background knowledge information corresponding to the keyword can be extracted from the knowledge base. Therefore, in the process of identifying the webpage type of the webpage to be identified, the identification can be performed according to the background knowledge information and the text information simultaneously, and compared with the identification according to the text information only, the accuracy of the identification can be improved.

[0007] In an optional implementation, the encoding and representing the background knowledge information and the text information to obtain a background knowledge vector and a text vector respectively comprises: encoding and representing the background knowledge information and the text information by using a Bert pre-training model to obtain a background knowledge vector and a text vector respectively. In the foregoing solution, the encoding and representing of the background knowledge information and the text information can be implemented by using the Bert pre-training model, so that high-quality semantic expressions are obtained. Therefore, by using the high-quality semantic expressions of the background knowledge information and the text information, the accuracy of the identification can be improved.

[0008] In an optional implementation, the determining the webpage type of the webpage to be identified according to the background knowledge vector and the text vector comprises: splicing the background knowledge vector and the text vector to obtain a feature vector; and determining the webpage type of the webpage to be identified according to the feature vector. In the foregoing solution, the identification according to the background knowledge vector and the text vector simultaneously can be implemented by splicing the background knowledge vector and the text vector. Therefore, compared with the identification according to the text information only, the identification according to the background knowledge vector and the text vector simultaneously can improve the accuracy of the identification.

[0009] In an optional implementation, after the text content of the webpage to be identified is obtained, the method further comprises: performing data cleaning on the text information in the text content to remove special characters in the text information. In the foregoing solution, the special characters in the text information can be removed by performing data cleaning on the text information, so that the influence of the special characters on the identification process is avoided.

[0010] In an optional implementation, the webpage identification method is performed by a webpage identification model, and the webpage identification model is trained by: obtaining sample text content of a sample webpage and a corresponding label result; wherein the label result represents a webpage type of the sample webpage; inputting the sample text content into the webpage identification model to obtain a corresponding predicted webpage type; calculating a prediction loss representing a difference between the label result and the predicted webpage type according to a cross-entropy loss function; and updating parameters of the webpage identification model according to the prediction loss. In the foregoing solution, the webpage identification model can be used to identify a webpage to be identified according to a background knowledge vector and a text vector. Compared with identification only according to text information, identification according to a background knowledge vector and a text vector can improve the generalization ability of the webpage identification model, thereby improving the identification accuracy.

[0011] In a second aspect, the embodiments of the present application provide a webpage identification apparatus, comprising: a first obtaining module configured to obtain text content of a webpage to be identified; wherein the text content comprises keywords corresponding to the webpage to be identified and text information corresponding to webpage body in the webpage to be identified; a second obtaining module configured to obtain background knowledge information related to the webpage to be identified according to the keywords; an encoding module configured to encode the background knowledge information and the text information to obtain a background knowledge vector and a text vector, respectively; and a determining module configured to determine a webpage type of the webpage to be identified according to the background knowledge vector and the text vector.

[0012] In the foregoing solution, after obtaining the keywords corresponding to the webpage to be identified and the text information corresponding to the webpage body, the background knowledge information related to the webpage to be identified can be obtained according to the keywords, and the background knowledge information and the text information are converted into a background knowledge vector and a text vector, respectively. Therefore, in the process of identifying the webpage type of the webpage to be identified, the background knowledge vector corresponding to the background knowledge information and the text vector corresponding to the text information can be identified simultaneously, and compared with identification only according to the text information, the identification accuracy can be improved.

[0013] In an optional implementation, the second obtaining module is specifically configured to: extract background knowledge information corresponding to the keyword from a knowledge base; and the knowledge base stores a plurality of keywords and background knowledge information corresponding to each keyword. In the above solution, a plurality of keywords and background knowledge information corresponding to each keyword can be stored in the knowledge base in advance, so that when the keyword corresponding to the webpage to be recognized is obtained, the background knowledge information corresponding to the keyword can be extracted from the knowledge base. Therefore, in the process of identifying the webpage type of the webpage to be recognized, the identification can be performed according to the background knowledge information and the text information at the same time, and compared with the identification according to the text information only, the accuracy of the identification can be improved.

[0014] In an optional implementation, the encoding module is specifically configured to: encode and represent the background knowledge information and the text information by using a Bert pre-training model to obtain a background knowledge vector and a text vector respectively. In the above solution, the encoding and representation of the background knowledge information and the text information can be realized by using the Bert pre-training model, so as to obtain high-quality semantic expression. Therefore, by using the high-quality semantic expression of the background knowledge information and the text information, the accuracy of the identification can be improved.

[0015] In an optional implementation, the determining module is specifically configured to: splice the background knowledge vector and the text vector to obtain a feature vector; and determine the webpage type of the webpage to be recognized according to the feature vector. In the above solution, the identification according to the background knowledge vector and the text vector at the same time can be realized by splicing the background knowledge vector and the text vector. Therefore, compared with the identification according to the text information only, the identification according to the background knowledge vector and the text vector at the same time can improve the accuracy of the identification.

[0016] In an optional implementation, the webpage recognition apparatus further includes a data cleaning module configured to clean data of the text information in the text content to remove special characters in the text information. In the above solution, the special characters in the text information can be removed by cleaning the data of the text information, so as to avoid the influence of the special characters on the identification process.

[0017] In an optional implementation, the webpage identification apparatus further includes a training module configured to train the webpage identification model; the training module is specifically configured to: obtain sample text content of a sample webpage and a corresponding label result; the label result represents a webpage type of the sample webpage; input the sample text content into the webpage identification model to obtain a corresponding predicted webpage type; calculate a prediction loss representing a difference between the label result and the predicted webpage type according to a cross-entropy loss function; and update parameters of the webpage identification model according to the prediction loss. In the foregoing solution, the webpage identification model can be used to identify a webpage to be identified according to the background knowledge vector and the text vector. Compared with identification according to only text information, identification according to the background knowledge vector and the text vector can improve the generalization capability of the webpage identification model, thereby improving the identification accuracy.

[0018] In a third aspect, an embodiment of the present application provides a computer program product, including computer program instructions, which, when read and executed by a processor, perform the webpage identification method according to the first aspect.

[0019] In a fourth aspect, an embodiment of the present application provides an electronic device, including a processor, a memory and a bus; the processor and the memory complete mutual communication through the bus; the memory stores computer program instructions executable by the processor; the processor calling the computer program instructions can perform the webpage identification method according to the first aspect.

[0020] In a fifth aspect, an embodiment of the present application provides a computer readable storage medium, which stores computer program instructions; when the computer program instructions are run by a computer, the computer is caused to perform the webpage identification method according to the first aspect.

[0021] In order to make the above objectives, characteristics and advantages of the present application more apparent and easy to understand, the following will specifically describe embodiments of the present application with reference to the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments of the present application. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation to the scope, and for those skilled in the art, other related drawings can also be obtained without paying creative labor on the basis of these drawings.

[0023] Figure 1 A flowchart of a webpage identification method provided by an embodiment of the present application;

[0024] Figure 2A structural block diagram of a web page identification device provided by an embodiment of the present application is shown in the figure.

[0025] Figure 3 A structural block diagram of an electronic device provided by an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0026] The technical solutions in the embodiments of the present application will be described below with reference to the accompanying drawings.

[0027] Please refer to Figure 1 , Figure 1 A flow chart of a web page identification method provided by an embodiment of the present application is shown in the figure, which can include the following steps.

[0028] Step S101: Obtain the text content of a web page to be identified.

[0029] Step S102: Obtain background knowledge information related to the web page to be identified according to the keywords.

[0030] Step S103: Encode and represent the background knowledge information and the text information to obtain a background knowledge vector and a text vector respectively.

[0031] Step S104: Determine the web page type of the web page to be identified according to the background knowledge vector and the text vector.

[0032] Specifically, in the above step S101, the web page to be identified refers to a web page whose type needs to be identified at present, and the specific division method of the web page type is not limited in the embodiments of the present application, and can be adjusted appropriately by those skilled in the art according to the actual situation. For example, the web page type can include unhealthy web page and normal web page, or the web page type can include entertainment web page, learning web page, etc.

[0033] It can be understood that different operations can be performed after the web page to be identified is identified according to the different web page types, and the embodiments of the present application are not limited in this regard. For example, when it is identified that the web page to be identified is an unhealthy web page, the web page can be prevented from being opened and a prompt information can be outputted; or when it is identified that the web page to be identified is an entertainment web page, a prompt information can be outputted, etc.

[0034] The following will introduce a specific scenario of identifying a web page to be identified by using the web page identification method provided by the embodiments of the present application: a user opens a web page, in order to ensure that the content on the web page is healthy and positive, the web page type of the web page needs to be identified to determine whether the web page is an unhealthy web page; the web page identification method provided by the embodiments of the present application is executed to identify the above-mentioned web page to be identified, and it is determined that the web page is an unhealthy web page; the web page is prevented from being opened and a prompt information is outputted.

[0035] The text content includes keywords corresponding to the webpage to be identified and text information corresponding to webpage text in the webpage to be identified.

[0036] The keywords corresponding to the webpage to be identified refer to words that can represent the main content of the webpage to be identified. For example, when the webpage to be identified is a webpage related to advertisements, the keywords corresponding to the webpage to be identified can include "advertisement" and "consumption". It can be understood that the keywords corresponding to the webpage to be identified can be obtained from the name, content, pictures, and the like of the webpage to be identified.

[0037] It should be noted that the number of keywords in the text content obtained in the above step S101 can be one or more. When the number of keywords is more than one, the keywords can be associated or not associated. For example, when the webpage to be identified is a webpage for viewing a paper, but there is information related to advertisements around the paper, the keywords corresponding to the webpage to be identified can include "paper", "advertisement", and "consumption".

[0038] In addition, the specific implementation of obtaining the text content of the webpage to be identified is not specifically limited in the embodiments of the present application, and those skilled in the art can make appropriate adjustments according to the actual situation. For example, the text content of the webpage to be identified can be obtained by crawling through a web crawler; or the text content of the webpage to be identified sent by an external device can be received.

[0039] In the above step S102, the background knowledge information refers to text that explains the background knowledge of the keyword. Taking the keyword "network security" as an example, the background knowledge information can be "network security (Cyber Security) refers to the protection of hardware, software, and data in the system, and the system continues to run reliably and normally, and the network service is not interrupted due to accidental or malicious reasons".

[0040] It should be noted that for a keyword, the corresponding background knowledge information can have multiple expressions according to different sources, and the embodiments of the present application do not specifically limit this. As an implementation, in the embodiments of the present application, the background knowledge information corresponding to the keyword can be obtained from Baidu Encyclopedia.

[0041] In addition, the specific implementation of obtaining the background knowledge information related to the webpage to be identified according to the keyword is not specifically limited in the embodiments of the present application, and those skilled in the art can make appropriate adjustments according to the actual situation. For example, a knowledge base can be established in advance, and a plurality of keywords and the background knowledge information corresponding to each keyword can be stored in the knowledge base, so that the background knowledge information can be directly extracted from the knowledge base; or the background knowledge information corresponding to the keyword can be extracted from Baidu Encyclopedia.

[0042] As an implementation, the obtained background knowledge information can be represented as follows: {keyword 1: "text description of the keyword 1", keyword 2: "text description of the keyword 2",...}.

[0043] For example, the obtained background knowledge information corresponding to the keywords "advertisement" and "consumption" can be represented as follows: {advertisement: "advertisement, as the name implies, is to inform the public of something. In terms of its meaning, there are broad and narrow meanings.", consumption: "consumption is an important link in the process of social reproduction, and is the final link. It refers to the process of using social products to meet people's various needs. It is divided into production consumption and personal consumption. The former refers to the use and consumption of means of production and living labor in the process of material production. The latter is the behavior and process of people using the produced material and spiritual products to meet their personal life needs, which is "outside the production process to perform life functions". It is an essential condition for the restoration of labor and labor reproduction."}.

[0044] It can be understood that in some scenarios, there is no keyword in the knowledge base or the background knowledge information corresponding to the keyword cannot be obtained, at which time the background knowledge information corresponding to the keyword can be empty.

[0045] In the above step S103, the above background knowledge information can be encoded and represented to obtain a background knowledge vector; at the same time, the above text information can be encoded and represented to obtain a text vector.

[0046] It should be noted that the specific implementation of the above encoding and representation is not specifically limited by the embodiments of the present application, and is not specifically limited by those skilled in the art. For example, the Bert pre-training model can be used to encode and represent the background knowledge information and the text information; or a semantic expression model can be used to encode and represent the background knowledge information and the text information, etc.

[0047] In the above step S104, according to the above background knowledge vector and text vector, the web page type of the to-be-identified web page can be determined, so as to realize the identification of the web page type of the to-be-identified web page.

[0048] It should be noted that the specific implementation of determining the web page type of the to-be-identified web page is not specifically limited by the embodiments of the present application, and is not specifically limited by those skilled in the art. For example, the background knowledge vector and the text vector can be input into a neural network model to obtain the output web page type of the to-be-identified web page; or the similarity between the to-be-identified web page and the web pages of different web page types can be calculated to determine the web page type of the to-be-identified web page, etc.

[0049] In the above scheme, after obtaining the keywords corresponding to the to-be-identified webpage and the text information corresponding to the webpage text, the background knowledge information related to the to-be-identified webpage can be obtained according to the keywords, and the background knowledge information and the text information are converted into background knowledge vectors and text vectors respectively. Therefore, in the process of identifying the webpage type of the to-be-identified webpage, the background knowledge vectors corresponding to the background knowledge information and the text vectors corresponding to the text information can be identified at the same time, and compared with identification only according to the text information, the accuracy of identification can be improved.

[0050] Further, on the basis of the above embodiment, the step S102 can specifically include the following steps:

[0051] Extracting the background knowledge information corresponding to the keywords from the knowledge base.

[0052] Specifically, before performing the webpage identification method provided by the present application, a knowledge base can be established first, so that when the keywords corresponding to the to-be-identified webpage are obtained, the background knowledge information corresponding to the keywords can be directly extracted from the knowledge base. Wherein, the knowledge base can store a plurality of keywords and the background knowledge information corresponding to each keyword in advance.

[0053] It can be understood that the knowledge base can be updated continuously to obtain a more abundant knowledge base, and to ensure that each keyword can obtain corresponding background knowledge information as much as possible.

[0054] In the above scheme, a plurality of keywords and the background knowledge information corresponding to each keyword can be stored in the knowledge base in advance, so that when the keywords corresponding to the to-be-identified webpage are obtained, the background knowledge information corresponding to the keywords can be extracted from the knowledge base. Therefore, in the process of identifying the webpage type of the to-be-identified webpage, the background knowledge information and the text information can be identified at the same time, and compared with identification only according to the text information, the accuracy of identification can be improved.

[0055] Further, on the basis of the above embodiment, the step S103 can specifically include the following steps:

[0056] The background knowledge information and the text information are encoded and represented by using the Bert pre-training model, and the background knowledge vectors and the text vectors are obtained respectively.

[0057] Specifically, Bert is a pre-training language model, which uses the encoding (Encoder) layer of the Transformer model to extract features, adopts a pre-training plus fine-tuning training mode, learns deep word-level and sentence-level features through the Masked LM task and the Next Sentence Prediction task based on the Mask mechanism, and trains and tests through the Fine-Tuning mode on different downstream tasks to obtain the final model and experimental results.

[0058] As an implementation, the background knowledge information is encoded and represented as a background knowledge vector, which can be represented as BK=BERT (background knowledge sample), and the text information is encoded and represented as a text vector, which can be represented as TEXT=BERT (web page text sample).

[0059] In the above scheme, the Bert pre-training model can be used to realize the encoding representation of the background knowledge information and the text information, so as to obtain high-quality semantic expression. Therefore, by using the high-quality semantic expression of the background knowledge information and the text information, the recognition accuracy can be improved.

[0060] Further, on the basis of the above embodiment, the step S104 can specifically include the following steps:

[0061] Step 1), concatenating the background knowledge vector and the text vector to obtain a feature vector.

[0062] Step 2), determining the web page type of the to-be-recognized web page according to the feature vector.

[0063] Specifically, the background knowledge vector and the text vector are concatenated to obtain a feature vector, and the web page type of the to-be-recognized web page is determined according to the feature vector, so as to realize the determination of the web page type of the to-be-recognized web page according to the background knowledge vector and the text vector.

[0064] Taking the encoding representation of the background knowledge information and the text information by using the Bert pre-training model as an example, the above step 1) can be represented as follows:

[0065] F=BK⊕TEXT.

[0066] Wherein, F represents the feature vector, BK represents the background knowledge vector, TEXT represents the text vector, and represents concatenating the background knowledge vector and the text vector.

[0067] In the above scheme, the background knowledge vector and the text vector can be spliced to realize recognition according to the background knowledge vector and the text vector at the same time. Therefore, compared with recognition according to only text information, recognition according to the background knowledge vector and the text vector at the same time can improve the recognition accuracy.

[0068] Further, on the basis of the above embodiment, after the step S101, the webpage recognition method provided by the embodiment of the application can further include the following steps:

[0069] The text information in the text content is data cleaned to remove special characters in the text information.

[0070] In the above scheme, the text information can be data cleaned to remove special characters in the text information, so as to avoid the influence of special characters on the recognition process.

[0071] Further, on the basis of the above embodiment, the embodiment of the application further provides a webpage recognition model for executing the above webpage recognition method. The webpage recognition model can include a background knowledge acquisition model, a coding representation model, and a recognition model.

[0072] The background knowledge acquisition model is used to acquire background knowledge information related to the webpage to be recognized according to the keyword; the coding representation model is used to code and represent the background knowledge information and the text information to obtain the background knowledge vector and the text vector respectively, which can be realized by using the Bert pre-training model; and the recognition model is used to determine the webpage type of the webpage to be recognized according to the background knowledge vector and the text vector, which can be realized by using a neural network model (including multiple fully connected layers).

[0073] Therefore, the webpage recognition method provided by the embodiment of the application can further include the following steps, and the webpage recognition model can be trained by the following steps:

[0074] Step 1), obtaining sample text content of a sample webpage and corresponding label results.

[0075] Step 2), inputting the sample text content into the webpage recognition model to obtain corresponding predicted webpage types.

[0076] Step 3), calculating a prediction loss representing the difference between the label results and the predicted webpage types according to a cross-entropy loss function.

[0077] Step 4), updating parameters of the webpage recognition model according to the prediction loss.

[0078] Specifically, in the step 1), the sample web page refers to a web page of a known web page type, and the label result of the sample web page is used to represent the web page type of the sample web page. As an implementation manner, the sample text content of the sample web page can be obtained by web crawling; as another implementation manner, the sample text content of the sample web page can be directly obtained from a sample library.

[0079] In the step 2), after the sample text content is input into the web page recognition model, the sample background knowledge information related to the sample web page can be obtained according to the sample keywords in the sample text content; then the sample background knowledge information and the sample text information in the sample text content can be encoded to represent the sample background knowledge vector and the sample text vector respectively; finally, the predicted web page type of the sample web page can be obtained according to the sample background knowledge vector and the sample text vector.

[0080] In the steps 3) to 4), the cross-entropy loss function can be used as the loss function, the training epoch, the batch size and the learning rate (lr) are set, the parameters of the web page recognition model are iteratively updated until the target effect is achieved.

[0081] In the above scheme, the web page recognition model can be used to identify the to-be-identified web page according to the background knowledge vector and the text vector. Compared with the identification according to the text information only, the identification according to the background knowledge vector and the text vector can improve the generalization ability of the web page recognition model, thereby improving the identification accuracy.

[0082] Please refer to Figure 2 , Figure 2 A structural block diagram of a web page recognition device provided by an embodiment of the present application is shown in FIG. 2. The web page recognition device 200 can include a first obtaining module 201 configured to obtain the text content of a to-be-identified web page; wherein the text content includes keywords corresponding to the to-be-identified web page and text information corresponding to a web page body in the to-be-identified web page; a second obtaining module 202 configured to obtain background knowledge information related to the to-be-identified web page according to the keywords; an encoding module 203 configured to encode the background knowledge information and the text information to represent a background knowledge vector and a text vector respectively; and a determining module 204 configured to determine the web page type of the to-be-identified web page according to the background knowledge vector and the text vector.

[0083] In the embodiment of the present application, after the keyword corresponding to the to-be-identified webpage and the text information corresponding to the webpage text are acquired, the background knowledge information related to the to-be-identified webpage can be acquired according to the keyword, and the background knowledge information and the text information are converted into background knowledge vectors and text vectors respectively. Therefore, in the process of identifying the webpage type of the to-be-identified webpage, the background knowledge vectors corresponding to the background knowledge information and the text vectors corresponding to the text information can be identified at the same time, and compared with the identification only according to the text information, the accuracy of identification can be improved.

[0084] Further, on the basis of the above embodiment, the second acquisition module 202 is specifically configured to: extract the background knowledge information corresponding to the keyword from a knowledge base; wherein the knowledge base stores a plurality of keywords and the background knowledge information corresponding to each keyword.

[0085] In the embodiment of the present application, a plurality of keywords and the background knowledge information corresponding to each keyword can be stored in the knowledge base in advance, so that when the keyword corresponding to the to-be-identified webpage is acquired, the background knowledge information corresponding to the keyword can be extracted from the knowledge base. Therefore, in the process of identifying the webpage type of the to-be-identified webpage, the background knowledge information and the text information can be identified at the same time, and compared with the identification only according to the text information, the accuracy of identification can be improved.

[0086] Further, on the basis of the above embodiment, the encoding module 203 is specifically configured to: encode and represent the background knowledge information and the text information by using a Bert pre-training model, to obtain background knowledge vectors and text vectors respectively.

[0087] In the embodiment of the present application, the Bert pre-training model can be used to realize the encoding representation of the background knowledge information and the text information, so as to obtain high-quality semantic expression. Therefore, by using the high-quality semantic expression of the background knowledge information and the text information, the accuracy of identification can be improved.

[0088] Further, on the basis of the above embodiment, the determination module 204 is specifically configured to: splice the background knowledge vectors and the text vectors to obtain feature vectors; and determine the webpage type of the to-be-identified webpage according to the feature vectors.

[0089] In the embodiment of the present application, the background knowledge vectors and the text vectors can be spliced to realize the identification according to the background knowledge vectors and the text vectors at the same time. Therefore, compared with the identification only according to the text information, the identification according to the background knowledge vectors and the text vectors at the same time can improve the accuracy of identification.

[0090] Further, on the basis of the above-mentioned embodiments, the webpage recognition apparatus 200 further comprises a data cleaning module configured to clean data of the text information in the text content to remove special characters in the text information.

[0091] In the embodiments of the present application, the special characters in the text information can be removed by cleaning data of the text information, so as to avoid the influence of the special characters on the recognition process.

[0092] Further, on the basis of the above-mentioned embodiments, the webpage recognition apparatus 200 further comprises a training module configured to train the webpage recognition model; the training module is specifically configured to: obtain sample text content of a sample webpage and a corresponding label result; wherein the label result represents a webpage type of the sample webpage; input the sample text content into the webpage recognition model to obtain a corresponding predicted webpage type; calculate a prediction loss representing a difference between the label result and the predicted webpage type according to a cross-entropy loss function; and update parameters of the webpage recognition model according to the prediction loss.

[0093] In the embodiments of the present application, the webpage recognition model can be used to recognize the webpage to be recognized according to the background knowledge vector and the text vector. Compared with the recognition according to the text information only, the recognition according to the background knowledge vector and the text vector can improve the generalization ability of the webpage recognition model, thereby improving the recognition accuracy.

[0094] For the embodiments of the present application, refer to Figure 3 , Figure 3 A structural block diagram of an electronic device provided by the embodiments of the present application is shown in FIG. 3. The electronic device 300 comprises at least one processor 301, at least one communication interface 302, at least one memory 303 and at least one communication bus 304. The communication bus 304 is configured to realize direct connection communication among the components, the communication interface 302 is configured to communicate signaling or data with other node devices, and the memory 303 stores machine readable instructions executable by the processor 301. When the electronic device 300 is running, the processor 301 and the memory 303 communicate through the communication bus 304, and the machine readable instructions are executed by the processor 301 to perform the webpage recognition method described above.

[0095] For example, the processor 301 of the embodiment of the present application can implement the following method by reading a computer program from the memory 303 through the communication bus 304 and executing the computer program: step S101: obtaining text content of a webpage to be recognized. Step S102: obtaining background knowledge information related to the webpage to be recognized according to a keyword. Step S103: encoding and representing the background knowledge information and the text information to obtain a background knowledge vector and a text vector respectively. Step S104: determining a webpage type of the webpage to be recognized according to the background knowledge vector and the text vector.

[0096] The processor 301 includes one or more, which can be an integrated circuit chip having a signal processing capability. The processor 301 described above can be a general-purpose processor, including a central processing unit (CPU), a micro control unit (MCU), a network processor (NP) or other conventional processors; or can be a special-purpose processor, including a neural network processing unit (NPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. Moreover, when the processor 301 is multiple, a part of them can be general-purpose processors and the other part can be special-purpose processors.

[0097] The memory 303 includes one or more, which can be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read only memory (PROM), an erasable programmable read only memory (EPROM), an electrically erasable programmable read only memory (EEPROM) and the like.

[0098] Understandable. Figure 3 The structure shown is for illustrative purposes only; the electronic device 300 may also include components that are more advanced than those shown. Figure 3 The more or fewer components shown, or having the same Figure 3 The different configurations shown. Figure 3 The components shown can be implemented using hardware, software, or a combination thereof. In the embodiments of this application, electronic device 300 can be, but is not limited to, physical devices such as desktop computers, laptops, smartphones, smart wearable devices, and in-vehicle devices, or virtual devices such as virtual machines. Furthermore, electronic device 300 is not necessarily a single device; it can be a combination of multiple devices, such as a server cluster, etc.

[0099] This application also provides a computer program product, including a computer program stored on a computer-readable storage medium. The computer program includes computer program instructions. When the computer program instructions are executed by a computer, the computer can perform the steps of the webpage recognition method described in the above embodiments, such as: obtaining the text content of a webpage to be recognized; wherein the text content includes keywords corresponding to the webpage to be recognized and text information corresponding to the main body of the webpage to be recognized; obtaining background knowledge information related to the webpage to be recognized based on the keywords; encoding the background knowledge information and the text information to obtain a background knowledge vector and a text vector, respectively; and determining the webpage type of the webpage to be recognized based on the background knowledge vector and the text vector.

[0100] This application also provides a computer-readable storage medium that stores computer program instructions. When the computer program instructions are executed by a computer, the computer performs the webpage recognition method described in the foregoing method embodiments.

[0101] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0102] In addition, the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, i.e. may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment of the present application.

[0103] Furthermore, the functional modules in each embodiment of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0104] It should be noted that if the function is realized in the form of a software function module and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a magnetic disk or an optical disk, and various program code storage media.

[0105] In this paper, the relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations.

[0106] The above is only an embodiment of the present application and is not used to limit the protection scope of the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the present application shall be included in the protection scope of the present application.​

Claims

1. A webpage recognition method, characterized in that, include: Obtain the text content of the webpage to be identified; wherein, the text content includes keywords corresponding to the webpage to be identified and text information corresponding to the main text of the webpage to be identified; Obtain background knowledge information related to the webpage to be identified based on the keywords; The background knowledge information and the text information are encoded and represented to obtain background knowledge vectors and text vectors, respectively. The webpage type of the webpage to be identified is determined based on the background knowledge vector and the text vector; The step of obtaining background knowledge information related to the webpage to be identified based on the keywords includes: Extract background knowledge information corresponding to the keywords from the knowledge base; wherein, the knowledge base stores multiple keywords and background knowledge information corresponding to each keyword; The process of encoding and representing the background knowledge information and the text information to obtain background knowledge vectors and text vectors, respectively, includes: The background knowledge information and the text information are encoded and represented using a BERT pre-trained model to obtain background knowledge vectors and text vectors, respectively. Determining the webpage type of the webpage to be identified based on the background knowledge vector and the text vector includes: The background knowledge vector and the text vector are concatenated to obtain the feature vector; The webpage type to be identified is determined based on the feature vector.

2. The webpage recognition method according to claim 1, characterized in that, After obtaining the text content of the webpage to be identified, the method further includes: Data cleaning is performed on the text information in the text content to remove special characters.

3. The webpage recognition method according to claim 1, characterized in that, The webpage recognition method is executed through a webpage recognition model, which is trained through the following process: Obtain the sample text content of the sample webpage and the corresponding annotation results; wherein, the annotation results represent the webpage type of the sample webpage; The sample text content is input into the webpage recognition model to obtain the corresponding predicted webpage type; The prediction loss, which characterizes the difference between the annotation results and the predicted web page type, is calculated based on the cross-entropy loss function. The parameters of the webpage recognition model are updated based on the predicted loss.

4. A webpage recognition device, characterized in that, include: The first acquisition module is used to acquire the text content of the webpage to be identified; wherein, the text content includes keywords corresponding to the webpage to be identified and text information corresponding to the main text of the webpage to be identified; The second acquisition module is used to acquire background knowledge information related to the webpage to be identified based on the keywords; An encoding module is used to encode and represent the background knowledge information and the text information to obtain background knowledge vectors and text vectors, respectively. The determination module is used to determine the webpage type of the webpage to be identified based on the background knowledge vector and the text vector; The second acquisition module is specifically used for: Extract background knowledge information corresponding to the keywords from the knowledge base; wherein, the knowledge base stores multiple keywords and background knowledge information corresponding to each keyword; The encoding module is specifically used for: The background knowledge information and the text information are encoded and represented using a BERT pre-trained model to obtain background knowledge vectors and text vectors, respectively. The determining module is specifically used for: The background knowledge vector and the text vector are concatenated to obtain the feature vector; The webpage type to be identified is determined based on the feature vector.

5. A computer program product, characterized in that, It includes computer program instructions, which, when read and executed by a processor, perform the webpage recognition method as described in any one of claims 1-3.

6. An electronic device, characterized in that, include: Processor, memory, and bus; The processor and the memory communicate with each other via the bus; The memory stores computer program instructions that can be executed by the processor, and the processor can execute the webpage recognition method as described in any one of claims 1-3 by calling the computer program instructions.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when executed by a computer, cause the computer to perform the webpage recognition method as described in any one of claims 1-3.

Images