Website information processing method and device, electronic equipment and storage medium

Abstract

The invention relates to a website information processing method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining website information, extracting candidate keyword information from the website information, carrying out the keyword extraction based on the candidate keyword information, obtaining target keyword information, and obtaining the target keyword information according to the target keyword information. The method comprises the following steps: performing data enhancement in combination with preset hot word list information to obtain embedded vector information corresponding to target keyword information, and then performing joint semantic analysis according to the embedded vector information to obtain joint feature vector information corresponding to the target keyword information, so that the target keyword information is obtained on the basis of the joint feature vector information. The website fingerprint information corresponding to the website information is generated, that is, the website fingerprint information corresponding to the website information is generated after the website information is subjected to multiple processing, so that the validity of the website fingerprint information is not influenced even if the website information is partially changed; the problem that the effectiveness is low when original field information of a website is used as a website fingerprint in the prior art is solved.

Description

Technical Field

[0001] The present application relates to the field of computers, and in particular to a website information processing method, device, electronic device and storage medium. Background Art

[0002] In recent years, with the rapid development of the Internet, online website services have become an indispensable part of people's daily lives. From information query, social entertainment to online shopping and financial transactions, website services cover all aspects, providing users with a convenient and efficient Internet experience. However, along with this come the problems of network security and information credibility, which bring security risks and information leakage risks to users. In this context, website fingerprint generation technology came into being. Website fingerprint generation technology can effectively identify the type of website by generating a website fingerprint as the unique identifier of the website, thereby helping users accurately identify and distinguish different websites, so as to ensure that they are visiting legitimate and credible website content.

[0003] At present, the relevant technology directly extracts part of the single original field information from the website as the fingerprint of the website, such as using the website domain name as the fingerprint of the website, or using the website Internet Protocol (IP) address and other information as the fingerprint of the website, etc.; although the fingerprint determined by the above method can play a role in identifying the website, since it uses a single information in the website, the content of the information may change during the operation of the website, thereby making the fingerprint invalid, that is, the website fingerprint in the existing relevant technology has the problem of low effectiveness. Summary of the invention

[0004] In order to solve the problem of low effectiveness of using original field information of a website as a website fingerprint in the existing related technology, the present application provides a website information processing method, device, electronic device and storage medium.

[0005] In a first aspect, the present application provides a website information processing method, comprising:

[0006] Get website information;

[0007] Extracting candidate keyword information from the website information;

[0008] Perform keyword extraction based on the candidate keyword information to obtain target keyword information;

[0009] According to the target keyword information, combined with the preset hot word list information, data enhancement is performed to obtain the embedding vector information corresponding to the target keyword information;

[0010] Perform joint semantic analysis based on the embedded vector information to obtain the joint feature vector information corresponding to the target keyword information;

[0011] Generate the website fingerprint information corresponding to the website information based on the joint feature vector information.

[0012] Optionally, the obtaining of the embedded vector information corresponding to the target keyword information by combining the preset hot word list information according to the target keyword information includes:

[0013] Perform one-hot encoding according to the target keyword information in combination with the hot word list information to obtain the binary vector information corresponding to the target keyword information;

[0014] Perform probability analysis based on the binary vector information in combination with the hot word list information to obtain the embedded vector information.

[0015] Optionally, the performing of one-hot encoding according to the target keyword information in combination with the hot word list information to obtain the binary vector information corresponding to the target keyword information includes:

[0016] Extract at least one target keyword from the target keyword information;

[0017] For each of the target keywords, determine the word list position information corresponding to the target keyword in combination with the hot word list information;

[0018] Generate the binary vector information based on the word list position information.

[0019] Optionally, the performing of probability analysis based on the binary vector information in combination with the hot word list information to obtain the embedded vector information includes:

[0020] Determine at least one word list word information included in the hot word list information;

[0021] Perform probability analysis based on the binary vector information in combination with each of the word list word information to obtain the probability distribution information;

[0022] Perform weighted summation using the binary vector information and the probability distribution information to obtain the embedded vector information.

[0023] Optionally, the performing of joint semantic analysis based on the embedded vector information to obtain the joint feature vector information corresponding to the target keyword information includes:

[0024] Perform joint semantic analysis based on the embedded vector information to obtain the joint semantic information;

[0025] Based on the combined semantic information, vector processing is performed in combination with a preset vector space to generate combined feature vector information.

[0026] Optionally, after generating the website fingerprint information corresponding to the website information based on the combined feature vector information, the following further includes:

[0027] Based on the website fingerprint information, search is performed in combination with a preset website fingerprint vector library to obtain at least one target website fingerprint vector information. The website fingerprint vector library is a database storing website fingerprint vector information, and the website fingerprint vector information and the website fingerprint information belong to the same vector space;

[0028] Based on the target website fingerprint vector information, website type analysis is performed to obtain the target website type corresponding to the website information.

[0029] Optionally, the performing website type analysis based on the target fingerprint vector to obtain the target website type corresponding to the website information includes:

[0030] Determine the preset website type corresponding to each of the target fingerprint vectors;

[0031] Based on the preset website types, website type statistics are performed to obtain website type distribution information;

[0032] Based on the website type distribution information, determine the target website type.

[0033] In a second aspect, the present application provides a website information processing device, including:

[0034] An acquisition module, configured to acquire website information;

[0035] A first extraction module, configured to extract candidate keyword information from the website information;

[0036] A second extraction module, configured to perform keyword extraction based on the candidate keyword information to obtain target keyword information;

[0037] A data enhancement module, configured to perform data enhancement based on the target keyword information in combination with preset hot word list information to obtain embedding vector information corresponding to the target keyword information;

[0038] A combined semantic analysis module, configured to perform combined semantic analysis based on the embedding vector information to obtain combined feature vector information corresponding to the target keyword information;

[0039] A generation module, configured to generate website fingerprint information corresponding to the website information based on the combined feature vector information.

[0040] In a third aspect, an electronic device is provided, including a processor, a communication interface, a memory, and a communication bus. Among them, the processor, the communication interface, and the memory complete communication with each other through the communication bus;

[0041] The memory is used to store a computer program;

[0042] The processor is used to implement the website information processing method described in any item of the first aspect when executing the program stored on the memory.

[0043] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the website information processing method described in any item of the first aspect is implemented.

[0044] In the embodiments of the present application, by obtaining website information, extracting candidate keyword information from the website information, performing keyword extraction based on the candidate keyword information to obtain target keyword information, and combining the preset hot word list information according to the target keyword information for data enhancement to obtain the embedding vector information corresponding to the target keyword information, and then performing joint semantic analysis based on the embedding vector information to obtain the joint feature vector information corresponding to the target keyword information, so as to generate the website fingerprint information corresponding to the website information based on the joint feature vector information. That is, the present application generates the corresponding website fingerprint information by performing multiple processes on the website information, so that even if the website information changes partially, it will not affect the validity of the website fingerprint information, solving the problem of low validity in the related art of using the original field information of the website as the website fingerprint, and can effectively improve the validity of the website fingerprint. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The drawings here are incorporated into the description and form a part of this description, showing embodiments consistent with the present invention and used together with the description to explain the principles of the present invention.

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, other drawings can also be obtained based on these drawings without creative efforts.

[0047] Figure 1 It is a schematic flowchart of a website information processing method provided by an embodiment of the present application;

[0048] Figure 2 It is another schematic flowchart of a website information processing method provided by an embodiment of the present application;

[0049] Figure 3Schematic diagram of an application scenario of a website information processing method provided by an embodiment of the present application;

[0050] Figure 4 Another schematic diagram of an application scenario of a website information processing method provided by an embodiment of the present application;

[0051] Figure 5 Schematic diagram of the structure of a website information processing device provided by an embodiment of the present application;

[0052] Figure 6 Schematic diagram of the structure of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0053] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are some, but not all, of the embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative efforts shall fall within the protection scope of the present application.

[0054] With the development of Internet technology, the types and categories of websites are also increasing. A website usually contains various types of information, such as page titles, domain names, displayed texts, displayed images, links, IPs, and other information. To improve the recognition of websites, website fingerprints are usually used to identify websites. In the existing related technologies, the original field information in website information is usually used as the website fingerprint to play the role of identifying the website, such as the website domain name, website IP, etc. Although using the original field information in website information as the website fingerprint can play the role of identifying the website, this method can only identify the website and is difficult to play the role of classifying the website. In addition, since the website information is directly used as the website fingerprint, if the original field information of the website is found to change during the operation of the website, the website fingerprint will become invalid, that is, the problem of low effectiveness exists in using the original field information of the website as the website fingerprint in the existing related technologies.

[0055] To solve the problem of low effectiveness in using the original field information of a website as a website fingerprint in the existing related technologies, the present application provides a website information processing method, apparatus, electronic device, and storage medium. By obtaining website information, extracting candidate keyword information from the website information, performing keyword extraction based on the candidate keyword information to obtain target keyword information, enhancing data based on the target keyword information in combination with a preset hot word list information to obtain embedded vector information corresponding to the target keyword information, then performing joint semantic analysis based on the embedded vector information to obtain joint feature vector information corresponding to the target keyword information, and thus generating website fingerprint information corresponding to the website information based on the joint feature vector information. That is, the present application generates corresponding website fingerprint information by performing multiple processes on the website information, so that even if the website information changes partially, it will not affect the effectiveness of the website fingerprint information, solving the problem of low effectiveness in using the original field information of a website as a website fingerprint in the existing related technologies and being able to effectively improve the effectiveness of the website fingerprint.

[0056] Figure 1 The flowchart of a website information processing method provided by an embodiment of the present application. This method can be applied to one or more electronic devices such as a client, a server, etc. In addition, the execution subject of this method can be hardware or software. When the above execution subject is hardware, the execution subject can be one or more of the above electronic devices. For example, a single electronic device can execute this method, or multiple electronic devices can cooperate with each other to execute this method. When the above execution subject is software, this method can be implemented as multiple software or software modules, or can be implemented as a single software or software module. No specific limitation is made here.

[0057] As Figure 1 shown, the present application discloses an embodiment and provides a website information processing method, which specifically may include the following steps:

[0058] Step S110: Obtain website information.

[0059] Among them, the website information may refer to the information corresponding to the target website, where the target website may represent the website for which website fingerprint needs to be generated, and the website information may include but is not limited to information such as the HyperText Markup Language (HTML) of the website, page identifier, icon, display image, certificate, domain name, IP address, Whois information, etc.; and the specific acquisition method of the website information may be through script crawling, algorithm acquisition, etc., and this embodiment does not make specific limitations on this.

[0060] Step S120: Extract candidate keyword information from the website information.

[0061] Specifically, after obtaining website information, since the website information contains a large amount of information, at this time, candidate keyword information can be extracted from the website information. The candidate keyword information represents the information contained in the pre-configured candidate keyword fields, where the candidate fields can be configured differently according to different target websites, and this embodiment does not make specific limitations on this.

[0062] It should be noted that the extraction method of the candidate keyword information can determine the preset candidate keyword fields corresponding to the website information, extract the information belonging to the preset candidate keyword fields in the website information, and obtain the candidate keyword information. For example, the candidate keyword information can include but is not limited to domain name information, certificates, DNS data, Whois information, website IP, HTML information, etc. Of course, the above is only for illustrative purposes, and this embodiment does not make specific limitations on this.

[0063] Step S130: Extract keywords based on the candidate keyword information to obtain target keyword information.

[0064] Specifically, after obtaining the candidate keyword information, keywords can be extracted based on the candidate keyword information to obtain target keyword information. The target keyword information can represent the keywords contained in the candidate keyword information.

[0065] In addition, since the candidate keyword information can include one or more types of information, different keyword extraction methods can be adopted for different types of information at this time; that is, in the process of extracting keywords based on the candidate keyword information to obtain target keyword information, the information type of the candidate keyword information can be determined, the extraction method corresponding to the information type can be determined, and the candidate keyword information can be extracted using this extraction method to obtain target keyword information.

[0066] In a specific example, when the candidate keyword information is website domain name information, it is determined that the information type of the candidate keyword information is the website domain name type. At this time, the extraction method corresponding to the website domain name type is to extract the domain name text as the keyword, that is, the domain name text in the website domain name information can be directly extracted to obtain the target keyword information.

[0067] When the candidate keyword information is website certificate information, it is determined that the information type of the candidate keyword information is the website certificate type. At this time, the extraction method corresponding to the website certificate type is to extract the information corresponding to the organization keyword, that is, the information corresponding to the organization keyword in the website domain name information can be extracted to obtain the target keyword information.

[0068] When the candidate keyword information is DNS data information, determine that the information type of the candidate keyword information is DNS type. At this time, the extraction method corresponding to the DNS type is to extract the information of the CNAME field and the subdomain field, that is, the information of the CNAME field and the subdomain field in the website domain name information can be extracted to obtain the target keyword information.

[0069] When the candidate keyword information is Whois information, determine that the information type of the candidate keyword information is Whois type. At this time, the extraction method corresponding to the Whois type is to extract information such as the registrant's email and ANS number, that is, the information such as the registrant's email and ANS number in the website domain name information can be extracted to obtain the target keyword information.

[0070] When the candidate keyword information is the website IP, determine that the information type of the candidate keyword information is website IP type. At this time, the extraction method corresponding to the website IP type is to extract the first 3 segments of the string in the IP, that is, the first 3 segments of the string in the IP in the website domain name information can be extracted to obtain the target keyword information.

[0071] When the candidate keyword information is HTML information, determine that the information type of the candidate keyword information is HTML type. At this time, the extraction method corresponding to the HTML type is to extract the content corresponding to the tag, the website filing information and the website text information, that is, the content corresponding to the tag, the website filing information and the website text information in the website domain name information can be extracted to obtain the target keyword information. Of course, the above is only for illustrative purposes, and this implementation does not make specific limitations.

[0072] Specifically, after obtaining the target data, the target data can be input into a preset target model. The target model is a model that has been pre-trained to process the target data. More specifically, the target model can be a model for feature embedding of text data, such as the BERT (Bidirectional Encoder Representations from Transformers) model. Then, the intermediate layer output vector of the target model can be obtained, and the intermediate layer output vector is a feature representation vector generated by the target model based on the target data. The feature representation vector is a vector obtained by the target model for feature embedding of the target data.

[0073] Step S140: According to the target keyword information, combine the preset hot word list information for data augmentation to obtain the embedding vector information corresponding to the target keyword information.

[0074] Specifically, after obtaining the target keyword information, data augmentation can be performed in combination with the preset hot word list information to obtain the embedding vector information corresponding to the target keyword information. The embedding vector information can represent the vector information corresponding to the embedded representations of each keyword in the target keyword information. Embedded representation is a process of converting high-dimensional data such as text or images into lower-dimensional vectors. In addition, the preset hot word list information can represent a pre-configured list information containing many high-frequency words, where the high-frequency words can include but are not limited to Chinese words, English words, symbol words, etc. This embodiment does not make specific limitations in this regard. It should be noted that the target keyword information can include one or more keywords. At this time, the embedding vector information can include the vector information corresponding to the embedded representation of each keyword.

[0075] Since the target keyword information in this embodiment is directly extracted from the website information, this process may fail to extract effective keywords due to the overly rich website content and the lack of prominent theme words. At this time, data augmentation is a process of expanding the dimensions of each keyword in the target keyword information. For example, when the target keyword information contains the keyword "love", when performing data augmentation in combination with the hot word list information, a mapping relationship can be established between "love" and high-frequency words such as "like" and "preference" in the hot word list information. Then, based on the relationship between "love" and "like" and "preference", the corresponding embedded representation is generated as the embedding vector information. Of course, the above is only for illustrative purposes, and this embodiment does not make specific limitations in this regard.

[0076] Step S150: Perform joint semantic analysis based on the embedding vector information to obtain the joint feature vector information corresponding to the target keyword information.

[0077] Specifically, since the embedding vector information can represent the vector information corresponding to the embedded representations of each keyword, after obtaining the embedding vector information, joint semantic analysis can be performed based on the embedding vector information to obtain the joint feature vector information corresponding to the target keyword information. Among them, joint semantic analysis is to perform context semantic analysis by combining the vector information corresponding to the embedded representation of each keyword, obtain the feature vector of the joint embedded representation of the website information corresponding to the target keyword information, and determine this feature vector as the joint feature vector information. That is, the joint feature vector information can represent the feature vector that combines each embedding vector information.

[0078] In one example, for the specific manner of performing joint semantic analysis based on the embedded vector information to obtain the joint feature vector information corresponding to the target keyword information, a neural network model can be used. For example, the vector information of the embedded representation corresponding to each keyword included in the embedded vector information is used as the input of the Bert-base model. In the bidirectional stacked Transformer Encoder, the Bert-base model jointly learns the upstream features and downstream features of the vector information of the embedded representation corresponding to each input keyword. After forward learning and backward learning, at the position of the [CLS] token in the Bert-base model, the joint embedded representation of all input features of the current website is output, that is, a feature vector, and this feature vector is used as the joint feature vector information. Of course, the above is only for illustrative purposes, and this embodiment does not make specific limitations on this.

[0079] Step S150: Generate website fingerprint information corresponding to the website information based on the joint feature vector information.

[0080] Specifically, after obtaining the joint feature vector information, based on this joint feature vector information, website fingerprint information corresponding to the website information can be generated. It can be directly using this joint feature vector information as the website fingerprint information, or converting the joint feature vector information to obtain target format information and using the converted target format information as the website fingerprint information. Of course, other methods can also be used to generate the website fingerprint information, and this embodiment does not make specific limitations on this.

[0081] It can be seen that in this embodiment, the original field information in the website information is not directly used as the website fingerprint. For example, features such as the organization field in the website information, the registrant email in Whois, the ASN number, and the IP are not used as the website fingerprint. Instead, it is the joint feature vector information obtained after keyword extraction, feature addition, and joint semantic analysis of the entire website information. Subsequently, based on this joint feature vector information, website fingerprint information corresponding to the website information is generated. In this process, various types of information included in the website information are processed multiple times, enabling the website fingerprint information to identify the website in the form of a feature vector. At this time, even if part of the website information changes, it will not affect the validity of this website fingerprint information, solving the problem of low validity in using the original field information of the website as the website fingerprint in the existing related technologies, and effectively improving the validity of the website fingerprint.

[0082] On the other hand, when generating the combined feature vector information corresponding to each website information as website fingerprint information through this embodiment, all website fingerprint information can be made to be in the same vector space, which can facilitate the analysis of the similarity between websites or the similarity of website categories. That is to say, it solves the problem in the existing related technologies that website fingerprints can only identify websites and cannot classify and identify website types, and can effectively improve the efficiency of website classification and identification.

[0083] As Figure 2 shown, in an alternative embodiment of the present application, step S140 performs data enhancement based on the target keyword information in combination with the preset hot word list information to obtain the embedding vector information corresponding to the target keyword information, which may specifically include the following sub-steps:

[0084] Step S141: Perform one-hot encoding based on the target keyword information in combination with the hot word list information to obtain the binary vector information corresponding to the target keyword information;

[0085] Step S142: Perform probability analysis based on the binary vector information in combination with the hot word list information to obtain the embedding vector information.

[0086] After obtaining the target keyword information in this embodiment, one-hot encoding can be performed based on the target keyword information in combination with the hot word list information to obtain the binary vector information corresponding to the target keyword information; among them, one-hot encoding is to convert categorical variables into vector information that is easier for machine learning algorithms to process, and the binary vector information can represent the binary vectors corresponding to each keyword in the target keyword information; thus, probability analysis can be performed based on the binary vector information in combination with the hot word list information to obtain the embedding vector information, where the probability analysis can represent the probability analysis process of analyzing which word in the word list represented by the hot word list information each word corresponding to the binary vector belongs to, and the embedding vector information can represent the vector information corresponding to the embedding representation of each keyword in the target keyword information.

[0087] In an alternative embodiment of the present application, performing one-hot encoding based on the target keyword information in combination with the hot word list information to obtain the binary vector information corresponding to the target keyword information may specifically include the following sub-steps: Extract at least one target keyword from the target keyword information; for each target keyword, determine the word list position information corresponding to the target keyword in combination with the hot word list information; generate the binary vector information based on the word list position information.

[0088] In the process of performing one-hot encoding based on the target keyword information and in combination with the hot word list information to obtain the binary vector information corresponding to the target keyword information, at least one target keyword can be extracted from the target keyword information, where the target keyword can represent the keyword contained in the target keyword information. Thus, for each target keyword, the position information of the word list corresponding to the target keyword can be determined in combination with the hot word list information. Since the hot word list information indicates that there can be many high-frequency words in the word list, at this time, the target keyword can be compared with each high-frequency word in the hot word list information to determine the high-frequency word that matches the target keyword, and the position information of the word list of the high-frequency word that matches the target keyword can be obtained, so that the position information of the word list can represent the position of the target keyword in the word list corresponding to the hot word list information. Furthermore, the binary vector information can be generated based on the position information of the word list. The binary vector information can be used to indicate the vector of the specific target keyword. Among them, in this embodiment, the binary vector can be a vector with the dimension of the size of the word list corresponding to the hot word list information. Subsequently, the binary vectors corresponding to each target keyword in the target keyword information can be integrated to obtain the binary vector information corresponding to the target keyword information.

[0089] Specifically, the method of performing one-hot encoding in this embodiment can be to adopt One-hot encoding. One-hot encoding will create a new binary feature for each category value of the categorical variable, where only one position is 1 indicating "hot" and the rest are 0. The categorical variable corresponds to each target keyword in the target keyword information of this embodiment. The encoding principle of One-hot encoding is the preset hot word list information, which indicates that there are many high-frequency words in the word list. In the process of encoding each target keyword using One-hot encoding, the position of the target keyword in the word list corresponding to the hot word list information can be determined, and binary encoding can be performed based on this position information of the word list. For example, if the word list corresponding to the hot word list information contains 5,000 high-frequency words, when the target keyword is "love", the position information of the word list of the high-frequency word "love" that matches this target keyword is determined to be the 253rd position. At this time, binary encoding can generate a binary vector with the 253rd position encoded as "1" and the remaining 4,757 positions encoded as "0", that is, this binary vector is a vector with the dimension of the size of the word list corresponding to the hot word list information, which is 5,000. The above steps can be executed for each target keyword. Subsequently, the binary vectors corresponding to each target keyword in the target keyword information can be integrated to obtain the binary vector information corresponding to the target keyword information. The above is only for illustrative purposes, and this embodiment does not make specific limitations on this.

[0090] In an alternative embodiment of the present application, based on the binary vector information, probability analysis is performed in combination with the hot word list information to obtain the embedded vector information, which may specifically include the following sub-steps: determining at least one word list word information included in the hot word list information; based on the binary vector information, performing probability analysis in combination with each word list word information to obtain probability distribution information; and performing weighted summation using the binary vector information and the probability distribution information to obtain the embedded vector information.

[0091] In this embodiment, in the process of performing probability analysis based on the binary vector information and in combination with the hot word list information to obtain the embedded vector information, at least one word list word information included in the hot word list information can be determined, and the word list word information represents the specific words included in the hot word list information; thus, probability analysis can be performed based on the binary vector information and in combination with each word list word information to obtain probability distribution information, where the probability analysis can represent analyzing the mapping probability between the binary vector information corresponding to the target keyword and each word list word information in the word list, so as to obtain the probability distribution information, that is, the probability distribution information can represent the distribution of the mapping probabilities between the binary vector information corresponding to the target keyword and each word list word information; furthermore, weighted summation can be performed using the binary vector information and the probability distribution information to obtain the embedded vector information, where the specific weighted summation can be in accordance with a preset formula or model, etc., and this embodiment does not make specific limitations in this regard.

[0092] In an example, in the process of performing probability analysis based on the binary vector information and in combination with each word list word information to obtain the probability distribution information, a non-linear transformation can be performed on the binary vector of each input target keyword in the Transformer Block part of the Text Smoothing model; the output of the Transformer Block part will pass through the MLM network part to calculate the probability that each input target keyword belongs to each word list word information in the hot word list information; for example, when the binary vector information represents the binary vector of the target keyword "love", performing probability analysis in combination with each word list word information, the obtained probability distribution information can represent that the probability corresponding to "love" is 1, the probability corresponding to "like" is 0.8, the probability corresponding to "preference" is 0.5, etc.; that is, for each binary vector corresponding to each target keyword in the input binary vector information, a probability distribution of the size of the word list will be output, representing the probability that the model predicts that the current input target keyword belongs to each word in the word list; furthermore, it can play a role in data augmentation for the binary vector information, avoiding the problem that the extracted target keywords are not prominent due to the overly rich website content during the keyword extraction process, and can effectively enhance the extracted target keywords.

[0093] In another example, as Figure 3As shown, in the process of obtaining the embedded vector information by performing weighted summation on the zero-one vector information and the probability distribution information, the following formula can be used for weighted summation:

[0094]

[0095] Among them, λ represents a pre-configured weighting coefficient, and t i represents the zero-one vector corresponding to the i-th target keyword in the current zero-one vector information obtained by one-hot, and MLM(t i ) represents the predicted probability of the current target keyword in the vocabulary dimension obtained by Transformer.

[0096] In an alternative embodiment of the present application, step S150 performs joint semantic analysis based on the embedded vector information to obtain joint feature vector information corresponding to the target keyword information, which may specifically include the following sub-steps: performing joint semantic analysis based on the embedded vector information to obtain joint semantic information; based on the joint semantic information, combining a preset vector space to perform vector processing to generate joint feature vector information.

[0097] In this embodiment, after obtaining the embedded vector information, joint semantic analysis can be performed based on the embedded vector information to obtain joint semantic information; since the embedded vector information can represent the vector information corresponding to each keyword in the target keyword information, performing joint semantic analysis at this time can represent the process of combining the above-context features and the below-context features of each vector information in the embedded vector information, and performing forward learning and backward learning analysis. Specifically, a preset natural language processing model can be used for joint semantic analysis, and the joint semantic information can represent the semantic features corresponding to each vector information in the embedded vector information; furthermore, based on the joint semantic information, combining a preset vector space to perform vector processing to generate joint feature vector information, where the vector space represents the space to which the pre-configured generated feature vector belongs, that is, the joint semantic information can be vector-processed according to the vector processing method corresponding to the vector space to obtain the joint feature vector information corresponding to the joint semantic information, and the vector processing method can be a neural network model, a calculation model, etc., and this embodiment does not make specific limitations on this.

[0098] Specifically, such as Figure 4As shown, in this embodiment, the Bert-base model can be used to perform the above steps. The embedded vector information can be used as the input of the Bert-base model, and semantic analysis can be performed through the Bert-base model. And at the position of the [CLS] token of the Bert-base model, a joint embedded representation of all input features corresponding to the current embedded vector information is output, that is, a feature vector, and this feature vector is used as the joint feature vector information. In addition, during the training of the Bert-base model, classification tasks can be performed for different types of websites to enable the model to identify whether it is a XX / XX / XX website, etc. Therefore, the model will learn the fingerprint features of websites in a specific domain. Of course, the above is only for illustrative purposes, and this embodiment does not make specific limitations on this.

[0099] Although the website fingerprints in the existing related technologies can play a role in identifying websites, since the website fingerprints usually use the original field information in the website information, during the process of website classification and recognition, it is still necessary to manually classify and recognize the website fingerprints, that is, there is a problem of low efficiency in website classification and recognition.

[0100] In an optional embodiment of the present application, after step S160 generates the website fingerprint information corresponding to the website information based on the joint feature vector information, the following steps may specifically further include: based on the website fingerprint information, combined with a preset website fingerprint vector library for searching, obtaining at least one target website fingerprint vector information, where the website fingerprint vector library is a database storing website fingerprint vector information, and the website fingerprint vector information and the website fingerprint information belong to the same vector space; based on the target website fingerprint vector information, performing website type analysis to obtain the target website type corresponding to the website information.

[0101] Since the joint feature vector information in this embodiment is generated through a preset unified vector processing method, the feature vectors represented by subsequent joint feature vector information are in the same vector space. Thus, after generating the website fingerprint information corresponding to the website information, based on the website fingerprint information, combined with a preset website fingerprint vector library for searching, at least one target website fingerprint vector information can be obtained. Among them, the website fingerprint vector library can represent a database for storing website fingerprint vector information, and the website fingerprint vector information and the website fingerprint information belong to the same vector space, and the target website fingerprint vector information can represent a vector similar to the type of the website fingerprint information; thus, website type analysis can be performed based on the target website fingerprint vector information to obtain the target website type corresponding to the website information, and the target website type can represent the website type of the target website corresponding to the website information.

[0102] It should be noted that in the process of searching based on website fingerprint information and combining with a preset website fingerprint vector library to obtain at least one piece of target website fingerprint vector information, an Approximate Nearest Neighbor Search (ANN) algorithm can be used. Through the ANN algorithm, K vectors most similar to the website fingerprint information are searched from the website fingerprint vector library, and these K vectors are determined as the target website fingerprint vector information. The above is only for illustrative purposes, and this embodiment does not make specific limitations on this.

[0103] It can be seen that after the website fingerprint information is generated in this embodiment, if it is necessary to determine the type of the website, such as determining whether it is a website of XX / XX / XX type, etc., only the website fingerprint information corresponding to the current website needs to be used for approximate nearest neighbor search in the website fingerprint vector library, and the target website fingerprint vector information corresponding to the top K similar websites is searched out. Then, by analyzing the types of the similar websites through the target website fingerprint vector information, the target website type corresponding to the website information can be determined. Additionally, if it is necessary to find websites similar to the current website, then the top K websites searched out can be directly used as the similar websites; thus, it avoids the problem of low efficiency in website classification and recognition that exists in the prior related technologies where manual classification and recognition of website fingerprints are required, and can effectively improve the efficiency of website classification and recognition.

[0104] In an optional embodiment of the present application, for website type analysis based on the target fingerprint vector to obtain the target website type corresponding to the website information, it may specifically include the following sub-steps: determining the preset website type corresponding to each target fingerprint vector; performing website type statistics based on the preset website type to obtain website type distribution information; and determining the target website type according to the website type distribution information.

[0105] In the process of this embodiment for website type analysis based on the target fingerprint vector to obtain the target website type corresponding to the website information, the preset website type corresponding to each target fingerprint vector can be determined. The preset website type can represent the website type to which the website corresponding to the target fingerprint vector belongs; thus, website type statistics can be performed based on the preset website type to obtain website type distribution information, where website type statistics can represent counting the quantities of various preset website types included in the target fingerprint vector, so that the website type distribution information can represent the distribution of the quantities of various preset website types; furthermore, the target website type can be determined according to the website type distribution information, where the target website type can represent the preset network type with the largest quantity.

[0106] In a specific example, the preset website types corresponding to each target fingerprint vector are determined. The preset website types may include XX website types, video website types, and game website types. When the number of target fingerprint vectors is 10, based on the preset website types, website type statistics are performed, and the obtained website type distribution information may be that the number of XX website types is 5, the number of video website types is 2, and the number of game website types is 3. Furthermore, the preset network type with the largest number can be determined as the target website type, that is, the XX website type is determined as the target website type. Of course, the above is only for illustrative purposes, and this embodiment does not make specific limitations in this regard.

[0107] As Figure 5 shown, the present application also discloses an embodiment, which provides a website information processing device, specifically including the following modules:

[0108] An acquisition module 510, configured to acquire website information;

[0109] A first extraction module 520, configured to extract candidate keyword information from the website information;

[0110] A second extraction module 530, configured to perform keyword extraction based on the candidate keyword information to obtain target keyword information;

[0111] A data enhancement module 540, configured to perform data enhancement based on the target keyword information in combination with preset hot word list information to obtain embedding vector information corresponding to the target keyword information;

[0112] A joint semantic analysis module 550, configured to perform joint semantic analysis based on the embedding vector information to obtain joint feature vector information corresponding to the target keyword information;

[0113] A generation module 560, configured to generate website fingerprint information corresponding to the website information based on the joint feature vector information.

[0114] In an optional embodiment of the present application, the data enhancement module 540 may include:

[0115] A one-hot encoding unit, configured to perform one-hot encoding based on the target keyword information in combination with the hot word list information to obtain zero-one vector information corresponding to the target keyword information;

[0116] A probability analysis unit, configured to perform probability analysis based on the zero-one vector information in combination with the hot word list information to obtain the embedding vector information.

[0117] In an optional embodiment of the present application, the one-hot encoding unit may include:

[0118] A first extraction subunit, configured to extract at least one target keyword from the target keyword information;

[0119] A first determination subunit, configured to determine, for each of the target keywords, a vocabulary position information corresponding to the target keyword in combination with the hot word list information;

[0120] A first generation subunit, configured to generate the binary vector information based on the vocabulary position information.

[0121] In an optional embodiment of the present application, the probability analysis unit may include:

[0122] A second determination subunit, configured to determine at least one vocabulary word information included in the hot word list information;

[0123] A probability analysis subunit, configured to perform probability analysis based on the binary vector information in combination with each piece of the vocabulary word information to obtain probability distribution information;

[0124] A weighted summation subunit, configured to perform weighted summation using the binary vector information and the probability distribution information to obtain embedded vector information.

[0125] In an optional embodiment of the present application, the joint semantic analysis module 550 may include:

[0126] A joint semantic analysis unit, configured to perform joint semantic analysis based on the embedded vector information to obtain joint semantic information;

[0127] A vector processing unit, configured to perform vector processing based on the joint semantic information in combination with a preset vector space to generate joint feature vector information.

[0128] In an optional embodiment of the present application, the website information processing device may further include:

[0129] A search module, configured to perform a search based on the website fingerprint information in combination with a preset website fingerprint vector library to obtain at least one target website fingerprint vector information, where the website fingerprint vector library is a database storing website fingerprint vector information, and the website fingerprint vector information and the website fingerprint information belong to the same vector space;

[0130] A website type analysis module, configured to perform website type analysis based on the target website fingerprint vector information to obtain a target website type corresponding to the website information.

[0131] In an optional embodiment of the present application, the website type analysis module may include:

[0132] A first determination unit, configured to determine a preset website type corresponding to each of the target fingerprint vectors;

[0133] A website type statistics unit, configured to perform website type statistics based on the preset website types to obtain website type distribution information;

[0134] A second determination unit, configured to determine the target website type according to the website type distribution information.

[0135] For the implementation processes of the functions and roles of each module in the above device, please refer to the implementation processes of the corresponding steps in the above method for details, which will not be elaborated here.

[0136] As Figure 6 shown, an embodiment of the present application provides an electronic device, including a processor 610, a communication interface 620, a memory 630, and a communication bus 640. Among them, the processor 610, the communication interface 620, and the memory 630 complete mutual communication through the communication bus 640;

[0137] The memory 630 is used to store a computer program;

[0138] In an embodiment of the present application, when the processor 610 is configured to execute the program stored on the memory 630, it implements the website information processing method provided by any one of the foregoing method embodiments. By obtaining website information, extracting candidate keyword information from the website information, performing keyword extraction based on the candidate keyword information to obtain target keyword information, and then combining the preset hot word list information based on the target keyword information for data enhancement to obtain embedding vector information corresponding to the target keyword information. Subsequently, joint semantic analysis is performed based on the embedding vector information to obtain joint feature vector information corresponding to the target keyword information. Thus, based on the joint feature vector information, website fingerprint information corresponding to the website information is generated. That is, the present application generates its corresponding website fingerprint information by performing multiple processes on the website information, so that even if the website information changes partially, it will not affect the validity of the website fingerprint information, solving the problem of low validity in using the original field information of the website as the website fingerprint in the existing related technologies and being able to effectively improve the validity of the website fingerprint.

[0139] The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the website information processing method provided in any of the foregoing method embodiments. By obtaining website information, extracting candidate keyword information from the website information, performing keyword extraction based on the candidate keyword information to obtain target keyword information, and combining the preset hot word list information according to the target keyword information for data enhancement to obtain the embedding vector information corresponding to the target keyword information, then performing joint semantic analysis based on the embedding vector information to obtain the joint feature vector information corresponding to the target keyword information, and thus generating the website fingerprint information corresponding to the website information based on the joint feature vector information. That is, the present application generates the corresponding website fingerprint information by performing multiple processes on the website information, so that even if the website information changes partially, it will not affect the validity of the website fingerprint information, solving the problem of low validity existing in using the original field information of the website as the website fingerprint in the existing related technologies, and being able to effectively improve the validity of the website fingerprint.

[0140] It should be noted that in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "including a..." does not exclude the existence of additional identical elements in the process, method, article or device including the said element.

[0141] The above describes specific embodiments of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in a different order than in the embodiments and still achieve the desired result. Additionally, the processes depicted in the drawings do not necessarily require the specific order or sequential order shown to achieve the desired result. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0142] The above are only specific embodiments of the present invention, enabling those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be obvious to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to these embodiments shown herein, but rather to the broadest scope consistent with the principles and novel features claimed herein.

Claims

1. A website information processing method, characterized in that: include: Get website information; Extracting candidate keyword information from the website information; Perform keyword extraction based on the candidate keyword information to obtain target keyword information; According to the target keyword information, combined with the preset hot word list information, data enhancement is performed to obtain the embedding vector information corresponding to the target keyword information; Performing joint semantic analysis based on the embedded vector information to obtain joint feature vector information corresponding to the target keyword information; Based on the joint feature vector information, website fingerprint information corresponding to the website information is generated.

2. The website information processing method according to claim 1, characterized in that: The step of performing data enhancement based on the target keyword information and combining the preset hot word list information to obtain the embedding vector information corresponding to the target keyword information includes: According to the target keyword information, combined with the hot word list information, one-hot encoding is performed to obtain zero-one vector information corresponding to the target keyword information; Based on the zero-one vector information, probability analysis is performed in combination with the hot word list information to obtain the embedded vector information.

3. The website information processing method according to claim 2, characterized in that: The one-hot encoding is performed based on the target keyword information and combined with the hot word list information to obtain zero-one vector information corresponding to the target keyword information, including: Extracting at least one target keyword from the target keyword information; For each of the target keywords, determine the keyword position information corresponding to the target keyword in combination with the hot word list information; Based on the vocabulary position information, the zero-one vector information is generated.

4. The website information processing method according to claim 2, characterized in that: The performing probability analysis based on the zero-one vector information and combining the hot word list information to obtain the embedded vector information includes: Determine at least one vocabulary word information included in the hot word list information; Based on the zero-one vector information, probability analysis is performed in combination with each word information of the vocabulary to obtain probability distribution information; The zero-one vector information and the probability distribution information are weightedly summed to obtain embedded vector information.

5. The website information processing method according to claim 1, characterized in that: The performing joint semantic analysis based on the embedded vector information to obtain joint feature vector information corresponding to the target keyword information includes: Performing joint semantic analysis based on the embedded vector information to obtain joint semantic information; Based on the joint semantic information, vector processing is performed in combination with a preset vector space to generate joint feature vector information.

6. The website information processing method according to any one of claims 1 to 5, characterized in that: After generating the website fingerprint information corresponding to the website information based on the joint feature vector information, the method further includes: Based on the website fingerprint information, a search is performed in combination with a preset website fingerprint vector library to obtain at least one target website fingerprint vector information, wherein the website fingerprint vector library is a database storing website fingerprint vector information, and the website fingerprint vector information and the website fingerprint information belong to the same vector space; A website type analysis is performed based on the target website fingerprint vector information to obtain the target website type corresponding to the website information.

7. The website information processing method according to claim 6, characterized in that: The performing website type analysis based on the target fingerprint vector to obtain the target website type corresponding to the website information includes: Determine a preset website type corresponding to each of the target fingerprint vectors; Performing website type statistics based on the preset website types to obtain website type distribution information; The target website type is determined according to the website type distribution information.

8. A website information processing device, characterized in that: include: Acquisition module, used to obtain website information; A first extraction module, used to extract candidate keyword information from the website information; A second extraction module, configured to extract keywords based on the candidate keyword information to obtain target keyword information; A data enhancement module, used to perform data enhancement based on the target keyword information and in combination with preset hot word list information to obtain embedded vector information corresponding to the target keyword information; A joint semantic analysis module, used to perform joint semantic analysis based on the embedded vector information to obtain joint feature vector information corresponding to the target keyword information; A generating module is used to generate website fingerprint information corresponding to the website information based on the joint feature vector information.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; The processor is used to implement the website information processing method described in any one of claims 1 to 7 when executing the program stored in the memory.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the website information processing method according to any one of claims 1 to 7 is implemented.

Images