A method to fight malicious spider pools based on large models

Abstract

The present invention relates to the field of network security technology, and in particular to a method for combating a malicious spider pool based on a large model, wherein the method comprises: obtaining all long-tail keywords of the current web page data packet based on the current web page returned by the crawler in the current working cycle, inputting a pre-trained semantic large model to obtain all target keywords of the current web page data packet, thereby detecting whether the current web page meets a preset similarity condition, and when the preset similarity condition is met, determining that the crawler is in a malicious spider pool, and stopping the crawler from crawling the current web page. The embodiment of the present invention can use the pre-trained large model to extract the corresponding target keywords of all long-tail keywords of the current page, and construct a web page feature vector, detect the keyword similarity between the current web page and the previous web page, thereby accurately identifying the long-tail keywords, judging the similarity of unknown web pages, and quickly diagnosing the crawler trapped in the spider pool, ensuring the efficiency and security of network search, and being more practical.

Description

Technical Field

[0001] The present invention relates to the field of network security technology, and in particular to a method for combating a malicious spider pool based on a large model. Background Art

[0002] Black Hat SEO (BlackHat Search Engine Optimization) is a new type of business used to attract traffic at a low price. Operators build a large number of dynamic, simple web pages containing long-tail keywords, and have several links that jump to other simple websites containing long-tail keywords. A closed loop is formed through web page jumps, and a complex loop structure is constructed to trap search crawlers in the loop. This closed-loop network is called a spider pool.

[0003] In the related technology, there are widespread websites on the Internet that use black hat SEO to maliciously improve rankings, which has a negative impact on the maintenance of the network environment. Among them, part-of-speech analysis algorithms can currently be used to extract the target keywords that users actually search for from long-tail keywords to help crawlers identify similar pages.

[0004] However, in the related art, because the attributives of long-tail keywords usually include multiple parts of speech and have complex lexical structures, the error rate of the part-of-speech analysis algorithm is relatively high when processing long-tail keywords. The part-of-speech recognition of the target keyword may be located at the noun in the attributive, resulting in recognition errors and recognition biases, which affects the recognition accuracy of the target keyword, making it impossible for the crawler to accurately identify similar pages and escape from the corresponding spider pool, resulting in a waste of network search resources and affecting the user's search experience. This problem needs to be solved urgently. Summary of the invention

[0005] The present invention provides a method for fighting malicious spider pools based on a large model, so as to solve the problems in the related art that, because the attributives of long-tail keywords usually include multiple parts of speech and have a complex vocabulary structure, the error rate of the part-of-speech analysis algorithm is high when processing long-tail keywords, and the part-of-speech recognition of the target keyword may be located at the noun in the attributive, thereby causing recognition errors and recognition deviations, affecting the recognition accuracy of the target keyword, making it impossible for the crawler to accurately identify similar pages and escape from the corresponding spider pool, resulting in a waste of network search resources, affecting the user's search experience, and the like.

[0006] The first aspect of the present invention provides a method for combating a malicious spider pool based on a large model, comprising the following steps: based on the current web page returned by the crawler in the current working cycle, cutting the plain text content of the current web page data packet to obtain all long-tail keywords of the current web page data packet; based on the pre-trained semantic large model, identifying the target keywords corresponding to each long-tail keyword respectively to output all target keywords of the current web page data packet; confirming the current page feature vector of the current web page data packet according to all the target keywords, and detecting whether the current web page meets a preset similarity condition based on the current page feature vector; when it is detected that the current web page meets the preset similarity condition, determining that the crawler is in a malicious spider pool, and stopping the crawler from crawling the current web page.

[0007] Optionally, in one embodiment of the present invention, before identifying the target keyword corresponding to each long-tail keyword based on the pre-trained semantic big model, it also includes: obtaining a target training data set of the target semantic big model, grouping each long-tail keyword in the target training data set into adjacent positions to obtain all vocabulary groups; iteratively training the target semantic big model based on all vocabulary groups and the labels corresponding to the target keywords in each long-tail keyword until the target semantic big model meets the iteration stop condition, and outputting the pre-trained semantic big model.

[0008] Optionally, in one embodiment of the present invention, the obtaining of the target training data set of the target semantic big model includes: generating a bad product long tail keyword library based on a bad product target keyword library and a bad product attributive word library, confirming the bad product label of each bad product long tail keyword according to the position of each bad product target keyword in the bad product long tail keyword library, and constructing a bad product data set based on all bad product long tail keywords and all bad product labels; crawling a user attributive word library based on a user target keyword library, generating a user long tail keyword library according to the user target keyword library and the user attributive word library, confirming the user label of each user long tail keyword according to the position of each user target keyword in the user long tail keyword library, and constructing a user data set based on all user long tail keywords and all user labels; obtaining a black hat long tail keyword library based on crawled content of at least one malicious spider pool, confirming the position of each black hat target keyword in the black hat long tail keyword library using a preset big model, obtaining the black hat label of each black hat long tail keyword, and constructing a black hat data set based on all black hat long tail keywords and all black hat labels; and obtaining the target training data set from the bad product data set, the user data set, and the black hat data set.

[0009] Optionally, in one embodiment of the present invention, before outputting the pre-trained semantic big model, it also includes: generating a target test set based on the black hat long-tail keyword library, and using the target test set to determine whether the output accuracy of the trained target semantic big model is greater than or equal to a preset accuracy threshold; if the output accuracy is greater than or equal to the preset accuracy threshold, outputting the pre-trained semantic big model according to the trained target semantic big model, otherwise, adjusting the proportion of the black hat data set in the target training data set, and retraining the target semantic big model using the adjusted target training data set until the pre-trained semantic big model is output.

[0010] Optionally, in one embodiment of the present invention, detecting whether the current web page satisfies a preset similarity condition based on the current page feature vector includes: obtaining all historical page feature vectors of the historical work cycle corresponding to the current work cycle based on a preset backtracking range; respectively calculating the page similarity of each historical page feature vector and the current page feature vector, and confirming the number of counts for which the page similarity is greater than or equal to a preset similarity threshold; and when the number of counts reaches a preset count threshold, determining that the current web page satisfies the preset similarity condition.

[0011] The second aspect of the present invention provides an apparatus for combating a malicious spider pool based on a large model, comprising: a cutting module, for cutting the plain text content of a current web page data packet based on a current web page returned by a crawler in a current working cycle, to obtain all long-tail keywords of the current web page data packet; an identification module, for identifying the target keywords corresponding to each long-tail keyword based on a pre-trained semantic large model, to output all target keywords of the current web page data packet; a detection module, for confirming a current page feature vector of the current web page data packet based on all the target keywords, and detecting whether the current web page meets a preset similarity condition based on the current page feature vector; a determination module, for determining that the crawler is in a malicious spider pool and stopping the crawler from crawling the current web page when it is detected that the current web page meets the preset similarity condition.

[0012] Optionally, in one embodiment of the present invention, the device also includes: an acquisition module, which is used to acquire a target training data set of a target semantic big model before identifying the target keywords corresponding to each long-tail keyword based on a pre-trained semantic big model, and group each long-tail keyword in the target training data set into adjacent positions to obtain all vocabulary groups; an iteration module, which is used to iteratively train the target semantic big model based on all vocabulary groups and the labels corresponding to the target keywords in each long-tail keyword until the target semantic big model meets the iteration stop condition, and output the pre-trained semantic big model.

[0013] Optionally, in one embodiment of the present invention, the acquisition module includes: a first construction unit, which is used to generate a bad product long-tail keyword library based on the bad product target keyword library and the bad product attributive word library, and confirm the bad product label of each bad product long-tail keyword according to the position of each bad product target keyword in the bad product long-tail keyword library, and construct a bad product data set based on all bad product long-tail keywords and all bad product labels; a second construction unit, which is used to crawl a user attributive word library based on the user target keyword library, generate a user long-tail keyword library according to the user target keyword library and the user attributive word library, and construct a bad product data set based on each user in the user long-tail keyword library. The method comprises the following steps: a first construction unit, a second construction unit, and a third construction unit, for obtaining a black hat long-tail keyword library based on the crawled content of at least one malicious spider pool, and using a preset large model to confirm the position of each black hat target keyword in the black hat long-tail keyword library, obtain the black hat tag of each black hat long-tail keyword, and construct a black hat dataset based on all black hat long-tail keywords and all black hat tags; a confirmation unit, for obtaining the target training dataset from the malicious dataset, the user dataset and the black hat dataset.

[0014] Optionally, in one embodiment of the present invention, the iteration module also includes: a generation unit, used to generate a target test set according to the black hat long-tail keyword library before outputting the pre-trained semantic big model, and use the target test set to determine whether the output accuracy of the trained target semantic big model is greater than or equal to a preset accuracy threshold; an adjustment unit, used to output the pre-trained semantic big model according to the trained target semantic big model when the output accuracy is greater than or equal to the preset accuracy threshold, otherwise, adjust the proportion of the black hat data set in the target training data set, and retrain the target semantic big model using the adjusted target training data set until the pre-trained semantic big model is output.

[0015] Optionally, in one embodiment of the present invention, the detection module includes: an acquisition unit, used to acquire all historical page feature vectors of the historical work cycle corresponding to the current work cycle based on a preset backtracking range; a calculation unit, used to respectively calculate the page similarity of each historical page feature vector and the current page feature vector, and confirm the number of counts of the page similarity being greater than or equal to a preset similarity threshold; a determination unit, used to determine that the current web page satisfies a preset similarity condition when the number of counts reaches a preset count threshold.

[0016] A third aspect of the present invention provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method for combating a malicious spider pool based on a large model as described in the above embodiment.

[0017] A fourth aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, which, when executed by a processor, implements the above method for combating a malicious spider pool based on a large model.

[0018] A fifth aspect of the present invention provides a computer program, which, when executed, implements the above method for combating malicious spider pools based on a large model.

[0019] The embodiment of the present invention can use the pre-trained large model to extract the corresponding target keywords of all long-tail keywords of the current page, and construct a web page feature vector to detect the similarity between the keywords of the current web page and the previous web page, so as to accurately identify the long-tail keywords, judge the similarity of unknown web pages, and quickly diagnose the crawler trapped in the spider pool, thereby ensuring the efficiency and security of network search and being more practical. Thus, the problem in the related art that the attributive of the long-tail keyword usually includes multiple parts of speech and has a complex vocabulary structure, resulting in a high error rate when the part-of-speech analysis algorithm processes the long-tail keyword, and the part-of-speech recognition of the target keyword may be located to the noun in the attributive, thereby generating recognition errors and recognition deviations, affecting the recognition accuracy of the target keyword, making it impossible for the crawler to accurately identify similar pages and escape the corresponding spider pool, resulting in a waste of network search resources and affecting the user's search experience.

[0020] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0022] Figure 1 A flowchart of a method for combating a malicious spider pool based on a large model according to an embodiment of the present invention;

[0023] Figure 2 A schematic diagram of training a semantic large model according to an embodiment of the present invention;

[0024] Figure 3 A schematic diagram of generating a malicious product data set according to an embodiment of the present invention;

[0025] Figure 4 A schematic diagram of a long-tail keyword structure according to an embodiment of the present invention;

[0026] Figure 5 A schematic diagram of generating a user data set for an embodiment of the present invention;

[0027] Figure 6 A schematic diagram of generating a black hat data set according to an embodiment of the present invention;

[0028] Figure 7 A schematic diagram of the application of a semantic big model according to an embodiment of the present invention;

[0029] Figure 8 This is a schematic diagram of an application scenario of a semantic big model in a normal working condition according to an embodiment of the present invention;

[0030] Fig. 9 This is a schematic diagram of an application scenario of a semantic big model in an embodiment of the present invention when entering a spider pool;

[0031] Fig.10 A schematic diagram of the operation logic of fighting against malicious spider pools based on a large model according to an embodiment of the present invention;

[0032] Fig.11 A schematic diagram of the structure of a device for combating a malicious spider pool based on a large model according to an embodiment of the present invention;

[0033] Fig.12 FIG. 4 is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0034] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0035] The following describes a method for combating malicious spider pools based on a large model according to an embodiment of the present invention with reference to the accompanying drawings. In view of the related art mentioned in the above background technology, since the attributive of long-tail keywords usually includes multiple parts of speech and has a complex vocabulary structure, the error rate of the part-of-speech analysis algorithm when processing long-tail keywords is high, and the part-of-speech recognition of the target keyword may be located at the noun in the attributive, thereby generating recognition errors and recognition deviations, affecting the recognition accuracy of the target keyword, making it impossible for the crawler to accurately identify similar pages and escape from the corresponding spider pool, resulting in a waste of network search resources and affecting the user's search experience. The present invention provides a method for combating malicious spider pools based on a large model, in which the pre-trained large model can be used to extract the corresponding target keywords of all long-tail keywords of the current page, and a web page feature vector can be constructed to detect the similarity of keywords between the current web page and the previous web page, so as to accurately identify long-tail keywords, judge the similarity of unknown web pages, and quickly diagnose crawlers trapped in the spider pool, thereby ensuring the efficiency and security of network search, and being more practical. Thereby, a problem in the related art is solved, that is, since the attributives of long-tail keywords usually include multiple parts of speech and have a complex lexical structure, the error rate of the part-of-speech analysis algorithm is high when processing long-tail keywords, and the part-of-speech recognition of the target keyword may be located at the noun in the attributive, thereby causing recognition errors and recognition biases, affecting the recognition accuracy of the target keyword, making it impossible for the crawler to accurately identify similar pages and escape from the corresponding spider pool, resulting in a waste of network search resources and affecting the user's search experience.

[0036] Specifically, Figure 1 A flowchart of a method for combating malicious spider pools based on a large model provided by an embodiment of the present invention.

[0037] like Figure 1 As shown, the method for combating malicious spider pools based on a large model includes the following steps:

[0038] In step S101, based on the current web page returned by the crawler in the current working cycle, the plain text content of the current web page data packet is cut to obtain all long-tail keywords of the current web page data packet.

[0039] It is understandable that in the embodiment of the present invention, the crawler can periodically return a web page data packet to obtain the current web page data packet corresponding to the current web page in the current working cycle, and temporarily store it in the data range server. Then, the server side cleans the page data returned by the crawler, and filters through the text tags in the page, such as the title h tag, the text p tag, etc., to obtain the plain text, and forms multiple long-tail keywords through basic cutting, and generates an array containing all the long-tail keywords of the current page for the page returned in each working cycle, wherein the long-tail keyword can be a keyword with multiple attributives, and the attributive can be composed of multiple verbs, nouns, adjectives and other phrases, which are often used in low-cost drainage web pages of black hat SEO business to limit the crawler to the spider pool.

[0040] In step S102, based on the pre-trained semantic big model, the target keyword corresponding to each long-tail keyword is identified respectively to output all the target keywords of the current web page data packet.

[0041] It can be understood that in an embodiment of the present invention, the semantic recognition capability of a pre-trained semantic large model can be utilized to specifically identify target keywords in long-tail keywords, and return the target keywords corresponding to each long-tail keyword in the current page to form a keyword array.

[0042] Optionally, in one embodiment of the present invention, before identifying the target keyword corresponding to each long-tail keyword based on the pre-trained semantic big model, it also includes: obtaining a target training data set of the target semantic big model, grouping each long-tail keyword in the target training data set by adjacent positions, and obtaining all vocabulary groups; iteratively training the target semantic big model based on all vocabulary groups and the labels corresponding to the target keywords in each long-tail keyword until the target semantic big model meets the iteration stop condition, and outputting the pre-trained semantic big model.

[0043] In the actual execution process, the vector of each long-tail keyword can be generated according to each long-tail keyword and its corresponding label in the target training data set, where the position of the attribute is 0 and the position of the keyword is 1. Then, for each long-tail keyword and vector obtained, the vectors and attributes of every two adjacent positions are grouped together, and the model is trained through the labels in the vector to determine the position of the keyword (i.e., 1). After multiple rounds of iterative optimization, the model is output after the number of iterations is reached.

[0044] Specifically, Figure 2As shown, it is a training diagram of a semantic big model of an embodiment of the present invention. According to the long-tail keywords and their labels, the vector of each long-tail keyword is obtained, and the words in all adjacent positions of a single long-tail keyword are grouped into groups of two for the training of the target semantic big model. Among them, the crawler data in actual production is analyzed by using a specially trained semantic big model instead of using an existing language big model for analysis, because compared with local operation, calling a big model on the market has higher network requirements, larger delays, and low efficiency, and calling an existing big model often requires the purchase of corresponding ports, and the frequency of calling is high during the application process, resulting in an increase in economic costs.

[0045] Optionally, in one embodiment of the present invention, obtaining a target training data set of a target semantic big model includes: generating a bad product long tail keyword library based on a bad product target keyword library and a bad product attributive word library, confirming the bad product label of each bad product long tail keyword according to the position of each bad product target keyword in the bad product long tail keyword library, and constructing a bad product data set based on all bad product long tail keywords and all bad product labels; crawling a user attributive word library based on a user target keyword library, generating a user long tail keyword library according to the user target keyword library and the user attributive word library, confirming the user label of each user long tail keyword according to the position of each user target keyword in the user long tail keyword library, and constructing a user data set based on all user long tail keywords and all user labels; obtaining a black hat long tail keyword library based on crawled content of at least one malicious spider pool, confirming the position of each black hat target keyword in the black hat long tail keyword library using a preset big model, obtaining a black hat label of each black hat long tail keyword, and constructing a black hat data set based on all black hat long tail keywords and all black hat labels; and obtaining a target training data set from the bad product data set, the user data set, and the black hat data set.

[0046] It should be noted that the preset large model can be set by those skilled in the art according to actual conditions and is not specifically limited here.

[0047] In the actual implementation process, Figure 3 As shown, it is a schematic diagram of generating a malicious product data set according to an embodiment of the present invention. The malicious product target keyword library may be a commonly used keyword library of malicious web page industries engaged in illegal activities. Since the use of spider pools is usually highly purposeful, the range of malicious product target keywords is relatively small. Therefore, a malicious product target keyword library may be constructed for commonly used malicious product keywords. The malicious product attributive word library may be a commonly used attributive word library provided by a third-party supplier (such as a search engine manufacturer, etc.).

[0048] Then, the words in the bad product target keyword library and the bad product attributive word library are randomly arranged and combined to obtain several groups of long-tail keywords, such as Figure 4As shown, it is a schematic diagram of the long-tail keyword structure of an embodiment of the present invention. Each long-tail keyword is composed of several attributes and a single target keyword, and it is ensured that the position of the target keyword in the entire long-tail keyword is not unique. The corresponding target keyword is used as the label of the long-tail keyword to obtain a malicious data set.

[0049] For user datasets, such as Figure 5 As shown in FIG. 1 , a schematic diagram of generating a user data set according to an embodiment of the present invention is shown. The user target keyword library can be constructed by external means such as questionnaire surveys according to the search engine usage of different users (for example, search keywords with a high frequency of popping up irrelevant pages). A crawler is performed based on the obtained user target keywords to extract attributives related to the words in the user keyword library to obtain a user attributive word library, and then based on the user target keyword library and the user attributive word library, the following is obtained: Figure 4 The user long-tail keywords and the user labels corresponding to each long-tail keyword are shown to construct a user data set.

[0050] For black hat datasets, such as Figure 6 As shown, a schematic diagram of generating a black hat data set according to an embodiment of the present invention can be used to crawl the spider pool content to obtain a black hat long-tail keyword library, and call an existing semantic big model as a preset big model to analyze black hat long-tail keywords, determine their target keywords, and obtain corresponding black hat labels, thereby constructing a black hat data set.

[0051] Optionally, in one embodiment of the present invention, before outputting the pre-trained semantic big model, it also includes: generating a target test set based on the black hat long-tail keyword library, and using the target test set to determine whether the output accuracy of the trained target semantic big model is greater than or equal to a preset accuracy threshold; if the output accuracy is greater than or equal to the preset accuracy threshold, outputting the pre-trained semantic big model according to the trained target semantic big model, otherwise, adjusting the proportion of the black hat data set in the target training data set, and retraining the target semantic big model using the adjusted target training data set until the pre-trained semantic big model is output.

[0052] It should be noted that the preset accuracy threshold can be set by those skilled in the art according to actual conditions and is not specifically limited here.

[0053] In the actual execution process, the target semantic big model after iterative training is tested based on the black hat long tail keywords and corresponding black hat labels in the black hat long tail keyword library. If the output accuracy is higher than the preset accuracy threshold of 95%, the model training is considered successful and the model is stored in the server. Otherwise, the proportion of the black hat data set in the target training data set is adjusted, and the target semantic big model is retrained until the accuracy meets the requirements, and the pre-trained semantic big model is output.

[0054] In step S103, the current page feature vector of the current web page data packet is confirmed according to all target keywords, and whether the current web page meets the preset similarity condition is detected based on the current page feature vector.

[0055] It should be noted that the preset similarity conditions can be set by those skilled in the art according to actual conditions and are not specifically limited here.

[0056] It is understandable that in the embodiment of the present invention, the feature vector of the current web page data packet is confirmed based on all target keywords to confirm whether the previous web page meets the preset similarity condition, that is, whether the crawler falls into the spider pool.

[0057] Optionally, in one embodiment of the present invention, detecting whether the current web page meets a preset similarity condition based on the current page feature vector includes: based on a preset backtracking range, obtaining all historical page feature vectors of the historical work cycle corresponding to the current work cycle; respectively calculating the page similarity of each historical page feature vector and the current page feature vector, and confirming the number of counts for which the page similarity is greater than or equal to a preset similarity threshold; when the number of counts reaches the preset count threshold, determining that the current web page meets the preset similarity condition.

[0058] It should be noted that the preset backtracking range, the preset similarity threshold and the preset counting threshold can be set by those skilled in the art according to actual conditions and are not specifically limited here.

[0059] In actual execution, the preset backtracking range may be a predetermined historical page coverage range for comparing the similarity with the current page, for example Figure 7 As shown, it is a schematic diagram of the application of the semantic big model of an embodiment of the present invention. The feature vector of the current page can be compared with the feature vectors of the 10 historical pages returned by the crawler before, the number of repeated target keywords can be calculated, and the page similarity γ can be calculated according to the formula:

[0060]

[0061] Among them, γ is the page similarity, For the current page, is the page to compare with the current page, The number of target keywords repeated on the two pages. is the total number of keywords on the current page, The total number of keywords in the comparison page. The independent counter of the current page determines the number of times that the page similarity between the current page and the historical page is greater than or equal to the preset similarity threshold. When the number reaches the preset counting threshold of 10 times, it is determined that the current web page meets the preset similarity condition.

[0062] In step S104, when it is detected that the current webpage meets the preset similarity condition, it is determined that the crawler is in a malicious spider pool, and the crawler is stopped from crawling the current webpage.

[0063] It is understandable that in the embodiments of the present invention, when it is detected that the current webpage meets the preset similarity condition, the server can issue an instruction to the current crawler to stop crawling after completing the crawling of the current page, jump out of the spider pool or abandon the crawler, and continue crawling from other directions. Figure 8 As shown, a schematic diagram of an application scenario of a semantic big model of an embodiment of the present invention under normal working conditions is shown in FIG. Fig. 9 The figure is a schematic diagram of an application scenario of a semantic big model according to an embodiment of the present invention when entering a spider pool.

[0064] like Fig.10 As shown, the working content of the embodiment of the present invention is described in detail below with a specific embodiment.

[0065] Among them, in the model training stage, the commonly used target keywords of malicious products and the attributive library provided by the third party can be obtained to construct the malicious long-tail keywords of malicious products through permutations and combinations, and the malicious keyword labels of the malicious long-tail keywords are confirmed according to the commonly used target keywords of malicious products to obtain the malicious product data set; the commonly used target keywords of users are investigated, and the attributive library of the commonly used target keywords of users is extracted by crawlers, and the user long-tail keywords are constructed through permutations and combinations, and the user keyword labels of the user long-tail keywords are confirmed according to the commonly used target keywords of users to obtain the user data set; the black hat long-tail keywords in the spider pool are crawled, the target keywords are manually marked to confirm the black hat keyword labels of the black hat long-tail keywords to obtain the black hat data set, and the data set used for model training is confirmed by the malicious data set, the user data set and the black hat data set, and the language model is trained, and then the test set is obtained according to the black hat data set to verify whether the accuracy of the trained language model is higher than 95%. If it is higher, the training is completed, otherwise the proportion of the black hat data set is adjusted for retraining.

[0066] In the application process stage, the text data in the website is obtained by screening the periodic return data of the crawler, and the text data is processed to extract multiple corresponding long-tail keywords to form an array containing all the long-tail keywords of the current page. The pre-trained semantic recognition model is used to analyze each long-tail keyword in the array, and the target keyword corresponding to each long-tail keyword is returned to form a keyword array. The web page feature vector is constructed using the array to calculate the keyword similarity between the current web page and the previous web pages. If the similarity is higher than the preset ratio of 60% for 10 times, the crawler should be abandoned and crawled along other web pages, thereby achieving accurate judgment of the similarity of unknown web pages and avoiding the waste of search resources caused by the crawler falling into the spider pool.

[0067] According to the method for fighting malicious spider pools based on a large model proposed in an embodiment of the present invention, the pre-trained large model can be used to extract the corresponding target keywords of all long-tail keywords of the current page, and a web page feature vector can be constructed to detect the similarity of keywords between the current web page and the previous web page, thereby accurately identifying long-tail keywords to judge the similarity of unknown web pages, and quickly diagnose crawlers trapped in the spider pool, thereby ensuring the efficiency and security of network search and being more practical. Thus, the problem in the related art that the attributive of long-tail keywords usually includes multiple parts of speech and has a complex vocabulary structure, resulting in a high error rate when the part-of-speech analysis algorithm processes long-tail keywords, and the part-of-speech recognition of the target keyword may be located at the noun in the attributive, thereby generating recognition errors and recognition deviations, affecting the recognition accuracy of the target keyword, making it impossible for the crawler to accurately identify similar pages and escape from the corresponding spider pool, resulting in a waste of network search resources and affecting the user's search experience.

[0068] Next, a device for combating a malicious spider pool based on a large model according to an embodiment of the present invention is described with reference to the accompanying drawings.

[0069] Fig.11 4 is a schematic diagram of the structure of a device for combating malicious spider pools based on a large model according to an embodiment of the present invention.

[0070] like Fig.11 As shown, the device 10 for combating malicious spider pools based on a large model includes: a cutting module 100 , a recognition module 200 , a detection module 300 and a determination module 400 .

[0071] The cutting module 100 is used to cut the plain text content of the current web page data packet based on the current web page returned by the crawler in the current working cycle to obtain all long-tail keywords of the current web page data packet.

[0072] The identification module 200 is used to identify the target keywords corresponding to each long-tail keyword based on the pre-trained semantic large model, so as to output all the target keywords of the current web page data packet.

[0073] The detection module 300 is used to confirm the current page feature vector of the current web page data packet according to all target keywords, and detect whether the current web page meets the preset similarity condition based on the current page feature vector.

[0074] The determination module 400 is used to determine that the crawler is in a malicious spider pool and stop the crawler from crawling the current web page when it is detected that the current web page meets a preset similarity condition.

[0075] Optionally, in one embodiment of the present invention, the device 10 further includes: an acquisition module and an iteration module.

[0076] Among them, the acquisition module is used to obtain the target training data set of the target semantic big model before identifying the target keyword corresponding to each long-tail keyword based on the pre-trained semantic big model, and group each long-tail keyword in the target training data set into adjacent positions to obtain all vocabulary groups.

[0077] The iteration module is used to iteratively train the target semantic big model based on all vocabulary groups and the labels corresponding to the target keywords in each long-tail keyword until the target semantic big model meets the iteration stopping condition, and output the pre-trained semantic big model.

[0078] Optionally, in one embodiment of the present invention, the acquisition module includes: a first construction unit, a second construction unit, a third construction unit and a confirmation unit.

[0079] Among them, the first construction unit is used to generate a bad product long-tail keyword library based on the bad product target keyword library and the bad product attributive vocabulary library, and confirm the bad product label of each bad product long tail keyword according to the position of each bad product target keyword in the bad product long tail keyword library, and construct a bad product data set based on all bad product long tail keywords and all bad product labels.

[0080] The second construction unit is used to crawl the user attributive vocabulary based on the user target keyword library, generate a user long-tail keyword library based on the user target keyword library and the user attributive vocabulary library, and confirm the user tag of each user long-tail keyword according to the position of each user target keyword in the user long-tail keyword library, and construct a user data set based on all user long-tail keywords and all user tags.

[0081] The third construction unit is used to obtain a black hat long-tail keyword library based on the crawled content of at least one malicious spider pool, and use a preset large model to confirm the position of each black hat target keyword in the black hat long-tail keyword library, obtain the black hat label of each black hat long-tail keyword, and construct a black hat data set based on all black hat long-tail keywords and all black hat labels.

[0082] The confirmation unit is used to obtain a target training data set from a malicious data set, a user data set and a black hat data set.

[0083] Optionally, in one embodiment of the present invention, the iteration module further includes: a generation unit and an adjustment unit.

[0084] Among them, the generation unit is used to generate a target test set according to the black hat long-tail keyword library before outputting the pre-trained semantic big model, and use the target test set to determine whether the output accuracy of the trained target semantic big model is greater than or equal to the preset accuracy threshold.

[0085] The adjustment unit is used to output the pre-trained semantic big model according to the trained target semantic big model when the output accuracy is greater than or equal to the preset accuracy threshold; otherwise, adjust the proportion of the black hat data set in the target training data set, and retrain the target semantic big model with the adjusted target training data set until the pre-trained semantic big model is output.

[0086] Optionally, in one embodiment of the present invention, the detection module 300 includes: an acquisition unit, a calculation unit and a determination unit.

[0087] The acquisition unit is used to acquire all historical page feature vectors of the historical working cycle corresponding to the current working cycle based on a preset backtracking range.

[0088] The calculation unit is used to calculate the page similarity of each historical page feature vector and the current page feature vector respectively, and confirm the count number of times that the page similarity is greater than or equal to a preset similarity threshold.

[0089] The determination unit is used to determine that the current web page meets a preset similarity condition when the counted times reaches a preset count threshold.

[0090] It should be noted that the aforementioned explanation of the embodiment of the method for combating a malicious spider pool based on a large model is also applicable to the device for combating a malicious spider pool based on a large model in this embodiment, and will not be repeated here.

[0091] According to the device for fighting malicious spider pools based on a large model proposed in an embodiment of the present invention, the pre-trained large model can be used to extract the corresponding target keywords of all long-tail keywords of the current page, and a webpage feature vector can be constructed to detect the similarity of keywords between the current webpage and the previous webpage, thereby accurately identifying long-tail keywords to judge the similarity of unknown webpages, and quickly diagnose crawlers trapped in the spider pool, thereby ensuring the efficiency and security of network search and being more practical. Thus, the problem in the related art that the attributive of long-tail keywords usually includes multiple parts of speech and has a complex vocabulary structure, resulting in a high error rate when the part-of-speech analysis algorithm processes long-tail keywords, and the part-of-speech recognition of the target keyword may be located at the noun in the attributive, thereby generating recognition errors and recognition deviations, affecting the recognition accuracy of the target keyword, making it impossible for the crawler to accurately identify similar pages and escape from the corresponding spider pool, resulting in a waste of network search resources and affecting the user's search experience.

[0092] Fig.12 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. The electronic device may include:

[0093] A memory 1201 , a processor 1202 , and a computer program stored in the memory 1201 and executable on the processor 1202 .

[0094] When the processor 1202 executes the program, the method for combating a malicious spider pool based on a large model provided in the above embodiment is implemented.

[0095] Furthermore, the electronic device further comprises:

[0096] The communication interface 1203 is used for communication between the memory 1201 and the processor 1202 .

[0097] The memory 1201 is used to store computer programs that can be executed on the processor 1202 .

[0098] The memory 1201 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0099] If the memory 1201, the processor 1202 and the communication interface 1203 are implemented independently, the communication interface 1203, the memory 1201 and the processor 1202 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.12 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0100] Optionally, in a specific implementation, if the memory 1201, the processor 1202 and the communication interface 1203 are integrated on a chip, the memory 1201, the processor 1202 and the communication interface 1203 can communicate with each other through an internal interface.

[0101] The processor 1202 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention.

[0102] This embodiment also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above method for combating a malicious spider pool based on a large model.

[0103] This embodiment also provides a computer program, which, when executed, implements the above method for combating a malicious spider pool based on a large model.

[0104] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0105] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "N" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0106] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present invention belong.

[0107] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection with one or N wirings (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways as necessary and then storing it in a computer memory.

[0108] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0109] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0110] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0111] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present invention. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A method for combating malicious spider pools based on a large model, characterized in that: The following steps are involved: Based on the current web page returned by the crawler in the current working cycle, the plain text content of the current web page data packet is cut to obtain all long-tail keywords of the current web page data packet; Based on the pre-trained semantic big model, the target keywords corresponding to each long-tail keyword are identified respectively to output all the target keywords of the current web page data packet; Confirming a current page feature vector of the current web page data packet according to all the target keywords, and detecting whether the current web page meets a preset similarity condition based on the current page feature vector; When it is detected that the current webpage satisfies the preset similarity condition, it is determined that the crawler is in a malicious spider pool, and the crawler is stopped from crawling the current webpage; Before identifying the target keywords corresponding to each long-tail keyword based on the pre-trained semantic model, the following steps are also included: Obtain a target training data set for a target semantic large model, and group each long-tail keyword in the target training data set by adjacent positions to obtain all vocabulary groups; Based on the labels corresponding to the target keywords in all the vocabulary groups and each long-tail keyword, the target semantic big model is iteratively trained until the target semantic big model meets the iteration stopping condition, and the pre-trained semantic big model is output.

2. The method for combating malicious spider pools based on a large model according to claim 1, characterized in that: The step of obtaining a target training data set for a target semantic large model includes: Based on the bad product target keyword library and the bad product attributive word library, a bad product long tail keyword library is generated, and the bad product label of each bad product long tail keyword is confirmed according to the position of each bad product target keyword in the bad product long tail keyword library, and a bad product data set is constructed based on all bad product long tail keywords and all bad product labels; Crawling a user attributive word library based on a user target keyword library, generating a user long-tail keyword library based on the user target keyword library and the user attributive word library, and confirming a user tag of each user long-tail keyword based on the position of each user target keyword in the user long-tail keyword library, and constructing a user data set based on all user long-tail keywords and all user tags; A black hat long tail keyword library is obtained based on the crawled content of at least one malicious spider pool, and a preset large model is used to confirm the position of each black hat target keyword in the black hat long tail keyword library, and a black hat label of each black hat long tail keyword is obtained, and a black hat data set is constructed based on all black hat long tail keywords and all black hat labels; The target training data set is obtained from the malicious data set, the user data set and the black hat data set.

3. The method for combating malicious spider pools based on a large model according to claim 2, characterized in that: Before outputting the pre-trained semantic model, the method further includes: Generate a target test set according to the black hat long-tail keyword library, and use the target test set to determine whether the output accuracy of the trained target semantic large model is greater than or equal to a preset accuracy threshold; If the output accuracy is greater than or equal to the preset accuracy threshold, the pre-trained semantic big model is output according to the trained target semantic big model; otherwise, the proportion of the black hat data set in the target training data set is adjusted, and the target semantic big model is retrained using the adjusted target training data set until the pre-trained semantic big model is output.

4. The method for combating malicious spider pools based on a large model according to claim 1, characterized in that: The detecting whether the current webpage satisfies a preset similarity condition based on the current page feature vector includes: Based on a preset backtracking range, all historical page feature vectors of the historical working cycle corresponding to the current working cycle are obtained; Calculating the page similarity between each historical page feature vector and the current page feature vector respectively, and confirming the number of times the page similarity is greater than or equal to a preset similarity threshold; When the counted number reaches a preset count threshold, it is determined that the current webpage meets a preset similarity condition.

5. A device for combating malicious spider pools based on a large model, characterized in that: include: A cutting module, for cutting the plain text content of the current web page data packet based on the current web page returned by the crawler in the current working cycle, to obtain all long-tail keywords of the current web page data packet; An identification module, for identifying the target keywords corresponding to each long-tail keyword based on a pre-trained semantic large model, so as to output all the target keywords of the current web page data packet; A detection module, configured to confirm a current page feature vector of the current web page data packet according to all the target keywords, and detect whether the current web page meets a preset similarity condition based on the current page feature vector; A determination module, configured to determine that the crawler is in a malicious spider pool and stop the crawler from crawling the current web page when it is detected that the current web page meets the preset similarity condition; An acquisition module is used to acquire a target training data set of a target semantic big model before identifying the target keyword corresponding to each long-tail keyword based on the pre-trained semantic big model, and group each long-tail keyword in the target training data set by adjacent positions to obtain all vocabulary groups; An iteration module is used to iteratively train the target semantic big model based on the labels corresponding to the target keywords in all the vocabulary groups and each long-tail keyword until the target semantic big model meets the iteration stopping condition, and output the pre-trained semantic big model.

6. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method for combating a malicious spider pool based on a large model as described in any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the method for combating a malicious spider pool based on a large model as described in any one of claims 1 to 4.

8. A computer program product, comprising a computer program, characterized in that The computer program is executed to implement the method for combating malicious spider pools based on a large model as described in any one of claims 1 to 4.

Images