A web crawler protection method based on device fingerprint and dynamic protection

Abstract

The present invention relates to the field of crawler protection, and in particular to a Web crawler protection method based on device fingerprint and dynamic protection, comprising: determining a characteristic state of a target website based on a page link distribution value and a related page difference value of the target website; determining, based on the characteristic state of the target website, a crawler analysis protection mode as performing analysis protection on a user operation fingerprint or performing dynamic protection on the target website; in the analysis protection on the user operation fingerprint, determining whether to perform a crawler protection warning based on an information-related reference value and a search depth reference value or based on a mouse track anomaly reference value and a mouse sliding anomaly reference value according to a comparison result of a link information difference degree with a preset link information difference degree; in the dynamic protection on the target website, determining a dynamic protection requirement state based on the number of sensitive access pages and resource utilization, and determining a dynamic protection strategy based on the dynamic protection requirement state; the present invention improves the accuracy of crawler protection.

Description

Technical Field

[0001] The present invention relates to the field of crawler protection, and in particular to a Web crawler protection method based on device fingerprint and dynamic protection. Background Art

[0002] A web crawler is an automated program or script that typically crawls links from a web page and recursively accesses them until a certain condition is met. To combat malicious crawler attacks, many websites implement a series of crawler protection measures, such as device fingerprint monitoring, verification codes, and access frequency restrictions. These measures aim to protect the security and stability of websites and prevent malicious crawlers from excessively crawling and damaging them. However, current crawler protection measures are relatively poor. For example, current device fingerprint monitoring can determine whether there is a crawler risk based on the user's mouse trajectory and operating patterns. However, it is impossible to tailor device fingerprint monitoring strategies based on the actual web page, resulting in poor protection.

[0003] Patent publication number CN109189660A discloses a crawler identification method based on user mouse interaction behavior, comprising the following steps: step (1), sampling user behavior data; step (2), sparsely encrypting the sampled data in step (1); step (3), performing trajectory analysis; step (4), after discovering that the user is a malicious user, performing a ban process; wherein, in the trajectory analysis step, a multi-dimensional analysis is performed, including whether the user mouse movement trajectory is smooth and continuous, and whether the page scrolling distance per unit time is reasonable, a threshold and a weight value are set for each dimension, and when the sampling information of a single dimension exceeds the threshold, the user is determined to be a malicious user, or when the result value after weight calculation of multiple dimensions is lower than a certain threshold, the user is determined to be a malicious user; thus, the comparative document has the following problems: it does not take into account the different link distribution states corresponding to different pages, for example, pages with different link distribution densities have different degrees of smoothness and continuity of the actual user mouse movement trajectory and the page scrolling distance per unit time, which is prone to misjudgment or wrong judgment, and cannot make active crawler protection. Summary of the Invention

[0004] To this end, the present invention provides a web crawler protection method based on device fingerprint and dynamic protection, so as to overcome the problem in the prior art that different targeted crawler protection strategies cannot be selected according to the actual page situation, resulting in poor crawler protection efficiency.

[0005] To achieve the above objectives, the present invention provides a web crawler protection method based on device fingerprint and dynamic protection, comprising:

[0006] Determine the characteristic status of the target website based on the page link distribution value and the related page difference value of the target website;

[0007] Determine the crawler analysis protection method based on the characteristic status of the target website, which is to analyze and protect the user operation fingerprint or dynamically protect the target website;

[0008] In the analysis and protection of user operation fingerprints, the comparison result of the link information difference with the preset link information difference is used to determine whether to issue a crawler protection warning based on the information-related reference value and the search depth reference value or the mouse track abnormality reference value and the mouse sliding abnormality reference value;

[0009] In dynamic protection of the target website, the dynamic protection demand status is determined according to the number of sensitive access pages and resource utilization, and the dynamic protection strategy is determined according to the dynamic protection demand status.

[0010] Furthermore, when the characteristic state of the target website is that the page link distribution value is within the first preset distribution value range or the related page difference value is within the first preset difference value range, analysis and protection are performed on the user operation fingerprint, including:

[0011] Detect the difference in link information;

[0012] If the link information difference is greater than the preset link information difference, determine whether to issue a crawler protection warning based on the information relevance reference value and the search depth reference value;

[0013] If the link information difference is less than or equal to the preset link information difference, whether to issue a crawler protection warning is determined based on the mouse track abnormality reference value and the mouse sliding abnormality reference value.

[0014] Furthermore, the confirmation method of the difference of link information includes:

[0015] Detect the validity of the link name of the target website;

[0016] If the link name validity is greater than the preset link name validity, the link information difference is determined based on the number of similarities between the main page link names;

[0017] If the link name validity is less than or equal to the preset link name validity, the link information difference is determined based on the number of similarities between the link names of the related pages.

[0018] Furthermore, the method for confirming the abnormal reference value of the mouse trajectory includes:

[0019] Detecting the distribution similarity between the target website and the historical records, wherein the distribution similarity is determined based on the link distance and the number of links;

[0020] If the distribution similarity is greater than the preset distribution similarity, the mouse trajectory abnormality reference value is determined according to the mouse trajectory similarity;

[0021] If the distribution similarity is less than or equal to the preset distribution similarity, the mouse trajectory abnormality reference value is determined according to the number of mouse disconnection times.

[0022] Furthermore, the confirmation method of the mouse sliding abnormal reference value includes:

[0023] Obtaining a number of use frames according to a preset selection rule;

[0024] Detect the number of images in each used frame;

[0025] If the number of images is within a first preset valid image number range, determining that the mouse sliding abnormality reference value is a preset response reference value;

[0026] If the number of images is within a second preset valid image number range, it is determined that the mouse sliding abnormality reference value is determined based on the image repetition value.

[0027] Furthermore, when the number of images is within a second preset valid image number range, the mouse sliding abnormality reference value is positively correlated with the image repetition value.

[0028] Furthermore, the information-related reference value is determined based on the search deviation degree;

[0029] There is a negative correlation between the search deviation and the number of keyword combinations in the user's requested page.

[0030] Furthermore, the search depth reference value is determined based on the number of vertical search pages;

[0031] The search depth reference value is positively correlated with the number of vertical search pages.

[0032] Furthermore, when the characteristic state of the target website is that the page link distribution value is within the second preset distribution value range and the related page difference value is within the second preset difference value range, dynamic protection is performed on the target website, including:

[0033] Detect the number of sensitive access pages and resource utilization;

[0034] Determine dynamic protection requirements based on the number of sensitive access pages and resource utilization;

[0035] Determine dynamic protection strategy based on dynamic protection demand status;

[0036] If the dynamic protection requirement state is that the number of sensitive access pages is within the first preset sensitive access page number range or the resource utilization is within the first resource utilization range, the dynamic protection strategy is to perform port hopping at a fixed frequency;

[0037] If the dynamic protection requirement state is that the number of sensitive access pages is within the second preset sensitive access page number range and the resource utilization is within the second resource utilization range, the dynamic protection strategy is to perform port hopping with an increasing hopping frequency.

[0038] Furthermore, the increasing jump frequency is positively correlated with the number of sensitive access pages;

[0039] The increasing jump frequency is negatively correlated with the resource utilization rate.

[0040] Compared with the prior art, the beneficial effect of the present invention lies in that the technical solution of the present invention determines the characteristic state of the target website based on the page link distribution value of the target website and the related page difference value, reflects the distribution of links of the pages in the target website through the characteristic state, and selects different crawler analysis and protection methods accordingly, so that the selection of crawler analysis and protection methods is more in line with the actual work scenario, thereby improving the crawler protection efficiency of the present invention.

[0041] Furthermore, in the present invention, in the analysis and protection of user operation fingerprints, according to the comparison result of the link information difference degree and the preset link information difference degree, it is selected to determine whether to issue a crawler protection warning based on the information-related reference value and the search depth reference value, or to determine whether to issue a crawler protection warning based on the mouse trajectory abnormality reference value and the mouse sliding abnormality reference value. The link information difference degree is used to reflect the degree of content difference between the corresponding contents of each link corresponding to the target website, thereby making the selection of the benchmark for determining whether to issue a crawler protection warning more accurate.

[0042] Furthermore, in the present invention, when the link information difference is greater than the preset link information difference, it is determined whether to issue a crawler protection warning based on the information relevance reference value and the search depth reference value. By taking advantage of the disadvantage that most crawlers cannot identify the relevance between pages, the information relevance reference value is used to reflect the relevance of each page in the user page request process, and then reflect whether there is a risk of crawler attack, thereby improving the crawler identification effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 Schematic diagram of the web crawler protection method based on device fingerprint and dynamic protection of the present invention;

[0044] Figure 2 A schematic diagram of the present invention determining the dynamic protection requirement state based on the number of sensitive access pages and resource utilization;

[0045] Figure 3 This is a flow chart of the present invention for analyzing and protecting user operation fingerprints;

[0046] Figure 4Flowchart of the method for confirming the difference of link information of the present invention. DETAILED DESCRIPTION

[0047] In order to make the objects and advantages of the present invention more clearly understood, the present invention is further described below in conjunction with embodiments; it should be understood that the specific embodiments described herein are merely used to explain the present invention and are not intended to limit the present invention.

[0048] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0049] It should be noted that, in the description of the present invention, terms such as "up", "down", "left", "right", "inside", and "outside" indicating directions or positional relationships are based on the directions or positional relationships shown in the accompanying drawings. This is only for the convenience of description and does not indicate or imply that the device or element must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it cannot be understood as a limitation on the present invention.

[0050] Furthermore, it should be noted that, in the description of the present invention, unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be understood in a broad sense. For example, they may refer to fixed connections, detachable connections, or integral connections; mechanical connections or electrical connections; direct connections or indirect connections through an intermediate medium; and internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0051] See also Figures 1 to 4 As shown, the present invention provides a web crawler protection method based on device fingerprint and dynamic protection, including:

[0052] Determine the characteristic status of the target website based on the page link distribution value and the related page difference value of the target website;

[0053] Determine the crawler analysis protection method based on the characteristic status of the target website, which is to analyze and protect the user operation fingerprint or dynamically protect the target website;

[0054] In the analysis and protection of user operation fingerprints, the comparison result of the link information difference with the preset link information difference is used to determine whether to issue a crawler protection warning based on the information-related reference value and the search depth reference value or the mouse track abnormality reference value and the mouse sliding abnormality reference value;

[0055] In dynamic protection of the target website, the dynamic protection demand status is determined according to the number of sensitive access pages and resource utilization, and the dynamic protection strategy is determined according to the dynamic protection demand status.

[0056] In the present invention, the target website is a website that needs to be protected against crawlers, the main page of the target website is the page currently being used, and the related pages of the target website are pages searched through the link of the main page of the target website or the link of the related pages. The single page described in the present invention can be a single main page or a related page. The link distribution value corresponding to the single page is confirmed by calculating the reference distance corresponding to each link of the page, and recording the average value of the reference distance as the page link distribution value. The link distribution value of the target website is the link distribution value corresponding to the main page of the target website. For a single link, its corresponding reference distance is the minimum value of the distance between the link and other links. The related page difference value is confirmed by detecting the link structure feature value of each related page of the target website, calculating the absolute value of the difference between each link structure feature value and the reference average value, and recording the absolute value as the link structure feature difference value. The number of link structure feature difference values ​​greater than the preset link structure feature difference value is recorded as the related page difference value. The reference average value is the average value of each link structure feature value. For a single page, its corresponding link structure feature value = the corresponding value of the page Number of links × α1 + image area ratio of the page × α2 + page link distribution value of the page × α3, where α1 is the first calculation coefficient, α2 is the second calculation coefficient, and α3 is the third calculation coefficient. It can be understood that the values ​​of α1, α2 and α3 can be determined based on the user's attention to the number of links corresponding to the page, the image area ratio or the link structure feature value. For example, the user's past experience determines that the number of links corresponding to the page has a greater impact on the page similarity. The larger the value of the first calculation coefficient, the greater the value of the first calculation coefficient. A value is provided, α1 = 0.4, α2 = 0.3, α3 = 0.3, and the preset link structure feature difference is 30% of the reference average value. It can be understood that the greater the user's judgment accuracy for page similarity, the greater the preset link structure feature difference. The present invention reflects the page similarity of the target website and the link distribution density of the page through the page link distribution value and the related page difference value, and then determines the corresponding protection method, avoiding the problem in the prior art that pages with large page similarity and large link distribution density cannot be effectively protected against crawlers based on user device operation fingerprints.

[0057] Specifically, when the characteristic state of the target website is that the page link distribution value is within the first preset distribution value range or the related page difference value is within the first preset difference value range, analysis and protection are performed on the user operation fingerprint, including:

[0058] Detect the difference in link information;

[0059] If the link information difference is greater than the preset link information difference, determine whether to issue a crawler protection warning based on the information relevance reference value and the search depth reference value;

[0060] If the link information difference is less than or equal to the preset link information difference, determining whether to issue a crawler protection warning based on the mouse track abnormality reference value and the mouse sliding abnormality reference value;

[0061] The page link distribution value is within the first preset distribution value range or the related page difference value is within the first preset difference value range, which is recorded as the first characteristic state.

[0062] Specifically, the values ​​within the first preset distribution value range of the present invention are all greater than the preset distribution value, the values ​​within the second preset distribution value range are all less than or equal to the preset distribution value, the values ​​within the first preset difference value range are all less than the preset difference value, and the values ​​within the second preset difference value range are all greater than or equal to the preset difference value. The preset distribution value and the preset difference value can be set by the user according to the actual application scenario. The data can be cleaned by the page link distribution value and the related page difference corresponding to the historical crawler protection record that meets the user's needs to remove the abnormal values. The method for confirming the abnormal value can be the Z-Score method or The IQR method records the average value of the page link distribution value after removing outliers as the preset distribution value, and records the average value of the difference between related pages after removing outliers as the preset difference; the present invention applies historical crawler protection records, and any historical record in the historical crawler protection record records at least one crawler protection usage process, and each historical crawler protection record corresponds to a qualified mark, and the qualified mark indicates whether the crawler protection effect meets user requirements. The qualified mark can be recorded manually, wherein it can be understood that the user can determine whether the crawler protection effect meets the requirements based on the accuracy of the crawler protection, which will not be elaborated here.

[0063] Specifically, the methods for confirming the difference of link information include:

[0064] Detect the validity of the link name of the target website;

[0065] If the link name validity is greater than the preset link name validity, the link information difference is determined based on the number of similarities between the link names on the main page;

[0066] If the link name validity is less than or equal to the preset link name validity, the link information difference is determined based on the number of similarities between the link names of the related pages.

[0067] Specifically, the number of recognizable keywords in the link name of the main page of the target website is recorded as the link name validity, all the link names of the main page of the target website are extracted, the keywords in the link name are identified through the deep learning network and recorded as recognizable keywords, the value of the link name validity is preset, and the user can set it according to actual needs, and the link name validity corresponding to the historical crawler protection record that meets the user's needs is cleaned to remove outliers. The confirmation method of the outlier can be the Z-Score method or the IQR method, and the average value of the link name validity after removing the outlier is recorded as the preset link name validity. It can be understood that the link name validity can reflect whether the link name of the main page of the target website can meet the calculation requirements of the link information difference. It can be understood that the smaller the link name validity, the smaller the accuracy of the link information difference determined according to the similar number of link names on the main page;

[0068] If the same keywords exist in both link names, the same keywords are recorded as repeated keywords, and the number of repeated keywords is the similar number;

[0069] The method for confirming the link information difference is that when it is determined based on the similar number of link names of the main page, (the number of identifiable keywords of the link name of the main page - the similar number of link names of the main page) / the number of identifiable keywords of the link name of the main page is recorded as the link information difference. When the link information difference is determined based on the similar number of link names of related pages, (the number of identifiable keywords of the link name of related pages - the similar number of link names of related pages) / the number of identifiable keywords of the link name of related pages is recorded as the link information difference. The value of the link information difference is preset, and the user can adjust it according to the actual situation. The crawler protection needs and past experience are used to set the value. The greater the user's demand for crawler protection performance, the greater the value of the preset link information difference. This avoids the phenomenon of misjudgment when determining whether to issue a crawler protection warning based on information-related reference values ​​and search depth reference values. A preset link information difference value is provided. The preset link information difference is 70% of the number of identifiable keywords in the link name of the main page or related page. Among them, in the confirmation of the preset link information difference, the selection of the main page and the related page needs to be determined according to the calculation method of the link information difference. This is content that is easy for technical personnel in this field to understand and will not be elaborated here.

[0070] Specifically, the methods for confirming the abnormal reference value of mouse trajectory include:

[0071] Detecting the distribution similarity between the target website and the historical records, wherein the distribution similarity is determined based on the link distance and the number of links;

[0072] If the distribution similarity is greater than the preset distribution similarity, the mouse trajectory abnormality reference value is determined according to the mouse trajectory similarity;

[0073] If the distribution similarity is less than or equal to the preset distribution similarity, the mouse trajectory abnormality reference value is determined according to the number of mouse disconnection times.

[0074] Specifically, the method for confirming the distribution similarity is to divide the main page of the target website into regions to obtain several sub-regions of the same area. For the main page and any reference page, if the number of similar sub-regions between the two pages is greater than the preset similarity number, then the reference page is a similar page of the main page, and the number of similar pages of the main page is recorded as the distribution similarity. For a sub-region and a reference historical sub-region, if the absolute value of the difference in the number of links corresponding to the two regions is less than 3 and the absolute value of the difference in the maximum distance of the links is less than 10% of the maximum distance of the links, then the sub-region and the reference historical sub-region are similar sub-regions. The preset similarity number = the number of sub-regions of the main page × ζ. The user The greater the demand for the accuracy of determining the similarity of pages, the larger the value of ζ. However, it is worth noting that ζ should be greater than 1 and less than 2. A value of ζ is provided, ζ = 1.6. The greater the user's demand for the accuracy of crawler protection based on historical records, the larger the value of the preset distribution similarity. In the present invention, a historical record includes a page of user historical request, namely a reference page, and its related information. The related information includes the link distance and the number of links in the historical sub-area of ​​the reference page and the mouse operation trajectory. The historical sub-area is a number of sub-areas of the same area obtained by dividing the reference page of the historical record. The recording of mouse operation trajectories can be achieved through tools with mouse tracking recording functions such as Mouse Tracks.

[0075] Specifically, the confirmation methods of the abnormal reference value of mouse sliding include:

[0076] Obtaining a number of use frames according to a preset selection rule;

[0077] Detect the number of images in each used frame;

[0078] If the number of images is within a first preset valid image number range, determining that the mouse sliding abnormality reference value is a preset response reference value;

[0079] If the number of images is within a second preset valid image number range, it is determined that the mouse sliding abnormality reference value is determined based on the image repetition value.

[0080] Among them, the preset selection rule is to capture the image of the mouse reference range in the page used by the current user once every 5 seconds and record it as the usage frame. The mouse reference range is a circle established with the current mouse position as the center, and the area of ​​the circle is 15% of the display area of ​​the current page. The area of ​​the mouse reference range can be determined after learning the historical crawler protection records through a deep learning network. How to set the training set and the verification set, and how to perform model learning and verification are contents that are easy for technical personnel in this field to understand. The values ​​within the first preset valid image number range are all less than the preset valid image number, and the values ​​within the second preset valid image number range are all greater than or equal to the preset valid image number. The number of images in the usage frame is the number of images that do not touch each other except links. The value of the preset valid image number can be set by the user according to the actual application scenario. The data can be cleaned by preset valid image numbers of pages corresponding to historical crawler protection records that meet user needs to remove outliers. The average value of the number of images after removing outliers is recorded as the preset valid image number, and the preset response reference value is 0.

[0081] Specifically, when the number of images is within a second preset valid image number range, the mouse sliding abnormality reference value is positively correlated with the image repetition value.

[0082] The image repetition value is the maximum number of times each image in the collected use frame appears repeatedly as of the current moment. It uses the fact that the mouse sliding rules during the crawler process cannot be changed according to the distribution of images to perform anomaly recognition, thereby improving the crawler recognition efficiency.

[0083] Specifically, the information-related reference value is determined based on the search deviation degree;

[0084] There is a negative correlation between the search deviation and the number of keyword combinations in the user's requested page.

[0085] Information-related reference value = 1 / search deviation. The user request page is the M pages requested by the user most recently as of the current moment. The method for confirming the number of keyword combinations is to extract the request link corresponding to each user request page, through which the user request page can be entered, extract the identifiable keywords of each request link, detect the number of times any two identifiable keywords appear at the same time in the historical health search records, and record it as the number of combinations. The average value of the number of combinations is recorded as the number of keyword combinations. The value of M can be set by the user according to the actual application scenario, which will not be elaborated here. M is preferably 10.

[0086] Historical crawler protection records in which no crawlers were detected are recorded as historical healthy search records.

[0087] Specifically, the search depth reference value is determined based on the number of vertical search pages;

[0088] The search depth reference value is positively correlated with the number of vertical search pages.

[0089] The method for confirming the number of vertical search pages is to detect the main page currently used by the user, detect the number of indirect links from the initial page of the target website to the main page, and record the number of indirect links as the number of vertical search pages. The number of indirect links is the number of links clicked from the initial page of the target website to the main page.

[0090] Specifically, when the characteristic state of the target website is that the page link distribution value is within the second preset distribution value range and the related page difference value is within the second preset difference value range, dynamic protection is performed on the target website, including:

[0091] Detect the number of sensitive access pages and resource utilization;

[0092] Determine dynamic protection requirements based on the number of sensitive access pages and resource utilization;

[0093] Determine dynamic protection strategy based on dynamic protection demand status;

[0094] If the dynamic protection requirement state is that the number of sensitive access pages is within the first preset sensitive access page number range or the resource utilization is within the first resource utilization range, the dynamic protection strategy is to perform port hopping at a fixed frequency;

[0095] If the dynamic protection requirement state is that the number of sensitive access pages is within the second preset sensitive access page number range and the resource utilization is within the second resource utilization range, the dynamic protection strategy is to perform port hopping with an increasing hopping frequency.

[0096] Specifically, whether a page is a sensitive page is set by the user. It is understandable that the user can decide whether a page is set as a sensitive page based on factors including but not limited to the value of information, the amount of information, and the number of links in the page. I will not go into details here. The number of sensitive access pages is the number of sensitive pages requested to be accessed by the user in the most recent monitoring cycle. Resource utilization = (CPU utilization / preset CPU utilization) + (memory utilization / preset memory utilization). The preset CPU utilization reference value and the preset memory utilization value can be set by the user. It is understandable that the greater the resource utilization, the impact on the service capability of the Web system. A value selection method is provided so that the user can set the CPU utilization reference value corresponding to the records that meet the user's requirements in the historical crawler protection records. The average value of the value and the average value of the memory utilization are respectively recorded as the preset CPU utilization reference value and the preset memory utilization; the values ​​within the first resource utilization range are all less than the preset resource utilization, and the values ​​within the second resource utilization range are all greater than or equal to the preset resource utilization. The user can set the preset resource utilization value according to actual needs, and provide a preset resource utilization value. The preset resource utilization value is preferably 1.2. The values ​​within the first preset sensitive access page number range are all less than the preset sensitive access page number, and the values ​​within the second preset sensitive access page number range are all greater than or equal to the preset sensitive access page number. The preferred value of the preset sensitive access page number is 20. The greater the user's demand for crawler protection, the smaller the value of the preset sensitive access page number.

[0097] In the present invention, the Web service system corresponding to the target website is continuously controlled to perform periodic port hopping. The value of the fixed frequency is set by the user according to the crawler protection demand. The greater the user's crawler protection demand, the greater the fixed frequency. A fixed frequency value is provided, and the fixed frequency is 1 time / 30s. When the port hopping is performed at a fixed frequency, the Web service system reallocates page resources every 30s and allocates a random or pseudo-random port number.

[0098] Specifically, the increasing jump frequency is positively correlated with the number of sensitive access pages;

[0099] The growth jump frequency is negatively correlated with resource utilization;

[0100] The page link distribution value being within the second preset distribution value range and the related page difference value being within the second preset difference value range is recorded as a second characteristic state.

[0101] It is worth noting that the minimum value of the increased hopping frequency should be greater than the fixed frequency. When the increased hopping frequency is used for port hopping, the Web service system allocates page resources according to the increased hopping frequency.

[0102] When determining the mouse trajectory abnormality reference value based on the mouse trajectory similarity, the mouse trajectory abnormality reference value is the mouse trajectory similarity, and the mouse trajectory similarity = the number of similar trajectories / the total number of trajectories. For two mouse trajectories, if the Euclidean distance reference value corresponding to the two mouse trajectories is less than the preset Euclidean distance reference value, the two mouse trajectories are similar trajectories. The present invention extracts 50 mouse trajectories in the user's most recent operation process and the mouse trajectories in the most recent 50 historical records. A single mouse trajectory is the trajectory of the mouse operation between two link requests. The number of similar trajectories in 100 mouse trajectories is recorded as the number of similar trajectories. For two mouse trajectories, the corresponding Euclidean distance reference value is confirmed by obtaining the coordinate data of the two mouse trajectories, recording the two trajectories as trajectory A and trajectory B respectively, and taking n coordinate points on each trajectory A and trajectory B respectively. The coordinate points are selected randomly. Note that there are n coordinate points on both trajectories. The left endpoints of the two trajectories are placed at the origin. Each coordinate point contains values ​​of two dimensions, x and y. For each corresponding point in trajectory A and trajectory B, the Euclidean distance between them is calculated, where

[0103]

[0104] Where XAi is the x-coordinate of the i-th point on track A, XBi is the x-coordinate of the i-th point on track B, YAi is the y-coordinate of the i-th point on track A, and YBi is the y-coordinate of the i-th point on track B. The calculation formula for the Euclidean distance reference value D0 is:

[0105]

[0106] It can be understood that the smaller D0 is, the more similar the two trajectories are. The value of the preset Euclidean distance reference value can be set by the user according to actual needs. For those skilled in the art, it is easy to understand to set the preset Euclidean distance reference value based on the data corresponding to the historical crawler protection records and the actual crawler protection needs, and on the premise that the Euclidean distance reference value calculation method has been provided. We will not go into details here and provide a preferred preset Euclidean distance reference value, which is (dmax+dmin) / 2×0.5, where dmax is the maximum value in di, dmin is the minimum value in di, and i=1,2,3,…,n.

[0107] When determining the mouse track abnormality reference value based on the number of mouse disconnection times, the mouse track abnormality reference value = the number of mouse disconnection times.

[0108] The information-related reference value, search depth reference value, mouse trajectory abnormality reference value and mouse sliding abnormality reference value are all individually set with preset comparison thresholds, and the corresponding preset comparison thresholds are different. Users can determine them based on the records in the historical crawler protection records that meet user needs. For example, the preset comparison threshold corresponding to the information-related reference value, the user can record the average value of the information-related reference values ​​corresponding to the records in the historical crawler protection records that meet user needs as the preset comparison threshold, and the user can adjust the preset comparison threshold according to the crawler protection accuracy requirements. The greater the user's demand for crawler protection, the smaller the preset comparison threshold.

[0109] When determining whether to issue a crawler protection warning based on the information-related reference value and the search depth reference value, if one of the information-related reference value and the search depth reference value is greater than the corresponding preset comparison threshold, a crawler protection warning is issued;

[0110] When determining whether to issue a crawler protection warning based on the mouse track abnormality reference value and the mouse sliding abnormality reference value, if one of the mouse track abnormality reference value and the mouse sliding abnormality reference value is greater than the corresponding preset comparison threshold, a crawler protection warning is issued;

[0111] The crawler protection warning is a Web service system that reminds administrators to verify the user's identity and check whether to manually ban the user. This is easy for technicians in this field to understand and will not be elaborated here.

[0112] Thus far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art may make equivalent changes or substitutions to the relevant technical features, and the technical solutions after such changes or substitutions will fall within the scope of protection of the present invention.

[0113] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that the present invention is susceptible to various modifications and variations. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A web crawler protection method based on device fingerprint and dynamic protection, characterized in that: include: Determine the characteristic status of the target website based on the page link distribution value and the related page difference value of the target website; Determine the crawler analysis protection method based on the characteristic status of the target website, which is to analyze and protect the user operation fingerprint or dynamically protect the target website; In the analysis and protection of user operation fingerprints, the comparison result of the link information difference with the preset link information difference is used to determine whether to issue a crawler protection warning based on the information-related reference value and the search depth reference value or the mouse track abnormality reference value and the mouse sliding abnormality reference value; In dynamic protection of target websites, the dynamic protection demand status is determined based on the number of sensitive access pages and resource utilization, and the dynamic protection strategy is determined based on the dynamic protection demand status; The link distribution value corresponding to a single page is confirmed by calculating the reference distance corresponding to each link of the page and recording the average value of the reference distance as the link distribution value of the page; The relevant page difference value is confirmed by detecting the link structure characteristic value of each relevant page of the target website, calculating the absolute value of the difference between each link structure characteristic value and the reference average value, recording the absolute value as the link structure characteristic difference value, and recording the number of link structure characteristic difference values ​​greater than the preset link structure characteristic difference value as the relevant page difference value; Methods for confirming the difference of link information include: Detect the validity of the link name of the target website; If the link name validity is greater than the preset link name validity, the link information difference is determined based on the number of similarities between the link names on the main page; If the link name validity is less than or equal to the preset link name validity, the link information difference is determined based on the number of similarities between the link names of the relevant pages; The information-related reference value is determined based on the search deviation degree; There is a positive correlation between the search deviation and the number of keyword combinations in the user's requested page; The search depth reference value is determined based on the number of vertical search pages; The search depth reference value is positively correlated with the number of vertical search pages.

2. The web crawler protection method based on device fingerprint and dynamic protection according to claim 1 is characterized in that: When the characteristic state of the target website is that the page link distribution value is within the first preset distribution value range or the related page difference value is within the first preset difference value range, analysis and protection are performed on the user operation fingerprint, including: Detect the difference in link information; If the link information difference is greater than the preset link information difference, determine whether to issue a crawler protection warning based on the information relevance reference value and the search depth reference value; If the link information difference is less than or equal to the preset link information difference, determining whether to issue a crawler protection warning based on the mouse track abnormality reference value and the mouse sliding abnormality reference value; The values ​​within the first preset distribution value range are all greater than the preset distribution value, and the values ​​within the first preset difference value range are all less than the preset difference value.

3. The web crawler protection method based on device fingerprint and dynamic protection according to claim 1 is characterized in that: The methods for confirming the abnormal reference value of mouse trajectory include: Detecting the distribution similarity between the target website and the historical records, wherein the distribution similarity is determined based on the link distance and the number of links; If the distribution similarity is greater than the preset distribution similarity, the mouse trajectory abnormality reference value is determined according to the mouse trajectory similarity; If the distribution similarity is less than or equal to the preset distribution similarity, the mouse trajectory abnormality reference value is determined according to the number of mouse disconnection times.

4. The web crawler protection method based on device fingerprint and dynamic protection according to claim 3 is characterized in that: The confirmation methods of abnormal mouse sliding reference values ​​include: Obtaining a number of use frames according to a preset selection rule; Detect the number of images in each used frame; If the number of images is within a first preset valid image number range, determining that the mouse sliding abnormality reference value is a preset response reference value; If the number of images is within a second preset valid image number range, determining a mouse sliding abnormality reference value based on the image repetition value; The values ​​within the first preset valid image number range are all smaller than the preset valid image number, and the values ​​within the second preset valid image number range are all greater than or equal to the preset valid image number.

5. The web crawler protection method based on device fingerprint and dynamic protection according to claim 4 is characterized in that: When the number of images is within the second preset valid image number range, the mouse sliding abnormality reference value is positively correlated with the image repetition value.

6. The web crawler protection method based on device fingerprint and dynamic protection according to claim 1 is characterized in that: When the characteristic state of the target website is that the page link distribution value is within the second preset distribution value range and the related page difference value is within the second preset difference value range, dynamic protection is performed on the target website, including: Detect the number of sensitive access pages and resource utilization; Determine dynamic protection requirements based on the number of sensitive access pages and resource utilization; Determine dynamic protection strategy based on dynamic protection demand status; If the dynamic protection requirement state is that the number of sensitive access pages is within the first preset sensitive access page number range or the resource utilization is within the first resource utilization range, the dynamic protection strategy is to perform port hopping at a fixed frequency; If the dynamic protection requirement state is that the number of sensitive access pages is within the second preset sensitive access page number range and the resource utilization is within the second resource utilization range, the dynamic protection strategy is to perform port hopping with an increasing hopping frequency; The values ​​within the second preset distribution value range are all less than or equal to the preset distribution value, the values ​​within the second preset difference value range are all greater than or equal to the preset difference value, the values ​​within the first preset sensitive access page number range are all less than the preset sensitive access page number, the values ​​within the first resource utilization range are all less than the preset resource utilization, the values ​​within the second preset sensitive access page number range are all greater than or equal to the preset sensitive access page number, and the values ​​within the second resource utilization range are all greater than or equal to the preset resource utilization.

7. The web crawler protection method based on device fingerprint and dynamic protection according to claim 6 is characterized in that: The increasing jump frequency is positively correlated with the number of sensitive access pages; The increasing jump frequency is negatively correlated with the resource utilization rate.

Images