A deep anti-crawl method, system, device, and medium based on TLS and webrtc

Abstract

The present invention relates to a deep anti-crawl method, system, device, and medium based on TLS and webrtc. The key points of the technical solution are as follows: the method comprises: obtaining a TCP handshake request carrying user information; generating a corresponding TLS fingerprint according to the TCP handshake request; comparing the TLS fingerprint and user information with a redis fingerprint library to determine whether it is a malicious access, and if so, denying access; returning a JavaScript file for detecting the environment according to the TCP handshake request, and obtaining canvas information, font information, audio fingerprint, and instant messaging address returned by the user; determining whether the instant messaging address is consistent with the IP information of the TCP handshake request, and if not, denying access; obtaining the access frequency of requests that are completely consistent with the canvas information, font information, audio fingerprint, or instant messaging address, comparing the access frequency with a preset access frequency, and denying access if the access frequency is greater than the preset access frequency; the application has the advantage of being able to prevent data from being captured by crawlers.

Description

Technical Field

[0001] The present invention relates to the field of anti-crawling technology, and more specifically, to a deep anti-crawling method, system, device, and medium based on TLS and webrtc. Background Art

[0002] Data security is a critical issue for companies. How can we ensure that data is not accessible to competitors and mitigate competition? A common approach is to integrate the security strategies of existing risk control platforms. However, most mainstream risk control strategies currently available focus on increasing obfuscation difficulty and are lax in fingerprint detection. Numerous circumvention schemes have emerged online, making them unacceptable.

[0003] During website development, security considerations necessitate security processing of interfaces to prevent data from being captured by crawlers. A common approach involves using free online obfuscation tools to obfuscate JavaScript files, making them harder to identify, and detecting IP access frequency, or employing paid security platforms' anti-crawling strategies. However, these approaches often suffer from incompleteness and lack of stringency. For example, canvas information, font information, and audio fingerprints are only extracted without verification, and HTTPS only covers the post-TCP handshake phase, ignoring the handshake phase. Therefore, there is room for improvement. Summary of the Invention

[0004] In view of the shortcomings of the existing technology, the purpose of the present invention is to provide a deep anti-crawl method, system, device, and medium based on TLS and webrtc to solve at least one technical problem raised in the background technology.

[0005] The above technical objectives of the present invention are achieved through the following technical solutions: a deep anti-crawl method based on TLS and WebRTC, comprising:

[0006] S1. Obtain a TCP handshake request carrying user information;

[0007] S2. Generate a corresponding TLS fingerprint according to the TCP handshake request;

[0008] S3. Compare the TLS fingerprint and user information with the Redis fingerprint database to determine whether the access is malicious. If not, execute S4. If yes, deny access.

[0009] S4. Return a JavaScript file for detecting the environment according to the TCP handshake request, and obtain the canvas information, font information, audio fingerprint, and instant messaging address returned by the user;

[0010] S5. Determine whether the instant messaging address is consistent with the IP information of the TCP handshake request. If so, execute S6; if not, deny access.

[0011] S6. Obtain the access frequency of requests that are completely consistent with the canvas information, font information, audio fingerprint, or instant messaging address, and compare the access frequency with a preset access frequency. If the access frequency is greater than the preset access frequency, deny access and block the instant messaging address or audio fingerprint.

[0012] Optionally, generating a corresponding TLS fingerprint according to the TCP handshake request includes:

[0013] Obtain corresponding TLS information according to the TCP handshake request;

[0014] The version, Cipher Suites, Extension length, Extension:elliptic_curves, and Extension:ec_pooint_formats in the TLS information are combined and encrypted using the MD5 algorithm to generate a TLS fingerprint.

[0015] Optionally, comparing the TLS fingerprint and user information with the Redis fingerprint database includes:

[0016] Get all browser fingerprints and their corresponding user agents;

[0017] Store each browser fingerprint as a value and the corresponding user agent as a key in the redis fingerprint library;

[0018] The TLS fingerprint is compared with each browser fingerprint in the redis fingerprint library; if the redis fingerprint library contains the same browser fingerprint as the TLS fingerprint, the user agent corresponding to the browser fingerprint is compared with the user information corresponding to the TLS fingerprint. If the user agent corresponding to the browser fingerprint is consistent with the user information corresponding to the TLS fingerprint, it is determined that it is not a malicious access; if the user agent corresponding to the browser fingerprint is inconsistent with the user information corresponding to the TLS fingerprint, it is determined to be a malicious access; if the redis fingerprint library does not contain the same browser fingerprint as the TLS fingerprint, it is determined to be a malicious access.

[0019] Optionally, the JavaScript file is used to detect the user's canvas information, font information, and audio fingerprint, and to perform WebRTC with the user to obtain the user's instant messaging address;

[0020] The acquisition of canvas information, font information, audio fingerprint and instant messaging address returned by the user includes:

[0021] Get the user's canvas information, font information, and audio fingerprint through JavaScript files;

[0022] Get the user's instant messaging address through WebRTC.

[0023] Optionally, after comparing the TLS fingerprint and user information with the redis fingerprint database, the method further includes:

[0024] Use an obfuscation tool to obfuscate JavaScript files.

[0025] Optionally, after determining whether the instant messaging address is consistent with the IP information of the TCP handshake request, if they are consistent, executing S6, the method further includes:

[0026] The canvas information, font information, audio fingerprint and instant messaging address returned by the user are stored in the Redis middleware.

[0027] Optionally, the comparing the access frequency with a preset access frequency further includes:

[0028] If the access frequency is less than the preset access frequency, the data in the redis middleware is stored in redis for authentication interface;

[0029] Returns the security parameters token.

[0030] A deep anti-crawl system based on TLS and webrtc, comprising: a request acquisition module for acquiring a TCP handshake request carrying user information;

[0031] A fingerprint generation module, configured to generate a corresponding TLS fingerprint according to the TCP handshake request;

[0032] The fingerprint comparison module is used to compare the TLS fingerprint and user information with the Redis fingerprint library to determine whether it is a malicious access. If not, the next step is executed; if yes, the access is denied.

[0033] The environment detection module is used to return a JavaScript file for detecting the environment according to the TCP handshake request, and obtain the canvas information, font information, audio fingerprint and instant messaging address returned by the user;

[0034] An IP comparison module is used to determine whether the instant messaging address is consistent with the IP information of the TCP handshake request. If they are consistent, the next step is executed; if they are inconsistent, the access is denied;

[0035] The frequency judgment module is used to obtain the access frequency of requests that are completely consistent with the canvas information, font information, audio fingerprint or instant messaging address, and compare the access frequency with the preset access frequency. If the access frequency is greater than the preset access frequency, access is denied and the instant messaging address or audio fingerprint is blocked.

[0036] A computer device includes a memory and a processor, wherein the memory stores a computer program and the processor implements the steps of the above method when executing the computer program.

[0037] A computer-readable storage medium stores a computer program, which implements the steps of the above method when executed by a processor.

[0038] In summary, the present invention has the following beneficial effects:

[0039] (1) By comparing the TLS fingerprint and user information with the fingerprint data in the redis fingerprint library and its corresponding userAgent, it is possible to determine whether it is a normal browser access, thereby preventing Python, NodeJs and other network libraries from accessing;

[0040] (2) By verifying whether the instant messaging address is consistent with the IP information of the TCP handshake request, it can be determined that the user has IP masquerading, thereby preventing the user with IP masquerading from accessing;

[0041] (3) By verifying the user's access frequency, it is possible to determine whether the user is a frequent visitor and block frequent users from accessing. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a schematic diagram of the process of the present invention;

[0043] Figure 2 This is a structural diagram of the present invention when assembled;

[0044] Figure 3 2 is a diagram showing the internal structure of a computer device in an embodiment of the present invention. DETAILED DESCRIPTION

[0045] To make the objectives, features, and advantages of the present invention more readily apparent, the following detailed description of the present invention is provided with reference to the accompanying drawings. The accompanying drawings illustrate several embodiments of the present invention. However, the present invention may be implemented in many different forms and is not limited to the embodiments described herein.

[0046] In the present invention, unless otherwise expressly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or an indirect connection through an intermediate medium, or it can be a communication between the two components. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to the specific circumstances. The terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of such features.

[0047] In the present invention, unless otherwise expressly specified and limited, a first feature being "above" or "below" a second feature may include the first and second features being in direct contact, or may include the first and second features being in contact not directly but through another feature between them. Moreover, a first feature being "above," "above," and "above" a second feature includes the first feature being directly above and obliquely above the second feature, or simply indicates that the first feature is higher in level than the second feature. A first feature being "below," "below," and "below" a second feature includes the first feature being directly below and obliquely below the second feature, or simply indicates that the first feature is lower in level than the second feature. The terms "vertical," "horizontal," "left," "right," "above," "below," and similar expressions are for illustrative purposes only and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore should not be understood as limiting the present invention.

[0048] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0049] The present invention provides a deep anti-crawl method based on TLS and webrtc, such as Figure 1 As shown, including:

[0050] S1. Obtain a TCP handshake request carrying user information;

[0051] S2. Generate a corresponding TLS fingerprint according to the TCP handshake request;

[0052] S3. Compare the TLS fingerprint and user information with the Redis fingerprint database to determine whether the access is malicious. If not, execute S4. If yes, deny access.

[0053] S4. Return a JavaScript file for detecting the environment according to the TCP handshake request, and obtain the canvas information, font information, audio fingerprint, and instant messaging address returned by the user;

[0054] S5. Determine whether the instant messaging address is consistent with the IP information of the TCP handshake request. If so, execute S6; if not, deny access.

[0055] S6. Obtain the access frequency of requests that are completely consistent with the canvas information, font information, audio fingerprint, or instant messaging address, and compare the access frequency with a preset access frequency. If the access frequency is greater than the preset access frequency, deny access and block the instant messaging address or audio fingerprint.

[0056] In practice, users need to perform a TCP handshake with the website through their browser. TCP (Transmission Control Protocol) is a connection-oriented, reliable, byte-stream-based transport layer communication protocol defined by IETF RFC 793. Therefore, during the TCP handshake, a TCP handshake request carrying user information is obtained. The connection established between the browser and the website is established using the TLS protocol. The Transport Layer Security (TLS) protocol is used to provide confidentiality and data integrity between two communicating applications. For the same browser, some parameters in its TLS information are fixed, so these parameters are combined to form the TLS fingerprint of the browser, and the TLS fingerprint corresponds to the user information userAgent one-to-one. However, there is no corresponding user information userAgent for the TLS fingerprint of network libraries such as Python and NodeJs. Therefore, it is easy to identify malicious requests through the redis fingerprint library. Redis (RemoteDictionary Server), that is, remote dictionary service, is an open source log-type, key-value database written in ANSI C language, supports network, can be based on memory or persistence, and provides APIs in multiple languages. If the TLS fingerprint and the redis fingerprint library can be successfully matched, JavaScript verification is required. JavaScript is a lightweight, interpreted or just-in-time compiled programming language with function priority. At this time, a JavaScript file with a detection environment is returned. In the JavaScript file, in addition to writing basic canvas information, font information, and audio fingerprint detection, we also need to obtain the real IP through webrtc, and WebRTC (Web Real-Time Even if a VPN proxy is used to access the Internet, the user's real IP address will still be exposed and used as the instant messaging address. In this embodiment, the canvas information is canvas, the font information is font, and the audio fingerprint is audio fingerprint, so the instant messaging address is the user's real IP. The user information carried in the TCP handshake request has its corresponding source IP, so the IP is compared. If the instant messaging address and the IP information are inconsistent, it is determined to be a malicious request and the IP is blocked for a period of time. If they are consistent, the access frequency is detected. If the IP or canvas information, font information, and audio fingerprint are exactly the same and have been accessed within 0.5 seconds, it is determined to be a frequent access and the IP or fingerprint is blocked for a period of time. If the IP or canvas information, font information, and audio fingerprint are exactly the same and have not been accessed within 0.5 seconds, it is not determined to be a frequent access and the TCP handshake request can be replied to.

[0057] Furthermore, generating a corresponding TLS fingerprint according to the TCP handshake request includes:

[0058] Obtain corresponding TLS information according to the TCP handshake request;

[0059] The version, Cipher Suites, Extension length, Extension:elliptic_curves, and Extension:ec_pooint_formats in the TLS information are combined and encrypted using the MD5 algorithm to generate a TLS fingerprint.

[0060] In actual applications, since the TLS version, CipherSuites, Extension length, Extension:elliptic_curves, and Extension:ec_pooint_formats parameters are fixed when the browser performs a TCP handshake with the website, they are combined together and encrypted with MD5 to generate a unique fingerprint, which serves as the TLS fingerprint.

[0061] Optionally, comparing the TLS fingerprint and user information with the Redis fingerprint database includes:

[0062] Get all browser fingerprints and their corresponding user agents;

[0063] Store each browser fingerprint as a value and the corresponding user agent as a key in the redis fingerprint library;

[0064] The TLS fingerprint is compared with each browser fingerprint in the redis fingerprint library; if the redis fingerprint library contains the same browser fingerprint as the TLS fingerprint, the user agent corresponding to the browser fingerprint is compared with the user information corresponding to the TLS fingerprint. If the user agent corresponding to the browser fingerprint is consistent with the user information corresponding to the TLS fingerprint, it is determined that it is not a malicious access; if the user agent corresponding to the browser fingerprint is inconsistent with the user information corresponding to the TLS fingerprint, it is determined to be a malicious access; if the redis fingerprint library does not contain the same browser fingerprint as the TLS fingerprint, it is determined to be a malicious access.

[0065] In actual applications, commonly used browser fingerprints on the market are stored as the value Value and the corresponding userAgent as the key Key in the redis fingerprint library; TLS fingerprints that do not match the browser fingerprint, and TLS fingerprints that match the browser fingerprint, but the user agent corresponding to the browser fingerprint is inconsistent with the user information corresponding to the TLS fingerprint are identified as malicious access.

[0066] Optionally, the JavaScript file is used to detect canvas information, font information, and audio fingerprint of the user, and to perform WebRTC with the user to obtain the user's instant messaging address;

[0067] The acquisition of canvas information, font information, audio fingerprint and instant messaging address returned by the user includes:

[0068] Get the user's canvas information, font information, and audio fingerprint through JavaScript files;

[0069] Get the user's instant messaging address through WebRTC.

[0070] In actual applications, after the fingerprint of the user visiting the website is identified and approved, the JavaScript file of our detection environment will be returned first. In the JavaScript file, in addition to writing basic canvas information, font information, and audio fingerprint detection, we also need to obtain the real IP through webrtc.

[0071] In actual applications, when deploying JavaScript files, configure nginx's ssl_certificate to the crt certificate path, ssl_certificate_key to the certificate private key file path, ssl_ciphers to the SSL encryption algorithm selected ECDHE-RSA-AES128-GCM-SHA256:HIGH:!aNULL:!MD5:!RC4:!DHE, ssl_protocols to the SSL protocol versions TLSv1.1 and TLSv1.2 supported, ssl_prefer_server_ciphers to the enable mode on, and use the uwsgi protocol configuration format: include uwsgi_params;

[0072] uwsgi_param fp$ssl_ciphers;

[0073] uwsgi_param Http2.0_fingerprint$server_protocol;

[0074] uwsgi_param TLS_fingerprint_curves$ssl_curves.

[0075] Furthermore, after comparing the TLS fingerprint and user information with the redis fingerprint library, the method further includes:

[0076] Use an obfuscation tool to obfuscate JavaScript files.

[0077] In practical applications,

[0078] Furthermore, after determining whether the instant messaging address is consistent with the IP information of the TCP handshake request, if they are consistent, executing S6, the method further includes:

[0079] The canvas information, font information, audio fingerprint and instant messaging address returned by the user are stored in the Redis middleware.

[0080] In actual applications, obfuscation tools confuse JavaScript files to increase the difficulty of cracking, and the obfuscated program is deployed to the server together with the front-end JavaScript files. Common JS obfuscation tools include the following: YUICompressor, Google Closure Compiler, UglifyJS, JScrambler, and JShaman. In this embodiment, JShaman is selected.

[0081] Furthermore, the comparing the access frequency with a preset access frequency further includes:

[0082] If the access frequency is less than the preset access frequency, the data in the redis middleware is stored in redis for authentication interface;

[0083] Returns the security parameters token.

[0084] In actual applications, when the user's TCP handshake request passes all tests, the security parameter token can be returned to the user, which can provide authentication for users with normal access.

[0085] like Figure 2 As shown, the present invention also provides a deep anti-crawl system based on TLS and webrtc, including:

[0086] Request acquisition module 10, used to obtain the TCP handshake request carrying user information;

[0087] Fingerprint generation module 20, used to generate a corresponding TLS fingerprint according to the TCP handshake request;

[0088] Fingerprint comparison module 30, used to compare the TLS fingerprint and user information with the redis fingerprint library to determine whether it is a malicious access. If not, the next step is executed; if yes, the access is denied;

[0089] The environment detection module 40 is used to return a JavaScript file for detecting the environment according to the TCP handshake request, and obtain the canvas information, font information, audio fingerprint and instant messaging address returned by the user;

[0090] The IP comparison module 50 is used to determine whether the instant messaging address is consistent with the IP information of the TCP handshake request. If they are consistent, the next step is executed; if they are inconsistent, the access is denied;

[0091] The frequency determination module 60 is used to obtain the access frequency of requests that are completely consistent with the canvas information, font information, audio fingerprint, or instant messaging address, and compare the access frequency with a preset access frequency. If the access frequency is greater than the preset access frequency, access is denied and the instant messaging address or audio fingerprint is blocked.

[0092] Furthermore, the fingerprint generation module 20 includes:

[0093] An information acquisition unit, configured to acquire corresponding TLS information according to the TCP handshake request;

[0094] The information encryption unit is used to combine the version, Cipher Suites, Extension length, Extension:elliptic_curves and Extension:ec_pooint_formats in the TLS information and encrypt them through the MD5 algorithm to generate a TLS fingerprint.

[0095] Furthermore, the fingerprint comparison module 30 includes:

[0096] A browser information acquisition unit, used to obtain all browser fingerprints and their corresponding user agents;

[0097] Fingerprint storage unit, used to store each browser fingerprint as a value and the corresponding user agent as a key in the redis fingerprint library;

[0098] The fingerprint comparison unit is used to compare the TLS fingerprint with each browser fingerprint in the redis fingerprint library; if the redis fingerprint library contains the same browser fingerprint as the TLS fingerprint, the user agent corresponding to the browser fingerprint is compared with the user information corresponding to the TLS fingerprint; if the user agent corresponding to the browser fingerprint is consistent with the user information corresponding to the TLS fingerprint, it is determined that it is not a malicious access; if the user agent corresponding to the browser fingerprint is inconsistent with the user information corresponding to the TLS fingerprint, it is determined that it is a malicious access; if the redis fingerprint library does not contain the same browser fingerprint as the TLS fingerprint, it is determined that it is a malicious access.

[0099] Furthermore, the environment detection module 40 includes:

[0100] User information detection module, used to obtain user canvas information, font information and audio fingerprint through JavaScript files;

[0101] The user IP acquisition module is used to obtain the user's instant messaging address through WebRTC.

[0102] Furthermore, the system further comprises:

[0103] The obfuscation module is used to obfuscate JavaScript files using obfuscation tools.

[0104] Furthermore, the system further comprises:

[0105] The user information storage module is used to store the canvas information, font information, audio fingerprint and instant messaging address returned by the user in the redis middleware.

[0106] Furthermore, the frequency determination module 60 further includes:

[0107] A data storage unit is used to store the data in the redis middleware into redis for the authentication interface when the access frequency is less than the preset access frequency;

[0108] The parameter return unit is used to return the security parameter token.

[0109] For the specific definition of a deep anti-crawl system based on TLS and webrtc, please refer to the definition of a deep anti-crawl method based on TLS and webrtc above, which will not be repeated here. Each module in the above-mentioned deep anti-crawl system based on TLS and webrtc can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to the above modules.

[0110] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 3 As shown. The computer device includes a processor, memory, network interface, and database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. When the computer program is executed by the processor, a deep anti-crawl method based on TLS and WebRTC is implemented.

[0111] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0112] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the following steps are implemented: S1. Obtain a TCP handshake request carrying user information;

[0113] S2. Generate a corresponding TLS fingerprint according to the TCP handshake request;

[0114] S3. Compare the TLS fingerprint and user information with the Redis fingerprint database to determine whether the access is malicious. If not, execute S4. If yes, deny access.

[0115] S4. Return a JavaScript file for detecting the environment according to the TCP handshake request, and obtain the canvas information, font information, audio fingerprint, and instant messaging address returned by the user;

[0116] S5. Determine whether the instant messaging address is consistent with the IP information of the TCP handshake request. If so, execute S6; if not, deny access.

[0117] S6. Obtain the access frequency of requests that are completely consistent with the canvas information, font information, audio fingerprint, or instant messaging address, and compare the access frequency with a preset access frequency. If the access frequency is greater than the preset access frequency, deny access and block the instant messaging address or audio fingerprint.

[0118] In one embodiment, generating a corresponding TLS fingerprint according to the TCP handshake request includes:

[0119] Obtain corresponding TLS information according to the TCP handshake request;

[0120] The version, Cipher Suites, Extension length, Extension:elliptic_curves, and Extension:ec_pooint_formats in the TLS information are combined and encrypted using the MD5 algorithm to generate a TLS fingerprint.

[0121] In one embodiment, comparing the TLS fingerprint and user information with the redis fingerprint library includes:

[0122] Get all browser fingerprints and their corresponding user agents;

[0123] Store each browser fingerprint as a value and the corresponding user agent as a key in the redis fingerprint library;

[0124] The TLS fingerprint is compared with each browser fingerprint in the redis fingerprint library; if the redis fingerprint library contains the same browser fingerprint as the TLS fingerprint, the user agent corresponding to the browser fingerprint is compared with the user information corresponding to the TLS fingerprint. If the user agent corresponding to the browser fingerprint is consistent with the user information corresponding to the TLS fingerprint, it is determined that it is not a malicious access; if the user agent corresponding to the browser fingerprint is inconsistent with the user information corresponding to the TLS fingerprint, it is determined to be a malicious access; if the redis fingerprint library does not contain the same browser fingerprint as the TLS fingerprint, it is determined to be a malicious access.

[0125] In one embodiment, the JavaScript file is used to detect canvas information, font information, and audio fingerprint of the user, and to perform WebRTC with the user to obtain the user's instant messaging address;

[0126] The acquisition of canvas information, font information, audio fingerprint and instant messaging address returned by the user includes:

[0127] Get the user's canvas information, font information, and audio fingerprint through JavaScript files;

[0128] Get the user's instant messaging address through WebRTC.

[0129] In one embodiment, after comparing the TLS fingerprint and user information with the redis fingerprint database, the method further includes:

[0130] Use an obfuscation tool to obfuscate JavaScript files.

[0131] In one embodiment, after determining whether the instant messaging address is consistent with the IP information of the TCP handshake request, if they are consistent, executing S6, the method further includes:

[0132] The canvas information, font information, audio fingerprint and instant messaging address returned by the user are stored in the Redis middleware.

[0133] In one embodiment, the comparing the access frequency with a preset access frequency further includes:

[0134] If the access frequency is less than the preset access frequency, the data in the redis middleware is stored in redis for authentication interface;

[0135] Returns the security parameters token.

[0136] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0137] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0138] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiment. All technical solutions based on the concept of the present invention are within the scope of protection of the present invention. It should be noted that for those skilled in the art, various improvements and modifications that do not depart from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A deep anti-crawl method based on TLS and WebRTC, characterized in that: include: S1. Obtain a TCP handshake request carrying user information; S2. Generate a corresponding TLS fingerprint according to the TCP handshake request; S3. Compare the TLS fingerprint and user information with the Redis fingerprint database to determine whether the access is malicious. If not, execute S4. If yes, deny access. S4. Return a JavaScript file for detecting the environment according to the TCP handshake request, and obtain the canvas information, font information, audio fingerprint, and instant messaging address returned by the user; S5. Determine whether the instant messaging address is consistent with the IP information of the TCP handshake request. If so, execute S6; if not, deny access. S6. Obtain the access frequency of requests that are completely consistent with the canvas information, font information, audio fingerprint, or instant messaging address, compare the access frequency with a preset access frequency, and if the access frequency is greater than the preset access frequency, deny access and block the instant messaging address or audio fingerprint. Among them, the canvas information is canvas, the font information is font, and the audio fingerprint is audio fingerprint, so the instant messaging address is the user's real IP; the user information carried in the TCP handshake request has its corresponding source IP, so the IP is compared. If the instant messaging address is inconsistent with the IP information, it is determined to be a malicious request and the IP end is blocked for a period of time; if they are consistent, the access frequency is detected. If the IP or canvas information, font information, and audio fingerprint are completely identical and have been accessed within 0.5 seconds, it is considered a frequent access and the IP or fingerprint is blocked for a period of time; if the IP or canvas information, font information, and audio fingerprint are completely identical and have not been accessed within 0.5 seconds, it is not considered a frequent access and the TCP handshake request can be replied to at this time; Generating a corresponding TLS fingerprint according to the TCP handshake request includes: Obtain corresponding TLS information according to the TCP handshake request; Combine the version, Cipher Suites, Extension length, Extension:elliptic_curves and Extension:ec_pooint_formats in the TLS information and encrypt them using the MD5 algorithm to generate a TLS fingerprint; The data comparison between the TLS fingerprint and user information and the redis fingerprint library includes: Get all browser fingerprints and their corresponding user agents; Store each browser fingerprint as a value and the corresponding user agent as a key in the redis fingerprint library; The TLS fingerprint is compared with each browser fingerprint in the redis fingerprint library; if the redis fingerprint library contains the same browser fingerprint as the TLS fingerprint, the user agent corresponding to the browser fingerprint is compared with the user information corresponding to the TLS fingerprint. If the user agent corresponding to the browser fingerprint is consistent with the user information corresponding to the TLS fingerprint, it is determined that it is not a malicious access; if the user agent corresponding to the browser fingerprint is inconsistent with the user information corresponding to the TLS fingerprint, it is determined to be a malicious access; if the redis fingerprint library does not contain the same browser fingerprint as the TLS fingerprint, it is determined to be a malicious access.

2. The method according to claim 1, characterized in that The JavaScript file is used to detect the user's canvas information, font information and audio fingerprint, and to conduct WebRTC with the user to obtain the user's instant messaging address; The acquisition of canvas information, font information, audio fingerprint and instant messaging address returned by the user includes: Obtain user canvas information, font information, and audio fingerprint through JavaScript files; Get the user's instant messaging address through WebRTC.

3. The method according to claim 1, characterized in that After comparing the TLS fingerprint and user information with the redis fingerprint database, the following steps are also included: Use an obfuscation tool to obfuscate JavaScript files.

4. The method according to claim 3, characterized in that After determining whether the instant messaging address is consistent with the IP information of the TCP handshake request, if they are consistent, executing S6, the method further includes: The canvas information, font information, audio fingerprint and instant messaging address returned by the user are stored in the Redis middleware.

5. The method according to claim 3, characterized in that The comparing the access frequency with the preset access frequency further includes: If the access frequency is less than the preset access frequency, the data in the redis middleware is stored in redis for authentication interface; Returns the security parameters token.

6. A deep anti-crawl system based on TLS and webrtc, characterized in that: include: Request acquisition module, used to obtain TCP handshake requests carrying user information; A fingerprint generation module, configured to generate a corresponding TLS fingerprint according to the TCP handshake request; The method includes: obtaining corresponding TLS information according to the TCP handshake request; combining version, Cipher Suites, Extension length, Extension:elliptic_curves and Extension:ec_pooint_formats in the TLS information and encrypting them through MD5 algorithm to generate a TLS fingerprint; The fingerprint comparison module is used to compare the TLS fingerprint and user information with the redis fingerprint library to determine whether it is a malicious access. If not, the next step is executed; if yes, the access is denied. The module includes: obtaining all browser fingerprints and their corresponding user agents; storing each browser fingerprint as a value and the corresponding user agent as a key in the redis fingerprint library; comparing the TLS fingerprint with each browser fingerprint in the redis fingerprint library; if the redis fingerprint library contains the same browser fingerprint as the TLS fingerprint, comparing the user agent corresponding to the browser fingerprint with the user information corresponding to the TLS fingerprint; if the user agent corresponding to the browser fingerprint is consistent with the user information corresponding to the TLS fingerprint, it is determined that it is not a malicious access; if the user agent corresponding to the browser fingerprint is inconsistent with the user information corresponding to the TLS fingerprint, it is determined to be a malicious access; if the redis fingerprint library does not contain the same browser fingerprint as the TLS fingerprint, it is determined to be a malicious access; The environment detection module is used to return a JavaScript file for detecting the environment according to the TCP handshake request, and obtain the canvas information, font information, audio fingerprint and instant messaging address returned by the user; An IP comparison module is used to determine whether the instant messaging address is consistent with the IP information of the TCP handshake request. If they are consistent, the next step is executed; if they are inconsistent, the access is denied; The frequency judgment module is used to obtain the access frequency of requests that are completely consistent with the canvas information, font information, audio fingerprint or instant messaging address, and compare the access frequency with the preset access frequency. If the access frequency is greater than the preset access frequency, access is denied and the instant messaging address or audio fingerprint is blocked.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images