Anti-crawler method, system, computer device and storage medium

Abstract

The application relates to an anti-crawler method and system, a computer device and a storage medium, and the technical scheme points are as follows: receiving a network request sent by a client, wherein the network request comprises a request parameter and an encrypted parameter; parsing the network request to obtain the request parameter and the encrypted parameter; performing an encryption process on the request parameter according to a preset sorting rule and a preset non-standard hash function to obtain result data, and taking the result data as verification data; comparing the verification data and the encrypted parameter to obtain a comparison result, if the comparison result is the same, sending data required by the network request to the client, and if the comparison result is different, not sending the data required by the network request to the client; the application has the effects of increasing the difficulty of crawler reverse cracking, improving the anti-crawler capability, and effectively blocking crawler attacks.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of network security, more particularly, it relates to an anti-crawler method and system, a computer device and a storage medium. BACKGROUND

[0002] With the rapid development of network technology, more and more attention is paid to the data collection capability, therefore, the network crawler emerges as the times require, and the network crawler is a program and script that automatically captures web page information according to certain rules.

[0003] However, the abuse of the network crawler causes a large number of IP accesses to the target website, and the request frequency of the target website is too high, which can interfere with the normal operation of the website, and even cause the server to be paralyzed, and the user privacy and other contents are captured, resulting in user privacy data leakage, so it is necessary to prevent the network crawler from obtaining the target website information in batches through the anti-crawler technology.

[0004] The current website anti-crawler ability is weak, such as only setting IP ban, character verification code, CSS (Cascading Style Sheet) anti-crawler, and the front-end code is not confused and is easy to be debugged and cracked, and the anti-crawler effect is poor. SUMMARY

[0005] In view of the defects in the prior art, the present application aims to provide an anti-crawler method, system, computer device and storage medium, which has the advantages of increasing the difficulty of crawler reverse cracking, improving the anti-crawler ability, and effectively blocking the crawler attack.

[0006] The above technical purpose of the present application is realized by the following technical scheme:

[0007] An anti-crawler method comprises:

[0008] Receiving a network request sent by a client, the network request comprising: a request parameter and an encrypted parameter; wherein the client is used for performing an encryption process on the request parameter according to a preset sorting rule and a preset non-standard hash function to obtain result data, taking the result data as the encrypted parameter, and performing confusion on the code of the encryption process to obtain a confused code;

[0009] Parsing the network request to obtain the request parameter and the encrypted parameter;

[0010] Performing an encryption process on the request parameter according to a preset sorting rule and a preset non-standard hash function to obtain result data, taking the result data as verification data;

[0011] Comparing the verification data and the encryption parameter to obtain a comparison result, and if the comparison result is the same, sending the data required by the network request to the client, and if the comparison result is different, not sending the data required by the network request to the client.

[0012] Optionally, the encryption process of the request parameter according to the preset sorting rule and the preset non-standard hash function obtains result data, including:

[0013] The request parameter is sorted according to the sorting rule and spliced into a string;

[0014] The string is input into the non-standard hash function to obtain result data.

[0015] Optionally, the result data obtained by inputting the string into the non-standard hash function includes:

[0016] The string is converted into a bit string;

[0017] The bit string is padded to obtain a padding value, and an additional length value used to represent the length of the input value is appended to the rear of the padding value to obtain an additional padding value;

[0018] The additional padding value is divided to obtain at least one data block, and each data block is divided and expanded to obtain a plurality of data groups;

[0019] A plurality of initialization magic numbers are defined to obtain a plurality of custom magic numbers, and each custom magic number is stored in a corresponding first buffer area. The magic number is a fixed-length data added in each first buffer area;

[0020] A plurality of constants are obtained according to a preset constant calculation function. The number of constants is the same as the number of data groups of each data block, and the constants have a one-to-one mapping relationship with the data groups of each data block;

[0021] The custom magic numbers, each data group of each data block, and the corresponding constants are input into a preset loop function to obtain a plurality of calculation values corresponding to the plurality of custom magic numbers. The preset loop function includes a loop right shift function;

[0022] Each custom magic number and the corresponding calculation value are added to obtain an addition operation value, and all addition operation values are spliced according to a preset second order to obtain a spliced operation value;

[0023] The middle value of the spliced operation value is selected as the result data.

[0024] Optionally, the preset constant calculation function is K i= int ( | sin ( i ) | * 4294967296 ), wherein, i is an integer selected from 1 to 80 in turn, K i The output value of the calculation function is a preset constant.

[0025] Optionally, the request parameter comprises a parameter name, and the sorting rule is to sort the parameter names in ascending order according to ASCII codes.

[0026] Optionally, the code of the encryption process is subjected to variable name obfuscation to obtain intermediate code.

[0027] The code of the encryption process is subjected to variable name obfuscation to obtain intermediate code.

[0028] The intermediate code is subjected to string extraction and encryption to obtain encrypted code.

[0029] The encrypted code is subjected to flattening processing to obtain flattened control code.

[0030] The flattened control code is injected with zombie code to obtain obfuscated code.

[0031] A method for anti-crawler, comprising:

[0032] Obtaining a request parameter;

[0033] According to a preset sorting rule and a preset non-standard hash function, the request parameter is subjected to an encryption process to obtain result data, the result data is taken as encrypted parameter, and the code of the encryption process is subjected to obfuscation to obtain obfuscated code.

[0034] Generating a network request according to the request parameter and the encrypted parameter;

[0035] Sending the network request to a server, so that the server is used for parsing the network request to obtain the request parameter and the encrypted parameter, according to a preset sorting rule and a preset non-standard hash function, the request parameter is subjected to an encryption process to obtain result data, the result data is taken as verification data, and a comparison result is obtained by comparing the verification data and the encrypted parameter.

[0036] In the case that the comparison result is the same, receiving return data required by the network request sent by the server.

[0037] An anti-crawler system, comprising:

[0038] A receiving module is configured to receive a network request sent by a client, wherein the network request comprises a request parameter and an encrypted parameter; and the client is configured to obtain result data by subjecting the request parameter to an encryption process according to a preset sorting rule and a preset non-standard hash function, take the result data as the encrypted parameter, and obtain obfuscated code by subjecting the code of the encryption process to obfuscation.

[0039] The analysis module is used for analyzing the network request to obtain a request parameter and an encryption parameter;

[0040] The encryption module is used for performing an encryption process on the request parameter according to a preset sorting rule and a preset non-standard hash function to obtain result data, and taking the result data as verification data;

[0041] The comparison module is used for comparing the verification data and the encryption parameter to obtain a comparison result, and if the comparison result is the same, sending data required to be acquired by the network request to the client, and if the comparison result is different, not sending the data required to be acquired by the network request to the client.

[0042] A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and wherein the processor implements the steps of the method when executing the computer program.

[0043] A computer readable storage medium, which stores a computer program, wherein the computer program is executed by a processor to implement the steps of the method.

[0044] In summary, the present application has the following advantages: the code of the encryption process of the client is obfuscated to obtain obfuscated code, so as to increase the difficulty of code reading, reduce the readability of the code, and increase the difficulty of reverse cracking of the crawler; the encryption parameter is obtained by performing the encryption process on the client, the verification data is obtained by performing the encryption process on the server, and then the encryption parameter is verified by the verification data to determine whether the network request sent by the client is legal, so as to improve the anti-crawler ability and effectively block the crawler attack. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 is the main flow diagram of the anti-crawler method of example one provided by the present application;

[0046] Figure 2 is the main flow diagram of the anti-crawler method of example two provided by the present application;

[0047] Figure 3 is the main module diagram of the anti-crawler system of example three provided by the present application;

[0048] Figure 4 is the internal structure diagram of the computer device in the embodiment of the present application. DETAILED DESCRIPTION

[0049] In order to make the objects, features and advantages of the present application more clear, the specific embodiments of the present application are described in detail below with reference to the drawings. The present application is illustrated by several embodiments in the drawings. However, the present application can be realized in many different forms and is not limited to the embodiments described herein.

[0050] In the present application, unless otherwise explicitly specified and limited, the terms "first", "second" are only used for description purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features limited by "first", "second" can explicitly or implicitly include one or more of the features.

[0051] The present application is described in detail below with reference to the drawings and embodiments.

[0052] Figure 1 is the main flowchart of the anti-crawler method of the first embodiment provided by the present application. As shown in Figure 1 the anti-crawler method of the first embodiment of the present application comprises:

[0053] Step 100, receiving a network request sent by a client, the network request comprising: a request parameter and an encrypted parameter; wherein the client is used to perform an encryption process on the request parameter according to a preset sorting rule and a preset non-standard hash function to obtain result data, take the result data as the encrypted parameter, and perform code obfuscation on the code of the encryption process to obtain obfuscated code;

[0054] Step 200, parsing the network request to obtain the request parameter and the encrypted parameter;

[0055] Step 300, performing an encryption process on the request parameter according to a preset sorting rule and a preset non-standard hash function to obtain result data, taking the result data as verification data;

[0056] Step 400, comparing the verification data and the encrypted parameter to obtain a comparison result, if the comparison result is the same, sending the data required to be acquired by the network request to the client, if the comparison result is different, not sending the data required to be acquired by the network request to the client.

[0057] In actual application, the same sorting rule and the same non-standard hash function are configured in advance in the client and the server, the client encrypts the request parameter according to the sorting rule and the non-standard hash function to obtain result data, and the result data is taken as encrypted parameter, the code of the encryption process is obfuscated to obtain obfuscated code, the client also generates a network request according to the request parameter and the encrypted parameter, and sends the network request to the server, so as to increase the difficulty of code reading, reduce the readability of the code, and increase the difficulty of reverse cracking of the crawler; after receiving the network request sent by the client, the server parses the network request to obtain the request parameter and the encrypted parameter, the server encrypts the request parameter according to the pre-configured sorting rule and the pre-configured non-standard hash function to obtain result data, and the result data is taken as verification data, then the verification data and the encrypted parameter are compared, so as to judge whether the network request is legal, that is, in the case that the verification data and the encrypted parameter are the same (the comparison result is the same), it is judged that the network request is legal, and the data required by the network request is sent to the client, in the case that the verification data and the encrypted parameter are different (the comparison result is different), it is judged that the network request is illegal, that is, the crawler request, at this time, the server does not send the data required by the network request to the client; the encrypted parameter is obtained by the encryption process in the client, the verification data is obtained by the encryption process in the server, and then the encrypted parameter is verified by the verification data to judge whether the network request sent by the client is legal, so as to improve the anti-crawler ability and effectively block the crawler attack.

[0058] Further, the result data obtained by the encryption process of the request parameter according to the pre-configured sorting rule and the pre-configured non-standard hash function comprises:

[0059] The request parameter is sorted according to the sorting rule, and then spliced into a string;

[0060] The string is input into the non-standard hash function to obtain the encrypted parameter.

[0061] In actual application, the client sorts the request parameter according to the sorting rule, splices the request parameter into a string according to the sorting order of the request parameter, and then inputs the string into the non-standard hash function to encrypt the string and obtain result data, and the result data is taken as encrypted parameter, and then a network request is generated according to the request parameter and the encrypted parameter, and the network request is sent to the server; the server receives the network request and parses to obtain the encrypted parameter and the request parameter, the server also sorts the request parameter according to the sorting rule, splices the request parameter into a string, inputs the string into the non-standard hash function to obtain result data, and takes the result data as verification data, so that the server compares the verification data with the encrypted parameter to verify the encrypted parameter and judge whether the network request is legal.

[0062] Further, the inputting the character string into the non-standard hash function to obtain result data comprises:

[0063] Converting the character string into a bit string; specifically, converting the input value into ASCII code according to the ASCII table, and then converting the ASCII code of the input value into binary to obtain a bit string; if the character string is "abc", the ASCII code obtained according to the ASCII table is 97 98 99, and the bit string obtained by converting 979899 into binary is 01100001 01100010 01100011; wherein the ASCII (American Standard Code for Information Interchange) is a set of computer coding system based on Latin letters, mainly used for displaying modern English and other Western European languages;

[0064] Padding the bit string to obtain a padded value, and appending an additional length value used to represent the length of the character string to the rear of the padded value to obtain an additional padded value; specifically, padding the bit string, that is, padding the bit string to make the data length of the bit string modulo 512 equal to 448, padding the highest bit with 1 and the remaining bits with 0; in the case of the bit string being 01100001 01100010 01100011, the padded value is 01100001 0110001001100011 10......0 with a total of 448 bits; the additional length value usually uses 64-bit data to represent the length of the character string; since abc occupies 3 bytes, that is, the data length is 24 bits, the additional length value corresponding to abc is 0......000011000 with a total of 64 bits; appending the additional length value to the padded value to obtain an additional padded value of 01100001 0110001001100011 10......0 00011000 with a total of 512 bits; the additional padded value is represented in hexadecimal; the additional padded value corresponding to abc is 61626380 0......0 00000018, wherein 110 0s are omitted;

[0065] The additional padding value is divided to obtain at least one data block, each data block is divided and expanded to obtain a plurality of data groups, and all data groups of each data block are arranged in a first preset order; in actual application, the additional padding value can be a multiple of 512 bits, that is, more than 512 bits, so the additional padding value is grouped according to 512 bits to obtain at least one data block, and all data blocks are recorded as M1, M2,..., Mn in order from high bit to low bit, in the case of an input value of 'abc', a data block is obtained as 61626380 0......0 00000018, and then the data block is divided according to 32 bits to obtain 16 data groups W1=61626380, W2=00000000,..., W16=00000018, according to the function Wi=Sn(W(i-3)⊕W(i-8)⊕W(i-14)⊕W(i-16)), 17≤i≤80, the data groups of the data block are expanded to obtain 80 data groups, ⊕ represents XOR, Sn represents cyclic left shift n bits, and if n is 1, W17=C2C4C700 is obtained according to the function Wi.

[0066] A plurality of custom magic numbers are defined to obtain a plurality of custom magic numbers, and each custom magic number is stored in a corresponding first buffer, the magic number being a fixed-length data added in each first buffer; specifically, the number of custom magic numbers is 5, denoted as H0, H1, H2, H3 and H4, the data length is 32 bits, the custom magic number is also expressed in hexadecimal, in addition, there are five second buffers, and the data stored on the five second buffers is denoted as A, B, C, D and E, respectively, the second buffer is used to save the intermediate calculation results in subsequent loop calculation, and initially A=H0, B=H1, C=H2, D=H3 and E=H4.

[0067] A plurality of constants are obtained according to a preset constant calculation function, the number of constants being the same as the number of data groups of each data block, and the constants having a one-to-one mapping relationship with the data groups of each data block;

[0068] The custom magic numbers, each data group of each data block and the corresponding constants are input into a preset loop function to obtain a plurality of calculation values corresponding to the plurality of custom magic numbers; wherein the preset loop function includes a loop right shift function; specifically, the loop right shift function includes Yx1(A) and Yx2(B), Y represents loop right shift, Yx1(A) represents that data A is looped right shifted by x1 bits, and Yx2(B) represents that data B is looped right shifted by x2 bits; if H0=67452302, A=H0, and x1=5, then Yx1(A)=33A2918, and the preset loop function is TEMP=Yx1(A)+fi(B,C,D)+E+Wi+K i, E = D, D = C, C = Yx2(B), B = A, A = TEMP, to achieve a loop calculation, in the case of multiple data blocks, the data groups of all data blocks of M1, M2,..., Mn are input to the loop function according to the order from high to low, and the final calculation value corresponding to multiple custom magic numbers is obtained after the last data group is calculated, that is, the final A, the final B, the final C, the final D and the final E are obtained, wherein, & represents an and operation, ~ represents a not operation, | represents an or operation, if B = EFCDAB87, C = 98BADCFB, D = 10325475, i = 1, then f1(B, C, D) = 98BA9CF3, K i represents the output value of the preset constant calculation function, and the complexity of data encryption is ensured through the calculation of the preset loop function. The constant has a one-to-one mapping relationship with the data groups corresponding to each data block, so that the calculation constant of the data group corresponding to each data block is different when the preset loop function is calculated on all data groups of each data block, further increasing the complexity of the calculation, and making the encryption effect better;

[0069] The custom magic numbers and the calculation values corresponding thereto are added to obtain addition operation values, and all the addition operation values are spliced according to a preset second order to obtain spliced operation values. Specifically, after obtaining the final calculation value, the custom magic numbers and the calculation values corresponding thereto are added, that is, H0 and the final A are added, H1 and the final B are added, H2 and the final C are added, H3 and the final D are added, and H4 and the final E are added, to obtain the final H0, the final H1, the final H2, the final H3 and the final H4, that is, the addition operation values are obtained. Then, the final H0, the final H1, the final H2, the final H3 and the final H4 are spliced in the order of H0 to H4 to obtain the spliced operation values, and the spliced operation values are 160-bit data. Since the spliced operation values of the present application are expressed in hexadecimal, the spliced operation values are also 40 hexadecimal numbers.

[0070] The middle value of the spliced operation values is selected as the result data. In the present application, the middle 20 hexadecimal numbers of the spliced operation values are selected as the result data, so that the characteristics of the hash function can be better hidden, and the anti-crawler difficulty is increased.

[0071] Further, the preset constant calculation function is K i = int(|sin(i)| * 4294967296), wherein i is an integer selected from 1 to 80 in turn, K iThe preset constant calculation function is represented as the integral of the product of the absolute value of sin(i) and 4294967296. The constant corresponding to each data group of the data block is calculated by the preset constant calculation function, so that the constants corresponding to each data group of the data block are different, further increasing the complexity of the calculation; in the case of i=1, K1=74957514, expressed in hexadecimal as 0477C2CA, if A=00000001, B=00000002, C=00000003, D=00000004, E=00000005, W1=61626380, x1=5, x2=30, then Y5(A)=00000000, f1(B,C,D)=00000006, the first cycle calculation result of the cycle function is TEMP=65DA2655, E=00000004, D=00000003, C=00000000, B=00000001, A=65DA2655, the cycle function will perform 80 operations on all data blocks in turn until the 80th operation on the last data block is completed, and the final operation result is obtained.

[0072] Further, the request parameter includes a parameter name, and the sorting rule is to sort the ASCII codes of the parameter names in ascending order to ensure that the same network request input string of the non-standard hash function is unique.

[0073] Further, the code of the encryption process is obfuscated to obtain an obfuscated code, including:

[0074] The code of the encryption process is subjected to variable name obfuscation to obtain intermediate code.

[0075] The intermediate code is subjected to string extraction and encryption to obtain encrypted code.

[0076] The encrypted code is subjected to flattening processing to obtain flattened control code.

[0077] The flattened control code is injected with a zombie code to obtain an obfuscated code.

[0078] In practical application, the variable name confusion is to confuse the variable name, extract and encrypt the intermediate code, thereby improving the reading difficulty of the code of the client, performing the flattening processing on the encrypted code, obtaining the flattened control code, and recombining the code flow in the encrypted code into a code flow of a switch case structure which is difficult to understand, so as to improve the reading difficulty. The switch case is a statement in the code flow control, and the zombie code represents the code which is never called and executed in the code running process. Specifically, a zombie code library can be established, a plurality of zombie codes are selected from the zombie code library and injected into the flattened control code to obtain the confusion code. In addition, the anti-debugging processing can be performed on the confusion code, so as to strengthen the protection of the code of the encryption process of the client.

[0079] The anti-crawler method of the present application confuses the code of the encryption process of the client to obtain the confusion code, so as to increase the reading difficulty of the code and reduce the readability of the code, thereby increasing the difficulty of the reverse cracking of the crawler. The encryption parameter is obtained by performing the encryption process on the client, the verification data is obtained by performing the encryption process on the server, and then the encryption parameter is verified by the verification data to determine whether the network request sent by the client is legal, so as to improve the anti-crawler ability and effectively block the crawler attack.

[0080] Figure 2 Fig. 1 is a main flowchart of the anti-crawler method of the embodiment two provided by the present application. As shown in Fig. 1, the anti-crawler method of the embodiment two of the present application comprises the following steps. Figure 2

[0081] Step 10, obtaining the request parameter;

[0082] Step 20, performing the encryption process on the request parameter according to the preset sorting rule and the preset non-standard hash function to obtain the result data, taking the result data as the encryption parameter, and confusing the code of the encryption process to obtain the confusion code;

[0083] Step 30, generating the network request according to the request parameter and the encryption parameter;

[0084] Step 40, sending the network request to the server, so that the server is used to parse the network request to obtain the request parameter and the encryption parameter, performing the encryption process on the request parameter according to the preset sorting rule and the preset non-standard hash function to obtain the result data, taking the result data as the verification data, and comparing the verification data and the encryption parameter to obtain the comparison result;

[0085] Step 50, in the case that the comparison result is the same, receiving the return data required by the network request sent by the server.

[0086] Figure 3 ​is the main module schematic diagram of the anti-crawler system of the third embodiment provided by the present application. As shown in Figure 1 The anti-crawler system of the third embodiment of the present application comprises:

[0087] The receiving module 1 is configured to receive a network request sent by a client, wherein the network request comprises a request parameter and an encrypted parameter; wherein the client is configured to perform an encryption process on the request parameter according to a preset sorting rule and a preset non-standard hash function to obtain result data, use the result data as the encrypted parameter, and perform code obfuscation on the code of the encryption process to obtain an obfuscated code.

[0088] The parsing module 2 is configured to parse the network request to obtain the request parameter and the encrypted parameter.

[0089] The encryption module 3 is configured to perform an encryption process on the request parameter according to a preset sorting rule and a preset non-standard hash function to obtain result data, and use the result data as verification data.

[0090] The comparison module 4 is configured to compare the verification data and the encrypted parameter to obtain a comparison result, and if the comparison result is the same, send data required to be acquired by the network request to the client, and if the comparison result is different, do not send the data required to be acquired by the network request to the client.

[0091] The specific limitation of the anti-crawler system can refer to the limitation of the anti-crawler method of the first embodiment, which will not be repeated here. The above anti-crawler system can be realized by software, hardware and their combination. The above modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory in the computer device in software form, so as to call and execute the operations of the above modules by the processor.

[0092] In one embodiment, a computer device is provided, which can be a server, and the internal structure diagram thereof can be as shown in Figure 4 The computer device comprises a processor, a memory, a network interface and a database connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is configured to communicate with an external terminal through a network connection. The computer program is executed by the processor to implement an anti-crawler method.

[0093] Those skilled in the art can understand that, Figure 4The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0094] In one embodiment, a computer device is provided, comprising a memory and a processor, the memory storing a computer program, and the processor implementing the following steps when executing the computer program:

[0095] receiving a network request sent by a client, the network request comprising: a request parameter and an encrypted parameter; wherein the client is configured to perform an encryption process on the request parameter according to a preset sorting rule and a preset non-standard hash function to obtain result data, take the result data as the encrypted parameter, and perform obfuscation on the code of the encryption process to obtain obfuscated code;

[0096] parsing the network request to obtain the request parameter and the encrypted parameter;

[0097] performing an encryption process on the request parameter according to a preset sorting rule and a preset non-standard hash function to obtain result data, taking the result data as verification data;

[0098] comparing the verification data and the encrypted parameter to obtain a comparison result, and if the comparison result is the same, sending data required to be acquired by the network request to the client, and if the comparison result is different, not sending the data required to be acquired by the network request to the client.

[0099] In one embodiment, the encryption process on the request parameter according to the preset sorting rule and the preset non-standard hash function to obtain result data comprises:

[0100] sorting the request parameter according to the sorting rule and splicing it into a string;

[0101] inputting the string into the non-standard hash function to obtain result data.

[0102] In one embodiment, the inputting the string into the non-standard hash function to obtain result data comprises:

[0103] converting the string into a bit string;

[0104] padding the bit string to obtain a padding value, appending an additional length value used to represent the length of the input value to the back of the padding value to obtain an additional padding value;

[0105] dividing the additional padding value to obtain at least one data block, and dividing and expanding each data block to obtain a plurality of data groups;

[0106] The plurality of initialization magic numbers are customized to obtain a plurality of customized magic numbers, each of the customized magic numbers is stored in a corresponding first buffer, and the magic number is a fixed-length data added in each first buffer;

[0107] A plurality of constants are calculated according to a preset constant calculation function, the number of the constants is the same as the number of data groups of each data block, and the constants have a one-to-one mapping relationship with the data groups of each data block;

[0108] The customized magic numbers, each data group of each data block and the corresponding constants are input into a preset loop function to obtain a plurality of calculation values corresponding to the plurality of customized magic numbers; wherein the preset loop function includes a loop right shift function;

[0109] Each of the customized magic numbers and the corresponding calculation value are added to obtain an addition operation value, and all the addition operation values are spliced according to a preset second order to obtain a spliced operation value;

[0110] The middle value of the spliced operation value is selected as the result data.

[0111] In an embodiment, the preset constant calculation function is K i = int(sin(i)*4294967296), wherein i is an integer selected from 1 to 80 in turn, and K i is an output value of the preset constant calculation function.

[0112] In an embodiment, the request parameter includes a parameter name, and the sorting rule is to sort the parameter names in ascending order according to ASCII codes.

[0113] In an embodiment, the code of the encryption process is obfuscated to obtain obfuscated code, including:

[0114] The code of the encryption process is subjected to variable name obfuscation to obtain intermediate code;

[0115] The intermediate code is subjected to string extraction and encryption to obtain encrypted code;

[0116] The encrypted code is subjected to flattening processing to obtain flattened control code;

[0117] The flattened control code is injected with zombie code to obtain obfuscated code.

[0118] In an embodiment, including:

[0119] Obtaining a request parameter;

[0120] According to the preset sorting rule and the preset non-standard hash function, the request parameter is subjected to an encryption process to obtain result data, the result data is taken as an encryption parameter, and the code of the encryption process is subjected to obfuscation to obtain obfuscated code;

[0121] According to the request parameter and the encryption parameter, a network request is generated;

[0122] The network request is sent to a server, so that the server is used for parsing the network request to obtain the request parameter and the encryption parameter, according to the preset sorting rule and the preset non-standard hash function, the request parameter is subjected to an encryption process to obtain result data, the result data is taken as verification data, and the comparison result is obtained by comparing the verification data and the encryption parameter;

[0123] In the case that the comparison result is the same, the return data required to be obtained by the network request sent by the server is received.

[0124] In one embodiment, a computer readable storage medium is provided, and the computer readable storage medium has a computer program stored thereon, and the computer program is executed by a processor to implement the following steps:

[0125] A network request sent by a client is received, and the network request includes a request parameter and an encryption parameter; wherein the client is used for obtaining result data by subjecting the request parameter to an encryption process according to a preset sorting rule and a preset non-standard hash function, taking the result data as the encryption parameter, and obtaining obfuscated code by subjecting the code of the encryption process to obfuscation;

[0126] The network request is parsed to obtain the request parameter and the encryption parameter;

[0127] According to the preset sorting rule and the preset non-standard hash function, the request parameter is subjected to an encryption process to obtain result data, and the result data is taken as verification data;

[0128] The comparison result is obtained by comparing the verification data and the encryption parameter, if the comparison result is the same, data required to be obtained by the network request is sent to the client, and if the comparison result is different, data required to be obtained by the network request is not sent to the client.

[0129] In one embodiment, the result data obtained by subjecting the request parameter to the encryption process according to the preset sorting rule and the preset non-standard hash function includes:

[0130] According to the sorting rule, the request parameter is sorted and spliced into a string;

[0131] The string is input into the non-standard hash function to obtain result data.

[0132] In one embodiment, the inputting the character string into the non-standard hash function to obtain result data comprises:

[0133] converting the character string into a bit string;

[0134] padding the bit string to obtain a padding value, appending an additional length value for representing the length of the input value to the rear of the padding value to obtain an additional padding value;

[0135] dividing the additional padding value to obtain at least one data block, and dividing and extending each data block to obtain a plurality of data groups;

[0136] defining a plurality of initialization magic numbers to obtain a plurality of custom magic numbers, and storing each custom magic number in a corresponding first buffer, the magic number being a fixed-length data added in each first buffer;

[0137] obtaining a plurality of constants according to a preset constant calculation function, the number of the constants being the same as the number of data groups of each data block, and the constants having a one-to-one mapping relationship with the data groups of each data block;

[0138] inputting the custom magic numbers, each data group of each data block, and the corresponding constants into a preset loop function to obtain a plurality of calculation values corresponding to the plurality of custom magic numbers, wherein the preset loop function comprises a loop right shift function;

[0139] adding each custom magic number and the corresponding calculation value to obtain an addition operation value, and splicing all addition operation values according to a preset second order to obtain a spliced operation value;

[0140] selecting a middle value of the spliced operation value as the result data.

[0141] In one embodiment, the preset constant calculation function is K i = int(|sin(i)|*4294967296), wherein i is an integer selected from 1 to 80 in sequence, and K i is an output value of the preset constant calculation function.

[0142] In one embodiment, the request parameter comprises a parameter name, and the sorting rule is to sort the parameter names in ascending order according to their ASCII codes.

[0143] In one embodiment, the code of the encryption process is obfuscated to obtain an obfuscated code, comprising:

[0144] variable name obfuscation is performed on the code of the encryption process to obtain an intermediate code;

[0145] String extraction and encryption are performed on the intermediate code to obtain encrypted code;

[0146] Flattening processing is performed on the encrypted code to obtain flattened control code;

[0147] The flattened control code is injected with zombie code to obtain obfuscated code.

[0148] In one embodiment, comprising:

[0149] Obtaining a request parameter;

[0150] According to the preset sorting rule and the preset non-standard hash function, the request parameter is encrypted to obtain result data, and the result data is used as an encrypted parameter. The code of the encryption process is obfuscated to obtain obfuscated code;

[0151] Generating a network request according to the request parameter and the encrypted parameter;

[0152] The network request is sent to the server, so that the server is used to parse the network request to obtain the request parameter and the encrypted parameter, according to the preset sorting rule and the preset non-standard hash function, the request parameter is encrypted to obtain result data, and the result data is used as verification data. The comparison result is obtained by comparing the verification data and the encrypted parameter;

[0153] In the case of the same comparison result, receiving the return data required by the network request sent by the server.

[0154] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, storage, database or other medium used in the embodiments provided in the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0155] The technical features of the above embodiments can be combined in any manner. In order to make the description simple, not all possible combinations of the technical features in the above embodiments are described, however, as long as the combinations of the technical features do not contradict, they should be considered as the scope of the present disclosure.

Claims

1. An anti-crawler method, characterized in that, include: The client receives a network request from a client, the network request including: request parameters and encryption parameters; wherein, the client is used to encrypt the request parameters according to a preset sorting rule and a preset non-standard hash function to obtain result data, use the result data as encryption parameters, and obfuscate the code of the encryption process to obtain obfuscated code; Parse the network request to obtain the request parameters and encryption parameters; The request parameters are encrypted according to a preset sorting rule and a preset non-standard hash function to obtain the result data, which is then used as the verification data. The verification data and encryption parameters are compared to obtain a comparison result. If the comparison result is the same, the data required for the network request is sent to the client. If the comparison result is different, the data required for the network request is not sent to the client. The process of encrypting the request parameters according to a preset sorting rule and a preset non-standard hash function to obtain the result data includes: The request parameters are sorted according to the sorting rules and then concatenated into a string. The string is input into the non-standard hash function to obtain the result data; The step of inputting the string into the non-standard hash function to obtain the result data includes: Convert the string into a bit string; The bit string is padded to obtain a padding value, and an additional length value used to represent the length of the input value is appended to the padding value to obtain an additional padding value. The additional padding value is divided to obtain at least one data block, and each data block is divided and expanded to obtain multiple data groups; Multiple custom magic numbers are obtained by defining multiple initial magic numbers, and each custom magic number is stored in a corresponding first buffer. The magic number is a fixed-length data added to each first buffer. Multiple constants are obtained according to a preset constant calculation function. The number of constants is the same as the number of data groups in each data block, and the constants have a one-to-one mapping relationship with the data groups in each data block. The custom magic number, each data group of each data block, and their corresponding constants are input into a preset loop function to obtain multiple calculated values ​​that correspond one-to-one with the multiple custom magic numbers; wherein the preset loop function includes: a loop right shift function; Each of the custom magic numbers and its corresponding calculated value is added together to obtain the sum value. All the sum values ​​are then concatenated in a preset second order to obtain the concatenated value. The middle value of the concatenated values ​​is selected as the result data.

2. The anti-crawler method according to claim 1, characterized in that, The preset constant calculation function is K. i =int(|sin(i)|*4294967296), where i is an integer selected sequentially from 1 to 80, K i The output value of the function is calculated based on preset constants.

3. The anti-crawler method according to claim 1, characterized in that, The request parameters include: parameter name; the sorting rule is to sort the parameter names in ascending order of their ASCII codes.

4. The anti-crawler method according to claim 1, characterized in that, The obfuscated code obtained by obfuscating the code of the encryption process includes: Obfuscate the variable names in the encryption process code to obtain intermediate code; The intermediate code is extracted and encrypted to obtain encrypted code; The encrypted code is flattened to obtain flattened control code; Zombie code is injected into the flattening control code to obtain obfuscated code.

5. An anti-crawler system, characterized in that, include: The receiving module is used to receive network requests sent by a client. The network request includes request parameters and encryption parameters. The client is used to encrypt the request parameters according to a preset sorting rule and a preset non-standard hash function to obtain result data, use the result data as encryption parameters, and obfuscate the code of the encryption process to obtain obfuscated code. The parsing module is used to parse the network request to obtain request parameters and encryption parameters; An encryption module is used to encrypt request parameters according to a preset sorting rule and a preset non-standard hash function to obtain result data. This includes: sorting the request parameters according to the sorting rule and concatenating them into a string; converting the string into a bit string; padding the bit string to obtain a padding value, and appending an additional length value representing the length of the input value to the padding value to obtain an additional padding value; dividing the additional padding value to obtain at least one data block, and dividing and expanding each data block to obtain multiple data groups; defining multiple initial magic numbers to obtain multiple custom magic numbers, and storing each custom magic number in a corresponding first buffer, wherein the magic number is obtained by adding a specific initial magic number to each first buffer. A fixed-length data is input; multiple constants are obtained according to a preset constant calculation function, the number of constants being the same as the number of data groups in each data block, and the constants having a one-to-one mapping relationship with the data groups in each data block; the custom magic number, each data group in each data block, and their corresponding constants are input into a preset loop function to obtain multiple calculated values ​​corresponding one-to-one with the multiple custom magic numbers; wherein the preset loop function includes: a loop right shift function; each of the custom magic numbers and their corresponding calculated values ​​are added to obtain an sum operation value, and all the sum operation values ​​are concatenated according to a preset second order to obtain a concatenated operation value; the middle value of the concatenated operation value is selected as the result data; the result data is used as verification data; The comparison module is used to compare the verification data and the encryption parameters to obtain a comparison result. If the comparison results are the same, the module sends the data required for the network request to the client. If the comparison results are different, the module does not send the data required for the network request to the client.

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 4.

Images