Anti-crawler processing method, device, electronic device, and storage medium

Abstract

The present application discloses an anti-crawler processing method, apparatus, electronic device, and storage medium. The method includes: receiving an external request; according to the external request, before returning the request result and / or after receiving the external request, transforming the entry ID in the request result obtained by the external request, where the entry ID includes a replacement ID, and there is a mapping relationship between the replacement ID and the original ID. Anti-crawling is achieved based on dynamic ID transformation in the present application. The present application can be used to prevent web crawlers in the financial field.

Description

Technical Field

[0001] This application relates to the field of network security technology, and in particular, to an anti-crawler processing method, apparatus, electronic device, and storage medium. Background Art

[0002] Web crawlers have now become an important means of network data collection. However, with the development of web crawlers, problems such as data security and customer privacy have become increasingly prominent.

[0003] In the financial field, information exposed on the network should have stronger anti-crawler capabilities. Summary of the Invention

[0004] Embodiments of this application provide an anti-crawler processing method, apparatus, electronic device, and storage medium to achieve anti-crawler through dynamic ID transformation.

[0005] Embodiments of this application adopt the following technical solutions:

[0006] In a first aspect, embodiments of this application provide an anti-crawler processing method, where the method includes:

[0007] Receive an external request;

[0008] According to the external request, before returning the request result and / or after receiving the external request, transform the entry ID in the request result obtained from the external request, where the entry ID includes a replacement ID, and there is a mapping relationship between the replacement ID and the original ID.

[0009] In some embodiments, there is a mapping relationship between multiple replacement IDs and one original ID. The step of, according to the external request, before returning the request result and / or after receiving the external request, transforming the entry ID in the request result obtained from the external request includes:

[0010] According to an external call request, before returning the call request result, transform the entry ID of the return parameter into a replacement ID and isolate the original ID internally;

[0011] and / or,

[0012] According to an external call request, after receiving the external request, transform the entry IDs of all requests of the external call into replacement IDs and isolate the original ID internally.

[0013] In some embodiments, the step of, according to an external call request, before returning the call request result, transforming the entry ID of the return parameter into a replacement ID and isolating the original ID internally includes:

[0014] Judge the type of potential crawler external call request;

[0015] In the case where it is determined that the type of the potential crawler external call request is a query type, the original ID is internally used for business query during the business processing;

[0016] After the business processing is completed and before returning the call request result, the original ID of the returned data is transformed into a replacement ID and then returned.

[0017] In some embodiments, according to the external call request, after receiving the external request, transforming the entry ID of all requests of the external call into a replacement ID and internally isolating the original ID includes:

[0018] In the case where it is determined that the type of the potential crawler external call request is an operation type, the replacement ID is transformed into the original ID and then the original ID is internally used for business operations during the business processing.

[0019] In some embodiments, the method further includes:

[0020] In the case where it is determined that the type of the potential crawler external call request is to query other content, the replacement ID is transformed into the original ID and then the original ID is internally used for business operations during the business processing;

[0021] The original ID is internally used for business query during the business processing;

[0022] After the business processing is completed and before returning the call request result, the original ID of the returned data is transformed into a replacement ID and then returned.

[0023] In some embodiments, transforming the entry ID in the request result obtained by the external request includes:

[0024] Configuring the base numbers n, m of the replacement ID and the original ID;

[0025] Randomly selecting a value s in the base number of the replacement ID;

[0026] According to the algorithm corresponding to the base number, the overlap degree k, where k represents how many bits are taken for each decryption calculation;

[0027] According to the overlap degree k, generating a multi-digit number z, and using the value s as the highest digit and taking 0 for the other digits;

[0028] Taking the remainder of the value with respect to m, subtracting this remainder from the first digit of the original ID + a * m, and after converting to base n, filling the last few digits of the multi-digit number z;

[0029] Take the least significant bit of the padded multi - digit number z as the most significant bit of the new multi - digit number, and repeat the above steps to generate a new multi - digit number until all the original ID digits are calculated;

[0030] Overlap and connect all the multi - digit numbers k end - to - end to obtain the transformed replacement ID, where n, m, and k are set values, and s and a are random values.

[0031] In some embodiments, transforming the entry ID in the request result obtained by the external request further includes:

[0032] According to the pre - set number of digits k of the multi - digit number, overlap and intercept the replacement ID into multiple k - digit numbers;

[0033] Obtain a specific value according to the set replacement base number n, take the remainder of the specific value modulo the target function base m, and splice the remainders to obtain the original ID.

[0034] In a second aspect, an embodiment of the present application further provides an anti - crawler processing device, where the device includes:

[0035] A receiving module, configured to receive an external request;

[0036] A dynamic transformation module, configured to transform the entry ID in the request result obtained by the external request before returning the request result and / or after receiving the external request according to the external request, where the entry ID includes a replacement ID, and there is a mapping relationship between the replacement ID and an original ID.

[0037] In a third aspect, an embodiment of the present application further provides an electronic device, including: a processor; and a memory arranged to store computer - executable instructions, where the executable instructions, when executed, cause the processor to execute the above - mentioned method.

[0038] In a fourth aspect, an embodiment of the present application further provides a computer - readable storage medium, where the computer - readable storage medium stores one or more programs, and when the one or more programs are executed by an electronic device including a plurality of application programs, the electronic device is caused to execute the above - mentioned method.

[0039] The above - mentioned at least one technical solution adopted in the embodiments of the present application can achieve the following beneficial effects: First, receive an external request, and then, according to the external request, transform the entry ID in the request result obtained by the external request before returning the request result and / or after receiving the external request. Since the entry ID includes a replacement ID, and there is a mapping relationship between the replacement ID and the original ID, it is ensured that the replacement IDs generated from the same original ID are one - to - many and unstable, and the replacement IDs obtained for each request of the same ID are not necessarily the same. Description of the Drawings

[0040] The accompanying drawings described herein are used to provide a further understanding of the present application and form a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation of the present application. In the drawings:

[0041] Figure 1 It is a schematic flowchart of the anti-crawler processing method in an embodiment of the present application;

[0042] Figure 2 It is a schematic diagram of the implementation principle of the anti-crawler processing method in an embodiment of the present application;

[0043] Figure 3(a) is a schematic diagram of the implementation principle of a type of service request of the anti-crawler processing method in an embodiment of the present application;

[0044] Figure 3(b) is a schematic diagram of the implementation principle of another type of service request of the anti-crawler processing method in an embodiment of the present application;

[0045] Figure 3(c) is a schematic diagram of the implementation principle of yet another type of service request of the anti-crawler processing method in an embodiment of the present application;

[0046] Figure 4 It is a schematic structural diagram of the anti-crawler processing device in an embodiment of the present application;

[0047] Figure 5 It is a schematic structural diagram of an electronic device in an embodiment of the present application. Detailed implementation manners

[0048] To make the objectives, technical solutions, and advantages of the present application clearer, the technical solutions of the present application will be clearly and completely described below in conjunction with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts shall fall within the protection scope of the present application.

[0049] When the inventor was conducting research, it was found that the anti-crawler solutions in the related technologies mainly fall into three categories: The first category is the verification of request data, such as request headers, cookies, request IPs, verification codes, etc. This type of method is easily bypassed by simulation or may cause collateral damage. The second category is to follow the reboot specification and specify the search logic for crawlers according to the default specification during website construction. However, this solution can only restrict crawlers of regular manufacturers, and this specification has no restrictive effect on irregular crawlers. The third category is to use encryption technology to transform and encrypt IDs, data, etc., thereby affecting the readability of the data captured by crawlers. This solution can solve the problem of crawler parsing to a certain extent. However, since the encryption technology commonly used is still the one-to-one method of plaintext and ciphertext, it has little impact on data statistics, and the statistical significance still exists.

[0050] In view of the above deficiencies, the anti-crawler processing method in the embodiments of the present application, based on the third category of solutions, proposes a one-to-many dynamic ID transformation scheme, so that the ID obtained for each request of the same project is different, but is the same for the background server. Thus, it confuses crawlers, hides customer information, and makes it impossible for crawlers to analyze the request parameters for the next round well, creating a trap of accessing the same data multiple times, and preventing commercial data from being crawled and analyzed by crawlers.

[0051] Different from most of the related technologies that are designed in terms of whether they can be crawled or whether the data is readable after being crawled. The solution in the embodiments of the present application, while ensuring that it is difficult to identify and use after being crawled, adds a one-to-many confusion of dynamic IDs, so that the crawler itself cannot identify the fixed project information, thereby affecting the generation of the next round of crawling, and causing a large amount of garbage data in the crawling results of the crawler. It not only affects readability, but also makes the statistical significance of the crawling results lost.

[0052] The following will describe in detail the technical solutions provided by the embodiments of the present application with reference to the accompanying drawings.

[0053] The embodiments of the present application provide an anti-crawler processing method, as Figure 1 shown in the schematic flowchart of the anti-crawler processing method in the embodiments of the present application. The method at least includes the following steps S110 to step S120:

[0054] Step S110, receive an external request.

[0055] As Figure 2 shown, an "external request" is a request relative to the internal business system and database. An external request may include, but is not limited to, a separate query request, an operation request, or other query requests (based on a separate query), etc. Usually, a web crawler may be carried in the external request to collect and count relevant business data in the internal business system.

[0056] It can be understood that for all external requests, the transformation of the entry ID will be performed, and the replaced entry ID will be returned.

[0057] Step S120, according to the external request, before returning the request result and / or after receiving the external request, transform the entry ID in the request result obtained by the external request, where the entry ID includes a replacement ID, and there is a mapping relationship between the replacement ID and the original ID.

[0058] The "entry ID" includes, but is not limited to, product IDs, project IDs, etc. on the external request page. Usually, after being collected by web crawlers, the business form or product characteristics can be obtained through statistical analysis. Therefore, it is necessary to continuously perform dynamic transformation on the "entry ID", that is, use the replacement ID to replace it, and the real ID is not exposed. Since the entry ID includes the replacement ID, and there is a mapping relationship between the replacement ID and the original ID, replacement IDs can be continuously generated to replace the entry ID.

[0059] Please refer to Figure 2 , the external requests are divided into different request types. Before returning the request result, transform the entry ID in the request result obtained by the external request. At the same time, after receiving the external request, transform the entry ID in the request result obtained by the external request.

[0060] In addition, since an ID transformation component is introduced at the request entry, all requests and return parameters of external calls are replaced, isolating the ID systems inside and outside the system. The real ID is made invisible to external calls, ensuring that the crawler system cannot crawl real and valid information through the call of external interfaces.

[0061] Through the above method, the dynamic one-to-many confusion items of the ID are increased, so that the crawler itself cannot recognize the fixed item information, affecting the generation of the next round of crawling, and causing a large amount of garbage data in the crawler's crawling results. It not only affects readability, but also makes the statistical significance of the crawling results lost.

[0062] Through the above method, before returning the request result and / or after receiving the external request, transform the entry ID in the request result obtained by the external request. When the crawler wants to obtain the list to be crawled from the request result, the content obtained is unstable, thus affecting data readability. And the same piece of data may return multiple results, and the crawler cannot perform deduplication well, generating a large amount of random garbage data. As a result, the statistical results of the crawled data are also unavailable.

[0063] In one embodiment of the present application, there is a mapping relationship between multiple said replacement IDs and an original ID. Transforming the entry ID in the request result obtained by the external request before and / or after receiving the external request includes: according to an external call request, before returning the call request result, transforming the entry ID of the return parameter into a replacement ID and isolating the original ID internally; and / or, according to an external call request, after receiving the external request, transforming the entry IDs of all requests for external calls into replacement IDs and isolating the original ID internally.

[0064] One case is transformation before returning the request result: according to an external call request, before returning the call request result, transforming the entry ID of the return parameter into a replacement ID and isolating the original ID internally.

[0065] Another case is replacement after receiving the external request: according to an external call request, after receiving the external request, transforming the entry IDs of all requests for external calls into replacement IDs and isolating the original ID internally.

[0066] It should be noted that since there may be multiple or a large number of external requests, the above two cases may occur separately or alternately.

[0067] In one embodiment of the present application, according to an external call request, before returning the call request result, transforming the entry ID of the return parameter into a replacement ID and isolating the original ID internally includes: determining the type of a potential crawler external call request; in the case where the type of the potential crawler external call request is a query type, using the original ID for business query internally during the business processing; after the business processing is completed and before returning the call request result, transforming the original ID of the return data into a replacement ID and then returning it.

[0068] It should be noted that the normal logic for processing external requests is to directly reach the business system, and the business system will operate according to the business logic. As shown in Figure 3(a), by setting up an ID transformation module, the entry ID will be transformed after receiving the external request and before returning the data.

[0069] Please continue to refer to Figure 3(a). Specifically, in implementation, according to different request types, when the request is a single query type request: when the request (which may be a crawler request) reaches the business system, the business system processes it, and the real ID in the business system is used throughout the processing. After the business processing is completed and before returning the data, the real ID is transmitted to the ID transformation module to obtain the transformed ID, and the original ID of the return data is replaced and then returned to the customer.

[0070] It can be understood that for a "single query type request", it means that the external request (by the crawler) only performs a query request operation and does not perform other requests.

[0071] Furthermore, when the next external request is received (at this time, it is already a request with the replaced ID), the returned ID is still the transformed ID, and the internal real ID is never exposed to the external interface. It should be noted that for different external requests, the transformed IDs returned each time may be the same or different, and are randomly generated, which can affect the statistical results of the crawled data, making the statistical results unavailable.

[0072] In an embodiment of the present application, the method of transforming the entry ID of all requests for external calls into a replacement ID and isolating the original ID internally according to the external call request after receiving the external request includes: in the case where it is determined that the type of the potential crawler external call request is an operation type, transforming the replacement ID into the original ID and then internally using the original ID for business operations during the business processing process.

[0073] It can be understood that for an "operation type request", it means that the external request (by the crawler) performs various operation type request operations, such as clicking, dragging, double-clicking, exiting, etc.

[0074] Please refer to FIG. 3(b). For an operation type request, since the ID returned to the customer in the early stage is the replaced ID, when the business system receives the request, it needs to first call the ID transformation module to obtain the real business ID using the transformed ID, and then use the real business ID to replace the ID in the request for business operations.

[0075] In an embodiment of the present application, the method further includes: in the case where it is determined that the type of the potential crawler external call request is to query other content, transforming the replacement ID into the original ID and then internally using the original ID for business operations during the business processing process; internally using the original ID for business queries during the business processing process; after the business processing is completed and before returning the result of the call request, transforming the original ID of the returned data into the replacement ID and then returning it.

[0076] It can be understood that for a "query other content" request, it means that the external request (by the crawler) performs multiple query request operations and does not perform operations other than queries.

[0077] Please refer to FIG. 3(c) for the scenario of obtaining query other content according to the ID (which is also a scenario often used by crawlers). First, a request with a replacement ID is received. The business system obtains the real ID through the ID transformation module for query operations, and then the query result needs to call the transformation module again to replace the ID and then return the data.

[0078] In one embodiment of the present application, transforming the entry ID in the request result obtained from the external request includes: configuring the base numbers n and m of the replacement ID and the original ID; randomly selecting a value s in the base number of the replacement ID; according to the algorithm corresponding to the base number, the replacement overlap k, where k represents how many bits are taken for each decryption calculation; generating a multi-digit number z according to the overlap k, and taking the value s as the highest bit and the other bits as 0; taking the remainder of the value with respect to m, subtracting this remainder from the first bit of the original ID + a*m, converting it to base n, and filling the last few bits of the multi-digit number z; taking the lowest bit of the filled multi-digit number z as the highest bit of the new multi-digit number, repeating the above steps to generate a new multi-digit number until all the bits of the original ID are calculated; connecting the last k digits of all the multi-digit numbers end to end in an overlapping manner to obtain the transformed replacement ID, where n, m, and k are set values, and s and a are random values.

[0079] Specifically, the logic for generating the transformed ID ensures that the replacement ID generated from the same original ID is one-to-many and unstable. The replacement ID obtained for each request of the same ID is not necessarily the same. However, the replacement ID can be restored to a definite real business ID. This algorithm is performed in the background and is invisible to the user. Therefore, when the crawler wants to obtain the list to be crawled from the request result, the content obtained is unstable, thus affecting data readability. And the same piece of data may return multiple results, and the crawler cannot perform deduplication well, generating a large amount of random garbage data. As a result, the statistical results of the crawled data are also unavailable.

[0080] Exemplarily, the specific algorithm steps for generating the transformed ID are as follows:

[0081] For the process of generating the replacement ID from the original ID (assuming the original ID is: 2023):

[0082] Step S1, configure the base numbers of the replacement ID and the original ID, marked as n and m respectively. (For example, the replacement ID is n = hexadecimal, and the original ID is m = decimal).

[0083] Step S2, randomly select a value s in the base number of the replacement ID (taking hexadecimal as an example, any one of 0 - F can be selected. Assume that s = C is selected here).

[0084] Step S3, configure the appropriate algorithm replacement overlap k according to the base selected in Step S1 (when n > m, it is recommended to select 2; when n <= m, it is recommended to select 3). K represents how many bits are taken for each decryption calculation. (Select 2 here)

[0085] Step S4: Generate a multi-digit number z, where the number of digits is derived from the K value in step S3. Use the s value obtained in step S2 as the highest digit, and set the other digits to 0 (the example is C0).

[0086] Step S5: Take the remainder of the value generated in step S4 with respect to m (C0 = 192, the remainder when divided by 10 is 2). Then subtract this remainder from the sum of the first digit of the original ID and a*m, and after converting to base n, fill the lower digits of the multi-digit number generated in step S4. Here, a is a random number, and its range is between 1 and [floor(n^(k - 1) / m)], which can ensure that there is no carry to the highest digit.

[0087] Since in the example it is assumed that a is randomly 1, 2 + 1 * 10 - 2 = 10, and the hexadecimal of 10 is A, so the two-digit number in step S4 is filled as CA.

[0088] Step S6: Use the lowest digit of the multi-digit number filled in the fifth step as the highest digit of the new two-digit number, and repeat steps S4 and S5 (when taking the original ID, move backward in sequence) to generate a new two-digit number until all digits of the original ID are calculated. (A0, 02, 2B)

[0089] Step S7: Overlap and connect the heads and tails of all K-digit numbers to obtain the transformed ID (CA02B).

[0090] It can be understood that in the algorithm, N, M, and K are set values, and S and A are random values. Due to the existence of the random values S and A, even for the same original ID, the results obtained each time are different, achieving the expectation of the algorithm.

[0091] In an embodiment of the present application, for transforming the entry ID in the request result obtained from the external request, it further includes: According to the pre-set number of digits k of the multi-digit number, overlappingly intercept and replace the ID into multiple K-digit numbers; Obtain the specific value according to the set replacement base number n, and take the remainder of the specific value with respect to the target function base m, and splice the remainders to be the original ID.

[0092] For the process of restoring the original ID from the replacement ID (taking CA02CB as an example of the replacement ID):

[0093] Step S1: Since the number of digits K of the multi-digit number is pre-set. Then overlappingly intercept and replace the ID into multiple K-digit numbers (CA, A0, 02, 2B).

[0094] Step S2: Obtain the specific value according to the set replacement base number n, (192, 160, 2, 43).

[0095] Step S3: Take the remainder of the values in step S2 with respect to the target function base m (2, 0, 2, 3).

[0096] Step S4: Concatenate the remainders to obtain the real ID (2023).

[0097] Based on the above method for generating dynamic one-to-many ID transformation, through 7 steps of operations including base conversion and remainder concatenation, one-to-many forward and reverse ID transformations are achieved. Thus, based on this transformation method, the shielding and hiding of the original object ID can be realized, thereby preventing crawlers from identifying the captured objects. Multiple white energies are set in the algorithm, and based on the adjustment of these variables, ID transformations of different lengths can be adapted, and by adjusting relevant parameters, ID transformation results of different complexities or confusion levels can be achieved. In addition, the transformed IDs are non-uniformly distributed, making it difficult for artificial intelligence to perform data training.

[0098] The embodiment of the present application also provides an anti-crawler processing device 400, as Figure 4 shown, which provides a structural schematic diagram of the anti-crawler processing device in the embodiment of the present application. The anti-crawler processing device 200 at least includes: a receiving module 410 and a dynamic transformation module 440, wherein:

[0099] In an embodiment of the present application, the receiving module 410 is specifically configured to: receive an external request.

[0100] As Figure 2 shown, the "external request" is a request relative to the internal business system and database. The external request may include, but is not limited to, a separate query request, an operation request, or other query requests (based on a separate query), etc. Usually, a web crawler may be carried in the external request to collect and count relevant business data in the internal business system.

[0101] It can be understood that for all external requests, the item ID will be transformed, and the replaced item ID will be returned.

[0102] In an embodiment of the present application, the dynamic transformation module 440 is specifically configured to: according to the external request, before returning the request result and / or after receiving the external request, transform the item ID in the request result obtained from the external request. The item ID includes a replacement ID, and there is a mapping relationship between the replacement ID and the original ID.

[0103] The "item ID" includes, but is not limited to, the product ID, project ID, etc. on the external request page. Usually, after being collected by a web crawler, the business form or product characteristics can be obtained through statistical analysis. Therefore, the "item ID" needs to be continuously transformed dynamically, that is, replaced with a replacement ID, and the real ID is not exposed. Since the item ID includes a replacement ID, and there is a mapping relationship between the replacement ID and the original ID, replacement IDs can be continuously generated to replace the item ID.

[0104] Please refer to Figure 2 , the external requests are divided into different request types. Before returning the request result, the entry ID in the request result obtained from the external request is transformed. At the same time, after receiving the external request, the entry ID in the request result obtained from the external request is transformed.

[0105] In addition, since an ID transformation component is introduced at the request entry, all requests and return parameters for external calls are replaced, isolating the ID systems inside and outside the system. The real ID is made invisible to external calls, ensuring that the crawler system cannot crawl real and valid information through the external interface calls.

[0106] It can be understood that the above anti-crawler processing device can implement each step of the anti-crawler processing method provided in the foregoing embodiments. The relevant explanations regarding the anti-crawler processing method are applicable to the anti-crawler processing device and will not be elaborated here.

[0107] Figure 5 is a schematic structural diagram of an electronic device according to an embodiment of the present application. Please refer to Figure 5 , at the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. Among them, the memory may include a memory, such as a high-speed random access memory (Random-Access Memory, RAM), and may also include a non-volatile memory, such as at least one disk memory, etc. Of course, the electronic device may also include other hardware required for other services.

[0108] The processor, network interface, and memory can be interconnected through the internal bus, and the internal bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of representation, Figure 5 only a bidirectional arrow is used in

[0109] Memory, for storing programs. Specifically, the program may include program code, and the program code includes computer operation instructions. The memory may include a memory and a non-volatile memory, and provide instructions and data to the processor.

[0110] The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming an anti-crawler processing device at the logical level. The processor executes the program stored in the memory and is specifically used to perform the following operations:

[0111] Receive an external request;

[0112] According to the external request, before returning the request result and / or after receiving the external request, transform the entry ID in the request result obtained by the external request, where the entry ID includes a replacement ID, and there is a mapping relationship between the replacement ID and the original ID.

[0113] The method executed by the anti-crawler processing device disclosed in the above embodiments of the present application Figure 1 can be applied to or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit in the hardware of the processor or by instructions in the form of software. The above processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as being completed by the hardware decoding processor, or by a combination of the hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. This storage medium is located in the memory, and the processor reads the information in the memory and combines its hardware to complete the steps of the above method.

[0114] The electronic device can also execute Figure 1 the method executed by the anti-crawler processing device in Figure 1 and implement the functions of the anti-crawler processing device in the embodiments shown. The embodiments of the present application will not be elaborated here.

[0115] An embodiment of the present application also provides a computer-readable storage medium storing one or more programs, the one or more programs including instructions that, when executed by an electronic device including a plurality of application programs, enable the electronic device to execute Figure 1 the method executed by the anti-crawler processing device in the illustrated embodiment, and specifically used to execute:

[0116] Receive an external request;

[0117] According to the external request, before returning the request result and / or after receiving the external request, transform the entry ID in the request result obtained by the external request, where the entry ID includes a replacement ID, and there is a mapping relationship between the replacement ID and the original ID.

[0118] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0119] The present invention is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for implementing the specified functions in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0120] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device, and the instruction device implements the specified functions in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0121] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus, causing a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, so that the instructions executed on the computer or other programmable apparatus provide steps for implementing the functions specified in one flow Figure 1 one or more flows and / or boxes Figure 1 or steps for implementing the functions specified in a box or boxes.

[0122] In a typical configuration, a computing device includes one or more processors (CPUs), an input / output interface, a network interface, and memory.

[0123] The memory may include non-permanent memory in the computer-readable medium, in the form of random access memory (RAM) and / or non-volatile memory such as read-only memory (ROM) or flash memory (flash RAM). The memory is an example of a computer-readable medium.

[0124] Computer-readable media includes both permanent and non-permanent, removable and non-removable media implemented by any method or technology for storing information. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0125] It should also be noted that the term "comprising", "including" or any other variation thereof is intended to cover a non-exclusive inclusion, such that a process, method, article or apparatus comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or apparatus. Without further limitation, an element defined by the statement "comprising a..." does not exclude the presence of additional identical elements in the process, method, article or apparatus comprising the element.

[0126] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0127] The above are only the embodiments of the present application and are not intended to limit the present application. For those skilled in the art, various changes and modifications can be made to the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included within the scope of the claims of the present application.

Claims

1. A method for anti-crawler processing, wherein, The method includes: Receiving an external request; According to the external request, before returning the request result and / or after receiving the external request, transforming the entry ID in the request result obtained from the external request, where the entry ID includes a replacement ID, and there is a mapping relationship between the replacement ID and the original ID; The transforming the entry ID in the request result obtained from the external request includes: Configuring the base numbers n and m of the replacement ID and the original ID; Randomly selecting a value s in the base number of the replacement ID; According to the algorithm corresponding to the base number, replacing the overlap degree k, where k represents how many bits are taken for each decryption calculation; According to the overlap degree k, generating a multi-digit number z, and using the value s as the highest digit and taking 0 for the other digits; Taking the remainder of the value pair with respect to m, subtracting this remainder from the first digit of the original ID + a * m, and after converting to base n, filling the last few digits of the multi-digit number z; Taking the lowest digit of the filled multi-digit number z as the highest digit of a new multi-digit number, repeating the above steps to generate a new multi-digit number until all the digits of the original ID are calculated; Connecting the first and last digits of all the k-digit numbers end to end to obtain the transformed replacement ID, where n, m, and k are set values, and s and a are random values; According to the pre-set overlap degree k, overlapping and intercepting the replacement ID into multiple k-digit numbers; Obtaining a specific value according to the set replacement base number n, taking the remainder of the specific value with respect to the target function base m, and concatenating the remainders to obtain the original ID.

2. The method according to claim 1, wherein There is a mapping relationship between multiple replacement IDs and one original ID. The transforming the entry ID in the request result obtained from the external request according to the external request, before returning the request result and / or after receiving the external request, includes: According to an external call request, before returning the call request result, transforming the entry ID of the return parameter into a replacement ID and isolating the original ID internally; And / or, According to an external call request, after receiving the external request, transforming the entry IDs of all requests of the external call into replacement IDs and isolating the original ID internally.

3. The method according to claim 2, wherein The transforming the entry ID of the return parameter into a replacement ID and isolating the original ID internally according to an external call request, before returning the call request result, includes: Judging the type of a potential crawler external call request; In the case where the type of the potential crawler external call request is judged to be a query type, using the original ID for business query internally during the business processing; After the business processing is completed and before returning the call request result, transforming the original ID of the return data into a replacement ID and then returning it.

4. The method according to claim 3, wherein The transforming the entry IDs of all requests of the external call into replacement IDs and isolating the original ID internally according to an external call request, after receiving the external request, includes: In the case where the type of the potential crawler external call request is judged to be an operation type, transforming the replacement ID into the original ID and using the original ID for business operations internally during the business processing.

5. The method according to claim 4, wherein The method further includes: When it is determined that the type of the potential external call request of the crawler is to query other content, the replacement ID is transformed into the original ID, and the original ID is internally used for business operations during the business processing process; The original ID is internally used for business queries during the business processing process; After the business processing is completed and before returning the result of the call request, the original ID of the returned data is transformed into the replacement ID and then returned.

6. An anti-crawler processing device, wherein, The device includes: A receiving module, configured to receive an external request; A dynamic transformation module, configured to transform the entry ID in the request result obtained by the external request before returning the request result and / or after receiving the external request according to the external request, where the entry ID includes a replacement ID, and there is a mapping relationship between the replacement ID and an original ID; The transforming the entry ID in the request result obtained by the external request includes: Configuring the base numbers n and m of the replacement ID and the original ID; Randomly selecting a value s in the base number of the replacement ID; According to the algorithm corresponding to the base number, the overlap degree k, where k represents how many bits are taken for each decryption calculation; Generating a multi-digit number z according to the overlap degree k, and using the value s as the highest bit and taking 0 for other bits; Taking the remainder of the value pair with respect to m, subtracting this remainder from the first bit of the original ID + a * m, and after converting to base n, filling the last few bits of the multi-digit number z; Taking the lowest bit of the multi-digit number z after filling as the highest bit of the new multi-digit number, and repeating the above steps to generate a new multi-digit number until all the bits of the original ID are calculated; Connecting the first and last bits of all the k-digit numbers end to end to obtain the transformed replacement ID, where n, m, and k are set values, and s and a are random values; According to the pre-set overlap degree k, overlapping and intercepting the replacement ID into multiple k-digit numbers; Obtaining a specific value according to the set replacement base number n, taking the remainder of the specific value with respect to the target function base m, and splicing the remainders to obtain the original ID.

7. An electronic device, including: A processor; And A memory arranged to store computer-executable instructions, where the executable instructions, when executed, cause the processor to execute the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, where the computer-readable storage medium stores one or more programs, and when the one or more programs are executed by an electronic device including a plurality of application programs, the electronic device is caused to execute the method according to any one of claims 1 to 5.

Images