An anti-crawler method and system combining WASM and data encryption

Abstract

The present invention relates to an anti-crawler method and system combining WASM with data encryption. The method comprises the following steps: initializing a WASM bytecode cache pool, the WASM bytecode cache pool generates a random number through the SecureRandom class of Java and inputs it into the WASM bytecode cache pool, and then generates a key through the AES algorithm based on the random number in the WASM bytecode cache pool; dynamically generating JAVA code containing decryption logic, and embedding the key into the code in the form of an array; translating the dynamically generated JAVA code to generate WASM decryption bytecode, and saving the WASM decryption bytecode and the key together in the WASM cache pool; before a webpage backend service returns request data, obtaining a key and a corresponding WASM decryption bytecode from the cache pool, encrypting the data using the key, and then returning the ciphertext data and the WASM decryption bytecode.

Description

Technical Field

[0001] The present invention relates to an anti-crawler method and system combining WASM and data encryption, belonging to the technical field of network security. Background Art

[0002] Anti-web crawler technology is in a stage of rapid development and continuous evolution. It is becoming more intelligent, dynamic and comprehensive, while also seeking a balance between user experience, laws and regulations, and website performance; as the value of data continues to increase, anti-crawler technology is likely to continue to maintain a high-speed development trend in the future.

[0003] Existing anti-crawler technologies usually include verification codes, IP blocking, user browser fingerprint detection, frequency restrictions and other methods to prevent crawlers from accessing websites. However, these technologies still have some shortcomings. The development of artificial intelligence technology allows crawlers to automatically solve most verification codes, reducing the defense effect of verification codes; IP addresses can be easily changed or proxy servers can be used to avoid blocking, making IP blocking methods relatively easy to bypass; crawlers can disguise themselves as ordinary users' browser fingerprints, thereby avoiding simple browser fingerprint detection;

[0004] The invention patent with the announcement number "CN114553541B" proposes a method for graded verification and anti-crawling, including: accepting user requests, counting the number of horizontal time visits, using the horizontal time visits to calculate the risk coefficient, judging whether the risk coefficient is within the preset verification abnormality range, if the risk coefficient is within the preset verification abnormality range, then performing risk value assessment according to the risk coefficient, judging the risk classification of the user request, when the risk classification of the user request is high risk, performing login interception verification, when the risk classification of the user request is medium risk, performing verification code verification, when the risk classification of the user request is low risk, allowing the request, the present invention avoids the mistaken blocking of normal user IP addresses, does not affect the access of normal users, and also prevents the risk of system downtime;

[0005] The sampling frequency limiting method in the above scheme is prone to misblocking normal users, and is not very effective for distributed crawlers. Summary of the invention

[0006] In order to solve the above problems existing in the prior art, the present invention proposes an anti-crawler method and system combining WASM and data encryption.

[0007] The technical solution of the present invention is as follows:

[0008] On the one hand, the present invention provides an anti-crawler method combining WASM and data encryption, comprising the following steps:

[0009] Initialize the WASM bytecode cache pool, generate a random number through Java's SecureRandom class and input it into the WASM bytecode cache pool, and then generate a key based on the random number in the WASM bytecode cache pool through the AES algorithm;

[0010] Dynamically generate JAVA code containing decryption logic and embed the key in the code in the form of an array;

[0011] Translate the dynamically generated JAVA code into WASM decryption bytecode, and save the WASM decryption bytecode and key together into the WASM cache pool;

[0012] Before the web backend service returns the requested data, it obtains a key and the corresponding WASM decryption bytecode from the cache pool, encrypts the data with the key, and then returns the ciphertext data and WASM decryption bytecode.

[0013] As a preferred embodiment of the present invention, the decryption steps of the ciphertext data are:

[0014] The web front end determines whether the data returned by the background needs to be decrypted. If necessary, JavaScript is used to load and instantiate the WASM decryption bytecode. Based on the decryption logic in the WASM decryption bytecode, the embedded key is used to decrypt the ciphertext data to obtain the original data.

[0015] As a preferred implementation manner of the present invention, the key is a 256-bit AES key.

[0016] As a preferred embodiment of the present invention, the WASM bytecode cache pool will generate new WASM decryption bytecode and key at fixed intervals.

[0017] On the other hand, the present invention also provides an anti-crawler system combining WASM and data encryption, including an initialization module, a key embedding module, a decryption bytecode generation module, and a data encryption module;

[0018] The initialization module is used to initialize the WASM bytecode cache pool, generate a random number through the Java SecureRandom class and input it into the WASM bytecode cache pool, and then generate a key based on the random number in the WASM bytecode cache pool through the AES algorithm;

[0019] The key embedding module is used to dynamically generate JAVA code containing decryption logic and embed the key into the code in the form of an array;

[0020] The decryption bytecode generation module is used to translate the dynamically generated JAVA code into WASM decryption bytecode, and save the WASM decryption bytecode and the key together into the WASM cache pool;

[0021] The data encryption module is used to obtain a key and a corresponding WASM decryption bytecode from the cache pool before the web page background service returns the requested data, encrypt the data using the key, and then return the ciphertext data and the WASM decryption bytecode.

[0022] As a preferred embodiment of the present invention, the system further comprises a front-end data decryption module;

[0023] The front-end data decryption module is used to determine whether the data returned by the background needs to be decrypted. If necessary, JavaScript is used to load and instantiate the WASM decryption bytecode. Based on the decryption logic in the WASM decryption bytecode, the ciphertext data is decrypted using the embedded key to obtain the original data.

[0024] As a preferred implementation manner of the present invention, the key is a 256-bit AES key.

[0025] As a preferred embodiment of the present invention, the WASM bytecode cache pool will generate new WASM decryption bytecode and key at fixed intervals.

[0026] On the other hand, the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in any embodiment of the present invention when executing the program.

[0027] In yet another aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any embodiment of the present invention.

[0028] The present invention has the following beneficial effects:

[0029] 1. The present invention uses WASM bytecode for decryption and utilizes the high efficiency of WASM to significantly improve the speed and performance of the decryption process.

[0030] 2. The key generated by the present invention is a unique key, which ensures the uniqueness of the data encryption result. Each WASM decryption bytecode is bound to a specific unique key, and the generated WASM decryption bytecode is also unique. Each WASM bytecode can only decrypt data encrypted with the corresponding key bound to it, effectively preventing the risk of malicious attacks and data leakage.

[0031] 3. The present invention provides a caching mechanism for keys and WASM decryption bytecodes. The WASM construction service generates bytecodes and key information, and provides a timed refresh cache mechanism, which not only improves the operating efficiency of the system, but also ensures a smooth experience for users during the data verification process. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0034] It should be understood that the step numbers used in this document are only for convenience of description and are not intended to limit the order in which the steps are executed.

[0035] It should be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0036] The terms “include” and “comprising” indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0037] The term "and / or" means and includes any and all possible combinations of one or more of the associated listed items.

[0038] Embodiment 1:

[0039] See also Figure 1 , an anti-crawler method combining WASM and data encryption, including the following steps:

[0040] Initialize the WASM bytecode cache pool, generate a random number through Java's SecureRandom class and input it into the WASM bytecode cache pool, and then generate a key based on the random number in the WASM bytecode cache pool through the AES algorithm;

[0041] Dynamically generate JAVA code containing decryption logic and embed the key in the code in the form of an array;

[0042] Translate the dynamically generated JAVA code into WASM decryption bytecode, and save the WASM decryption bytecode and key together into the WASM cache pool;

[0043] Before the web backend service returns the requested data, it obtains a key and the corresponding WASM decryption bytecode from the cache pool, encrypts the data with the key, and then returns the ciphertext data and WASM decryption bytecode.

[0044] When the web page background service is about to return data, a key and the corresponding WASM decryption bytecode are obtained from the cache pool, the data is encrypted using the key, and the ciphertext data and the WASM decryption bytecode are returned. Specifically, in this embodiment, when the user queries information, the background service initiates a data query request, and the background performs a data query. After obtaining the query result, for the interface that needs to be encrypted, the method of obtaining the WASM decryption bytecode and the encrypted data in the anti-crawler SDK is called. The anti-crawler SDK will encrypt the data according to the key and return the encrypted data and the WASM decryption bytecode to the browser.

[0045] As a preferred implementation of this embodiment, the decryption steps of the ciphertext data are:

[0046] The web front end determines whether the data returned by the background needs to be decrypted. If necessary, JavaScript is used to load and instantiate the WASM decryption bytecode. Based on the decryption logic in the WASM decryption bytecode, the embedded key is used to decrypt the ciphertext data, and the web page is rendered after the original data is obtained.

[0047] As a preferred implementation of this embodiment, the key is a 256-bit AES key to ensure the uniqueness of key generation.

[0048] As a preferred implementation of this embodiment, the WASM bytecode cache pool will generate new WASM decryption bytecode and key at fixed intervals to ensure that the cache pool is always in the best state.

[0049] Embodiment 2:

[0050] An anti-crawler system combining WASM and data encryption, including an initialization module, a key embedding module, a decryption bytecode generation module, and a data encryption module;

[0051] The initialization module is used to initialize the WASM bytecode cache pool, generate a random number through the Java SecureRandom class and input it into the WASM bytecode cache pool, and then generate a key based on the random number in the WASM bytecode cache pool through the AES algorithm;

[0052] The key embedding module is used to dynamically generate JAVA code containing decryption logic and embed the key into the code in the form of an array;

[0053] The decryption bytecode generation module is used to translate the dynamically generated JAVA code into WASM decryption bytecode, and save the WASM decryption bytecode and the key together into the WASM cache pool;

[0054] The data encryption module is used to obtain a key and a corresponding WASM decryption bytecode from the cache pool before the web page background service returns the requested data, encrypt the data using the key, and then return the ciphertext data and the WASM decryption bytecode.

[0055] As a preferred implementation of this embodiment, the system further includes a front-end data decryption module;

[0056] The front-end data decryption module is used to determine whether the data returned by the background needs to be decrypted. If necessary, JavaScript is used to load and instantiate the WASM decryption bytecode. Based on the decryption logic in the WASM decryption bytecode, the ciphertext data is decrypted using the embedded key to obtain the original data.

[0057] As a preferred implementation of this embodiment, the key is a 256-bit AES key to ensure the uniqueness of key generation.

[0058] As a preferred implementation of this embodiment, the WASM bytecode cache pool will generate new WASM decryption bytecode and key at fixed intervals to ensure that the cache pool is always in the best state.

[0059] This system is used to implement the functions in the first embodiment, which will not be described in detail here.

[0060] Embodiment three:

[0061] This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in any embodiment of the present invention when executing the program.

[0062] Embodiment 4:

[0063] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the method described in any embodiment of the present invention is implemented.

[0064] In the embodiments of the present application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone. Among them, A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c can be represented by: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, c can be single or multiple.

[0065] Those of ordinary skill in the art will appreciate that the various units and algorithm steps described in the embodiments disclosed herein can be implemented in a combination of electronic hardware, computer software, and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0066] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0067] In several embodiments provided in the present application, if any function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory; hereinafter referred to as: ROM), random access memory (Random Access Memory; hereinafter referred to as: RAM), disk or optical disk, and other media that can store program codes.

[0068] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. An anti-crawler method combining WASM and data encryption, characterized in that: The following steps are involved: Initialize the WASM bytecode cache pool, generate a random number through Java's SecureRandom class and input it into the WASM bytecode cache pool, and then generate a key based on the random number in the WASM bytecode cache pool through the AES algorithm; Dynamically generate JAVA code containing decryption logic and embed the key into the code in the form of an array; translate the dynamically generated JAVA code into WASM decryption bytecode, and save the WASM decryption bytecode and the key together into the WASM cache pool; Before the web backend service returns the requested data, it obtains a key and the corresponding WASM decryption bytecode from the cache pool, encrypts the data with the key, and returns the ciphertext data and WASM decryption bytecode.

2. According to claim 1, the anti-crawler method combining WASM and data encryption is characterized in that: The decryption steps of ciphertext data are: The web front end determines whether the data returned by the background needs to be decrypted. If necessary, JavaScript is used to load and instantiate the WASM decryption bytecode. Based on the decryption logic in the WASM decryption bytecode, the embedded key is used to decrypt the ciphertext data to obtain the original data.

3. According to claim 1, the anti-crawler method combining WASM and data encryption is characterized in that: The key is a 256-bit AES key.

4. According to claim 1, the anti-crawler method combining WASM and data encryption is characterized in that: At regular intervals, the WASM bytecode cache pool will generate new WASM decryption bytecode and keys.

5. An anti-crawler system combining WASM and data encryption, characterized in that: It includes an initialization module, a key embedding module, a decryption bytecode generation module and a data encryption module; The initialization module is used to initialize the WASM bytecode cache pool, generate a random number through the Java SecureRandom class and input it into the WASM bytecode cache pool, and then generate a key based on the random number in the WASM bytecode cache pool through the AES algorithm; The key embedding module is used to dynamically generate JAVA code containing decryption logic and embed the key into the code in the form of an array; The decryption bytecode generation module is used to translate the dynamically generated JAVA code into WASM decryption bytecode, and save the WASM decryption bytecode and the key together into the WASM cache pool; The data encryption module is used to obtain a key and a corresponding WASM decryption bytecode from the cache pool before the web page background service returns the requested data, encrypt the data using the key, and then return the ciphertext data and the WASM decryption bytecode.

6. The anti-crawler system combining WASM and data encryption according to claim 5, characterized in that: The system also includes a front-end data decryption module; The front-end data decryption module is used to determine whether the data returned by the background needs to be decrypted. If necessary, JavaScript is used to load and instantiate the WASM decryption bytecode. Based on the decryption logic in the WASM decryption bytecode, the ciphertext data is decrypted using the embedded key to obtain the original data.

7. The anti-crawler system combining WASM and data encryption according to claim 5, characterized in that: The key is a 256-bit AES key.

8. The anti-crawler system combining WASM and data encryption according to claim 5, characterized in that: At regular intervals, the WASM bytecode cache pool will generate new WASM decryption bytecode and keys.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 4 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images