A CUDA-based sequence collision fast search method and system under resource constraints

Abstract

The application provides a CUDA-based sequence collision fast search method and system under resource limitation, relates to the technical field of information security, and performs initial parameter setting, data loading, matching statistics and sorting, and outputs a search result. The application can perform efficient search of overlapping collisions under limited computing resources, greatly improves the upper limit of the detection data volume, solves the problem that it is difficult to efficiently search overlapping collisions under limited computing resources, and can better serve the RNG random number quality detection work.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of information security, and particularly relates to a CUDA-based sequence collision fast search method and system under resource restriction. BACKGROUND

[0002] With the development of information technology, the application demand of the cryptographic system is more and more extensive, and the security of the modern cryptographic system is closely related to the quality of the random number generated by the random number generator (RNG). Randomness statistical test is widely used in the randomness test of the RNG to test the uniform randomness of the random number generated thereby. Many countries and organizations have issued their own random number detection standards and specifications, such as the NIST standard of the United States, the AIS standard of Germany, the random number detection specification of China and the like, so as to better serve the development of the information industry. Among various random number detection standards, the detection of the fixed length template always occupies an important position, mainly including: disjoint detection, non-overlapping template matching detection, overlapping template matching detection, approximate entropy detection, sequence detection and the like. Among them, the non-overlapping template matching detection and the overlapping template matching detection are the frequency detection of the fixed template, the approximate entropy detection and the sequence detection are the frequency detection of all templates, and the disjoint detection is the detection of the template collision in the non-overlapping manner, that is, whether any template is repeatedly present in the detection sequence. The present application mainly focuses on the collision of the long template in the overlapping manner.

[0003] The collision is a statistical index of the random sequence, and is an important technical index of the hash function. The well-known "birthday collision problem" develops into the birthday attack or collision attack commonly used in the cryptographic analysis. First of all, the related definitions of the collision need to be clarified.

[0004] Collision: when sampling with replacement is performed from a population of size n, the collision number is defined as the difference between the sampling number n and the number of different elements n1 in the sample, that is, the collision number = n-n1. Specifically in the long sequence duplication detection, there are n m-bit templates, and the number of different templates is n1, and the long sequence collision number = n-n1.

[0005] Repeated logarithm: when sampling with replacement is performed from a population of size n, the repeated logarithm is defined as the number of n samples that are equal to each other, that is, wherein Specifically in the long sequence duplication detection, there are n m-bit templates, and the long sequence collision number = n-n1.

[0006] Non-overlapping statistics: in the long sequence template statistics, the templates are not allowed to overlap with each other, and there are N-bit sequences, that is, m-bit templates, that is, the long sequence is divided into m-bit segments for statistics.

[0007] Overlap statistics: In the long sequence template statistics, each template is allowed to overlap, and let N be the sequence, then there are n = N - m + 1 m-bit templates, that is, the long sequence is counted by shifting 1 bit at a time in m-bit width.

[0008] Non-trivial collision: In the long sequence template statistics, if there are two templates that form a collision, and the 1-bit before the start position and the 1-bit after the end position of the two templates are not equal, then it is called a non-trivial collision.

[0009] These definitions can be explained by an example. Let there be a 12-bit sequence 110110101101, then the sequence has 3 non-overlapping 4-bit templates: 1101, 1010, 1101; 9 overlapping 4-bit templates: 1101, 1011, 0110, 1101, 1010, 0101, 1011, 0110, 1101; among them, there is 1 collision and 1 repeated pair in non-overlapping statistics, there are 4 collisions and 5 repeated pairs in overlapping statistics, among them, there are 2 non-trivial 4-bit collisions and 1 non-trivial 6-bit collision.

[0010] Here, a non-overlapping duplicate detection detection in the existing standard is introduced - disjoint detection, that is, the generated sequence is counted in a non-overlapping manner to detect collisions, and the probability of occurrence of duplicates is obtained based on the birthday collision principle, and a small probability event is rejected.

[0011] The birthday collision model is as follows:

[0012] Let there be m balls, and randomly take out n balls, the probability of collision is:

[0013]

[0014] where m (n) = n(n-1)…(n-r+1). When and m→∞, then P can be approximated as

[0015]

[0016] In the disjoint detection proposed in AIS31, a random bit string X is tested, its length is N, the random number bit width b, and whether it passes the non-overlapping collision detection is tested. The parameters used are m = 2 b = 2 48 , then the probability of occurrence of duplicates is P = 1 - exp(-2 -17 ) ≈ 2 -17 . It can be seen that in a 2 16 *48-bit long random bit string, it is divided into a random string with 48-bit templates, a total of 2 16 , the probability of occurrence of 48-bit string template duplicates is 2-17 , which is a small probability event, if it occurs, the hypothesis that it is a random string is rejected. The detection strategy of disjoint detection (non-overlapping detection) is summarized as follows:

[0017] Disjoint detection model: n-bit long sequence, detection template length is m bits, and the repetition probability is

[0018]

[0019] Disjoint detection strategy example:

[0020] Table 1

[0021]

[0022] Six kinds of classic collision problems can be extended through the birthday problem, and more rich statistical test methods can be obtained based on overlapping and non-overlapping queries. With non-overlapping detection, it is natural to consider more powerful overlapping collisions, that is, to detect template collision in an overlapping manner. Taking n long 4 bits as an example, there are n / 4 templates in the non-overlapping manner, and there are n-3 4-bit templates in the overlapping manner.

[0023] First, for an n-bit long sequence, the detection template length is m bits, and the expectation of the logarithm of the repetition (≥ collision number) can be easily obtained:

[0024]

[0025] According to the formula, the following table (column: length of the sequence to be detected; row: length of the fixed length template; value: logarithmic value of the probability estimate of the occurrence of repetition

[0026] Table 2

[0027]

[0028] The present application mainly focuses on the overlapping collision search method of long bit length (optional length range) in the sequence.

[0029] First, the most trivial implementation method is: general implementation. Directly read the data into the memory and compare it according to the m-bit overlapping template. The complexity is O(n 2 ). Taking 1GB search 72-bit repetition as an example, this implementation method requires at least 8·72GB memory; when the data amount reaches 10GB, at least 80·72GB memory is required; when the data amount reaches 100GB, at least 800·72GB memory is required. Obviously, this implementation method requires super large memory, and is not suitable for general computers, and its complexity of pair-by-pair comparison is also high.

[0030] In order to accelerate the realization speed, a more efficient method is: fast sorting implementation. Directly read data into memory, and sort and compare according to m-bit overlapping template. The complexity is O (nlogn). Compared with the general implementation method, the implementation method greatly reduces the complexity, but the required memory remains unchanged. This implementation method is more suitable for high-performance computers, but the high demand for memory also greatly reduces the data volume range of high-performance computers.

[0031] By utilizing the parallel capabilities of computers (CPU and GPU), the search can be greatly accelerated, but this does not change the limitation of computing resources, thereby greatly limiting the detectable data volume. Therefore, in actual work, in order to balance computing resources and efficiency, an efficient overlapping collision search method is urgently needed, which undoubtedly increases the design difficulty of the search algorithm. SUMMARY

[0032] In order to solve the problems of limited computing resources and low efficiency, that is, to solve the problem of efficient search for overlapping collisions under limited computing resources, the present application provides a CUDA-based sequence collision fast search method and system under resource constraints, which can efficiently search for overlapping collisions under limited computing resources, greatly improve the upper limit of the detection data volume, and better serve the RNG random number quality detection work.

[0033] The CUDA-based sequence collision fast search method under resource constraints provided by the present application comprises the following steps:

[0034] 1) Initial parameter setting step: divide the detection template into prefix and suffix, set the detection template length, prefix length, and suffix length; set the CUDA parallel thread, which includes thread block and thread bundle; set the output collision template length range;

[0035] 2) Data loading step: load the data to be detected into the GPU device;

[0036] 3) Matching statistics step: when performing collision fast search, perform CUDA parallel matching on the data to be detected to obtain the frequency of all prefixes;

[0037] 4) Matching sorting step: use CUDA parallelism to traverse all prefixes, obtain all suffix data corresponding to the matched prefix, and sort them to obtain all collisions that meet the length requirement and their positions and lengths as results;

[0038] 5) Output step: transfer the above results to the CPU end for result processing and output {collision length; collision template; collision position}.

[0039] Further, in the initial parameter setting step, the theoretical expectation of the number of repetitions is determined according to the computing resources The detection template is divided into a prefix and a suffix.

[0040] Further, in the data loading step, the data to be detected is first read by the CPU and then copied (cudaMemcpyHostToDevice) to the GPU device.

[0041] Further, in the data loading step, for large data exceeding the GPU memory load, multiple data transmissions between the CPU and the GPU are performed to complete data loading.

[0042] Further, in the matching and sorting step, the method for traversing the prefix is as follows: according to the prefix frequency, the device memory size, and the number of parallel threads, the sorting of the suffix data corresponding to multiple prefixes (≥1) is planned and processed at one time.

[0043] Further, in the matching and sorting step, the radix sorting method suitable for CUDA parallelism is used for suffix sorting, and the steps include: parallel first byte or multi-byte (depending on the parallel thread allocation), and then byte block sorting through offset statistics.

[0044] Further, in the output step, if the collision length reaches the set upper limit of the search, the CPU side performs a longer collision determination search on the collision position. For searching a larger length collision (such as 120 bits), considering the calculation resources and implementation efficiency limitations, it is not directly searched, but first searches a smaller length (at the same time, the theoretical expectation value is also smaller) collision (such as 80 bits), and the larger length and the smaller length are determined according to the calculation resources; then, on this basis, the collision position obtained is used to continue searching for the possibility of a longer collision length, and finally the entire length collision search is completed. For example, according to the calculation resources (memory) and efficiency considerations, the longest detection length is set to 110 bits, but there is a demand for 180-bit collision detection, at this time, without modifying the program and setting, only 110-bit collision needs to be detected, and on this basis, a search is performed to find whether there is a 180-bit collision. On the other hand, if 180-bit length is directly detected, it will bring resource pressure and efficiency reduction, so the smaller length collision should be detected first, and then the longer length collision is detected on this basis.

[0045] Further, the output collision template length range needs to be set according to the calculation resources, and the lower limit is too small to have no meaning, and the upper limit is too large to affect the algorithm efficiency and excessive resource occupation, and the setting of the upper and lower limits needs to be considered according to the theoretical expectation result.

[0046] A resource-constrained CUDA-based sequence collision fast search system includes a memory and a processor, and a computer program is stored on the memory. The processor implements the steps of the above method when executing the program.

[0047] The present application mainly optimizes two aspects of resource occupation and search efficiency: (1) the data to be detected is classified and matched, and then collision search is performed, so as to reduce resource occupation by reducing the search space; (2) the CUDA parallel technology based on computer GPU is utilized to improve search efficiency.

[0048] Compared with the existing common method for overlapping collision search, the method of the present application has the following advantages:

[0049] 1. The present application innovatively proposes a prefix classification matching search method for the problem of super-large resource occupation in overlapping collision search, that is, the template is divided into prefix and suffix templates, and the prefix data is matched to perform collision search on the suffix data. Assuming that the prefix is set to b bits, the resource occupation in the search stage can be reduced by at least 1 / 2 b times, and the basic idea is to sacrifice complexity and highly parallelize;

[0050] 2. The present application innovatively proposes to use the radix sort algorithm to perform collision search on the suffix data for the problem of search efficiency, which can effectively utilize the CUDA parallel technology to realize efficient collision search, and simultaneously record the collision length by using the characteristics of the algorithm for offset statistics.

[0051] 3. The present application does not directly search for super-long collisions, but sets an upper limit of search length according to the computing resources and theoretical expectation calculation, and further searches for collisions that reach the upper limit length, which can greatly reduce the resource occupation of super-long template overlap statistics (according to the theoretical expectation calculation, the collision length with smaller expected number will be selected as the upper limit, and the search for longer collisions on this basis can be ignored in the sense of great probability), and can also avoid sacrificing computing performance (sorting super-long templates in the sorting stage will cause the sorting efficiency to decrease, and the iteration depth is increased in the algorithm level). BRIEF DESCRIPTION OF DRAWINGS

[0052] Figure 1 is a flowchart of a fast overlapping collision search method based on CUDA under resource limitation proposed by the present application;

[0053] Figure 2 is a flowchart of 64-112 bit overlapping collision search on 10 GB data according to the first embodiment of the present application. DETAILED DESCRIPTION

[0054] The present application will be further described in detail below in combination with the drawings and one embodiment, but the scope of the present application is not limited in any way.

[0055] The present application is mainly aimed at the overlap collision search of super large data scale, and can be applied to the data range of 1GB-100GB or more (without limiting the upper limit of data amount), so as to meet the application requirements of collision search on a general-purpose computer. Taking 10GB data to be detected as an example, 64-112 bit collision templates, positions, lengths and the like are searched. It is noted that according to the above table, for 1GB-100GB data, the theoretical expected value of collision number of 64 bits or more is small, which belongs to a reasonable range.

[0056] The computer used in the embodiment is configured as Intel Xeon Gold 5218, 16 cores, 32 threads, 2.3 main frequency, 32 GB memory, graphics card RTX 3090 24 GB, and power supply 1400 W.

[0057] The embodiment adopts a CUDA-based sequence collision fast search method under resource restriction proposed by the present application to perform overlap collision search on 10GB data to be detected, as shown in the following formula (1): Figures 1-2 wherein Figure 1 is a basic flowchart of the method of the present application, Figure 2 is a flowchart of the specific processing of the embodiment, including the following main steps:

[0058] S1: initial parameter setting. Set the check template length PATTERN_LEN to 14 (bytes), the prefix length PATTERN_PREFIX_LEN to 2 (bytes), and the suffix length PATTERN_SUFFIX_LEN to 12 (bytes), and PATTERN_LEN=PATTERN_PREFIX_LEN+PATTERN_SUFFIX_LEN. Set the CUDA parallel threads: 1024 matching thread blocks, each block containing 32 threads; 256 sorting thread blocks, each block containing 256 threads. Set the output collision template length range: 64-112 bits.

[0059] S2: data loading. Copy 10GB data to the GPU device.

[0060] S3: matching statistics. Perform CUDA parallel matching (<<1024, 32>>), and obtain the frequency of 2 16 prefixes.

[0061] S4: matching sorting. Traverse the first prefix byte (a total of 256), and perform CUDA parallel sorting (<<256, 256>>). Each thread block is directed to a second prefix byte (a total of 256), and 256 threads are used to perform parallel radix sorting on the suffix 12 byte data set (the specific flow is: first, the offset of the first byte of the suffix data is counted, and then each thread is directed to the radix sorting of the suffix data for 11 bytes in series).

[0062] S5: output. The result is transmitted to the CPU end for result processing and output of {collision length; collision template; collision position}.

[0063] The specific experimental effect is that, by using the search method of the application, the output of all 64-112 bit overlap collision templates and their positions and lengths for 10 GB data can be completed in about 4200 seconds, which shows that the search method of the application can efficiently meet the actual large data detection requirements on a general-purpose computer. It can be seen that, on the one hand, the existing method requires a large amount of memory space, and it is only possible to be implemented in a high-performance computing cluster, and it is impossible to complete the method on a general-purpose computer, while the method of the application greatly reduces the requirement for computing resources and can meet the actual detection requirements on a general-purpose computer; on the other hand, even if it is implemented on a high-performance computer, it is necessary to design an algorithm to enable it to complete the sorting of long sequences, and the existing method directly sorts long sequences, which takes too long and is not feasible, while the method of the application can significantly improve the detection efficiency and meet the actual requirements.

[0064] Obviously, those skilled in the art can make various modifications and variations to the application without departing from the spirit and scope of the application. Thus, if these modifications and variations of the application belong to the scope of the claims of the application and their equivalent technologies, the application also intends to include these modifications and variations.

Claims

1. A fast sequence collision search method based on CUDA under resource constraints, characterized in that, Includes the following steps: 1) Initial parameter setting steps: Divide the detection template into prefix and suffix, and set the detection template length, prefix length, and suffix length; set the CUDA parallel thread, which includes thread blocks and thread bundles; set the output collision template length range; 2) Data loading step: Load the data to be detected into the GPU device; 3) Matching statistics step: When performing collision fast search, CUDA parallel matching is performed on the data to be detected to obtain the frequency of all prefixes; 4) Matching and sorting steps: Using CUDA in parallel, traverse all prefixes, obtain all corresponding suffix data by matching prefixes, sort them, and obtain all collisions that meet the length requirements, their positions and lengths, and use them as the results; 5) Output steps: Transmit the above results to the CPU for result processing and output {collision length; collision template;} Collision location}.

2. The method as described in claim 1, characterized in that, In the initial parameter setting step, the theoretical expectation of the repeated logarithm is taken into account. , m represents the bit length of the detection template, n represents the total bit length of the long sequence to be detected, and the detection template is divided into a prefix and a suffix.

3. The method as described in claim 1, characterized in that, In the data loading step, the data to be tested is first read in by the CPU, and then copied to the GPU device by cudaMemcpyHostToDevice.

4. The method as described in claim 1, characterized in that, In the data loading step, for extremely large amounts of data that exceed the GPU's memory capacity, the data is loaded by multiple data transfers between the CPU and GPU.

5. The method as described in claim 1, characterized in that, In the matching and sorting step, the method for traversing the prefixes is to sort the suffix data corresponding to multiple prefixes in a unified manner based on the prefix frequency, device memory size, and number of parallel threads.

6. The method as described in claim 1, characterized in that, In the matching sorting step, a radix sorting method suitable for CUDA parallelism is used for suffix sorting. The steps include: parallel sorting of the first byte or multiple bytes, followed by sorting byte-by-byte blocks through offset statistics.

7. The method as described in claim 1, characterized in that, In the output step, if the collision length reaches the set search limit, the CPU will perform a longer collision determination search for the corresponding collision position.

8. The method as described in claim 7, characterized in that, For searching for collisions of a larger length, first search for collisions of a smaller length, where the larger and smaller lengths are determined based on computational resources; then, based on the collision positions obtained from the search, continue searching to find possible collisions of even longer lengths, and finally complete the search for collisions of all lengths.

9. The method as described in claim 1, characterized in that, The output collision template length range is set according to computing resources.

10. A CUDA-based fast sequence collision search system under resource constraints, characterized in that, It includes a memory and a processor, wherein a computer program is stored in the memory, and the processor executes the program to implement the steps of the method according to any one of claims 1-9.

Images