Address conversion-based capacity expansion method, system, device, storage medium and program product for fixed-length counting array counting overflow
Patent Information
- Application Number
- CN202610620492.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-05-08
- Publication Date
- 2026-08-04
AI Technical Summary
传统的方案的缺点在于，虽然降低了数据溢出的可能性，但是整个数组的内存占用的内存就从200个字节扩大为了400字节
[0023] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: (1) It saves more memory space. Assuming the data overflow probability is 5%, it can save about 45% of memory space; (2) The time complexity is still O(1), and there will be no obvious performance difference; (3) There is no limit to the upper limit of the count, and the memory efficiency is higher for data with large variance. Assuming that 95% of the counts are less than the maximum value of uint16 and 5% of the counts are greater than the maximum value of uint64, it can save about 78% of memory space.
Smart Images
Figure CN122507656A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of database technology, and in particular to a method, system, device, storage medium, and program product for expanding the capacity of a fixed-length counting array based on address translation to address overflow. Background Technology
[0002] Arrays in various programming languages require standardized memory allocation during initialization based on their data type and length. For example, in C, a uint16 array allocates 2 bytes to each element. If the array is initialized with 100 uint16 arrays, each occupying 2 bytes, it will require a total of 200 bytes of memory. Reading the array involves accessing the starting address for the first element, the offset for the second element (starting address plus 2 bytes), the offset for the third element (starting address plus 4 bytes), and so on. The maximum count that can be recorded for each uint16 element is 2^16 - 1, or 65535.
[0003] Suppose a software system is unaware of whether using the counting array will cause overflow during array initialization. The traditional solution is to increase the array's data type to minimize overflow. For example, increasing the data type of the 100-byte uint16 array in the example above to uint32 would allow each uint32 to count a value of 2^32 - 1, or 4294467295. The drawback of this traditional approach is that while it reduces the possibility of overflow, it increases the memory usage of the entire array from 200 bytes to 400 bytes. If the count value of most data in the array does not exceed 65535, and only a very small portion does, then the traditional expansion method results in wasted memory.
[0004] In some extreme cases, the variance of the count values can be very large. Suppose that most of the count values in an array do not exceed the maximum value of uint16, but a small number of data even exceed the maximum value of uint64. In this case, traditional solutions may not be able to solve the problem. Summary of the Invention
[0005] Purpose of the invention: This invention provides a method, system, device, storage medium, and program product for expanding the capacity of a fixed-length counting array based on address translation to address overflow. It provides an address translation-based solution that can continue to collect counts and avoid copying the original memory, thus avoiding memory waste, and provides efficient statistical and query performance.
[0006] Technical solution: The present invention provides a method for expanding the capacity of a fixed-length counting array based on address translation to address overflow, comprising the following steps:
[0007] Step 1: Determine the data type and length of the array counting based on the possible upper limit and maximum value of most counts, and allocate contiguous memory; taking 100 uint16 elements as an example, allocate 200 bytes of memory and initialize the counting array;
[0008] Step 2: Check if the highest column of the array has overflowed. If it has not overflowed, increment the count. If it has overflowed, proceed to step 3.
[0009] Step 3: Check the non-most significant bit and determine if the lower 15 bits of the address are all 0s. If they are not all 0s, jump to step 4; if they are all 0s, jump to step 5.
[0010] Step 4: Calculate the new address position based on the offset of the lower 15 bits of the address, index the new address position, and jump to step 6;
[0011] Step 5: Allocate a new memory block with data type uint32, record the offset between the new address and the old address in the lower 15 bits of the original address, and jump to step 6.
[0012] Step 6: Increment the uint32 type counter at the new address by 1, check if the highest bit is 1. If the highest bit is 1, overflow again, and repeat steps 5 to 6.
[0013] Furthermore, in step 4, the new address position is calculated by adding the offset p to the initial address p0 of the old array and the allocated address m of the array. The calculation formula is new address p1 = p0 + p + m.
[0014] Furthermore, in step 5, the size of each memory expansion is doubled; taking uint16 as an example, the first expansion is to uint32, and the second is to uint64.
[0015] Furthermore, in step 5, the offset of the lower 15 bits of the old address is calculated by subtracting the initial address p0 of the array and the allocated memory m from the new address p1. The calculation formula is offset p = p1 - p0 - m.
[0016] Furthermore, it also has a memory reclamation function. It checks each element of the counting array. If the highest bit flag is overflow, it needs to calculate the offset to obtain the expanded memory address and reclaim the memory at the current address. This operation is repeated until there is no memory space left to reclaim.
[0017] Correspondingly, a fixed-length counting array overflow-based expansion system based on address translation includes: a counting array, which determines whether an overflow occurs by using the highest bit of the count; if an overflow occurs, the non-highest bit of the count is used as the relative offset of the newly requested memory, and the original count is used as an index to point to the new memory address; if the newly requested address overflows again, the above steps are repeated.
[0018] Correspondingly, a capacity expansion device for a fixed-length counter array based on address translation for counting overflow includes: one or more processors;
[0019] Storage device for storing one or more programs or user data;
[0020] When the one or more programs are executed by one or more processors, the one or more processors implement a method for expanding the capacity of a fixed-length counter array based on address translation to prevent overflow.
[0021] Correspondingly, a storage medium for expanding the capacity of a fixed-length counter array based on address translation to prevent overflow is provided, wherein a computer program is stored thereon, which, when executed by a processor, implements a method for expanding the capacity of a fixed-length counter array based on address translation to prevent overflow.
[0022] Correspondingly, a fixed-length counter array overflow expansion program product based on address translation includes a computer program that, when executed by a processor, implements a fixed-length counter array overflow expansion method based on address translation.
[0023] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: (1) It saves more memory space. Assuming the data overflow probability is 5%, it can save about 45% of memory space; (2) The time complexity is still O(1), and there will be no obvious performance difference; (3) There is no limit to the upper limit of the count, and the memory efficiency is higher for data with large variance. Assuming that 95% of the counts are less than the maximum value of uint16 and 5% of the counts are greater than the maximum value of uint64, it can save about 78% of memory space. Attached Figure Description
[0024] Figure 1 This is a diagram showing the array memory layout of the present invention.
[0025] Figure 2 This is a schematic diagram of the counting array self-incrementing and expansion process of the present invention.
[0026] Figure 3 This is a schematic diagram of the counting array query process of the present invention.
[0027] Figure 4 This is a schematic diagram of the memory reclamation process for the counting array of the present invention.
[0028] Figure 5 This is a schematic diagram of the process for triggering a new task according to the present invention.
[0029] Figure 6 This is a schematic diagram of the counting and accumulation process of the present invention.
[0030] Figure 7 This is a schematic diagram of the counting query process of the present invention. Detailed Implementation
[0031] like Figure 1-4 As shown, a method for expanding the capacity of a fixed-length counting array based on address translation to address overflow includes the following steps:
[0032] Step 1: Determine the data type and length of the array counting based on the possible upper limit and maximum value of most counts, and allocate contiguous memory; taking 100 uint16 elements as an example, allocate 200 bytes of memory and initialize the counting array;
[0033] Step 2: Check if the highest column of the array has overflowed. If it has not overflowed, increment the count. If it has overflowed, proceed to step 3.
[0034] Step 3: Check the non-most significant bit and determine if the lower 15 bits of the address are all 0s. If they are not all 0s, jump to step 4; if they are all 0s, jump to step 5.
[0035] Step 4: Calculate the new address position based on the offset of the lower 15 bits of the address, index the new address position, and jump to step 6;
[0036] Step 5: Allocate a new memory block, twice the size of the original memory block, with a data type of uint32. Record the offset between the new address and the old address in the lower 15 bits of the original address. Jump to step 6.
[0037] Step 6: Increment the uint32 type counter at the new address by 1, check if the highest bit is 1. If the highest bit is 1, overflow again, and repeat steps 5 to 6.
[0038] Suppose a system needs a function to count the number of packets passing through each port per hour. The maximum number of ports is known to be 65535, and the services started on each port may be used at different frequencies at different times. Using a traditional method with a large initial count data type would waste memory. The expected variance of this count is also relatively large; therefore, the solution of this invention is adopted, with a default count array data size of uint16.
[0039] Example 1:
[0040] This embodiment can be used to implement the construction and recycling process of a fixed-length counting array overflow expansion scheme based on address translation, such as... Figure 5 The process for triggering a new task includes the following steps:
[0041] S101, the timer triggers a task every hour, starts a new thread to collect new counting information, and recycles the old data in the old thread.
[0042] S102, the new thread requests 128KB of memory to store the initial counting array based on the total number of port numbers and the initially set data structure.
[0043] S103: The old thread reads the old counting array in a loop, and reclaims the memory of the entire old counting array after it has been completely read.
[0044] S104: For each element of the count array being read, check if the first bit is marked as overflow. If overflow occurs, read the remaining non-highest bit for offset addressing.
[0045] S105: Read the newly addressed counter array, check the highest bit again, and if it overflows again, repeat S104 and reclaim the memory.
[0046] Example 2:
[0047] This embodiment can be used to implement the counting accumulation process of a fixed-length counting array overflow expansion scheme based on address translation, such as... Figure 6 The counting and accumulation process is shown in the figure. This process specifically includes the following steps:
[0048] S201: Locate the position of the initial count array based on the port number and read the initial count.
[0049] S202, check if the highest bit overflow flag has overflowed. If it has not overflowed, directly accumulate. If it has overflowed, add the initial array memory allocation of 12KB and the record offset address p1 to obtain the addressing address p = p0 + 12KB + p1.
[0050] S203: Address the new address obtained in S202. If it overflows again, repeat step S202 until a memory address that has not overflowed is reached, and then increment the count.
[0051] Example 3:
[0052] This embodiment can be used to implement the counting query process in a fixed-length counting array overflow expansion scheme based on address translation, such as... Figure 7 The counting query process, as shown, includes the following steps:
[0053] S301 locates the position of the initial count array based on the port number and reads the initial count.
[0054] S302: Check if the highest bit overflow flag has overflowed. Accumulate the unoverflowed portion. If no overflow occurs, the query ends. If overflow occurs, add the initial array allocated memory of 12KB and the record offset address p1 to obtain the address p = p0 + 12KB + p1.
[0055] S303: Address the new address obtained in S302, accumulate the unexploded portion, and check again whether the count has overflowed. If it overflows, repeat steps S302 to S303.
[0056] Correspondingly, a fixed-length counting array overflow-based expansion system based on address translation includes: a counting array, which determines whether an overflow occurs by using the highest bit of the count; if an overflow occurs, the non-highest bit of the count is used as the relative offset of the newly requested memory, and the original count is used as an index to point to the new memory address; if the newly requested address overflows again, the above steps are repeated.
Claims
1. A method for expanding the capacity of a fixed-length counting array based on address translation to address overflow, characterized in that, Includes the following steps: Step 1: Determine the data type and length of the array counting based on the possible upper limit and maximum value of most counts, and allocate contiguous memory. Step 2: Check if the highest column of the array has overflowed. If it has not overflowed, increment the count. If it has overflowed, proceed to step 3. Step 3: Check the non-most significant bit and determine if the lower 15 bits of the address are all 0s. If they are not all 0s, jump to step 4; if they are all 0s, jump to step 5. Step 4: Calculate the new address position based on the offset of the lower 15 bits of the address, index the new address position, and jump to step 6; Step 5: Allocate a new memory block with data type uint32, record the offset between the new address and the old address in the lower 15 bits of the original address, and jump to step 6. Step 6: Increment the uint32 type counter at the new address by 1, check if the highest bit is 1. If the highest bit is 1, overflow again, and repeat steps 5 to 6.
2. The method for expanding the capacity of a fixed-length counting array based on address translation for overflow as described in claim 1, characterized in that, In step 4, the new address position is calculated by adding the offset p to the initial address p0 of the old array and the allocated address m of the array. The calculation formula is new address p1 = p0 + p + m.
3. The method for expanding a fixed-length counting array based on address translation to address overflow as described in claim 1, characterized in that, In step 5, each memory expansion is twice the original size.
4. The method for expanding the capacity of a fixed-length counting array based on address translation for overflow as described in claim 1, characterized in that, In step 5, the offset of the lower 15 bits of the old address is calculated by subtracting the initial address p0 and the allocated memory m from the new address p1. The calculation formula is offset p = p1 - p0 - m.
5. The method for expanding the capacity of a fixed-length counting array based on address translation for overflow as described in claim 1, characterized in that, It also has a memory reclamation function, which checks each element of the counting array. If the highest bit flag is overflow, it needs to calculate the offset to obtain the expanded memory address and reclaim the memory at the current address. This operation is repeated until there is no memory space left to reclaim.
6. A capacity expansion system for implementing the capacity expansion method for a fixed-length counting array based on address translation as described in claim 1, characterized in that, include: The counting array is used to determine whether an overflow occurs by checking the highest bit of the count. If an overflow occurs, the non-highest bit of the count is used as the relative offset of the newly allocated memory, and the original count is used as an index to point to the new memory address. If the newly allocated address overflows again, the above steps are repeated.
7. A capacity expansion device for a fixed-length counter array based on address translation that handles count overflow, including: One or more processors; Storage device for storing one or more programs or user data; When the one or more programs are executed by one or more processors, the one or more processors implement the address translation-based method for expanding the capacity of a fixed-length counter array for counting overflow as described in any one of claims 1 to 5.
8. An expansion storage medium for a fixed-length counter array overflow based on address translation, wherein a computer program is stored thereon, which, when executed by a processor, implements the expansion method for a fixed-length counter array overflow based on address translation as described in any one of claims 1 to 5.
9. A fixed-length counter array expansion program product based on address translation for counting overflow, comprising a computer program that, when executed by a processor, implements the fixed-length counter array expansion method based on address translation for counting overflow as described in any one of claims 1 to 5.