On-demand compression IP library file generation method
By using an on-demand compression method to generate IP library files, the problems of large file size and low transmission efficiency of IP library files in resource-constrained environments are solved, enabling flexible adjustment of file size and improved efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-02
- Publication Date
- 2026-03-17
AI Technical Summary
Existing commercial or open-source IP libraries have large file sizes, resulting in large space consumption and low transmission efficiency in resource-constrained servers, embedded devices, and high-concurrency systems.
An on-demand compression method for generating IP library files is adopted. By defining core data structures, multi-level compression enumerations, and command-line interactive classes, the on-demand compression of IP library files is achieved, generating target format files of different sizes.
It effectively reduces the space occupied by the IP library file, improves transmission efficiency, and adapts to the needs of different resource-constrained scenarios.
Smart Images
Figure CN121681484A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of data compression, and in particular relates to a method for generating IP library files that can be compressed on demand. Background Technology
[0002] Existing commercial or open-source IP library files are typically large in size, ranging from hundreds of MB to several GB, requiring high storage space and loading speed from the operating environment; they also suffer from large space consumption and low transmission efficiency in resource-constrained servers, embedded devices, or high-concurrency systems. Summary of the Invention
[0003] The purpose of this embodiment is to provide a method for generating IP library files with on-demand compression, which can solve the problems of large space occupation and low transmission efficiency in resource-limited servers, embedded devices or high-concurrency systems.
[0004] A method for generating IP library files with on-demand compression, comprising:
[0005] S1: Define the core data structure used to carry IP segments and corresponding regional data. This structure includes at least an IP segment class that stores the start and end addresses of IP segments and regional association information, and a regional class that stores regional subdivision information.
[0006] S2: Define a multi-level compression enumeration, which includes at least two differentiated compression strategies, and each compression level is configured with corresponding IP segment merging logic and special address exclusion rules;
[0007] S3: Implement an IP library file generator class, which integrates functions for reading raw data, merging and compressing IP segments, and writing target format files, and includes data continuity verification;
[0008] S4: Implement a command-line interaction class, which includes parameters such as parsing input / output paths, compression levels, and authorization information, and can initialize a generator class based on the parameters to generate IP library files;
[0009] S5: Package the code corresponding to the above steps into an executable program, including generating IP library files of different sizes in target format by specifying the compression level via the command line.
[0010] Furthermore, in the core data structure, the IP segment class uses long integers to store the start and end addresses of the IP segment in order to realize the numerical calculation of the continuity of the IP segment; the region class includes fields such as country, province, city, and operator, and the IP segment class establishes an association with the region class through object references to form a one-to-one correspondence between IP data and region information, providing structured data support for subsequent compression logic.
[0011] Furthermore, the multi-level compression enumeration includes at least six compression levels: no compression, unified merging of foreign IP segments, merging of foreign IP segments by country, merging of domestic IP segments by prefecture-level city, merging of foreign IP segments by country and domestic IP segments by prefecture-level city, and unified merging of foreign IP segments and domestic IP segments by prefecture-level city. Each compression level defines its own merging logic by implementing an abstract merging method, and at the same time, it uses a special address exclusion method to filter out reserved addresses that include the identifiers "address", "LAN", and "broadcast" to prevent reserved addresses from participating in the merging and ensure the integrity of core IP data.
[0012] Furthermore, the IP library file generator class includes a raw data input stream, a target file output path, a data conversion function, a compression level, and a merged IP segment list attribute. The data conversion function is used to parse the raw text data into IP segment class objects. The core methods of the generator class include: a method for loading and merging IP segments, which reads the raw text through a buffer stream, calls the merging logic corresponding to the compression level to merge IP segments, and verifies the continuity of adjacent IP segments (the end address of the previous segment + 1 equals the start address of the current segment); and a method for generating the target file, which reuses the configuration and writing classes of a third-party IP positioning framework to write the merged IP segments into the target file in the format of "country|province|city|operator".
[0013] Furthermore, the command-line interactive class defines a data conversion function that splits each line of data in the original text into the format "start IP|end IP|country|province|city|carrier". If the number of fields after splitting is less than 6, a format exception is thrown. At the same time, it calls an IP conversion tool to convert the IP address into a long integer, and then constructs region class and IP segment class objects. The interactive class includes parsing the required input path parameters (including local path and HTTP path), the optional output path parameters (defaulting to the file name specified in the current directory), the optional compression level parameters (defaulting to the highest compression level, with a value range of 0-5), and the optional HTTP authorization header parameters. When a parameter parsing error occurs, it outputs an error message and help documentation and terminates the program.
[0014] Furthermore, the merging logic for each compression level in the multi-level compression enumeration is as follows: The no-compression level returns a no-merge flag and does not perform any IP segment merging operation; the unified merging level for foreign IP segments only merges non-Chinese IP segments, and after merging, the country, province, and city fields for the region are set to "foreign," and the operator field is set to "0"; the merging level for foreign IP segments by country only merges non-Chinese IP segments within the same country, and after merging, the province and city fields for the region are set to the corresponding country name, and the operator field is set to "0"; the merging level for domestic IP segments by prefecture-level city only merges Chinese IP segments with identical province, city, and operator, and only updates the end address of the IP segment during merging; the merging level for foreign segments by country and domestic segments by prefecture-level city prioritizes the foreign-by-country merging logic, and if merging fails, it executes the domestic-by-prefecture-level city merging logic; the unified merging level for foreign segments and domestic segments by prefecture-level city prioritizes the unified foreign merging logic, and if merging fails, it executes the domestic-by-prefecture-level city merging logic.
[0015] Furthermore, the IP library file generator class provides a default encoding constructor and a custom encoding constructor. The default encoding constructor uses UTF-8 encoding, which is achieved by calling the custom encoding constructor and passing in the UTF-8 encoding parameter. The custom encoding constructor receives the raw data input stream, the target file output path, the data conversion function, the compression level, and the character encoding parameter, and assigns the parameter to the corresponding attribute to complete the initialization, including reading raw data in different encoding formats.
[0016] Furthermore, the method for loading and merging IP segments in the IP library file generator class employs an automatic resource shutdown mechanism to manage buffer stream resources and prevent resource leaks. When an exception occurs during the reading of raw data, the "failed to load IP segment data" message and exception stack are logged, and the exception is thrown upwards. After the IP segment loading and merging is completed, logs including the number of merged IP segments and the loading time are recorded to facilitate users in tracing the data processing status.
[0017] Furthermore, the execution flow of the method for generating the target file in the IP library file generator class is as follows: First, the method for loading and merging IP segments is called to obtain the list of merged IP segments; then, a configuration class for the third-party framework is created to set the output path and file overwrite permissions for the target file; next, the file writing class is managed using an automatic resource shutdown mechanism, the IP segment list is traversed, the region information string is concatenated, and the write method is called to write the start and end addresses and region information of the IP segments to the file; finally, a data refresh operation is performed, and a log of successful file generation is recorded; if an exception occurs during the process, the message "XDB file generation failed" and the exception stack are recorded, and the exception is thrown upwards.
[0018] Furthermore, the executable program is in JAR format; when called via command line, based on a hierarchical compression strategy, different compression levels generate target format IP library files of different sizes to adapt to different resource constraint scenarios.
[0019] This invention provides a method for generating IP library files with on-demand compression, which solves the problems of large space occupation and low transmission efficiency in resource-limited servers, embedded devices, or high-concurrency systems.
[0020] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description
[0021] Figure 1 : A flowchart illustrating the method steps for generating an on-demand compressed IP library file according to an embodiment of the present invention. Detailed Implementation
[0022] The technical solutions of the present invention will now be described with reference to the accompanying drawings in the embodiments of the present invention.
[0023] This embodiment is based on the xdb format IP library of the open-source IP location framework Ip2region. Addressing the issues of large original IP library file sizes and inconvenient deployment in resource-constrained environments, it adds a read adapter to the process of reading the original txt format IP data, achieving on-demand compression of the IP library. Adaptable scenarios include lightweight servers, embedded devices, and high-concurrency systems (such as Nginx reverse proxy and Spark data cleaning), including generating xdb format IP library files of different sizes by specifying compression levels via command line.
[0024] This invention provides a method for generating IP library files with on-demand compression (see...). Figure 1 ),include:
[0025] S1: Define the core data structure (Segment and Region classes), including the Segment class (including start IP, end IP, and region information) which stores IP segment information, and the Region class (including country, province, city, and carrier) which stores region details. This serves as the carrier of IP data in memory, ensuring that subsequent compression logic can operate based on this structure.
[0026] The specific implementation corresponds to the Java code.
[0027]
[0028]
[0029] S2: Define a compression level enumeration (CompressLevel), including 6 compression levels (None, Foreign, ForeignCountry, SameCity, ForeignCountry_SameCity, All) based on business requirements. Each level implements the merge2Last method to define the IP segment merging logic, and uses the mustNotMerge method to exclude the merging of reserved addresses (such as intranet IPs and LAN addresses) to ensure the integrity of core IP segment data.
[0030] Corresponding Java code
[0031]
[0032]
[0033]
[0034]
[0035]
[0036]
[0037] S3: Implement an XDB file generator and a Maker class, including a core utility class called Maker responsible for reading raw txt format IP data, executing compression logic, verifying data continuity, and generating an xdb format IP library file. Core methods include:
[0038] -Constructor method: Initializes input stream, output path, line conversion function, and compression level;
[0039] The `-loadSegments` method reads a txt file and merges IP segments according to their compression level.
[0040] - Data continuity check: Ensure that there are no breaks in the IP segment after merging (the last IP of the previous segment + 1 = the first IP of the current segment).
[0041]
[0042]
[0043]
[0044]
[0045]
[0046]
[0047] S4: Implement the command-line utility class (Main class), which includes receiving command-line parameters (input path, output path, compression level, etc.) through the Main class, parsing the parameters, initializing the Maker instance, and executing xdb file generation. It accepts local files and HTTP URLs as input sources, meeting the command-line call requirements in the documentation. In this embodiment, the command-line utility class provides a default encoding constructor and a custom encoding constructor. The default encoding constructor uses UTF-8 encoding, implemented by calling the custom encoding constructor and passing in UTF-8 encoding parameters. The custom encoding constructor receives the raw data input stream, the target file output path, the data conversion function, the compression level, and character encoding parameters, assigns the parameters to the corresponding attributes to complete initialization, including reading raw data in different encoding formats.
[0048]
[0049]
[0050]
[0051]
[0052]
[0053]
[0054]
[0055] S5: Packaging and Command Line Call. Execute the code in the above steps to package it into an executable JAR file (xdb-0.1.jar). Generate the corresponding xdb file by specifying different compression levels through the command line and verify the compression effect (e.g., the original 329MB txt file can be compressed to the 3MB level).
[0056] #Level 0: No compression (generates an xdb file of approximately 58MB)
[0057] java-jar xdb-0.1.jar-iip_ip138.txt-o ip2region_c0.xdb-c 0
[0058] #Level 1: Merge foreign entries into "foreign" (generates an xdb file of approximately 6.36MB)
[0059] java-jar xdb-0.1.jar-iip_ip138.txt-o ip2region_c1.xdb-c 1
[0060] #Level 2: Merge foreign countries by country (generates an xdb file of approximately 9.26MB)
[0061] java-jar xdb-0.1.jar-iip_ip138.txt-o ip2region_c2.xdb-c 2
[0062] #Level 3: Merge cities within China by prefecture level (generates an xdb file of approximately 54.3MB)
[0063] java-jar xdb-0.1.jar-iip_ip138.txt-o ip2region_c3.xdb-c 3
[0064] #Level 4: Merge foreign countries by country + merge domestic countries by prefecture-level city (generates an xdb file of approximately 5.48MB)
[0065] java-jar xdb-0.1.jar-iip_ip138.txt-o ip2region_c4.xdb-c 4
[0066] #Level 5: Foreign entities are merged into "Foreign" + Domestic entities are merged by prefecture-level city (generating an xdb file of approximately 3.06MB)
[0067] java-jar xdb-0.1.jar-iip_ip138.txt-o ip2region_c5.xdb-c 5
[0068] This invention provides a method for generating IP library files with on-demand compression, which solves the problems of large space occupation and low transmission efficiency in resource-limited servers, embedded devices, or high-concurrency systems.
[0069] The above description is merely an embodiment of the present invention and is not intended to limit the scope of protection of the present invention. For those skilled in the art, the present invention can have various modifications and variations. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A method of generating an on-demand compressed IP library file, characterized by, The method comprises the following steps: S1: defining a core data structure for carrying IP segments and corresponding regional data, which at least includes an IP segment class for storing IP segment start and end addresses and regional association information, and a region class for storing regional subdivision information; S2: defining a multi-level compression enumeration, which includes at least two differentiated compression strategies, and each compression level is configured with corresponding IP segment merging logic and special address exclusion rules; S3: implementing an IP library file generator class, which integrates original data reading, IP segment merging and compression, and target format file writing functions, and includes data continuity verification; S4: implementing a command line interaction class, which includes parsing input and output paths, compression levels, authorization information and other parameters, and can initialize the generator class to execute IP library file generation according to the parameters; S5: encapsulating the codes corresponding to the above steps into an executable program, which includes generating target format IP library files of different sizes by specifying compression levels through the command line.
2. The method of claim 1, wherein, In the core data structure, the start and end addresses of the IP segment are stored in the long integer type to realize the numerical calculation of the continuity of the IP segment; the region class includes country, province, city and operator fields, and the IP segment class is associated with the region class through object reference to form a one-to-one correspondence between IP data and regional information, providing structured data support for subsequent compression logic.
3. The method of claim 1, wherein, The multi-level compression enumeration includes at least six compression levels: no compression, unified merging of foreign IP segments, merging of foreign IP segments by country, merging of domestic IP segments by prefecture-level city, merging of foreign IP segments by country and domestic IP segments by prefecture-level city, and unified merging of foreign IP segments and domestic IP segments by prefecture-level city; each compression level defines exclusive merging logic through an abstract merging method, and filters out reserved addresses including "address", "local area network" and "broadcast" identifiers through a special address exclusion method to avoid the participation of reserved addresses in merging and ensure the integrity of core IP data.
4. The method of claim 1, wherein, The IP library file generator class includes original data input stream, target file output path, data conversion function, compression level, and merged IP segment list attribute; the data conversion function is used to parse the original text data into IP segment class objects; The core method of the generator class includes: a method for loading and merging IP segments, which reads the original text through a buffer stream, calls the merging logic corresponding to the compression level to implement IP segment merging, and verifies the continuity of adjacent IP segments (the end address of the previous segment + 1 equals the start address of the current segment); a method for generating a target file, which reuses the configuration class and writing class of a third-party IP positioning framework to write the merged IP segments into the target file in the "country|province|city|operator" format.
5. The method of claim 1, wherein, The command line interaction class defines a data conversion function which splits the original text data of each line into the format of "start IP | end IP | country | province | city | operator", and throws a format exception if the number of fields after splitting is less than 6. Meanwhile, the IP conversion tool is called to convert the IP address into a long integer, and then the regional class and IP segment class objects are constructed. The interaction class includes parsing the required input path parameter (including the local path and HTTP path), the optional output path parameter (the default is the specified file name in the current directory), the optional compression level parameter (the default corresponds to the highest compression level, the value range is 0-5), the optional HTTP authorization header parameter, and outputs error information and help documents and terminates the program when the parameter parsing exception occurs.
6. The method of claim 3, wherein, The merging logic of each compression level in the multi-level compression enumeration is as follows: the non-compression level returns a non-merging identifier and does not perform any IP segment merging operation; the foreign IP segment uniform merging level only performs merging on non-Chinese IP segments, and after merging, the country, province and city fields of the regional class are set to "foreign", and the operator field is set to "0"; the foreign IP segment by country merging level only performs merging on non-Chinese IP segments of the same country, and after merging, the province and city fields of the regional class are set to the corresponding country name, and the operator field is set to "0"; the domestic IP segment by prefecture-level city merging level only performs merging on Chinese IP segments with consistent province, city and operator, and only updates the end address of the IP segment during merging; the foreign by country merging and domestic by prefecture-level city merging level preferentially performs the foreign by country merging logic, and performs the domestic by prefecture-level city merging logic when merging fails; the foreign uniform merging and domestic by prefecture-level city merging level preferentially performs the foreign uniform merging logic, and performs the domestic by prefecture-level city merging logic when merging fails.
7. The method of claim 4, wherein, The IP library file generator class provides a default encoding construction method and a custom encoding construction method; the default encoding construction method uses UTF-8 encoding, which is implemented by calling the custom encoding construction method and passing in the UTF-8 encoding parameter; the custom encoding construction method receives the original data input stream, the target file output path, the data conversion function, the compression level, and the character encoding parameter, assigns the parameters to the corresponding attributes to complete initialization, including reading the original data in different encoding formats.
8. The method of claim 4, wherein, The method of loading and merging IP segments in the IP library file generator class uses a resource automatic closing mechanism to manage buffer stream resources, avoiding resource leakage; when an exception occurs during reading the original data, the "loading IP segment data failed" information and exception stack are recorded through logging, and the exception is thrown upwards; After the IP segment loading and merging are completed, the logs including the number of merged IP segments and the loading time are recorded, which facilitates users to trace the data processing situation.
9. The method of claim 4, wherein, The method flow of generating target files in the IP library file generator class is as follows: firstly, a method of loading and merging IP segments is called to obtain a merged IP segment list; then, a configuration class of a third-party framework is created, and a target file output path and file overwrite permission are set; then, a resource automatic closing mechanism is used to manage a file writing class, a regional information string is spliced by traversing the IP segment list, and a writing method is called to write IP segment start and end addresses and regional information into a file; finally, a data refresh operation is performed, a file generation success log is recorded, if an exception occurs in the process, "XDB file generation failure" information and an exception stack are recorded, and an exception is thrown upwards.
10. The method of claim 1, wherein, The executable program is in JAR format; when called through a command line, different compression levels correspond to target format IP library files of different sizes generated based on a hierarchical compression strategy, which are used to adapt to different resource constraint scenarios.