Information processing method and apparatus, computer storage medium, electronic device

Abstract

The present disclosure relates to the technical field of computer, and provides an information processing method, an information processing device, a computer storage medium and an electronic device, wherein the information processing method comprises: acquiring a to-be-processed data table, wherein the to-be-processed data table comprises belonging place information corresponding to a plurality of IP segments, the to-be-processed data table comprises a plurality of rows, each row is used for storing a plurality of belonging place information contained by each IP segment, and the plurality of belonging place information is distributed in a plurality of columns of the to-be-processed data table; performing a deduplication processing on the belonging place information of each column, and generating a column file according to the belonging place information of each column after the deduplication processing; creating a reference table of a plurality of column files, and storing a storage pointer of each belonging place information in the reference table; and the storage pointer is used for indicating a storage position of the belonging place information in the column file. The present disclosure can quickly locate the belonging place information of each column, and improves the information loading speed.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to an information processing method, an information processing device, a computer storage medium, and an electronic device. Background Technology

[0002] IP addresses (Internet Protocol addresses) are widely used in fields such as internet advertising, fraud prevention and risk control, network security, and big data analytics, and have become an important source of fundamental data. After an IP address is purchased from a service provider, its associated geographic location information needs to be processed and stored before being loaded into memory for use.

[0003] In related technologies, after processing, the location information of each IP address is typically stored in rows using the DAT format. However, this storage method cannot quickly locate the location information of a specific column during subsequent queries.

[0004] Therefore, there is an urgent need in this field to develop a new information processing method and apparatus.

[0005] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure. Summary of the Invention

[0006] The purpose of this disclosure is to provide an information processing method, an information processing device, a computer storage medium, and an electronic device, thereby at least to some extent avoiding the shortcomings of related technologies in that they cannot quickly locate a certain column of location information.

[0007] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part by practice of this disclosure.

[0008] According to a first aspect of this disclosure, an information processing method is provided, comprising: obtaining a data table to be processed, the data table including location information corresponding to multiple IP segments, the data table including multiple rows, each row storing multiple location information contained in each IP segment, the multiple location information being distributed in multiple columns of the data table to be processed; performing deduplication processing on each column of location information, and generating a column file based on the deduplicated location information in each column; creating a reference table of multiple column files, storing a storage pointer for each location information in the reference table; the storage pointer indicating the storage location of the location information in the column file.

[0009] In an exemplary embodiment of this disclosure, generating a column file based on the deduplication-processed location information of each column includes: obtaining the length information of each remaining location information in each column after deduplication; and generating the column file based on the length information of each location information and the location information.

[0010] In an exemplary embodiment of this disclosure, creating a reference table of multiple column files includes: determining the number of rows in the reference table based on the number of rows in the data table to be processed; determining the number of bytes occupied by each column file based on the number of bytes occupied by each location information in the column file; determining the row storage space of the reference table based on the number of bytes occupied by each column file; and creating the reference table based on the number of rows in the reference table and the row storage space of the reference table.

[0011] In an exemplary embodiment of this disclosure, the number of bytes occupied by each of the location information in the column file includes the sum of the number of bytes occupied by each of the location information and the length information corresponding to each of the location information.

[0012] In an exemplary embodiment of this disclosure, determining the row storage space of the reference table based on the number of bytes occupied by each column file includes: determining the number of bits occupied by the storage pointer of each location information in the column file in each row of the reference table based on the number of bytes occupied by each column file; and determining the row storage space of the reference table based on the sum of the number of bits occupied by the storage pointers of the plurality of location information in each row of the reference table and a reserved specified number of bits.

[0013] In an exemplary embodiment of this disclosure, storing the storage pointer for each location information in the reference table includes: selecting a target row storage space from the reference table according to the row number of the target IP segment in the data table to be processed; obtaining the storage location of the target storage pointer in the target row storage space; the target storage pointer being the storage pointer for each location information contained in the target IP segment; and storing the target storage pointer in the storage location.

[0014] In an exemplary embodiment of this disclosure, the storage location of the target storage pointer in the target row storage space includes: the number of bits occupied by the target storage pointer in the target row storage space; the number of bits skipped by the target storage pointer in the first byte of the target row storage space; the starting byte position of the target storage pointer in the target row storage space; and the number of bytes occupied by the target storage pointer in the target row storage space.

[0015] In an exemplary embodiment of this disclosure, the number of bits occupied by the target storage pointer in the target row storage space is determined by: determining the number of bits occupied by the target storage pointer in the target row storage space based on the number of bytes occupied by the column file to which the location information belongs.

[0016] In an exemplary embodiment of this disclosure, the number of bits skipped by the target storage pointer in the first byte of the target row storage space is determined by: obtaining the column number n of the location information in the data table to be processed; when the column number n equals m, determining the preset number of bits as the number of bits skipped by the target storage pointer in the first byte of the target row storage space; when the column number n is not equal to m, determining the number of bits skipped by the target storage pointer in the first byte of the target row storage space based on the sum of the number of bits occupied by the storage pointer of each location information in the column file from column m to column n-1 in each row of the reference table and a first target value; wherein m and n are integers, m is less than or equal to n, and m is the column number corresponding to the first column of location information in the data table to be processed.

[0017] In an exemplary embodiment of this disclosure, determining the number of bits skipped by the target storage pointer in the first byte of the target row storage space based on the sum of the number of bits occupied by the storage pointer of each location information in the column file from column m to column (n-1) and a first target value includes: taking the remainder of the sum of the number of bits divided by the first target value to obtain the number of bits skipped by the target storage pointer in the first byte of the target row storage space; wherein m and n are integers, and m is less than or equal to n.

[0018] In an exemplary embodiment of this disclosure, the starting byte position of the target storage pointer in the target row storage space is determined by: obtaining the column number n of the location information in the data table to be processed; when the column number n is equal to m, determining the preset byte position as the starting byte position of the target storage pointer in the target row storage space; when the column number n is not equal to m, determining the starting byte position of the target storage pointer in the target row storage space based on the sum of the number of bits occupied by the storage pointer of each location information in the column file from column m to column (n-1) in the reference table and a first target value; wherein m and n are integers, m is less than or equal to n, and m is the column number corresponding to the first column of location information in the data table to be processed.

[0019] In an exemplary embodiment of this disclosure, determining the starting byte position of the target storage pointer in the target row storage space based on the sum of the number of bits occupied by the storage pointer of each location information in the column file from column m to column n-1 in the reference table and a first target value includes: rounding down the quotient obtained by dividing the sum of the number of bits by the first target value to obtain the starting byte position of the target storage pointer in the target row storage space.

[0020] In an exemplary embodiment of this disclosure, the number of bytes occupied by the target storage pointer in the target row storage space is determined by: determining whether the number of bits skipped by the target storage pointer in the first byte of the target row storage space is the preset number of bits; if it is the preset number of bits, then the quotient obtained by dividing the number of bits occupied by the target storage pointer in the target row storage space by the first target value is rounded up to obtain the number of bytes occupied by the target storage pointer in the target row storage space; if it is not the preset number of bits, then the number of bytes occupied by the target storage pointer in the target row storage space is determined based on the number of bits skipped by the target storage pointer in the first byte of the target row storage space, the number of bits occupied by the target storage pointer in the target row storage space, the first target value, and the second target value.

[0021] In an exemplary embodiment of this disclosure, the step of determining the number of bytes occupied by the target storage pointer in the target row storage space based on the number of bits skipped by the target storage pointer in the first byte of the target row storage space, the number of bits occupied by the target storage pointer in the target row storage space, the first target value, and the second target value if the number of bits is not the preset number of bits includes: if the number of bits is not the preset number of bits, obtaining a first difference between the first target value and the number of bits skipped by the target storage pointer in the first byte of the target row storage space; obtaining a second difference between the number of bits occupied by the target storage pointer in the target row storage space and the first difference; rounding up the quotient obtained by dividing the second difference by the first target value; and summing the second target value and the rounded value to obtain the number of bytes occupied by the target storage pointer in the target row storage space.

[0022] In an exemplary embodiment of this disclosure, the method further includes: upon receiving a location query request for an IP to be queried, obtaining the IP segment in which the IP to be queried is located; determining, based on the IP segment in which the IP to be queried is located, the row number of the storage pointer of the location information corresponding to the IP to be queried in the reference table; obtaining the storage pointer corresponding to the location information of the IP to be queried from the target row storage space corresponding to the row number; and reading the location information from the column file based on the storage pointer.

[0023] According to a second aspect of this disclosure, an information processing apparatus is provided, comprising: a data acquisition module for acquiring a data table to be processed, the data table including location information corresponding to multiple IP segments, the data table including multiple rows, each row storing multiple location information contained in each IP segment, the multiple location information being distributed in multiple columns of the data table; a columnar storage module for deduplicating each column of location information and generating a column file based on the deduplicated location information in each column; and a reference table creation module for creating a reference table of multiple column files, storing a storage pointer for each location information in the reference table; the storage pointer indicating the storage location of the location information in the column file.

[0024] According to a third aspect of this disclosure, a computer storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the information processing method described in the first aspect above.

[0025] According to a fourth aspect of this disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the information processing method described in the first aspect by executing the executable instructions.

[0026] As can be seen from the above technical solutions, the information processing method, information processing apparatus, computer storage medium, and electronic device in the exemplary embodiments of this disclosure have at least the following advantages and positive effects: In some embodiments of this disclosure, the technical solutions provide the following: First, a data table to be processed is obtained (the data table includes location information corresponding to multiple IP segments, the data table includes multiple rows, each row stores multiple location information contained in each IP segment, and the multiple location information is distributed in multiple columns of the data table to be processed). Second, each column of location information is deduplicated, and a column file is generated based on the deduplicated location information of each column. This solves the problem of duplicate information occupying a large amount of data and reduces the amount of data occupied by the column file. Third, a reference table of multiple column files is created, and a storage pointer of each location information is stored in the reference table. This facilitates the rapid loading of all location information of a certain column based on the storage pointer, solving the problem in related technologies where row-based storage requires reading all columns of a row each time data is read, making it impossible to quickly load location information of a certain column.

[0027] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0028] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0029] Figure 1 A flowchart illustrating the information processing method in an embodiment of this disclosure is shown; Figure 2 This diagram illustrates the process of determining the row storage space of the reference table in an embodiment of the present disclosure. Figure 3 This diagram illustrates a process for storing a pointer to each location information in a column file within a reference table, as shown in an embodiment of this disclosure. Figure 4 This illustration shows a flowchart of the process for determining the number of bits skipped by the target storage pointer in the first byte of the target row storage space in an embodiment of this disclosure; Figure 5 This illustration shows a flowchart of determining the starting byte position of the target storage pointer in the target row storage space according to an embodiment of the present disclosure; Figure 6 This illustration shows a flowchart of the process for determining the number of bytes occupied by the target storage pointer in the target row storage space in an embodiment of this disclosure; Figure 7 A schematic diagram of the information processing method in an embodiment of this disclosure is shown; Figure 8 This diagram illustrates a flowchart of a process for querying the location information corresponding to an IP address in an embodiment of this disclosure. Figure 9 This diagram illustrates the process of querying the location information corresponding to an IP address in an embodiment of this disclosure. Figure 10 This diagram illustrates the structure of the information processing apparatus in an exemplary embodiment of the present disclosure. Figure 11 A schematic diagram of the structure of an electronic device in an exemplary embodiment of this disclosure is shown. Detailed Implementation

[0030] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this disclosure more comprehensive and complete, and to fully convey the concept of the example embodiments to those skilled in the art. The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a full understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more of the specific details omitted, or other methods, components, apparatus, steps, etc., can be employed. In other instances, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0031] The terms “a,” “an,” “the,” and “the” are used in this specification to indicate the presence of one or more elements / components / etc.; the terms “including” and “having” are used to indicate an open-ended inclusion and to mean that there may be other elements / components / etc. in addition to the listed elements / components / etc.; the terms “first” and “second” are used only as markings and are not a limitation on the number of objects.

[0032] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities.

[0033] In related technologies, the DAT file format is commonly used to store the location information of IP addresses. The specific data format is shown in Table 1 below: Table 1

[0034] Referring to Table 1, the DAT file format includes a header, record area, index area, and description area. Among them: The file header occupies a fixed length of bytes and stores the start and end positions of the index area and the file size; The record area occupies a variable number of bytes and stores the IP address ending in the IP network segment, the length of each column of location information, and the location information. The size of the index area is determined by the file size, and it stores the starting IP address of the IP network segment and the starting position of each column of location information; The description area occupies a variable number of bytes and stores metadata related to the DAT, such as creation time and version.

[0035] However, the above storage method has at least the following drawbacks: First, although some compression has been performed (mainly for the same location value), as the information related to IP addresses becomes more complete, the space occupied after all of it is loaded into memory during use is still very large, and there is still room for further compression. Second, IP address data has many location columns, and storing it row by row cannot achieve on-demand loading and use, nor can it quickly locate the value of each column, resulting in poor scalability and ease of use; Third, regardless of the size of the IP address data, it is currently loaded entirely into memory, and only indexed mode reading is supported.

[0036] In the embodiments of this disclosure, an information processing method is first provided, which at least to some extent overcomes the defect in related technologies that cannot quickly load a column of location information.

[0037] Figure 1 The diagram shows a flowchart of an information processing method in an embodiment of this disclosure. The executing entity of this information processing method may be a server that processes the location information of IP addresses.

[0038] refer to Figure 1 An information processing method according to an embodiment of the present disclosure includes the following steps: Step S110: Obtain a data table to be processed. The data table to be processed includes location information corresponding to multiple IP segments. The data table to be processed includes multiple rows, each row is used to store multiple location information contained in each IP segment, and the multiple location information is distributed in multiple columns of the data table to be processed. Step S120: Perform deduplication on the location information in each column, and generate a column file based on the deduplicated location information in each column; Step S130: Create a reference table for multiple column files, and store a storage pointer for each location information in the reference table.

[0039] exist Figure 1In the technical solution provided by the illustrated embodiment, on the one hand, a data table to be processed is obtained (the data table to be processed includes location information corresponding to multiple IP segments, the data table to be processed includes multiple rows, each row is used to store multiple location information contained in each IP segment, and the multiple location information is distributed in multiple columns of the data table to be processed). On the other hand, deduplication processing is performed on the location information in each column, and a column file is generated based on the location information in each column after deduplication processing. This can solve the problem of duplicate information occupying a large amount of data and reduce the amount of data occupied by the column file. On the other hand, a reference table of multiple column files is created, and a storage pointer of each location information is stored in the reference table. This facilitates the rapid loading of all location information of a certain column based on the storage pointer, solving the problem in related technologies where row storage requires reading all columns of a row each time data is read, making it impossible to quickly load the location information of a certain column.

[0040] The following are Figure 1 The specific implementation process of each step in the process will be explained in detail: In step S110, the data table to be processed is obtained.

[0041] In this step, the aforementioned data table to be processed can be a data table containing relevant information about multiple IP addresses. For example, refer to Table 2: Table 2

[0042] Table 2 contains k rows of data. Each row stores multiple location information for each IP network segment (for example, the row number of each column from top to bottom can be 0, 1, 2, 3...k). Multiple location information are distributed in different columns (for example, the column number of each column from left to right can be 0, 1, 2, 3...).

[0043] For example, taking the data in the first row (row number 0) of Table 2 as an example, the first row represents the multiple location information contained in the IP network segment with the starting IP IP_a and the ending IP IP_d. China represents the country, Shanghai represents the municipality or province, Shanghai represents the city, xxxx.com represents the owner (e.g., school or company), and CN represents the English abbreviation of the country. For example, Table 2 may also include information such as the operator of each IP network segment, the latitude and longitude of the city / province / country center point, the representative city of the time zone, the time zone, the administrative division code, and the country code. These can all be set according to the actual situation, and this disclosure does not impose any special restrictions on them.

[0044] It should be noted that the above IP addresses can also be converted to unit32 integers (an IP address is itself a 32-bit binary number, but it is usually represented in the form of abcd. An IP address is divided into four segments, each of which is a number between 0 and 255. Each segment can be stored in 8 bits, which means that the IP address can be converted into a 32-bit integer). Then, the start IP data and end IP data of each IP network segment after unit32 conversion can be added to Table 1 above. This makes it easier to determine the network segment of an IP address based on the numerical range of the IP address, so as to quickly query the location information of the IP address.

[0045] In step S120, the location information in each column is deduplicated, and a column file is generated based on the deduplicated location information in each column.

[0046] In this step, after obtaining the above data table to be processed, the location information in each column can be deduplicated, and then a column file can be generated based on the deduplicated location information in each column.

[0047] For example, the location information in each column of Table 2 above can be deduplicated. Deduplication means deleting duplicate location information and recording the information only when it appears for the first time. This can avoid a large amount of duplicate and redundant information in the generated column file, thereby reducing the amount of data in the generated column file and improving the loading speed of the column file.

[0048] After deduplicating the location information in each column, the length information of each remaining location information in each column can be obtained. For example, referring to Table 2, taking the location information in the 4th column (column number 3) after deduplication as Shanghai, Beijing, and Hong Kong and Macau Special Administrative Regions as an example, the length information valen_len1 (e.g., 4) for "Shanghai", valen_len2 (e.g., 4) for "Beijing", and valen_len3 (e.g., 8) for "Hong Kong and Macau Special Administrative Regions" can be obtained respectively. Therefore, when generating the column file, the length information valen_len of each location information can be stored first, followed by the storage of each location information itself. For example, referring to Table 3, Table 3 shows the column file generated based on the location information in the 4th column (column number 3) of the data table to be stored. Table 3

[0049] As shown in Table 3, the length information 4 corresponding to "Shanghai" can be stored in value_len1, and then "Shanghai" can be stored in byte1-byte4. The length information 4 corresponding to "Beijing" can be stored in value_len2, and then "Beijing" can be stored in byte1-byte4. The length information 8 corresponding to "Hong Kong and Macao Special Administrative Region" can be stored in value_len3, and then "Hong Kong and Macao Special Administrative Region" can be stored in byte1-byte8 to generate the column file corresponding to the third column of location information.

[0050] It should be noted that the column number of each column file is the same as the column number of the location information in the aforementioned data table to be stored. Therefore, since the column number of the location information in the aforementioned data table to be stored starts from 2, the column number of the column file also starts from 2.

[0051] By storing the location information corresponding to IP network segments in a columnar manner, and since the type of location information in each column is the same, it is easier to perform data compression on the column file in the future, so as to further reduce the memory space occupied by the column file.

[0052] In step S130, a reference table for multiple column files is created, and a storage pointer for each location information is stored in the reference table.

[0053] In this step, in order to facilitate the rapid reading of each location information from the above column files, a reference table of the above column files needs to be created at the same time as generating the above column files, so as to record the storage pointer of each location information in the above column files in the reference table.

[0054] Specifically, we can first determine the number of rows in the reference table and the storage space for each row.

[0055] For example, the number of rows in the reference table can be determined based on the number of rows in the data table to be processed. Thus, referring to Table 2, it can be seen that the data table to be processed contains k rows (i.e., the data table to be processed contains k IP network segments), and thus, the number of rows in the reference table can also be determined to be k.

[0056] refer to Figure 2 , Figure 2 This diagram illustrates a flowchart of determining the row storage space of the reference table in an embodiment of the present disclosure, including steps S201-S203: In step S201, the number of bytes occupied by each column file is determined based on the number of bytes occupied by each location information in the column file.

[0057] In this step, for each column after deduplication, we can first count the number of bytes occupied by each remaining location information in the column file, and then determine the number of bytes occupied by each column file based on the number of bytes occupied by each remaining location information in the column file.

[0058] Specifically, referring to Table 3 above, the number of bytes occupied by each location information in the list file is determined by two parts: the number of bytes occupied by each location degree information and the number of bytes occupied by the length information (value_len) of each location information. For example, the number of bytes occupied by each location degree information can be a preset 1 byte. Therefore, the number of bytes occupied by each location information in the list file can be calculated based on the following formula 1: p(i)=bytelen(i)+1 Formula 1 Where i represents the sort number of the location information in the deduplication column (e.g., 0, 1, 2, 3, ...), p(i) represents the number of bytes occupied by the i-th location information in the column file, bytelen(i) represents the number of bytes occupied by the i-th location information, and 1 represents the number of bytes occupied by the length information of the location information. Thus, the number of bytes occupied by each location information in the column file is the sum of the number of bytes occupied by each location information and its length information.

[0059] After calculating the number of bytes occupied by each location information in the column file, the total number of bytes occupied by each column file can be determined by summing the total number of bytes occupied by all location information in that column, as shown in Formula 2 below: num_bytes= Formula 2 Where num_bytes represents the number of bytes occupied by the column file to which the nth column of location information belongs, where n can be 2, 3, 4, ..., and t represents the number of location information items contained in the above column file.

[0060] Taking Table 2 above, which contains 18 location columns, as an example, this will generate 18 column files, with column numbers 2-19. The number of bytes occupied by each column file can be found in Table 4. Table 4

[0061] As shown in Table 4, col represents the column number of each column file, and num_bytes represents the number of bytes occupied by each column file.

[0062] In step S202, the row storage space of the reference table is determined based on the number of bytes occupied by each column file.

[0063] In this step, after determining the number of bytes occupied by each column file, the number of bits occupied by the storage pointer for each location information in that column in each row of the reference table can be determined based on the number of bytes occupied by each column file. Specifically, refer to the following formula 3: need_bit_num=math.ceil(math.log(num_bytes,2)) Formula 3 Here, need_bit_num represents the number of bits occupied by the storage pointer of each location information in each row of the reference table, the math.log function returns the natural logarithm of the parameter, and the math.ceil function returns the value of the parameter after rounding up.

[0064] Next, referring to Table 4, taking the first row of data in Table 4 as an example, i.e., num_bytes is 11006, then math.log(11006, 2) = 13.426002613329361, math.ceil(13.426002613329361) = 14, that is, need_bit_num is 14 bits. Furthermore, the row storage space of the reference table can be determined based on the sum of the number of bits occupied by the storage pointers of the multiple location information in each row of the reference table, and the reserved number of bits. Specifically, referring to Table 4, the sum of the number of bits occupied by the storage pointers for the aforementioned multiple location information in each row of the reference table is: 14+15+18+20+19+18+18+13+8+12+10+10+5+4+4+10+3+4=205 bits. The start and end IP addresses of the IP network segment require a total of 8 bytes (1 byte = 8 bits), therefore, the reserved number of bits mentioned above is 8. Since 8 = 64 bits, the row storage space of the reference table can be determined to be 205 + 64 = 269 bits. Converting this to bytes and rounding up, we get 269 / 8 rounded up to 34 bytes.

[0065] In step S203, a reference table is created based on the number of rows in the reference table and the row storage space of the reference table.

[0066] In this step, after determining the number of rows and the row storage space of the reference table, the reference table can be created. For example, the created reference table can be found in Table 5: Table 5

[0067] After creating the reference table described above, a storage pointer for the location information of each row can be stored in the storage space of each row of the reference table. For example, refer to Table 6, which shows the reference table corresponding to Table 2 above: Table 6

[0068] Referring to Table 6, the first row of data in Table 6 represents the storage pointers corresponding to the location information in each column of the IP network segment (IP_a to IP_d). For example, pos_a1 to pos_a18 represent the storage pointers corresponding to the location information (China-Shanghai-Shanghai Municipality-xxxx.com-…-CN) in Table 2 with row number 0 and column number 2 to 19. pos_a1 to pos_a18 occupy a total of 34 bytes.

[0069] For details, please refer to Figure 3 , Figure 3 This diagram illustrates a process for storing a pointer to each home location information in a reference table according to an embodiment of this disclosure, including steps S301-S303: In step S301, the target row storage space is selected from the reference table according to the row number of the target IP segment in the data table to be processed.

[0070] In this step, the row number of the target IP segment in the data table to be processed can be obtained first. For example, the target IP segment can be IP_a to IP_d in Table 2. Thus, it can be determined that the row number of the target IP segment in the data table to be processed is 0.

[0071] After determining that the above row number is 1, it can be determined that the storage pointers corresponding to the multiple home location information of the target IP segment are all stored in the 34 bytes corresponding to row number 0 in the reference table.

[0072] In step S302, the location of the target storage pointer in the target row storage space is obtained.

[0073] In this step, the target storage pointer is the storage pointer of each home location information contained in the target IP segment mentioned above.

[0074] The aforementioned storage location specifically includes the following four elements: the number of bits occupied by the target storage pointer in the target row storage space (bit_len); the number of bits skipped by the target storage pointer in the first byte of the target row storage space (bit_start); the starting byte position of the target storage pointer in the target row storage space (byte_start); and the number of bytes occupied by the target storage pointer in the target row storage space (byte_len).

[0075] The number of bits (bit_len) occupied by the target storage pointer in the target row storage space can be determined in the following way: Taking the location information "China" in the 1st row and 3rd column (row number 0, column number 2) of the data table to be stored (Table 2) as an example, we can obtain the number of bytes occupied by the column file to which the location information "China" belongs. Referring to Table 4 above, we know that "China" is located in the column file with column number 2, and the column file with column number 2 occupies 11006 bytes. Furthermore, we can determine that the target storage pointer corresponding to the location information "China" occupies 14 bits in the above-mentioned 34-byte target row storage space.

[0076] Taking the location information "Shanghai" in the 1st row and 5th column (row number 0, column number 4) of the data table to be stored as an example, we can obtain the number of bytes occupied by the column file to which the location information "Shanghai" belongs. Referring to Table 4 above, we know that "Shanghai" is located in the column file with column number 4, and the column file with column number 4 occupies 237020 bytes. Furthermore, we can determine that the target storage pointer corresponding to the location information "Shanghai" occupies 18 bits in the above-mentioned 34-byte target row storage space.

[0077] The number of bits (bit_start) that the target storage pointer skips in the first byte of the target row storage space can be determined in the following way: refer to Figure 4 , Figure 4 This illustration shows a flowchart of determining the number of bits skipped by the target storage pointer in the first byte of the target row storage space in an embodiment of this disclosure, including steps S401-S403: In step S401, the column number n of the location information in the data table to be processed is obtained.

[0078] In step S402, when column number n equals m, the preset number of bits is determined as the number of bits that the target storage pointer skips in the first byte of the target row storage space.

[0079] In this step, m can be 2 (where m is the column number of the location information in the first column of the data table to be processed). Therefore, taking the location information "China" in the 1st row and 3rd column (row number 0, column number 2) of the data table to be stored as an example, its column number is 2, which determines that n = m. Furthermore, it can be determined that the number of bits skipped by the first byte of the target storage pointer in the target row storage space is 0.

[0080] In step S403, when column number n is not equal to m, the number of bits skipped by the target storage pointer in the first byte of the target row storage space is determined based on the sum of the number of bits occupied by the storage pointer of each location information in the column file from column m to column n-1 in each row of the reference table and the first target value.

[0081] In this step, when n is not equal to m, for example, referring to Table 2, taking the location information "Shanghai" in the 5th column of the 1st row (row number 0, column number 4) of the data to be stored as an example, we can obtain the number of bits occupied by the storage pointer of each location information in the reference table in the 2nd column file and the sum of the number of bits occupied by the storage pointer of each location information in the reference table in the 3rd column file, 14+15=29 bits. Taking the remainder of the sum of the bits 29 divided by the first target value 8, 29%8=5. Therefore, the number of bits skipped by the target storage pointer in the first byte of the target row storage space is 5 bits.

[0082] The starting byte position (byte_start) of the target storage pointer in the target row storage space can be determined in the following way: refer to Figure 5 , Figure 5 This diagram illustrates the flowchart for determining the starting byte position of the target storage pointer in the target row storage space according to an embodiment of the present disclosure, including steps S501-S503: In step S501, the column number n of the location information in the data table to be processed is obtained.

[0083] In step S502, when column number n equals m, the preset byte position is determined as the starting byte position of the target storage pointer in the target row storage space.

[0084] In this step, m can be 2. Therefore, taking the location information "China" in the 1st row and 3rd column (row number 0, column number 2) of the data table to be stored as an example, its column number is 2, so we can determine that n=m. Furthermore, we can determine that the starting byte position of the target storage pointer in the target row storage space is 0byte.

[0085] In step S503, when column number n is not equal to m, the starting byte position of the target storage pointer in the target row storage space is determined based on the sum of the number of bits occupied by the storage pointer of each location information in each row of the reference table from column m to column (n-1) and the first target value.

[0086] In this step, when n is not equal to m, for example, referring to Table 2, taking the location information "Shanghai" in the 1st row and 5th column (row number 0, column number 4) as an example, then n=4. At this time, we can obtain the number of bits occupied by the storage pointer of each location information in the reference table in the 2nd column file and the sum of the number of bits occupied by the storage pointer of each location information in the reference table in the 3rd column file, 14+15=29 bits. Divide the sum of bits 29 by the first target value 8 and round down, 29 / 8=3, to get the starting byte position of the target storage pointer in the target row storage space as the 3rd byte.

[0087] The number of bytes (byte_len) occupied by the target storage pointer in the target row storage space can be determined in the following way: Specifically, we can first determine whether the number of bits skipped by the target storage pointer in the first byte of the target row storage space is the preset number of bits. This preset number of bits can be 0. Therefore, if it is determined that the number of bits skipped by the target storage pointer in the first byte of the target row storage space is 0, we can divide the number of bits occupied by the target storage pointer in the target row storage space by the first target value 8 and round up the quotient to obtain the number of bytes occupied by the target storage pointer in the target row storage space. For example, taking the location information "China" in the 1st row and 3rd column (row number 0, column number 2) of the data table to be stored as an example, the number of bits occupied by its target storage pointer in the target row storage space is 14 bits. The quotient 14 / 8 rounded up is 2. Therefore, the location information "China" occupies 2 bytes in the 34-byte target row storage space.

[0088] If it is determined that the number of bits skipped in the first byte of the target storage pointer in the target row storage space is not 0, you can refer to... Figure 6 , Figure 6 This illustration shows a flowchart of determining the number of bytes occupied by the target storage pointer in the target row storage space in an embodiment of this disclosure, including steps S601-S604: In step S601, the first difference between the first target value and the number of bits skipped by the target storage pointer in the first byte of the target row storage space is obtained.

[0089] In this step, we take the location information "Shanghai" in the 5th column of the 1st row (row number 0, column number 4) of the data table to be stored as an example. Referring to the relevant explanation of step S402, we know that the number of bits skipped by the target storage pointer in the first byte of the target row storage space is 5 bits. Therefore, we can obtain the first difference between the first target value 8 and 5: 8-5=3.

[0090] In step S602, the second difference between the number of bits occupied by the target storage pointer in the target row storage space and the first difference is obtained.

[0091] In this step, referring to the relevant explanations in the above steps, we know that the target storage pointer corresponding to "Shanghai" occupies 18 bits in the target row storage space. Therefore, we can determine that the second difference is: 18-3=15.

[0092] In step S603, the quotient obtained by dividing the second difference by the first target value is rounded up.

[0093] In this step, after determining the second difference, the quotient obtained by dividing the second difference by the first target value 8 can be rounded up, so that the quotient obtained by 15 / 8 is rounded up to 2.

[0094] In step S604, the second target value is added to the rounded-up value to obtain the number of bytes occupied by the target storage pointer in the target row storage space.

[0095] In this step, the second target value 1 can be added to the above rounded-up value 2 to determine that the number of bytes occupied by the target storage pointer in the target row storage space is 1+2=3 bytes.

[0096] In summary, for the location information "China" in the 1st row and 3rd column (row number 0, column number 2) of the data to be stored, it can be determined that its target storage pointer is located in the 34th byte of the 0th row of the reference table, and specifically stored in the 14 bits after skipping the first 0 bits in the first 2 bytes of 8+0 (8 is the number of bytes occupied by the IP network segment, and 0 is the starting storage position).

[0097] The location information "Shanghai" of the 1st row and 5th column (row number 0, column number 4) in the data table to be stored can be determined. Its target storage pointer is located in the 34th byte of the 0th row of the reference table. Specifically, it is stored in the 18 bits after skipping the first 5 bits, starting from the 3 bytes occupied by the IP network segment (8 is the number of bytes occupied by the IP network segment, and 3 is the starting storage position).

[0098] Figure 7 The diagram illustrates an information processing method in an embodiment of this disclosure, specifically showing a diagram of storing a storage pointer corresponding to a certain type of information in the target row storage space of a certain row. (Refer to...) Figure 7Taking the storage location of a certain location information in the target row storage space of 34 bytes as an example: the number of bits (bit_len) occupied by the target storage pointer in the target row storage space is 20, the number of bits skipped by the target storage pointer in the first byte of the target row storage space (bit_start) is 7, the starting byte position (byte_start) of the target storage pointer in the target row storage space is 5, and the number of bytes occupied by the target storage pointer in the target row storage space (byte_len) is 4, then its storage location in the target row storage space of 34 bytes is: The first 8 bytes are used to store the start and end IP addresses of the IP network segment. After these 8 bytes, there are 5 more bytes to store the starting byte position of the target storage pointer. After this starting byte position, 7 bits are skipped, for a total of 20 bits.

[0099] After determining the location of the target memory pointer in the target row memory space, you can then refer to... Figure 3 In step S303, each target storage pointer is stored in its designated location.

[0100] After generating the column file and creating the reference table for the column file, the storage procedure for the aforementioned data table to be processed is complete.

[0101] It should be noted that when the number of IP network segments contained in the data table to be processed is large, multiple reference tables can be created. These reference tables can be used to store pointers to the location information contained in the multiple IP network segments. Furthermore, an index file can be created to store the mapping relationship between the identifier of the reference table and the IP network segments stored in it. In this way, only the index file and the reference tables are loaded into memory, while the column file is saved on disk. This supports reading the column file in disk + index mode, avoiding the problem of excessive memory consumption caused by loading a large number of column files into memory at once, which would affect the processing speed of other services.

[0102] After completing the stored procedure for the data table to be processed, it can receive location query requests for the IP address to be queried. Upon receiving such location query requests, it can refer to... Figure 8 , Figure 8 This document illustrates a flowchart of the process for querying the location information corresponding to the IP address to be queried, as shown in this embodiment, including steps S801-S804: In step S801, the IP segment to which the IP to be queried is located is obtained.

[0103] In this step, format conversion may be performed on the to-be-queried IP address first to convert it into unit32 integer data, and then the binary search method may be used to compare the IP segment where the to-be-queried IP is located. The basic idea of binary search is to divide n elements into two roughly equal parts, take a[n / 2] and compare it with x. If x=a[n / 2], x is found and the algorithm terminates; if x<a[n / 2], x only needs to be searched continuously in the left half of array a; if x>a[n / 2], x only needs to be searched in the right half of array a.

[0104] In step S802, the row number of the storage pointer of the home location information corresponding to the to-be-queried IP in the reference table is determined according to the IP segment where the to-be-queried IP is located.

[0105] In this step, after the IP segment where the to-be-queried IP is located is determined, since the reference table stores the storage pointers of the home location information included in different network segments, the row number of the storage pointer of the home location information corresponding to the to-be-queried IP in the reference table can be further determined.

[0106] In step S803, the storage pointer corresponding to the home location information of the to-be-queried IP is obtained from the target row storage space corresponding to the row number.

[0107] In this step, after the row number is determined, for example, if it is required to read the home location information of the to-be-queried IP in the second column, the storage position of the storage pointer corresponding to the home location information in the target row storage space may be calculated (including four elements: the number of bits occupied by the storage pointer in the target row storage space; the number of bits skipped by the storage pointer in the first byte of the target row storage space; the start byte position of the storage pointer in the target row storage space; the number of bytes occupied by the storage pointer in the target row storage space). After the storage position is determined, the above target storage pointer may be read from the storage position in 34 bytes of the corresponding row in the reference table.

[0108] In step S804, the home location information is read from the column file according to the storage pointer.

[0109] In this step, after the storage pointer is read out, the length information occupied by the home location information may be read from the column file first according to the above storage pointer. For example, if the length information is 4, the position where the length information is located may be moved backward by 4 bytes continuously, to read the above home location information.

[0110] Figure 9 shows a schematic flowchart of querying home location information corresponding to an IP address in an embodiment of the present disclosure, with reference to Figure 9 : When the input IP (i.e. the IP to be queried) is obtained, it can be converted into a unit32 value. Then, a binary search is used to determine the row number of the storage pointer of the input IP in the reference table. The data stored in the target row storage space corresponding to the row number is read. The storage location of the storage pointer corresponding to the location information in the target row storage space is calculated (i.e., the storage range determined according to the above four elements). Then, the storage pointer is read from the storage location. Based on the storage pointer, the length information of the location information is read from the column file. Then, the location information is read from after the length information.

[0111] By storing the location information of Internet Protocol (IP) data column-wise, multiple column files are obtained. Firstly, since each column file is independent and each location information in a column file has a corresponding storage pointer, it facilitates the rapid loading of all location information in a specific column. This solves the problem in related technologies where row-based storage requires reading all columns of a row each time data is retrieved, hindering the rapid loading of location information in a single column. Secondly, because the storage pointer for each location in the column file is not stored according to its actual size but rather a specified number of bits determined by the space occupied by each column file, there is excess compressible space. This allows for further compression, reducing the amount of data occupied by the reference table, improving disk utilization, and reducing disk I / O.

[0112] This disclosure also provides an information processing apparatus. Figure 10 This diagram illustrates the structure of an information processing apparatus in an exemplary embodiment of this disclosure; as shown below. Figure 10 As shown, the information processing device 1000 may include a data acquisition module 1010, a columnar storage module 1020, and a reference table creation module 1030. Wherein: The data acquisition module 1010 is used to acquire a data table to be processed. The data table to be processed includes location information corresponding to multiple IP segments. The data table to be processed includes multiple rows, each row is used to store multiple location information contained in each IP segment, and the multiple location information is distributed in multiple columns of the data table to be processed. The columnar storage module 1020 is used to perform deduplication processing on the location information in each column, and generate a column file based on the deduplicated location information in each column. The reference table creation module 1030 is used to create reference tables for multiple column files, and to store a storage pointer for each location information in the reference table; the storage pointer is used to indicate the storage location of the location information in the column file.

[0113] In an exemplary embodiment of this disclosure, the columnar storage module 1020 is configured as follows: Obtain the length information of each remaining location information in each column after the deduplication process; generate the column file based on the length information of each location information and the location information.

[0114] In an exemplary embodiment of this disclosure, the reference table creation module 1030 is configured to: In an exemplary embodiment of this disclosure, creating a reference table of multiple column files includes: determining the number of rows in the reference table based on the number of rows in the data table to be processed; determining the number of bytes occupied by each column file based on the number of bytes occupied by each location information in the column file; determining the row storage space of the reference table based on the number of bytes occupied by each column file; and creating the reference table based on the number of rows in the reference table and the row storage space of the reference table.

[0115] In an exemplary embodiment of this disclosure, the number of bytes occupied by each of the location information in the column file includes the sum of the number of bytes occupied by each of the location information and the length information corresponding to each of the location information.

[0116] In an exemplary embodiment of this disclosure, the reference table creation module 1030 is configured to: Based on the number of bytes occupied by each of the column files, determine the number of bits occupied by the storage pointer of each location information in each column file in each row of the reference table; based on the sum of the number of bits occupied by the storage pointers of the multiple location information in each row of the reference table and the reserved number of bits, determine the row storage space of the reference table.

[0117] In an exemplary embodiment of this disclosure, the reference table creation module 1030 is configured to: Based on the row number of the target IP segment in the data table to be processed, select the target row storage space from the reference table; obtain the storage location of the target storage pointer in the target row storage space; the target storage pointer is the storage pointer of each home location information contained in the target IP segment; store the target storage pointer in the storage location.

[0118] In an exemplary embodiment of this disclosure, the storage location of the target storage pointer in the target row storage space includes: the number of bits occupied by the target storage pointer in the target row storage space; the number of bits skipped by the target storage pointer in the first byte of the target row storage space; the starting byte position of the target storage pointer in the target row storage space; and the number of bytes occupied by the target storage pointer in the target row storage space.

[0119] In an exemplary embodiment of this disclosure, the reference table creation module 1030 is configured to: Based on the number of bytes occupied by the column file to which the location information belongs, determine the number of bits occupied by the target storage pointer in the target row storage space.

[0120] In an exemplary embodiment of this disclosure, the reference table creation module 1030 is configured to: Obtain the column number n of the location information in the data table to be processed; when the column number n is equal to m, determine the preset number of bits as the number of bits skipped by the target storage pointer in the first byte of the target row storage space; when the column number n is not equal to m, determine the number of bits skipped by the target storage pointer in the first byte of the target row storage space based on the sum of the number of bits occupied by the storage pointer of each location information in the column file from column m to column n-1 in the reference table and the first target value; where m and n are integers, m is less than or equal to n, and m is the column number corresponding to the first column of location information in the data table to be processed.

[0121] In an exemplary embodiment of this disclosure, the reference table creation module 1030 is configured to: The sum of the bits is modulo the first target value to obtain the number of bits that the target storage pointer skips in the first byte of the target row storage space; where m and n are integers, and m is less than or equal to n.

[0122] In an exemplary embodiment of this disclosure, the reference table creation module 1030 is configured to: Obtain the column number n of the location information in the data table to be processed; when the column number n is equal to m, determine the preset byte position as the starting byte position of the target storage pointer in the target row storage space; when the column number n is not equal to m, determine the starting byte position of the target storage pointer in the target row storage space according to the sum of the number of bits occupied by the storage pointer of each location information in the reference table from the mth column to the (n-1th)th column file and the first target value; where m and n are integers, m is less than or equal to n, and m is the column number corresponding to the first column of location information in the data table to be processed.

[0123] In an exemplary embodiment of this disclosure, the reference table creation module 1030 is configured to: The quotient obtained by dividing the sum of the bits by the first target value is rounded down to obtain the starting byte position of the target storage pointer in the target row storage space.

[0124] In an exemplary embodiment of this disclosure, the reference table creation module 1030 is configured to: Determine whether the number of bits skipped by the target storage pointer in the first byte of the target row storage space is the preset number of bits; if it is the preset number of bits, then divide the number of bits occupied by the target storage pointer in the target row storage space by the first target value and round up to obtain the number of bytes occupied by the target storage pointer in the target row storage space; if it is not the preset number of bits, then determine the number of bytes occupied by the target storage pointer in the target row storage space based on the number of bits skipped by the target storage pointer in the first byte of the target row storage space, the number of bits occupied by the target storage pointer in the target row storage space, the first target value, and the second target value.

[0125] In an exemplary embodiment of this disclosure, the reference table creation module 1030 is configured to: If it is not the preset number of bits, then obtain the first difference between the first target value and the number of bits skipped by the target storage pointer in the first byte of the target row storage space; obtain the second difference between the number of bits occupied by the target storage pointer in the target row storage space and the first difference; round up the quotient obtained by dividing the second difference by the first target value; sum the second target value and the rounded value to obtain the number of bytes occupied by the target storage pointer in the target row storage space.

[0126] In an exemplary embodiment of this disclosure, the reference table creation module 1030 is configured to: Upon receiving a location query request for an IP address to be queried, the system obtains the IP segment in which the IP address to be queried is located; based on the IP segment in which the IP address to be queried is located, the system determines the row number of the storage pointer for the location information corresponding to the IP address to be queried in the reference table; the system obtains the storage pointer corresponding to the location information of the IP address to be queried from the target row storage space corresponding to the row number; and the system reads the location information from the column file based on the storage pointer.

[0127] The specific details of each module in the above information processing device have been described in detail in the corresponding information processing methods, so they will not be repeated here.

[0128] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0129] Furthermore, although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0130] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, mobile terminal, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0131] This application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device.

[0132] Computer-readable storage media can be, for example—but not limited to—electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0133] A computer-readable storage medium can be sent, propagated, or transmitted for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable storage medium can be transmitted using any suitable medium, including but not limited to: wireless, wireline, optical fiber, RF, etc., or any suitable combination thereof.

[0134] A computer-readable storage medium carries one or more programs that, when executed by an electronic device, cause the electronic device to perform the methods described in the above embodiments.

[0135] Furthermore, this disclosure also provides an electronic device capable of implementing the above-described method.

[0136] Those skilled in the art will understand that various aspects of this disclosure can be implemented as a system, method, or program product. Therefore, various aspects of this disclosure can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, collectively referred to herein as a "circuit," "module," or "system."

[0137] The following reference Figure 11 To describe an electronic device 1100 according to such an embodiment of the present disclosure. Figure 11 The electronic device 1100 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0138] like Figure 11 As shown, the electronic device 1100 is manifested in the form of a general-purpose computing device. The components of the electronic device 1100 may include, but are not limited to: at least one processing unit 1110, at least one storage unit 1120, a bus 1130 connecting different system components (including storage unit 1120 and processing unit 1110), and a display unit 1140.

[0139] The storage unit stores program code that can be executed by the processing unit 1110, causing the processing unit 1110 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure. For example, the processing unit 1110 can perform actions such as... Figure 1 As shown: Step S110, obtain a data table to be processed. The data table includes location information corresponding to multiple IP segments. The data table includes multiple rows, each row storing multiple location information contained in each IP segment. The multiple location information is distributed in multiple columns of the data table to be processed; Step S120, perform deduplication processing on each column of location information, and generate a column file based on the deduplicated location information in each column; Step S130, create a reference table for multiple column files, and store a storage pointer for each location information in the reference table. The storage pointer is used to indicate the storage location of the location information in the column file.

[0140] Storage unit 1120 may include readable media in the form of volatile storage units, such as random access memory (RAM) 11201 and / or cache memory 11202, and may further include read-only memory (ROM) 11203.

[0141] Storage unit 1120 may also include a program / utility 11204 having a set (at least one) program module 11205, such program module 11205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0142] Bus 1130 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0143] Electronic device 1100 can also communicate with one or more external devices 1200 (e.g., keyboard, pointing device, Bluetooth device, etc.), one or more devices that enable a user to interact with electronic device 1100, and / or any device that enables electronic device 1100 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 1150. Furthermore, electronic device 1100 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 1160. As shown, network adapter 1160 communicates with other modules of electronic device 1100 via bus 1130. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 1100, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0144] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

Claims

1. An information processing method, characterized in that, include: Obtain a data table to be processed. The data table to be processed includes location information corresponding to multiple IP segments. The data table to be processed includes multiple rows, each row is used to store multiple location information contained in each IP segment, and the multiple location information is distributed in multiple columns of the data table to be processed. The location information in each column is deduplicated, and a column file is generated based on the deduplicated location information in each column. Create a reference table for multiple column files, and store a storage pointer for each location information in the reference table; The storage pointer is used to indicate the location where the attribution information is stored in the column file; The creation of a reference table for multiple column files includes: The number of rows in the referenced table is determined based on the number of rows in the data table to be processed; The number of bytes occupied by each column file is determined based on the number of bytes occupied by each of the aforementioned location information in the column file; The row storage space of the reference table is determined based on the number of bytes occupied by each column file; The reference table is created based on the number of rows in the reference table and the row storage space of the reference table; The step of determining the row storage space of the reference table based on the number of bytes occupied by each column file includes: Take the logarithm of the number of bytes occupied by each of the listed file columns, with the specified value as the base; Round the logarithm up to obtain the number of bits occupied by the storage pointer of each location information in the column file in each row of the reference table; The row storage space of the reference table is determined by the sum of the number of bits occupied by the storage pointers of the multiple location information in each row of the reference table and the reserved number of bits.

2. The method according to claim 1, characterized in that, The step of generating a column file based on the deduplication-processed location information of each column includes: Obtain the length information of each remaining location information in each column after the deduplication process; The column file is generated based on the length information of each location information and the location information itself.

3. The method according to claim 1, characterized in that, The number of bytes occupied by each of the aforementioned location information in the column file includes: The sum of the number of bytes occupied by each of the location information and the length information corresponding to each of the location information.

4. The method according to claim 1, characterized in that, The storage pointer for storing each location information in the reference table includes: Based on the row number of the target IP segment in the data table to be processed, select the target row storage space from the reference table; Obtain the location of the target storage pointer in the target row storage space; the target storage pointer is the storage pointer of each home location information contained in the target IP segment; Store the target storage pointer in the storage location.

5. The method according to claim 4, characterized in that, The location of the target storage pointer in the target row storage space includes: The number of bits occupied by the target storage pointer in the target row storage space; The number of bits that the target storage pointer skips in the first byte of the target row storage space; The target storage pointer is located at the beginning byte position in the target row storage space; The target storage pointer indicates the number of bytes occupied in the target row storage space.

6. The method according to claim 5, characterized in that, The number of bits occupied by the target storage pointer in the target row storage space is determined in the following way: Based on the number of bytes occupied by the column file to which the location information belongs, determine the number of bits occupied by the target storage pointer in the target row storage space.

7. The method according to claim 5, characterized in that, The number of bits that the target storage pointer skips in the first byte of the target row storage space is determined by the following method: Obtain the column number n of the location information in the data table to be processed; When the column number n equals m, the preset number of bits is determined as the number of bits that the target storage pointer skips in the first byte of the target row storage space; When the column number n is not equal to m, the number of bits skipped by the target storage pointer in the first byte of the target row storage space is determined based on the sum of the number of bits occupied by the storage pointer of each location information in the column file from column m to column n-1 in the reference table and the first target value. Where m and n are integers, m is less than or equal to n, and m is the column number corresponding to the first column of the location information in the data table to be processed.

8. The method according to claim 7, characterized in that, The step of determining the number of bits skipped by the target storage pointer in the first byte of the target row storage space based on the sum of the bits occupied by the storage pointers of each location information in the column file from column m to column (n-1) in the reference table and the first target value includes: The remainder of the sum of the bits divided by the first target value is used to obtain the number of bits that the target storage pointer skips in the first byte of the target row storage space; Where m and n are integers, and m is less than or equal to n.

9. The method according to claim 5, characterized in that, The starting byte position of the target storage pointer in the target row storage space is determined in the following way: Obtain the column number n of the location information in the data table to be processed; When the column number n equals m, the preset byte position is determined as the starting byte position of the target storage pointer in the target row storage space; When the column number n is not equal to m, the starting byte position of the target storage pointer in the target row storage space is determined based on the sum of the number of bits occupied by the storage pointer of each location information in the column file from column m to column (n-1) in the reference table and the first target value. Where m and n are integers, m is less than or equal to n, and m is the column number corresponding to the first column of the location information in the data table to be processed.

10. The method according to claim 9, characterized in that, The step of determining the starting byte position of the target storage pointer in the target row storage space based on the sum of the number of bits occupied by the storage pointers of each location information in the reference table in the column file from column m to column (n-1) and the first target value includes: The quotient obtained by dividing the sum of the bits by the first target value is rounded down to obtain the starting byte position of the target storage pointer in the target row storage space.

11. The method according to claim 7, characterized in that, The number of bytes occupied by the target storage pointer in the target row storage space is determined in the following way: Determine whether the number of bits skipped by the target storage pointer in the first byte of the target row storage space is the preset number of bits; If it is the preset number of bits, then the number of bits occupied by the target storage pointer in the target row storage space is divided by the first target value and the quotient is rounded up to obtain the number of bytes occupied by the target storage pointer in the target row storage space; If it is not the preset number of bits, then the number of bytes occupied by the target storage pointer in the target row storage space is determined based on the number of bits skipped by the target storage pointer in the first byte of the target row storage space, the number of bits occupied by the target storage pointer in the target row storage space, the first target value and the second target value.

12. The method according to claim 11, characterized in that, If the number of bits is not the preset number of bits, the number of bytes occupied by the target storage pointer in the target row storage space is determined based on the number of bits skipped by the target storage pointer in the first byte of the target row storage space, the number of bits occupied by the target storage pointer in the target row storage space, the first target value, and the second target value, including: If it is not the preset number of bits, then obtain the first difference between the first target value and the number of bits skipped by the target storage pointer in the first byte of the target row storage space; Obtain the second difference between the number of bits occupied by the target storage pointer in the target row storage space and the first difference; The quotient obtained by dividing the second difference by the first target value is rounded up. The second target value is added to the rounded-up value to obtain the number of bytes occupied by the target storage pointer in the target row storage space.

13. The method according to any one of claims 1 to 12, characterized in that, The method further includes: Upon receiving a location query request for the IP address to be queried, obtain the IP segment in which the IP address to be queried is located; Based on the IP segment in which the IP to be queried is located, determine the row number of the storage pointer for the location information corresponding to the IP to be queried in the reference table; Obtain the storage pointer corresponding to the location information of the IP to be queried from the target row storage space corresponding to the row number; The location information is read from the column file based on the storage pointer.

14. An information processing device, characterized in that, include: The data acquisition module is used to acquire a data table to be processed. The data table to be processed includes location information corresponding to multiple IP segments. The data table to be processed includes multiple rows, each row is used to store multiple location information contained in each IP segment, and the multiple location information is distributed in multiple columns of the data table to be processed. A columnar storage module is used to deduplicate the location information in each column and generate a column file based on the deduplicated location information in each column. The reference table creation module is used to create reference tables for multiple column files, and to store a storage pointer for each location information in the reference table; the storage pointer is used to indicate the storage location of the location information in the column file; The reference table creation module is also used to determine the number of rows in the reference table based on the number of rows in the data table to be processed; The number of bytes occupied by each column file is determined based on the number of bytes occupied by each of the aforementioned location information in the column file; The row storage space of the reference table is determined based on the number of bytes occupied by each column file; The reference table is created based on the number of rows in the reference table and the row storage space of the reference table; The reference table creation module is also used to take the logarithm of the number of bytes occupied by each of the column files, with a specified value as the base; Round the logarithm up to obtain the number of bits occupied by the storage pointer of each location information in the column file in each row of the reference table; The row storage space of the reference table is determined by the sum of the number of bits occupied by the storage pointers of the multiple location information in each row of the reference table and the reserved number of bits.

15. A computer storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the information processing method according to any one of claims 1 to 13.

16. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the information processing method according to any one of claims 1 to 13 by executing the executable instructions.

Images