A large-scale POI data processing and loading method for neural network training

Abstract

This paper proposes a large-scale POI data processing and loading method for neural network training, comprising the following steps: S1: defining basic parameters; S2: constructing auxiliary files; and S3: loading data, which is the actual working process of the system, and gradually parsing the auxiliary files in step S1 to obtain the results. Using this method, POI-related data is processed efficiently with low disk and memory usage; random reads are efficient with O(1) complexity and low memory usage; and the process of splitting into training, validation, and test sets is efficient and consumes low computing resources.

Description

Technical Field

[0001] The present invention belongs to the field of machine learning, and in particular relates to a large-scale POI data processing and loading method for neural network training. Background Art

[0002] With the development of information technology, online maps, recommendation platforms, and other information services are becoming increasingly integrated into people's lives. These services are providing increasing convenience. Points of Interest (POIs) are a key component of many platform services and a critical source of information for users. Platforms need to make targeted recommendations based on the characteristics of POIs, and users also need to quickly access relevant information about their target POIs. Standardized, objective, and informative high-quality POI descriptions can effectively support these services. Currently, a large number of new POIs are emerging rapidly, and the current POI base is already very large. However, most POIs lack such high-quality descriptions. Given the huge number of POIs, manually writing descriptions is practically unrealistic. Automating POI description generation models has enormous potential for application. POI description generation involves generating objective descriptions of POIs based on relevant data (reviews, categories, location context, etc.). Good descriptions facilitate users' quick access to POI information and improve the performance of POI recommendation systems. Currently, an effective approach to this task is to train deep neural network models using POI data. In the field of deep learning, using POI data to train relevant models requires processing and loading large-scale POI data. Processing refers to data preprocessing, including data screening and data feature tagging. Loading refers to reading data during the model training phase. Neural network training based on gradient descent often requires random reads. POI data is generally stored in distributed database systems and extracted locally as text-based tsv files (tab-delimited table files), where each column represents a POI data field and each row is a record. Large-scale POI data typically contains hundreds of millions of records. For a dataset, the above tsv files are split into multiple files by row. Each file has two key identification fields: POI ID (unique identifier of the POI) and ITEM ID (record identifier). Typically, multiple records correspond to one POI. Other fields contain POI-related data such as comments and POI categories. Deep neural network model training requires this loading and processing process to be efficient and resource-efficient.

[0003] Existing technology:

[0004] a) Tfrecord is a dataset storage method that facilitates loading and transmission. Tfrecord encodes each sample as a bytecode, along with a length and checksum, and stores them in a contiguous space. This makes it suitable for fast access to large amounts of data and highly efficient memory utilization. For each sample in a dataset, Tf record first converts it into bytecode. The bytecodes of each sample in the dataset are sequentially written into a file, forming a Tfrecord file. Fast sequential reading is supported, first reading the bytecode of a sample and then converting it back to the appropriate format based on the sample data type.

[0005] b) Pandas is a table file processing tool that supports reading and processing small table types (such as TSV). During processing, the entire data file is generally read into memory.

[0006] Dask is a tabular file processing tool that uses parallel computing to process large amounts of data.

[0007] c) Hash index technology: The basic implementation can use a dictionary to store data keys and corresponding data offsets. The offset can be used to quickly find the data location.

[0008] Disadvantages of existing technology:

[0009] a) POI data requires repeated processing, loading, and utilization. For large datasets, TFRecord processing consumes a lot of disk space (new data files must be constructed after processing). Training requires random reads, but TFRecord is more suitable for sequential reads. Random reads have a high complexity: O(n) (a term used in computer science to describe algorithm complexity).

[0010] b) Pandas is not suitable for large-scale data processing and cannot be applied to this task. Dask still takes a long time to process data and has low random read efficiency. This is mainly due to the large amount of unnecessary calculations and the lack of optimization for the characteristics of this task.

[0011] c) The hash table needs to be read into the memory, which still faces great memory pressure when the amount of POI data is too large. Summary of the Invention

[0012] To solve the problems existing in the prior art, the present invention provides a large-scale POI data processing and loading method for neural network training, which can support efficient, low memory and disk usage iterative processing and random reading, and is easy to segment.

[0013] To achieve the above object, the present invention adopts the following scheme:

[0014] The present invention provides a large-scale POI data processing and loading method for neural network training, comprising the following steps:

[0015] S1: Define basic parameters, including the structure definition of the POI data table;

[0016] S2: Auxiliary file construction: pre-process the data and store the processing results in the accompanying file; construct a search file and a mapping file, wherein the search file is used to record the location of the POI data in the original file and the processing results, and the mapping file is used to provide fast data positioning;

[0017] S3: Data loading: In actual use, the system first finds the location of the POI data in the search file according to the mapping file, then opens the search file, and reads the corresponding data row according to the offset indicated by the mapping file; wherein, the processing results in the accompanying file can provide a reference for further processing and utilization of the read data.

[0018] In some embodiments, the present invention further includes the following technical features:

[0019] Step S1 specifically includes: defining a POI data table, including several data table files T1, T2, ..., T N , N is the number of tables, the POI data table has 4 fields, namely POI identifier POIID, row identifier ITEMID, data field 1: D1, data field 2: D2; the data table file is in text format, wherein a series of rows corresponding to each POI in the POI data table are continuous and in the same data table file, and there are multiple POIs in the same data table file.

[0020] Step S2 includes the following steps:

[0021] S2-1: Data processing, the processing results are stored in the form of accompanying files; the data processing process can be performed multiple times, and the result files are recorded as Where P is the number of processing times, Indicates the table T n The accompanying file obtained by the i-th processing of n The accompanying file obtained from one processing is recorded as Contains two fields, the status field and the item ID field. The status field indicates the processing result of the row, and the item ID field stores the corresponding item ID for verification. The first row in T n The lth row in corresponds to;

[0022] S2-2: Construct the relevant auxiliary files required for loading based on the processed data.

[0023] In step S2-1, the data processing includes classifying the field content using a classification model, or n If the data field content of a row is classified, the content of the corresponding accompanying file is the classification result.

[0024] Data processing includes data length filtering. If T n If the first row of the D1 field in the It is 1 in row l and 0 otherwise.

[0025] Step S2-2 includes the following steps:

[0026] S2-2-1: After completing data processing, construct the search file:

[0027] The search files are denoted as S1, S2, ..., S N , where S n Corresponding T n , S n Each row of information corresponds to T n A POI in S n The information in any line can be divided into the following parts:

[0028] 3) The corresponding POI ID;

[0029] 4) The POI corresponds to T n Search string for all lines in

[0030] The specific form is as follows:

[0031] POIID\tL1#Q1\tL2#Q2\t…\tL K #Q K \n

[0032] Among them, L k ,Q k The above offset byte number and processing result are respectively expressed in string form; the characters \t and # are used as separators to facilitate parsing, and \n represents the line feed character, which is the position to separate two adjacent lines in the file;

[0033] S2-2-2: Construct a mapping file based on the search file, wherein the mapping file includes M T ,M D , M T It is a text list file, all lines of which correspond to all S n All the lines in , where each line k corresponds to a search file S nEach line in h has three fields: POI identifier, i.e. POI ID, a search file S n The file number n∈[1,N], and the file S n The starting position of h in a certain line is the number of bytes offset from the beginning of the file;

[0034] M D In terms of content, T Equivalent, but in dictionary form, with the POI ID as the key and the above file number and offset as the value;

[0035] During data loading, M D The role of M is to load according to the given POIID. T The function is to perform random reading during the training process; M is mainly used in model training T .

[0036] In step S2-2-1, each of the search strings includes the following two parts:

[0037] 3) The line is in T n The starting position in T n The offset in bytes from the beginning of the file;

[0038] 4) Results of data processing;

[0039] The above operation is performed once for all N files to obtain N search files.

[0040] In step S2-2-2, M T Each line of the program has the same number of bytes after excluding the newline character.

[0041] Step S3 includes the following steps:

[0042] S3-1: Locate the search file where the POI is located and its position in the search file according to the mapping file;

[0043] S3-2: Open the corresponding file S n , offset the file pointer by f bytes, read a line, and get a set of data corresponding to the POI;

[0044] S3-3: According to L k ,M k Read the original file T n Data in

[0045] S3-4: Using data processing results Q K Choose how to use data based on your needs.

[0046] Step S3-1 is specifically to use M when a given POIID is loaded. D , M D Load it into memory, find the file number n and offset f according to the POIID; when performing random reading during training, set the number of POIs used in the entire dataset as P, then the data sampler used in training will generate a set of random numbers from 0 to P, and then load the POI sample corresponding to the random number; this is equivalent to renumbering the POIs in the dataset from 0 to P; at this time, use M T More efficient (no need to load file M T to memory); due to M T The number of bytes in each row is equal. As long as the random number generated by the sampler is multiplied by the number of row bytes as an offset, the required data can be located. Let the generated random number be R and the number of row bytes be A. First, open the file M T , offset the pointer by R×A to read a line, that is, to obtain the file number n and offset f in string form;

[0047] In step S3-2, the set of data is:

[0048] POIID\tL1#Q1\tL2#Q2\t…\tL K #Q K \n;

[0049] Step S3-3 specifically includes first opening the file T n , offset the file pointer by L k bytes, and then read a line to get the data fields D1 and D2.

[0050] The beneficial effects of the present invention are:

[0051] POI-related data is processed efficiently with low disk and memory usage;

[0052] Random reading is efficient, with a complexity of O(1) and low memory usage;

[0053] The process of splitting into training, validation, and test sets is efficient and consumes low computational resources.

[0054] This invention addresses the processing and loading of large-scale POI (point of interest) data for neural network training, offering significant efficiency and resource utilization advantages. First, through a carefully designed data processing flow, it achieves efficient processing of POI data, significantly reducing disk and memory usage. This method optimizes data storage and access by constructing auxiliary and mapping files, thereby reducing the demand on system resources during data processing.

[0055] Furthermore, the present invention demonstrates excellent performance during the data loading phase. By utilizing a mapping file, the system can achieve random access to data with an O(1) time complexity. This means that regardless of the size of the dataset, the time required to read any POI data remains constant, significantly improving data access efficiency. Furthermore, this approach reduces memory usage because the system can quickly locate and read the required data without having to load the entire dataset into memory.

[0056] Furthermore, the present invention demonstrates its efficiency in the dataset segmentation process. In neural network training, a dataset is typically segmented into training, validation, and test sets. This invention provides an efficient segmentation solution that consumes minimal computing resources, allowing researchers and data scientists to train and evaluate models on different data subsets without consuming significant resources.

[0057] In summary, the main advantages of the present invention lie in its high efficiency and low resource usage in processing, loading and segmenting large-scale POI data, which make it an effective tool for data management in neural network training. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 is a system construction flow chart of the method in an embodiment of the present invention;

[0059] Figure 2 Schematic diagram of data correspondence of the method in an embodiment of the present invention;

[0060] Figure 3 4 is a data loading flow chart of the method in an embodiment of the present invention. DETAILED DESCRIPTION

[0061] To make the technical solutions and advantages of the present invention more clear, the technical solutions of the embodiments of the present invention will be fully described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of the present invention.

[0062] The following describes embodiments of the present invention in detail. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended only to explain the present invention and are not to be construed as limiting the present invention.

[0063] Using large-scale POI data to train a neural network requires a data processing system that meets the following three conditions:

[0064] 1. Support iterative processing with high efficiency, low memory and disk usage. POI-related data often requires multi-step processing. In many cases, the Nth processing requires the N-1th processing of the entire data set to be completed. In addition, in the actual research and development process, after using the processed data for training and evaluating the results, it may be necessary to process the processed data again to optimize performance. This involves repeated processing of the data set. If the results are saved after each processing, the entire processing flow will take up a lot of disk space due to the large amount of space occupied by large-scale POI data. During the processing process, in order to reduce memory utilization, it is necessary to support processing without completely loading the data;

[0065] 2. Support efficient, low-memory random reading. During the training phase, POI samples need to be read randomly repeatedly. To ensure training efficiency, the random loading time of POI samples should be controlled to reach O(1) (O(1) means that the resources (time or memory) consumed by extracting a POI are independent of the total number of POIs and are fixed values). Large-scale POI data is often difficult to load completely into memory. Even if it can be loaded on a machine with large memory, it will inevitably lead to excessive resource consumption. This requires support for random reading with low memory usage and incomplete data loading;

[0066] 3. Easy segmentation. During neural network training, it is often required to divide the dataset into training, test, and validation sets. The data processing system must have an efficient segmentation solution (low time and space consumption).

[0067] The existing methods for large-scale POI data cannot fully meet the above requirements. The present invention aims to realize a large-scale POI data processing and loading system that can simultaneously meet the above requirements and provide data support for POI-related neural network training.

[0068] This invention involves auxiliary file construction and data loading. Data processing is implemented in auxiliary file construction, while data loading is the actual working process of the system. This invention is a software system based on the Python language and Linux environment, and can also be extended to other languages ​​and environments.

[0069] See Figures 1 to 3 . Figure 1 The system construction process of the data processing and loading method in the present invention is demonstrated, starting from defining basic parameters, through the construction of auxiliary files, to the final data loading. Figure 2 The data correspondence is shown, and through the detailed data correspondence, it shows how to index and locate POI data by searching files and raw data files. The row data example shows that each POI consists of multiple data fields, and the data is accessed by searching strings and offsets. Figure 3 The data loading process is shown: opening the mapping file, locating the data according to the POI ID or random number, and finally opening the original data file and reading the data of a specific row, showing the complete process of data loading.

[0070] The following is further described in detail with reference to specific embodiments.

[0071] Name and symbol definitions

[0072] POI data table is defined as T1, T2, ..., T N , where N is the number of tables, and the value of N is determined according to the actual scenario, T N Represents the Nth data table file. Each table can be viewed as a row-by-row split from a continuous table, and is in text format. The table has four fields: POI identifier (POIID), row identifier (ITEMID), data field 1 (D1), and data field 2 (D2). Each POI corresponds to a series of continuous rows in the same data table file.

[0073] Auxiliary file construction

[0074] The auxiliary file construction process requires data processing first, and then the related auxiliary files required for loading are constructed based on the processed data. In order to avoid the problem of needing to save the processed data table during the processing process, the processing results here are stored in the form of accompanying files, which occupy very little space. Corresponding to table T n The accompanying file obtained from one processing is recorded as The first row in T n The first row in . It contains two fields, the status field and the ITEM ID field, where the status field indicates the processing result of the row (the status field is the data processing result and can be regarded as T n The ITEM ID field stores the corresponding ITEM ID (for verification only). For example, if the length of the data in the row is within a certain range, the ITEM ID field stores the corresponding ITEM ID (for verification only). n If the first row of the D1 field in the The first row is 1, otherwise it is 0. The data processing process can be repeated multiple times and in various forms. The result files are recorded as Where P is the number of processing times, Indicates the table T n The accompanying file obtained by the i-th processing of . The processing can take various forms, such as using a classification model to classify the field content, and the content of the corresponding accompanying file is the classification result.

[0075] After the file processing is completed, the search file is constructed. The search file is recorded as S1, S2, ..., S N , where S n Corresponding T n , for each search file S n , which is the same as the original data file T n Correspondingly, S n Each record in T n In T, all relevant rows of information of a POI are represented. n In the example, consecutive rows of data belonging to the same POI are grouped by S n Referenced by a single entry in (T n The number of rows does not need to be fixed and can be any number of rows depending on the scenario. The data of several consecutive rows belongs to one POI. n The number of rows is T n POI number in S n The information of any row can be divided into the following parts: 1) the corresponding POI ID; 2) the corresponding POI in T n Each search string consists of the following two parts: 1) the row in T n The starting position in T n The offset byte number of the beginning of the file; 2) The result of the data processing process (usually only the result of the last step is used). The specific form is as follows:

[0076] POIID\tL1#Q1\tL2#Q2\t…\tL K #Q K \n

[0077] Among them, L k ,Q k The above offset byte number and processing result are respectively expressed in string form. The characters \t and # are used as separators to facilitate parsing. The above operation is performed once for all N files, and N search files are obtained. In actual application, T may be discarded according to the processing results of the data. n Part of the rows in the construction of S n It can be omitted according to the specific situation, and it will not cause invalid calculation.

[0078] Next, build the mapping file M based on the search file T ,M D , where M T It is a text list file, where each line k corresponds to a search file S n Each line has three fields: POI ID, a search file Sn The file number n∈[0,N-1], and the file S n The starting position of h in a line is the number of bytes offset from the beginning of the file. T Each line of M is equal in number of bytes after excluding newline characters. D In terms of content, T Equivalent, but in dictionary form, with the POI ID as the key and the above file number and offset as the value.

[0079] During data loading, M D The role of M is to load according to the given POIID. T The function of is to perform random reading during the training process. The main use of M in model training is T .

[0080] Intermediate files generated during data construction: S n ,M T ,M D The space occupied is relative to the original file T N Both are small and do not take up much disk space. Cn contains only two fields and both are small. Sn corresponds to a row in the original Tn and only retains two pieces of information (offset and processing result). Mt holds the same number of rows as all Sns combined, but each row has fewer fields (only three fields and both are small). Md is similar to Mt.

[0081] Figure 1 and Figure 2 In the , Linex represents the xth line in the file, L k Indicates the offset. File C n M With T n Row-by-row correspondence, not shown in the figure.

[0082] Data loading

[0083] The data loading process is the actual application process of the system, which gradually analyzes the above auxiliary files to obtain the results. When loading a POI,

[0084] First, according to M T or M D Locate the search file where the POI is located and its position (offset) in the search file.

[0085] When a given POIID is loaded, it uses M D , M DLoad it into memory and find the file number n and offset f according to the POIID. When performing random reading during training, let the number of POIs used in the entire dataset be P. The data sampler used during training will generate a set of random numbers from 0 to P. At this time, the POI samples corresponding to the random numbers are loaded. This is equivalent to renumbering the POIs in the dataset from 0 to P. At this time, use M T More efficient (no need to load file M T to memory). Since M T The number of bytes in each row is equal. As long as the random number generated by the sampler is multiplied by the number of bytes in the row as an offset, the required data can be located. Let the generated random number be R, the number of bytes in the row be A, and first open the file M T , read a line by offsetting the pointer by R×A, and then get the file number n and offset f in string form. T Loaded into memory, which further reduces memory consumption.

[0086] Then open the corresponding file S n , offset the file pointer by f bytes, read a line, and get a set of data corresponding to the POI:

[0087] POIID\tL1#Q1\tL2#Q2\t…\tL K #Q K \n

[0088] The above content has the same meaning as described in 4.2.

[0089] Next, according to L k ,M k Read the original file T n The data in .

[0090] First open the file T n , offset the file pointer by L k bytes, and then read a line to get the data fields D1 and D2.

[0091] Then use the data processing result Q K Choose how to use data based on your needs.

[0092] At this point, the entire data loading process is completed. The loading process is actually the reverse process of auxiliary file construction. It can be seen that the entire process does not involve traversing any files, and random loading during training does not require loading files related to the number of POIs. The time and space complexity are both O(1), and the efficiency advantage is obvious. When using M DSince the dictionary in Python uses a hash table, its average reading time complexity is O(1) and space complexity is O(n), so the time and space complexity of the entire system are also O(1) and O(n) respectively. D Each POI contains only a small amount of data (usually no more than 50 bytes), so when processing large-scale POI datasets, acceptable memory usage can be guaranteed: 10 million POIs generate D Generally, it does not exceed 500M. This solution supports fast random reading and multiple processing of data with low resource consumption, which can meet the needs of POI-related neural network training.

[0093] For data loading, M T A line of M can completely trace the information location of a specific POI and provide an interface for loading the POI information. Therefore, in the data partitioning process, only the file M T Just divide it by row. T The space occupied is much smaller than that of the original data file, so the partitioning process takes up very little time and space, making it easy to modify the partitioning scheme.

[0094] For the sake of brevity and ease of understanding, the above process simplifies the actual usage scenarios to some extent. However, this does not affect the embodiment of the core ideas and the implementation of specific applications.

[0095] In actual applications, the loading speed test results of the technical solution of the present invention on a Docker server showed that when the total number of lines of the original file was approximately 1 billion, it took an average of about 0.018 seconds to load a POI. This test shows that the system is effective, while other existing tools using Dask are very slow and not on the same order of magnitude as the present application.

[0096] The main innovative ideas of the present invention include:

[0097] (1) A method for efficiently loading large-scale tabular POI data randomly using auxiliary files combined with byte offsets for POI-related neural network training scenarios;

[0098] (2) A method for saving the results of data processing with low resource consumption using accompanying files for POI-related neural network training scenarios;

[0099] (3) Using auxiliary files (such as M T ) records the storage file and byte offset of POI information on disk (completely provides the information required for POI data loading), thereby facilitating the segmentation and loading of large-scale POI datasets.

[0100] The technical advantages of the present invention include:

[0101] POI-related data is processed efficiently with low disk and memory usage;

[0102] Random reading is efficient, with a complexity of O(1) and low memory usage;

[0103] The process of splitting into training, validation, and test sets is efficient and consumes low computational resources.

[0104] In this specification, reference to terms such as "one embodiment" and "example" means that a specific feature, structure, or characteristic described in conjunction with the embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the exemplary description of the above terms does not necessarily mean that they are appropriately combined in the corresponding embodiment or example.

[0105] It must be pointed out that the description of the above embodiments is not intended to be limiting but only to help understand the core idea of ​​the present invention. For ordinary technicians in this technical field, any improvements to the present invention and alternatives equivalent to this product, without departing from the principles of the present invention, also fall within the scope of protection of the claims of the present invention.

Claims

1. A large-scale POI data processing and loading method for neural network training, characterized in that: The following steps are involved: S1: Define basic parameters, including the structure definition of the POI data table; S2: Auxiliary file construction: pre-process the data and store the processing results in the accompanying file; construct a search file and a mapping file, wherein the search file is used to record the location of the POI data in the original file and the processing results, and the mapping file is used to provide fast data positioning; S3: Data loading: In actual use, the system first finds the location of the POI data in the search file according to the mapping file, then opens the search file and reads the corresponding data row according to the offset indicated by the mapping file; the processing results in the accompanying file can provide a reference for further processing and utilization of the read data; Step S1 specifically includes: defining a POI data table, including several data table files , The POI data table has four fields, namely, POI identifier POI ID, row identifier ITEM ID, and data field 1: , data field 2: The data table file is in text format, wherein a series of rows corresponding to each POI in the POI data table are continuous and in the same data table file, and there are multiple POIs in the same data table file; Step S2 includes the following steps: S2-1: Data processing, the processing results are stored in the form of accompanying files; the data processing process can be performed multiple times, and the result files are recorded as ,in is the number of processing times, Indicates the table The accompanying file obtained by the i-th processing of The accompanying file obtained by one processing is recorded as , Contains two fields, the status field and the ITEM ID field, where the status field indicates the processing result of the row. Field storage corresponding Used for verification; The Line and The row correspondence; S2-2: Construct the relevant auxiliary files required for loading based on the processed data; Step S3 includes the following steps: S3-1: Locate the search file where the required POI is located and its position in the search file according to the mapping file; S3-2: Open the corresponding search file based on the positioning information obtained in step S3-1 , offset the file pointer Bytes, read a line, and get a set of data corresponding to POI; S3-3: According to the number of offset bytes and data processing results Reading raw files Data in S3-4: Using data processing results Choose how to use data based on your needs.

2. The large-scale POI data processing and loading method for neural network training according to claim 1, characterized in that: In step S2-1, the data processing includes classifying the field content using a classification model, or If the data field content of a row is classified, the content of the corresponding accompanying file is the classification result.

3. The large-scale POI data processing and loading method for neural network training according to claim 2 is characterized in that: Data processing includes data length filtering. If middle The first If the line is within a reasonable length range, then No. 1 if the row is in the range, 0 otherwise.

4. The large-scale POI data processing and loading method for neural network training according to claim 3 is characterized in that: Step S2-2 includes the following steps: S2-2-1: After completing data processing, construct the search file: Find the file record , in correspond , Each row of information corresponds to A POI in The information in any line is divided into the following parts: 1) The corresponding POI ID; 2) The POI corresponds to Search string for all lines in The specific form is as follows: ； in, and The above offset byte number and data processing result are respectively expressed in string form; the characters \t and # are used as separators to facilitate parsing. Indicates a line break character, which is the separation position between two adjacent lines in a file; S2-2-2: Construct a mapping file based on the search file, wherein the mapping file includes and , It is a text list file, all lines of which correspond to All rows in , where each row Corresponding to a search file A line in Each line has three fields: POI identifier, i.e. POI ID, a search file File number , and files In a row The starting position is the number of bytes relative to the beginning of the file; In terms of content Equivalently, in dictionary form, with the POI ID as the key and the above file number and offset as the value; During data loading, The function is to load according to the given POI ID. The purpose of is to perform random reads during training.

5. The large-scale POI data processing and loading method for neural network training according to claim 4 is characterized in that: In step S2-2-1, for each search string, perform the following two operations to form a string form: 1) Calculate and record the row in the corresponding original data file The starting position in the file is relative to the The offset byte number of the starting position; 2) Record the results of the row obtained during data processing; The above operations 1) and 2) are effective for all Each file is executed once, and the result is Search files.

6. The large-scale POI data processing and loading method for neural network training according to claim 5, characterized in that: In step S2-2-2, Each line of the program has the same number of bytes after excluding the newline character.

7. The large-scale POI data processing and loading method for neural network training according to claim 1, characterized in that: Step S3-1 is specifically to use the given POI ID to load ,Will Load it into memory and find the file number according to POIID and offset ; When performing random reading during training, the number of POIs used in the entire dataset is set to , the data sampler used during training will generate a set of The POI samples corresponding to the random numbers are loaded; the POIs in the dataset are renumbered as ; When performing random data access during training, use Mapping files can improve efficiency without having to The file is loaded into memory; since The number of bytes in each row is equal. As long as the random number generated by the sampler is multiplied by the number of bytes in the row as an offset, the required data can be located. Let the generated random number be , the number of bytes in a row is , first open the file , offset the pointer Read a line and get the file number in string form and offset ; In step S3-2, the set of data is: ； Step S3-3 is to first open the file , offset the file pointer Bytes, then read a line to get the data field .

Images