Roaming data processing method and roaming data processing system

Abstract

A roaming data processing method and a roaming data processing system. The method comprises: acquiring a file list of files in a data folder, the files being used for recording analysis data, and the analysis data being data reflecting roaming information; on the basis of a first database and the file list, determining a file to be downloaded, the first database being used for recording a downloaded file and a file being downloaded; adding to a message queue file information of the file to be downloaded; and successively transmitting to a plurality of nodes the file information in the message queue to be subjected to analysis processing.

Description

Roaming data processing method and roaming data processing system

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This disclosure is based on and claims the priority of Chinese patent application with application number 202410137844.4 and application date January 31, 2024. The entire content of the Chinese patent application is hereby incorporated into this disclosure by reference. Technical Field

[0003] The present application relates to the field of communication technology, and in particular to a roaming data processing method and a roaming data processing system. Background Art

[0004] Roaming refers to moving from a mobile phone's registered location to another service area within a certain period of time. By analyzing user roaming data, we can calculate the movement trajectories of a large number of users within a certain period of time. Combining historical data with corresponding algorithms, we can estimate users' roaming behavior within a certain period of time in the future, providing a basis for dynamic expansion or compression of the core network, ensuring smooth calls and Internet access for users.

[0005] The traditional processing method, shown in Figure 1, configures a scheduled task to retrieve files from a file server, parses the files, and stores them in a unified database. The data is then compiled according to corresponding statistical rules. However, due to the massive volume of files that need to be processed, using a single node results in excessive file pressure and slow processing speeds. While using multiple nodes improves data processing speed to a certain extent, repeated parsing of files cannot be avoided, resulting in data duplication and reduced processing efficiency.

[0006] Therefore, how to improve the processing efficiency of roaming data has become a technical problem that needs to be solved urgently. Summary of the Invention

[0007] The present application provides a roaming data processing method and a roaming data processing system to solve the technical problem of low roaming data processing efficiency in the prior art.

[0008] This application provides a roaming data processing method, including:

[0009] Obtain a file list of files in a data folder, wherein the files are used to record analysis data, wherein the analysis data is data reflecting roaming information;

[0010] Determining a file to be downloaded based on a first database and the file list, wherein the first database is used to record downloaded files and files being downloaded;

[0011] Add the file information of the file to be downloaded to the message queue;

[0012] The information of each file in the message queue is distributed to multiple nodes in sequence for parsing and processing.

[0013] In one embodiment, obtaining a file list of files in the data folder includes:

[0014] Based on a preset scanning period, the data folder is scanned periodically to obtain a file list of files in the data folder.

[0015] In one embodiment, periodically scanning the data folder based on a preset scanning period includes:

[0016] In the i-th scan, the i-th data folder is scanned, wherein the i-th data folder includes a file recording analysis data during the period from the i-1-th scan to the i-th scan, and i is a positive integer.

[0017] In one embodiment, the first database uses the file name as a key and the file download status as a value, and determining the file to be downloaded based on the first database and the file list includes:

[0018] For any file in the file list, if the key in the first database does not include the file name of the file, it is determined that the file is a file to be downloaded.

[0019] In one embodiment, when the file being downloaded is not successfully downloaded within a preset time period, the file being downloaded is deleted from the first database.

[0020] In one embodiment, the message queue includes file information of files in the data folder and file information of files being downloaded that are deleted from the first database.

[0021] In one embodiment, the analysis data is UDM data, and each of the nodes determines the user's home location and roaming location based on the AMF_AMFINSTANCEID field and MMEHOST field in the UDM data, where the AMF_AMFINSTANCEID field is a network element instance identification field, and the MMEHOST field is an MME host field.

[0022] In one embodiment, the analysis data is UDM data, and each of the nodes determines the user's home location and roaming location based on the GUAMI field in the UDM data, where the GUAMI field is a field that uniquely identifies the AMF.

[0023] In one embodiment, after distributing the file information in the message queue to multiple nodes in sequence for parsing, the method further includes:

[0024] Based on preset configuration information, the parsing results of a specific province are synchronized to the second database, and the preset configuration information includes the provinces for which the parsing results need to be synchronized.

[0025] The present application also provides a roaming data processing system, comprising:

[0026] A first device is configured to obtain a file list of files in a data folder, the file being used to record analysis data, the analysis data being data reflecting roaming information; determine a file to be downloaded based on a first database and the file list, the first database being used to record downloaded files and files being downloaded; add file information of the file to be downloaded to a message queue; and sequentially distribute each file information in the message queue to a plurality of nodes;

[0027] A plurality of second devices, each of the second devices is used to parse and process a corresponding file to be downloaded based on the distributed file information.

[0028] The roaming data processing method and system provided by the present application obtain a file list of files in a data folder, wherein the files are used to record analysis data, which is data reflecting roaming information; determine the files to be downloaded based on a first database and the file list, wherein the first database is used to record downloaded files and files being downloaded; add the file information of the files to be downloaded to a message queue; and distribute the file information in the message queue to multiple nodes for parsing and processing. Distributing the file information enables parallel processing by multiple nodes, and by expanding the number of nodes, the speed of data parsing and processing can be significantly improved. In addition, by recording the download and processing status of the files in the first database, each file is ensured to be processed only once, avoiding the problem of repeated file parsing, thereby avoiding resource waste and further improving data processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the present application or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0030] FIG1 is a schematic diagram of data processing of roaming data in the prior art;

[0031] FIG2 is a flow chart of a roaming data processing method provided by the present application;

[0032] FIG3 is a second flow chart of the roaming data processing method provided by the present application;

[0033] FIG4 is a schematic diagram of the coding rules of the network element instance identifier in the roaming data processing method provided in this application;

[0034] FIG5 is a schematic structural diagram of the roaming data processing system provided by the present application. DETAILED DESCRIPTION

[0035] To make the objectives, technical solutions, and advantages of this application more clear, the technical solutions in this application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this application.

[0036] Existing core network services do not conduct statistical analysis on user roaming data. Roaming refers to moving from a mobile phone's registered location to another service area within a certain period of time. By analyzing user roaming data, we can count the movement trajectories of a large number of users within a certain period of time. Combining historical data with corresponding algorithms, we can estimate users' roaming behavior within a certain period of time in the future, providing a basis for dynamic expansion or compression of the core network, and ensuring smooth user calls and Internet access.

[0037] According to statistics, the base of mobile users is close to 1 billion, and a large number of user activities will generate massive roaming data. Manufacturers will regularly report user roaming data in the form of files. The reporting cycle is once every 6 hours, and the amount of data in each batch is about 1 billion. It is necessary to complete the collection of roaming files and statistical analysis of data in a short period of time. The traditional processing method is to regularly pull files from the file server, store them in a unified manner after analysis, and then aggregate the data according to the corresponding algorithm after storage. This method has high requirements for server configuration and long processing time, and cannot meet the needs of real-time processing of massive data.

[0038] FIG1 is a schematic diagram of data processing of roaming data in the prior art. As shown in FIG1 , the traditional processing method configures a scheduled task to obtain files from a file server, parses the files and stores them in a unified database, and then counts the data according to corresponding statistical rules.

[0039] Traditional solutions have the following drawbacks that cannot be solved:

[0040] There are massive amounts of files that need to be processed. A single node is overloaded with processing files, and when multiple nodes are processing, repeated parsing of files is unavoidable, resulting in data duplication.

[0041] The problem of large amounts of data being stored in the database. According to estimates, 1 billion data items are generated every 6 hours. The peak write speed of the ElasticSearch cluster is 20W / s. Therefore, it takes 5000 seconds, or about 1 hour and 25 minutes, to store 1 billion data items in the database. Not only is the storage time long, but it also requires a large amount of machine storage. Assuming each data item is 1kb, about 2TB of data is consumed every day, which is extremely consuming storage.

[0042] Data aggregation is difficult, as it requires aggregating data from 1 billion pieces of data. This is time-consuming and puts a lot of pressure on ElasticSearch storage.

[0043] To address the above issues, the present application provides the following embodiments. FIG2 is a flow chart of a roaming data processing method provided by the present application. As shown in FIG2 , the present application provides a roaming data processing method, which may include:

[0044] S210: Obtain a file list of files in a data folder, where the files are used to record analysis data, and the analysis data is data reflecting roaming information.

[0045] It should be noted that the roaming data processing method provided in the embodiments of the present application can be executed by a server or a computer device, such as a tablet computer, a laptop computer, or an ultra-mobile personal computer (UMPC). For ease of understanding, this embodiment uses a server as the execution entity, referred to as a roaming data processing server.

[0046] Specifically, the roaming data generated by the user activities is stored in a data folder in the file server, and the file server is scanned to pull a file list in the data folder.

[0047] S220: Determine a file to be downloaded based on a first database and the file list, where the first database is used to record downloaded files and files being downloaded.

[0048] Here, the downloaded file is a file that has been downloaded and parsed, the downloading file is a file that is being downloaded and parsed, and the to-be-downloaded file is a file that has not been downloaded and parsed.

[0049] S230: Add the file information of the file to be downloaded to a message queue.

[0050] In some embodiments, the file information includes relevant information of the file to be downloaded, such as location information, file name, etc., so that the node can obtain the corresponding file to be downloaded based on the file information, that is, an element in the message queue includes relevant information of a file to be downloaded.

[0051] In other embodiments, the file information includes unique identification information of the file to be downloaded. Based on the unique identification information, a file to be downloaded can be uniquely determined. The unique identification of the file to be downloaded is used as the key and the relevant information of the file to be downloaded is stored as the value. The node can determine the unique identification information of the file to be downloaded based on the file information, and then obtain the relevant information of the file to be downloaded based on the unique identification information.

[0052] Optionally, the message queue adopts Kafka, which is a high-throughput distributed publish-subscribe message system. Each node can pull the file information in the message queue by subscribing to the corresponding topic.

[0053] S240: Distribute each file information in the message queue to multiple nodes in sequence for parsing and processing.

[0054] Here, the multiple nodes may be multiple threads in the roaming data processing server, or other servers that parse and process files to be downloaded, or multiple threads in the roaming data processing server and other servers.

[0055] It is understandable that due to the huge amount of roaming data, the hardware requirements for computing devices are relatively high, and multiple servers are usually used to analyze and process the data.

[0056] The roaming data processing method provided in the embodiment of the present application scans a data folder to obtain a file list of the data folder, wherein the data folder is a folder that records analysis data, and the analysis data is data that reflects roaming information; based on a first database and the file list, the file to be downloaded is determined, wherein the first database is used to record downloaded files and files being downloaded; the file information of the file to be downloaded is added to a message queue; and each file information in the message queue is distributed to multiple nodes in sequence for parsing and processing. Distributing file information enables parallel processing of multiple nodes, and by expanding the number of nodes, the data parsing and processing speed can be greatly improved; in addition, by recording the download and processing status of the file in the first database, it is ensured that each file is only processed once, avoiding the problem of repeated file parsing, thereby avoiding resource waste and further improving data processing efficiency.

[0057] In an optional embodiment, scanning a data folder to obtain a file list of the data folder includes:

[0058] Based on a preset scanning period, the data folder is scanned periodically to obtain a file list of the data folder.

[0059] Furthermore, the periodic scanning of the data folder based on the preset scanning period includes:

[0060] In the i-th scan, the i-th data folder is scanned, wherein the i-th data folder includes a file recording analysis data during the period from the i-1-th scan to the i-th scan, and i is a positive integer.

[0061] Specifically, the roaming data of each scanning cycle is stored in a file in a data folder. When scanning a data folder, only one data folder needs to be scanned, thereby greatly reducing the number of scanned files.

[0062] In a specific implementation process, scanning a specific data folder can be achieved by uniformly setting the name of the data folder. Specifically, when storing data, a corresponding data folder is created for each scanning cycle, and the data folder is named with a corresponding name based on a preset naming rule.

[0063] In some embodiments, data folders are named using the format of year, month, day, and hour, for example, yyyyMMddHH, where yyyy represents the year, MM represents the month, dd represents the day, and HH represents the time. For example, 2023090106 represents the data folder at 6:00 on September 1, 2023. If the scanning period is 6 hours, the next data folder is named 2023090112, representing the data folder at 12:00 on September 1, 2023. The roaming data processing server performs a scan every 6 hours and can determine the name of the data folder directly based on the time of the scan, or can calculate and determine the name of the data folder to be scanned this time based on the name of the data folder scanned last time.

[0064] In other embodiments, the data folders are named in serial numbers, that is, the data folders are named in sequence of 1, 2, 3... The first data folder is named 1, the second data folder is named 2, and so on. The serial number of the name of the data folder is increased by one each time it is scanned.

[0065] It is understandable that by scanning specific folders at regular intervals, the number of scanned files can be greatly reduced, repeated scanning can be avoided, and the efficiency of obtaining file lists can be improved.

[0066] In an optional embodiment, the first database uses the file name as a key and the file download status as a value, and determining the file to be downloaded based on the first database and the file list includes:

[0067] For any file in the file list, if the key in the first database does not include the file name of the file, it is determined that the file is a file to be downloaded.

[0068] Specifically, the download status of a file includes downloaded and downloading. Downloaded means that the file has been downloaded and parsed, and downloading means that the file is being downloaded and parsed (it means that it has been added to the download list, and may have been downloaded, is being parsed, is being downloaded, or is waiting in line for download). The first database records the downloaded files and the files being downloaded. Files not recorded in the first database are files to be downloaded.

[0069] Optionally, the first database uses Redis, which is an open source key-value database server. In Redis, the key is the file name, and the value is the download status of the file.

[0070] For example, 0 indicates downloading and 1 indicates downloaded. According to the file name of a certain file, if the corresponding value 0 is queried in Redis, the file is determined to be a file being downloaded. If the corresponding value 1 is queried in Redis, the file is determined to be a downloaded file. If the corresponding value is not queried in Redis, the file is determined to be a file to be downloaded.

[0071] Furthermore, when the file being downloaded is not successfully downloaded within a preset time period, the file being downloaded is deleted from the first database.

[0072] For example, if the preset duration is 1 hour, when a file is written to Redis, the initial value is 0. If the file is not processed within one hour and the corresponding value is still 0, the data record is determined to be expired, the file record is deleted from Redis, and the file is added to the message queue again in the next scheduled task for reprocessing.

[0073] Furthermore, the message queue includes file information of files in the data folder and file information of files being downloaded that are deleted from the first database.

[0074] Specifically, the files in the data folder are new data added in the most recent scanning cycle, and the file information of the files is added to the message queue for processing. The timed-out files deleted from the first database are also added to the message queue for reprocessing.

[0075] It should be understood that the file added back to the message queue can be assigned to the original node for parsing and processing, or can be assigned to other nodes for parsing and processing, and there is no limitation here.

[0076] It is understandable that the timed-out file may be caused by an abnormal error during the parsing process. In the next scheduled task, the timed-out file will be reprocessed. On the one hand, this will prevent abnormal data from continuously occupying memory resources and release system resources. On the other hand, it will reallocate and process the abnormal data to improve the stability and fault tolerance of the entire roaming data processing system.

[0077] In an optional embodiment, the file information includes the IP address of the server storing the data folder, the user name, the password, the folder location and the file name of the data folder.

[0078] Based on any of the above embodiments, the analysis data is UDM data, and each node determines the user's home location and roaming location based on the AMF_AMFINSTANCEID field and MMEHOST field in the UDM data, the AMF_AMFINSTANCEID field is the network element instance identification field, and the MMEHOST field is the MME host field.

[0079] Specifically, the UDM (Unified Data Management) data includes VLR_NUMBER (VLR number) field, MMEHOST (MME host) field, PURGEDONMME (elimination from MME) field, AMF_AMFINSTANCEID (AMF instance identifier) ​​field, AMF_PURGEFLAG (AMF elimination flag) field, MSPURGEDATMSC field and CURRENTSCSCFNAME field.

[0080] Table 1 UDM field identifier 1

[0081] As shown in Table 1, the user's home location is determined by the AMF_AMFINSTANCEID field in the UDM data, the roaming location is determined by the MMEHOST field, and the user's network standard can be determined by other fields.

[0082] FIG4 is a schematic diagram of the encoding rules of the network element instance identifier in the roaming data processing method provided by the present application. As shown in FIG4 , AMF_AMFINSTANCEID (network element instance identifier) ​​adopts the UUID version 4 format (RFC 4122) with a total of 128 bits, 32 hexadecimal numbers, and is divided into 5 segments with "-": time-low"-"time-mid"-"time-high-and-version"-"clock-seq-and-reserved clock-seq-low"-"node, where time-low represents the first 32 bits of the UUID, time-mid represents the 16 bits following time-low, time-high-and-version represents the 16 bits following time-mid, clock-seq-and-reserved represents the 8 bits following time-high-and-version, clock-seq-low represents the 8 bits following clock-seq-and-reserved, and node represents the last 48 bits of the UUID, which consists of 6 bytes. The registration location of the number corresponding to the data can be determined through the Region and Province dictionary tables.

[0083] Table 2 User registration home network determination process

[0084] As shown in Table 2, VLR_NUMBER and SGSNNUM (SGSN number flag) are both nationally numbered according to the rules of MSCID (Mobile Switching Center Identifier). The three digits M1M2M3 in MSCID are mainly used to determine the location of the VLR device (i.e. the province where the user is roaming). The location of the device can be confirmed by querying the standard dictionary table.

[0085] In an optional embodiment, the analysis data is UDM data, and each of the nodes determines the user's home location and roaming location based on the GUAMI field in the UDM data, where the GUAMI field is a field that uniquely identifies the AMF.

[0086] Specifically, the UDM data includes an MSCNUMBER field, an MSPURGEFLAG field, an EPCMMEHOST field, an EPCMMEPFG field, a GUAMI field, a PURGEFLAG field, and an IMSREGSTATUS field.

[0087] Table 3 UDM field identifier three

[0088] As shown in Table 3, in the 5G system, GUAMI (Globally Unique AMF Identifier) ​​identifies a unique AMF (Action Message Format) and is used by the 5G NR base station gNB (the next Generation Node B) to address the AMF. It has a total of 24 fixed bits and consists of MCC+MNC+AMFI (AMF Identifier):

[0089] MCC (Mobile Country Code): Mobile country code, 460 for China;

[0090] MNC (Mobile Network Code): mobile network code, using 00;

[0091] AMFI (AMF Identifier): AMF identification, including AMF Region ID, AMF Set ID, AMF Pointer: AMF Region ID (8bit), AMF region identifier, identifies the area that AMF is responsible for; AMF Set ID (10bit), AMF group identifier, uniquely identifies a group of AMFs within the AMF region; AMF Pointer (6bit), AMF pointer identifier, identifies one or more AMFs within the AMF group.

[0092] The user's registered location can be determined using the 18-bit AMF Region ID + AMF Set ID and the standard location dictionary table.

[0093] Table 4 Dictionary table of place of origin

[0094] Figure 4 shows an exemplary table of the location dictionary table provided in an embodiment of the present application. For example, in the 5G GUAI "460-00-062002", the upper 12 bits of the AMF Region ID + AMF Set ID are 062 (corresponding to L1L2L3). By looking up the table, it can be concluded that the user is registered with the Hebei Province AMF.

[0095] It can be understood that the specific data such as the mobile country number, mobile network number, and home location dictionary table in the above embodiment are only exemplary data and can be replaced with other numbers during the specific implementation process.

[0096] Optionally, each node asynchronously analyzes data from a single file in batches based on the above method. After determining the number home location, roaming location, and registered network of a single user's data, it performs data integration and calculation. The amount of data processed in each batch can be adaptively set based on actual usage requirements, such as processing 10,000 data items per batch. When all data in a file is processed, the value corresponding to that file in the first database is set to 1.

[0097] In an optional embodiment, after distributing the file information in the message queue to multiple nodes in sequence for parsing, the method further includes:

[0098] Based on preset configuration information, the parsing results of a specific province are synchronized to the second database, and the preset configuration information includes the provinces for which the parsing results need to be synchronized.

[0099] Optionally, the second database uses ElasticSearch (ES). After the data is written to the first database, it is synchronized to ElasticSearch for subsequent calculations.

[0100] Here, the configurable solution on the page is used to configure which provinces can be written into the second database. Due to the large amount of data, it is impossible to determine it by querying the database each time. The original data of a maximum of n provinces can be written to provide a basis for subsequent verification.

[0101] Specifically, by loading data into direct memory, each judgment is made by reading the data in memory, which improves the data processing efficiency by n times. If the user changes the province configuration in the interface, the changed configuration is first written to MySQL, and then each processing node is notified by broadcasting. The data is reloaded into the memory after the next scheduled task starts to ensure that the user's configuration takes effect.

[0102] For ease of understanding, the preferred roaming data processing method provided in this application is described below.

[0103] Figure 3 is the second flow chart of the roaming data processing method provided by the present application. As shown in Figure 3, the file download distribution program obtains a list of files in a specific data folder, and traverses the Redis server according to the file name to see if it has been downloaded. The key in Redis is the file name, and the value 0 represents that it has been added to the download list, and 1 represents successful download and parsing; if the record does not exist in Redis according to the file name query, the corresponding file information is sent to the message queue, and the message queue uses Kafka. At the same time, the file name is written to Redis to record the download status of the file. The key is the file name, the initial value is 0, and the expiration time is 1 hour. If the file is not processed within one hour, the next scheduled task will put the file into the download queue for reprocessing.

[0104] Multiple nodes (processors) subscribe to the corresponding topics in Kafka, pull data to the message queue, and then go to the file server in turn to pull data, parse the data and put it into the database. The node data can be seamlessly expanded horizontally according to demand.

[0105] Each node analyzes the data of a single file asynchronously in multiple batches. After determining the number location, roaming location, registration network, etc. of a single user's data, it performs data integration calculations. After all the data in this file is processed, the file name is written to Redis at the same time to record the processing status of the file. The key is the file name, the initial value is 1, and the expiration time is 10 hours. Then, when the folder is scanned at the next time, since the file has been processed, it will not be processed again. At the same time, the processed data of each batch is written to Kafka and synchronized to ElasticSearch for subsequent calculations. The province that can be written to EasticSearch is configured through the configurable solution on the page. By loading the data into direct memory, each judgment is made by reading the data in memory, which improves the data processing efficiency.

[0106] In summary, the roaming data processing method provided by this application uses a unified naming and upload method for files, which reduces the scope of file scanning from the source; in addition, file processing requests are distributed through message middleware, making distributed file processing possible; Redis is used to record the download and processing status of files, ensuring that files are only parsed and processed once, solving the problem of data duplication during the download and parsing of massive files; during the file processing process, relevant specifications and standards are used to determine the relevant attributes of user data, such as the user's registered network, registered location, roaming province, and other important information. The processing process is carried out in batches with multi-threaded asynchronous processing, which not only improves the calculation speed but also compresses the data volume by about 10,000 times, saving a lot of server storage and bandwidth, compressing the data volume for subsequent statistics and aggregation, reducing the difficulty of subsequent processing, and improving the data processing speed. The single batch processing time is compressed to less than 1 hour, which is 10 times more efficient than the traditional processing method. Compared with the traditional roaming data statistics method, this method has the advantages of fast processing, high efficiency, low data volume, and convenience for later aggregation processing. In addition, it supports dynamic expansion in the later stage and is easy to maintain.

[0107] The following describes a roaming data processing system provided in an embodiment of the present application. The roaming data processing system described below and the roaming data processing method described above can refer to each other.

[0108] FIG5 is a schematic diagram of the structure of the roaming data processing system provided by the present application. As shown in FIG5 , the roaming data processing system includes:

[0109] The first device 510 is configured to scan a data folder to obtain a file list of the data folder, wherein the data folder is a folder recording analysis data, wherein the analysis data is data reflecting roaming information; determine a file to be downloaded based on a first database and the file list, wherein the first database is configured to record downloaded files and files being downloaded; add file information of the file to be downloaded to a message queue; and sequentially distribute each file information in the message queue to multiple nodes;

[0110] Multiple second devices 520, each second device is used to parse and process the corresponding file to be downloaded based on the distributed file information.

[0111] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0112] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by hardware. Based on this understanding, the above technical solution, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product, which can also be called a computer program product, can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiment.

[0113] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A roaming data processing method, comprising: Obtain a file list of files in a data folder, wherein the files are used to record analysis data, wherein the analysis data is data reflecting roaming information; Determining a file to be downloaded based on a first database and the file list, wherein the first database is used to record downloaded files and files being downloaded; Add the file information of the file to be downloaded to the message queue; The information of each file in the message queue is distributed to multiple nodes in sequence for parsing and processing.

2. The roaming data processing method according to claim 1, wherein: Get a list of files in the data folder, including: Based on a preset scanning period, the data folder is scanned periodically to obtain a file list of files in the data folder.

3. The roaming data processing method according to claim 2, wherein: The periodic scanning of the data folder based on the preset scanning cycle includes: In the i-th scan, the i-th data folder is scanned, wherein the i-th data folder includes a file recording analysis data during the period from the i-1-th scan to the i-th scan, and i is a positive integer.

4. The roaming data processing method according to claim 1, wherein: The first database uses the file name as a key and the file download status as a value, and determining the file to be downloaded based on the first database and the file list includes: For any file in the file list, if the key in the first database does not include the file name of the file, it is determined that the file is a file to be downloaded.

5. The roaming data processing method according to claim 4, wherein: When the file being downloaded is not successfully downloaded within a preset time period, the file being downloaded is deleted from the first database.

6. The roaming data processing method according to claim 5, wherein: The message queue includes file information of files in the data folder and file information of files being downloaded that are deleted from the first database.

7. The roaming data processing method according to any one of claims 1 to 6, wherein: The analysis data is UDM data, and each node determines the user's home location and roaming location based on the AMF_AMFINSTANCEID field and the MMEHOST field in the UDM data. The AMF_AMFINSTANCEID field is a network element instance identification field, and the MMEHOST field is an MME host field.

8. The roaming data processing method according to any one of claims 1 to 6, wherein: The analysis data is UDM data, and each node determines the user's home location and roaming location based on the GUAMI field in the UDM data. The GUAMI field is a field that uniquely identifies the AMF.

9. The roaming data processing method according to any one of claims 1 to 6, wherein: After distributing the file information in the message queue to multiple nodes in sequence for parsing, the method further includes: Based on preset configuration information, the parsing results of a specific province are synchronized to the second database, and the preset configuration information includes the provinces for which the parsing results need to be synchronized.

10. A roaming data processing system, comprising: The first device obtains a file list of files in a data folder, where the files are used to record analysis data, where the analysis data is data reflecting roaming information; Determining a file to be downloaded based on a first database and the file list, wherein the first database is used to record downloaded files and files being downloaded; adding file information of the file to be downloaded to a message queue; and distributing each file information in the message queue to multiple nodes in sequence; A plurality of second devices, each of the second devices is used to parse and process a corresponding file to be downloaded based on the distributed file information.

11. A computer-readable storage medium, wherein computer program instructions are stored on the computer-readable storage medium, and when the computer program instructions are executed by a processor, the roaming data processing method according to any one of claims 1 to 9 is implemented.

12. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the roaming data processing method according to any one of claims 1 to 9 is implemented.

Images