A data distribution method and device based on a Hadoop distributed file system

Abstract

The present application relates to the technical field of data processing, and provides a data distribution method and device based on a Hadoop distributed file system. The method comprises the following steps: at the beginning of each period, calculating the size of data written into a server, calculating the space occupancy rate of the server according to the size of data written into the server, and determining the distribution strategy of the period according to the relationship between the space occupancy rates of the servers; in the corresponding period, using the distribution strategy of the period to distribute to-be-written data into one or more servers; wherein the distribution strategy is used to specify the servers required to write each type of to-be-written data. The present application makes each period correspond to a distribution strategy, so that the DataNode node used to write the to-be-written data of the corresponding type only needs to communicate with the server determined by the distribution strategy, and does not need to communicate with the servers other than the distribution strategy, thereby reducing the occupation of network resources, improving the speed of data distribution, and reducing lag.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a data distribution method and apparatus based on the Hadoop Distributed File System. Background Technology

[0002] The demand for big data analysis and processing is growing rapidly, requiring the use of extensive distributed processing software for collaborative analysis to improve computational efficiency. Hadoop, as an open-source distributed processing software framework, plays a crucial role in big data processing by offering unique methods for fast, highly available, and scalable data processing.

[0003] Greenplum is a high-performance data warehouse system with high parallelism and scalability. It can be used to extract valuable information from large amounts of data. It uses a massively parallel processing (MPP) architecture, which can process large amounts of data simultaneously, and its processing capacity can be expanded by adding more nodes.

[0004] When building a data acquisition, storage, and processing system using Greenplum and Hadoop, it is often necessary to write the data collected on HDFS into Greenplum. In existing technologies, a type of data is usually distributed evenly across Greenplum servers. However, tests have shown that when the data volume is large, this even distribution method inevitably requires one DataNode to communicate with multiple servers, resulting in the consumption of a large amount of network bandwidth and causing data writing to be slow. Furthermore, since data is distributed to multiple servers at the same time, data queries or data reads also require reading from multiple servers, which makes the data query process also consume a large amount of network bandwidth, causing lag.

[0005] Therefore, overcoming the shortcomings of the existing technology is an urgent problem to be solved in this technical field. Summary of the Invention

[0006] The technical problem that this invention aims to solve is that existing technologies typically distribute a type of data evenly across Greenplum's servers. However, this even distribution method inevitably requires a DataNode to communicate with multiple servers, resulting in the consumption of a large amount of network bandwidth and causing data writing lag. Furthermore, since data is distributed to multiple servers at the same time, data queries or data reads also require reading from multiple servers, which makes the data query process also consume a large amount of network bandwidth, leading to lag.

[0007] The present invention adopts the following technical solution: In a first aspect, this invention provides a data distribution method based on the Hadoop Distributed File System, which stores public call detail records (CDRs), HTTP CDRs, and signaling CDRs separately, and pre-divides the data into multiple periods. The method includes: At the beginning of each cycle, the size of the data written to the server is calculated. Based on the size of the data written to the server, the server's space utilization rate is calculated. Based on the relationship between the space utilization rates of each server, the distribution strategy for that cycle is determined. Within a given period, the distribution strategy for that period is used to distribute the data to be written to one or more servers; wherein, the data to be written includes public call detail records (CDRs), HTTP CDRs, and signaling CDRs, and the distribution strategy is used to specify the servers to which each type of data to be written should be written.

[0008] Preferably, when there are five servers, namely the first server, the second server, the third server, the fourth server, and the fifth server, the step of determining the distribution strategy for this period based on the space occupancy relationship among the servers specifically includes: When the space occupancy rates of both the first and second servers are less than the space occupancy rate of the first server, the public call detail records (CDRs) are distributed to the first and second servers, the HTTP CDRs are distributed to the third server, and the signaling CDRs are distributed to the fourth and fifth servers. When the space occupancy rate of the first server and / or the second server is greater than or equal to the first space occupancy rate, and the space occupancy rate of both the first server and the second server is less than the second space occupancy rate, the HTTP call detail records (CDRs) are distributed to the first server and the second server, the signaling CDRs are distributed to the third server, and the public CDRs are distributed to the fourth server and the fifth server. When the space occupancy rate of the first server and / or the second server is greater than or equal to the space occupancy rate of the second server, the signaling call detail records (CDRs) are distributed to the first server and the second server, the public call detail records (PDRs) are distributed to the third server, and the HTTP call detail records (CDRs) are distributed to the fourth server and the fifth server.

[0009] Preferably, the distribution strategy using the cycle distributes the data to be written to one or more servers, specifically including: Divide the data to be written into n groups according to the number of servers n to be written to; Include the IP address of the corresponding server in the name of each data group; Obtain the corresponding IP address from the name of each data group, and write the data group to the server with the corresponding IP address; wherein the difference in data size between the n data groups is less than a preset difference value.

[0010] Preferably, the distribution of the data to be written to one or more servers using the distribution strategy is achieved by executing Hadoop commands using gpfdist, specifically including: The Hadoop files are placed in the bin directory of the system beforehand to install the Hadoop command package on the system; wherein the Hadoop files include the Hadoop client program and the Hadoop configuration file; When executing a corresponding Hadoop function, a corresponding Hadoop command is generated according to a preset rule, a new process is created, and the process is used to execute the Hadoop command.

[0011] Preferably, when the corresponding Hadoop function being executed is to retrieve corresponding data, after generating the corresponding Hadoop command according to preset rules, the method further includes: A preset filter command is added after the Hadoop command to filter the execution result of the Hadoop command; wherein the preset filter command is "grep preset id"; Use getInputStream() to obtain the input stream of the process, use an InputStreamReader object to read the input stream of the process, and use a BufferedReader object to convert the read input stream of the process into a character stream to obtain the filtered execution result of the Hadoop command.

[0012] Preferably, the data is placed in a preset folder, and the folder name of the preset folder is used as the preset ID.

[0013] Preferably, the method further includes: Pre-place Hadoop files from multiple vendors and / or multiple versions in the system's bin directory; Configure configuration files and scripts to manage the vendors and versions of Hadoop command packages required by the system, thereby achieving compatibility with multiple vendors and / or multiple versions of Hadoop.

[0014] Preferably, the configuration file includes multiple configuration parameters, each of which is the filename of the Hadoop file for the corresponding vendor and / or version; wherein, Hadoop files from different vendors and different versions have different filenames. Set the configuration parameters corresponding to the Hadoop command packages required by the system to true, and set the values ​​of other configuration parameters to false.

[0015] Preferably, the configuration script is executed before the system starts Hadoop; The configuration script reads the configuration file, obtains the target file name corresponding to the Hadoop command package required by the system, copies the target file corresponding to the target file name in the system's bin folder to obtain a new target file, and modifies the file name of the new target file to the default name of the Hadoop command.

[0016] Secondly, the present invention also provides a data distribution device based on a Hadoop Distributed File System (Hadoop Distributed File System) for implementing the data distribution method based on a Hadoop Distributed File System described in the first aspect, the device comprising: At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor for performing the data distribution method based on the Hadoop Distributed File System as described in the first aspect.

[0017] Thirdly, the present invention also provides a non-volatile computer storage medium storing computer-executable instructions, which are executed by one or more processors to perform the data distribution method based on the Hadoop distributed file system described in the first aspect.

[0018] This invention assigns a distribution strategy to each period. DataNodes used to write the corresponding type of data only need to communicate with the server determined by the distribution strategy, eliminating the need to communicate with servers outside the strategy. This reduces network resource consumption, increases data distribution speed, and minimizes buffering. Furthermore, by adjusting the distribution strategy according to the period, a balance can be maintained among multiple servers even when the servers writing different types of data are inconsistent. Since the servers writing different types of data within a period are fixed (i.e., the vertical distribution described in subsequent embodiments), data retrieval is easier, thus reducing resource consumption during data queries. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments of the present invention will be briefly described below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0020] Figure 1 This is a flowchart illustrating the first data distribution method based on the Hadoop Distributed File System provided in this embodiment of the invention; Figure 2 This is a flowchart illustrating the second data distribution method based on the Hadoop Distributed File System provided in this embodiment of the invention; Figure 3 This is a schematic diagram of the first data distribution method based on the Hadoop Distributed File System provided in this embodiment of the invention; Figure 4 This is a schematic diagram of the second data distribution method based on the Hadoop Distributed File System provided in this embodiment of the invention; Figure 5 This is a schematic diagram of the third data distribution method based on the Hadoop Distributed File System provided in this embodiment of the invention; Figure 6 This is a flowchart illustrating the third data distribution method based on the Hadoop Distributed File System provided in this embodiment of the invention. Figure 7 This is a flowchart illustrating the fourth data distribution method based on the Hadoop Distributed File System provided in this embodiment of the invention. Figure 8 This is a flowchart illustrating the fifth data distribution method based on the Hadoop Distributed File System provided in this embodiment of the invention. Figure 9 This is a schematic diagram of the fourth data distribution method based on the Hadoop Distributed File System provided in this embodiment of the invention; Figure 10 This is a schematic diagram of the architecture of a data distribution device based on the Hadoop Distributed File System provided in an embodiment of the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0022] In this invention, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0023] Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0024] Example 1: Embodiment 1 of this invention provides a data distribution method based on the Hadoop Distributed File System, which stores public call detail records (CDRs), HTTP CDRs, and signaling CDRs separately, and pre-divides them into multiple periods, such as... Figure 1 As shown, the method includes: In step 201, at the beginning of each cycle, the size of the data written to the server is calculated; the server can be understood as a data node in Greenplum, and in this embodiment, the executor can be a Hadoop DataNode. The size of the data written to the server is obtained by recording the size of the data written each time data is written to the server, and then calculating the recorded data size at the beginning of the corresponding cycle.

[0025] In step 202, the server's space occupancy rate is calculated based on the size of the data already written to the server.

[0026] In step 203, the distribution strategy for the period is determined based on the relationship of space occupancy rates among the servers. The length of the period is determined by those skilled in the art based on experience, such as using a day as the period and using the early morning or late night of each day as the period switching time, that is, updating the distribution strategy in the early morning or late night.

[0027] In step 204, within the corresponding period, the data to be written is distributed to one or more servers using the distribution strategy for that period. The data to be written includes public call detail records (CDRs), HTTP CDRs, and signaling CDRs. The distribution strategy specifies the servers to which each type of data should be written, i.e., which servers to distribute public CDRs to, which servers to distribute HTTP CDRs to, and which servers to distribute signaling CDRs to. Each type of data to be written is distributed by one DataNode. HTTP CDRs refer to service CDRs transmitted via the HTTP protocol, signaling CDRs refer to CDRs formed by network configuration signaling, and public CDRs refer to CDRs formed by all services other than HTTP and signaling CDRs. Actual testing shows that public CDRs typically have the largest number and the largest file size.

[0028] Under each strategy, a type of call detail record (CDR) is distributed to one or more fixed servers. Furthermore, the servers on which CDRs of various types are written are switched in a round-robin fashion to form multiple distribution strategies. The specific switching is determined by the space occupancy rate of each server.

[0029] This embodiment assigns a distribution strategy to each period. The DataNode node used to write the corresponding type of data only needs to communicate with the server determined by the distribution strategy, and does not need to communicate with servers outside the distribution strategy. This reduces network resource consumption, improves data distribution speed, and reduces lag. Furthermore, by adjusting and determining the distribution strategy according to the period, a balance can be maintained among multiple servers even when the servers writing different types of data are inconsistent. Since the servers writing different types of data within a period are fixed (i.e., the vertical distribution described in subsequent embodiments), data retrieval is easier, thus reducing resource consumption during data queries.

[0030] For example, in a practical application scenario, when there are five servers, namely server 1, server 2, server 3, server 4, and server 5, the distribution strategy for that period is determined based on the space occupancy relationship between each server, such as... Figure 2 As shown, it specifically includes: In step 301, when the space occupancy rates of both the first server and the second server are less than the first space occupancy rate, the public call detail records (CDRs) are distributed to the first server and the second server, the HTTP CDRs are distributed to the third server, and the signaling CDRs are distributed to the fourth server and the fifth server.

[0031] In step 302, when the space occupancy rate of the first server and / or the second server is greater than or equal to the first space occupancy rate, and the space occupancy rates of the first server and the second server are both less than the second space occupancy rate, the HTTP call detail records (CDRs) are distributed to the first server and the second server, the signaling CDRs are distributed to the third server, and the public CDRs are distributed to the fourth server and the fifth server.

[0032] In step 303, when the space occupancy rate of the first server and / or the second server is greater than or equal to the second space occupancy rate, the signaling call detail records (CDRs) are distributed to the first and second servers, the public call detail records (PDRs) are distributed to the third server, and the HTTP call detail records (HTTP CDRs) are distributed to the fourth and fifth servers. The first and second space occupancy rates are both obtained by those skilled in the art based on experience. For example, if the first space occupancy rate is 25% and the second space occupancy rate is 50%, then step 301 is as follows: Figure 3 As shown, step 302 is as follows Figure 4 As shown, step 303 is as follows Figure 5As shown, servers 1, 2, 3, 4, and 5 refer to the first, second, third, fourth, and fifth servers, respectively. "Share" refers to public call detail records (PDRs), "http" to HTTP call detail records (CDRs), and "Signaling" to signaling call detail records (CDRs). Fragment 1 refers to the fragments corresponding to data storage below 25%, fragment 2 to data storage between 25% and 50%, and fragment 3 to data storage above 50%. Fragmentation can be understood as storing a specific type of data on a specific server. Figure 5 If the share segment 3 is located on server 3, it means that when the share exceeds 50%, the public call detail records (PDRs) will be distributed to the third server.

[0033] When the disk usage of servers 1 and 2 is less than 25%, the largest shared call detail records (CDRs) are assigned to servers 1 and 2; HTTP CDRs are assigned to server 3; and signaling CDRs are assigned to servers 4 and 5. When the disk usage of servers 1 and 2 reaches 25%, HTTP CDRs are assigned to servers 1 and 2; signaling CDRs are assigned to server 3; and shared CDRs are assigned to servers 4 and 5. When the disk usage of servers 1 and 2 exceeds 50%, signaling CDRs are assigned to servers 1 and 2; shared CDRs are assigned to server 3; and HTTP CDRs are assigned to servers 4 and 5. There is also a situation where, when the disk usage of a server reaches 75%, it is considered to have reached a system threshold, and expired data is deleted.

[0034] In one optional implementation, the distribution strategy using the cycle to distribute the data to be written to one or more servers specifically includes: dividing the data to be written into n data groups according to the number n servers to which it needs to be written; wherein, different types of data to be written may require different servers, so each type of data to be written is written separately; carrying the IP address of the corresponding server in the name of each data group; obtaining the corresponding IP address from the name of each data group, and writing the data group to the server with the corresponding IP address; wherein, the difference in data size between the n data groups is less than a preset difference. The preset difference is obtained by those skilled in the art based on experience, that is, the sizes of the data groups are almost the same.

[0035] For example, after gphdfs retrieves the file list via the command line, gpfdist executes the corresponding Hadoop commands to distribute the data into the database according to the call detail record (CDR) type. Assuming there are 5 servers in the database cluster, the largest volume of general CDRs (shared) are assigned to servers 1 and 2; HTTP and HTTPS CDRs are assigned to server 3; and all signaling CDRs are assigned to servers 4 and 5.

[0036] Data allocation implementation: The data import program scans files on HDFS to obtain a list and size of .share, .http, .https, and various signaling files. Taking .share call detail records (CDRs) as an example: .share CDRs are grouped by size. Files in group 1 are renamed to: share_server1IP_unique_number.txt, and files in group 2 are renamed to: share_server2IP_unique_number.txt. During writing, the data is written to the corresponding server according to the IP address in the filename. For example, by modifying the gphdfs code, the file scan automatically obtains the server IP, scans .share_serverIP*.txt files, and performs the data import operation, thus achieving vertical data distribution.

[0037] This embodiment achieves a high degree of concentration of similar data within a short period of time, providing excellent query performance; in particular, the customer service complaint support function carried by signaling backtracking, because it is a vertically segmented data, isolates the disk impact when sharing call detail records into the database, thereby providing a stable query server.

[0038] For reports covering a large time frame: monthly reports can be scheduled for the early morning. Taking the monthly report carried by shared call detail records as an example, the three shards of shared call detail records are evenly distributed across all servers, utilizing all cluster resources for large-scale computation, ensuring the successful completion of the large report task.

[0039] In summary, by developing and scheduling the gphdfs allocation strategy, a distribution of small-scale time data with vertical slicing and large-scale time data with horizontal slicing was achieved, which met the project's requirements for real-time querying and large-scale report calculation.

[0040] In practical use, gphdfs, as a component of Hadoop, typically requires first establishing a dependency on the Hadoop JAR file, and then implementing gphdfs's calls to Hadoop's Application Programming Interface (API). However, testing has shown that calling the API to execute Hadoop consumes a significant amount of resources, becoming one of the main reasons for data write lag. Furthermore, since the functions provided by different vendors or versions of Hadoop may differ, adapting to various vendors or versions of Hadoop using the JAR dependency method requires compatibility modifications to gphdfs itself, increasing code size, runtime resource consumption, and maintenance costs. To address this issue, in a preferred embodiment, the distribution strategy for distributing data to be written to one or more servers is implemented by executing Hadoop commands using gpfdist, such as... Figure 6 As shown, it specifically includes: In step 401, the Hadoop files are placed in the bin directory of the system in advance to install the Hadoop command package in the system; wherein, the Hadoop files include the Hadoop client program and the Hadoop configuration file.

[0041] In step 402, when executing the corresponding Hadoop function, the corresponding Hadoop command is generated according to preset rules, a new process is created, and the process is used to execute the Hadoop command. The specific Hadoop command to be executed by gpfdist or gphdfs is determined by those skilled in the art based on their needs.

[0042] This embodiment transforms the original method of calling the API using JAR dependencies into a method of directly calling the command line. Compared to the API method, this method can directly access the underlying layer without going through the API's internal processing, thereby reducing resource consumption, speeding up execution, and ensuring that data writing proceeds normally.

[0043] like Figure 7 As shown, the method further includes: In step 501, Hadoop files from multiple vendors and / or multiple versions are placed in the bin directory of the system beforehand.

[0044] In step 502, configuration files and configuration scripts are used to manage the vendors and versions of the Hadoop command packages required by the system, in order to achieve compatibility with multiple vendors and / or multiple versions of Hadoop.

[0045] In practice, Hadoop files from multiple vendors and / or multiple versions may include multiple versions of hadoop2, hadoop3, cdh and hdp.

[0046] The configuration file includes multiple configuration parameters, each of which is the filename of a Hadoop file from a specific vendor and / or version. Different vendors and versions of Hadoop files have different filenames. The configuration parameter corresponding to the Hadoop command package required by the system is set to true, and the values ​​of other configuration parameters are set to false.

[0047] Before the system starts Hadoop, the configuration script is executed in advance. The configuration script reads the configuration file, obtains the target file name corresponding to the Hadoop command package required by the system, copies the target file corresponding to the target file name in the system's bin folder to obtain a new target file, and modifies the file name of the new target file to the default name of the Hadoop command.

[0048] Since the functions provided by different vendors or versions of Hadoop may differ, but all different vendors or versions of Hadoop are based on the open-source Hadoop, meaning that the Hadoop commands available for use by different vendors or versions of Hadoop are consistent, this embodiment modifies gphdfs to change it from relying on Hadoop JAR packages and directly using the functions provided by Hadoop to a method that does not rely on Hadoop JAR packages but directly uses Hadoop commands. This achieves compatibility with multiple versions and vendors, reduces the amount of code required for compatibility, and facilitates later maintenance.

[0049] The preset rules are derived by those skilled in the art based on experience. For example, in a specific application scenario, for a given Hadoop function, the following code is executed: private String hadoop=" / opt / hadoop / bin / hadoop"; private void greenetBulkLoad(){ ArrayList <string>files=new ArrayList<>(4096); ArrayList <string>segFiles=new ArrayList<>(4096); String hdfsPath=input.replace("gphdfs","hdfs"); writeLog("greenetBulkLoad "+hdfsPath); try { / / 1. Assemble and use the Hadoop local commands provided by the Hadoop vendor to read the Hadoop cluster file list. String ls=hadoop+" fs -ls -C "+hdfsPath+" 2> / dev / null"; writeLog("cmd:"+ls); / / 2. JVM executes the assembled local commands Process lsPro = Runtime.getRuntime().exec(ls); / / Create a new process and use it to execute the Hadoop command. BufferedReader br1 = new BufferedReader(new InputStreamReader(lsPro.getInputStream(), "UTF-8")); String item = null; while ((item=br1.readLine())!=null){ files.add(item); } br1.close(); int total = files.size(); int perSeg = total / totalseg; writeLog("total:"+total+",seg num:"+totalseg+",perSeg:"+perSeg); / / get segfile for(int i=0; i <files.size(); i++) { if (i % totalseg == segid) { segFiles.add(files.get(i)); } } In practical use, when the corresponding Hadoop function being executed is to retrieve corresponding data, after generating the corresponding Hadoop command according to the preset rules, such as... Figure 8 As shown, the method further includes: In step 601, a preset filter command is added after the Hadoop command to filter the execution result of the Hadoop command; wherein, the preset filter command is "grep preset id".

[0050] In step 602, getInputStream() is used to obtain the input stream of the process, InputStreamReader object is used to read the input stream of the process, and BufferedReader object is used to convert the read input stream of the process into a character stream to obtain the filtered execution result of the Hadoop command.

[0051] One optional implementation of the preset filtering command is to place the data in a preset folder, using the folder name of the preset folder as the preset ID. For example, consider the following code: private String hadoop=" / opt / hadoop / bin / hadoop"; private String id="greenet"; / / Folder name of the default folder private void greenetBulkLoad(){ ArrayList <string>files=new ArrayList<>(4096); ArrayList <string>segFiles=new ArrayList<>(4096); String hdfsPath=input.replace("gphdfs","hdfs"); writeLog("greenetBulkLoad "+hdfsPath); try { / / 1. Assemble and use the Hadoop local commands provided by the Hadoop vendor to read the Hadoop cluster file list, while filtering out useless output. String ls=hadoop+" fs -ls -C "+hdfsPath+"| grep "+id+" 2> / dev / null"; writeLog("cmd:"+ls); / / 2. JVM executes the assembled local commands Process lsPro = Runtime.getRuntime().exec(ls); BufferedReader br1 = new BufferedReader(new InputStreamReader(lsPro.getInputStream(), "UTF-8")); String item = null; while ((item=br1.readLine())!=null){ files.add(item); } br1.close(); int total = files.size(); int perSeg = total / totalseg; writeLog("total:"+total+",seg num:"+totalseg+",perSeg:"+perSeg); / / get segfile for(int i=0; i <files.size(); i++) { if (i % totalseg == segid) { segFiles.add(files.get(i)); } } The execution result before adding the preset filter command is as follows: Figure 9 As shown, it contains, for example Figure 9 The unwanted content in the middle frame can be filtered using a preset ID to obtain the desired content, which is the filtered execution result of the Hadoop command.

[0052] Example 2: like Figure 10 The diagram shown is an architectural schematic of a data distribution device based on the Hadoop Distributed File System according to an embodiment of the present invention. The data distribution device based on the Hadoop Distributed File System in this embodiment includes one or more processors 21 and a memory 22. Figure 10 Take a processor 21 as an example.

[0053] Processor 21 and memory 22 can be connected via a bus or other means. Figure 10 Taking the example of a connection between China and Israel via a bus.

[0054] The memory 22, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs and non-volatile computer-executable programs, such as the data distribution method based on the Hadoop distributed file system in Embodiment 1. The processor 21 executes the data distribution method based on the Hadoop distributed file system by running the non-volatile software programs and instructions stored in the memory 22.

[0055] Memory 22 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 22 may optionally include memory remotely located relative to processor 21, which can be connected to processor 21 via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0056] The program instructions / modules are stored in the memory 22. When executed by one or more processors 21, they perform the data distribution method based on the Hadoop distributed file system in Embodiment 1.

[0057] It is worth noting that the information interaction and execution process between the modules and units in the above-mentioned device and system are based on the same concept as the processing method embodiment of the present invention. For details, please refer to the description in the method embodiment of the present invention, and will not be repeated here.

[0058] Those skilled in the art will understand that all or part of the steps in the various methods of the embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc.

[0059] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.< / string> < / string> < / string> < / string>

Claims

1. A data distribution method based on the Hadoop Distributed File System, characterized in that, Public call detail records (CDRs), HTTP CDRs, and signaling CDRs are stored separately and pre-divided into multiple periods. Methods include: At the beginning of each cycle, the size of the data written to the server is calculated. Based on the size of the data written to the server, the server's space utilization rate is calculated. Based on the relationship between the space utilization rates of each server, the distribution strategy for that cycle is determined. Within a given period, the distribution strategy for that period is used to distribute the data to be written to one or more servers; wherein, the data to be written includes public call detail records (CDRs), HTTP call detail records (CDRs), and signaling call detail records (SDRs), and the distribution strategy is used to specify the servers to which each type of data to be written should be written. Under each strategy, a type of call detail record (CDR) is distributed to one or more fixed servers. The CDRs of different types are written to different servers in rotation according to the changes in the space occupancy rate of the servers, so as to form multiple distribution strategies. The servers to which different types of data are written within a cycle are fixed, so that when querying the same type of data, only the fixed server needs to be read, reducing the resource consumption during data query.

2. The data distribution method based on the Hadoop Distributed File System according to claim 1, characterized in that, When there are five servers, namely the first server, the second server, the third server, the fourth server, and the fifth server, the distribution strategy for this period is determined based on the space occupancy relationship among the servers, specifically including: When the space occupancy rates of both the first and second servers are less than the space occupancy rate of the first server, the public call detail records (CDRs) are distributed to the first and second servers, the HTTP CDRs are distributed to the third server, and the signaling CDRs are distributed to the fourth and fifth servers. When the space occupancy rate of the first server and / or the second server is greater than or equal to the first space occupancy rate, and the space occupancy rate of both the first server and the second server is less than the second space occupancy rate, the HTTP call detail records (CDRs) are distributed to the first server and the second server, the signaling CDRs are distributed to the third server, and the public CDRs are distributed to the fourth server and the fifth server. When the space occupancy rate of the first server and / or the second server is greater than or equal to the space occupancy rate of the second server, the signaling call detail records (CDRs) are distributed to the first server and the second server, the public call detail records (PDRs) are distributed to the third server, and the HTTP call detail records (CDRs) are distributed to the fourth server and the fifth server.

3. The data distribution method based on the Hadoop Distributed File System according to claim 1, characterized in that, The distribution strategy using this cycle distributes the data to be written to one or more servers, specifically including: Divide the data to be written into n groups according to the number of servers n to be written to; Include the IP address of the corresponding server in the name of each data group; Obtain the corresponding IP address from the name of each data group, and write the data group to the server with the corresponding IP address; wherein the difference in data size between the n data groups is less than a preset difference value.

4. The data distribution method based on the Hadoop Distributed File System according to claim 1, characterized in that, The distribution of the data to be written to one or more servers using the aforementioned distribution strategy is achieved by executing Hadoop commands via gpfdist, specifically including: The Hadoop files are placed in the system's bin directory beforehand to install the Hadoop command package on the system; wherein the Hadoop files include the Hadoop client program and the Hadoop configuration file; When executing a corresponding Hadoop function, a corresponding Hadoop command is generated according to a preset rule, a new process is created, and the process is used to execute the Hadoop command.

5. The data distribution method based on the Hadoop Distributed File System according to claim 4, characterized in that, When the corresponding Hadoop function being executed is to retrieve corresponding data, after generating the corresponding Hadoop command according to preset rules, the method further includes: A preset filter command is added after the Hadoop command to filter the execution result of the Hadoop command; wherein, the preset filter command is "grep preset id"; Use getInputStream() to obtain the input stream of the process, use an InputStreamReader object to read the input stream of the process, and use a BufferedReader object to convert the read input stream of the process into a character stream to obtain the filtered execution result of the Hadoop command.

6. The data distribution method based on the Hadoop Distributed File System according to claim 5, characterized in that, The data is placed in a preset folder, and the folder name of the preset folder is used as the preset ID.

7. The data distribution method based on the Hadoop Distributed File System according to claim 4, characterized in that, The method further includes: Pre-place Hadoop files from multiple vendors and / or multiple versions in the system's bin directory; Configure configuration files and scripts to manage the vendors and versions of Hadoop command packages required by the system, thereby achieving compatibility with multiple vendors and / or multiple versions of Hadoop.

8. The data distribution method based on the Hadoop Distributed File System according to claim 7, characterized in that, The configuration file includes multiple configuration parameters, each of which is the file name of the Hadoop file for the corresponding vendor and / or version; wherein, Hadoop files from different vendors and different versions have different file names. Set the configuration parameters corresponding to the Hadoop command packages required by the system to true, and set the values ​​of other configuration parameters to false.

9. The data distribution method based on the Hadoop Distributed File System according to claim 7, characterized in that, The configuration script is executed before the system starts Hadoop; The configuration script reads the configuration file, obtains the target file name corresponding to the Hadoop command package required by the system, copies the target file corresponding to the target file name in the system's bin folder to obtain a new target file, and modifies the file name of the new target file to the default name of the Hadoop command.

10. A data distribution device based on the Hadoop Distributed File System, characterized in that, include: At least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the processor for performing the data distribution method based on the Hadoop distributed file system as described in any one of claims 1-9.

Images