Data analysis system

Abstract

The application relates to the field of data processing and provides a data analysis system. The system comprises a communication data management platform, a data analysis query system and a service management system; the communication data management platform and the data analysis query system are in communication connection; the service management system is connected with the analysis query system and the communication data management platform; wherein the communication data management platform is used for generating communication index information according to received user call logs, user network access logs and mobile terminal measurement reports, and the communication index information is stored in a distributed manner; wherein the data analysis query system is used for regional hotspot analysis, regional fault statistical analysis, user fault complaint analysis and call detail query; wherein the service management system is used for realizing service request receiving, service request processing and service request response, system configuration file reading and other service process synchronous notification.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a data analysis system. BACKGROUND

[0002] At present, the essence of the traditional mobile communication process is to establish data transmission for signals based on a long-distance wireless communication network or a wired network. In the process of communication request initiation and data transmission, communication data needs to be received and forwarded by multiple wireless base stations or wired relay stations. In order to prevent data packet loss and communication drop, each base station outputs a large amount of monitoring log data in the process of data transmission. Mobile operators can capture call abnormalities in time and analyze abnormal reasons through real-time analysis of log records, so as to realize fault monitoring of the mobile network. At the same time, in-depth analysis and mining of log data can obtain hot areas, hot users and fault concentration areas in the communication network, which can assist in optimizing the quality of communication services.

[0003] However, the huge data throughput, complex data structure and real-time task requirements make the data processing very difficult, but real-time communication log storage and analysis provides an opportunity to improve the quality of communication services and create business value. Therefore, the design and implementation of a communication data big data management platform is a work with great potential and challenges. The traditional communication data management system is mainly based on a single machine system or a distributed environment based on a hard disk. Due to the limited data storage and processing capacity, the current data processing method mainly uses data sampling and statistics, only the statistical information is saved in the database system, and the original data is discarded. Therefore, how to analyze the huge communication data has become a problem to be solved in the industry. SUMMARY

[0004] The embodiment of the present application provides a data analysis system, which solves the technical problem of how to analyze the huge communication data.

[0005] In a first aspect, the embodiment of the present application provides a data analysis system, comprising:

[0006] A communication data management platform, a data analysis query system and a service management system; the communication data management platform and the data analysis query system are in communication connection, and the service management system is connected with the analysis query system and the communication data management platform respectively;

[0007] The communication data management platform is used for generating communication index information according to the received user call log, user network access log and mobile terminal measurement report, and distributing the communication index information.

[0008] The data analysis query system is used for regional hotspot analysis, regional fault statistical analysis, user fault complaint analysis, and call detail query.

[0009] The service management system is used for receiving service requests, processing service requests, responding to service requests, reading system configuration files, and synchronously notifying other service processes.

[0010] In one embodiment, the communication data management platform comprises a data processing module, a distributed file storage system, a data analysis query module, and a real-time positioning query module; the distributed file storage system is connected with the data processing module, the data analysis query module, and the real-time positioning query module respectively; the data processing module, the data analysis query module, and the real-time positioning query module are connected with the data analysis query system;

[0011] The data processing module is used for generating communication index information according to the received user call logs, user network access logs, and mobile terminal measurement reports, and transmitting the communication index information to the distributed file storage system for data storage.

[0012] The data analysis query module is used for performing data grouping, summarizing, sorting, and other complex logical operations on the data stored in the distributed file storage system from multiple dimensions, and feeding back data analysis results to the distributed file storage system.

[0013] The real-time positioning query module is used for locating target information from the data stored in the distributed file storage system according to data filtering conditions.

[0014] The data analysis query system is used for regional hotspot analysis, regional fault statistical analysis, user fault complaint analysis, and call detail query.

[0015] In one embodiment, the data processing module comprises a data splicing unit and an index calculation unit; the index calculation unit is connected with the data splicing unit and the distributed file storage system respectively.

[0016] The data splicing unit is used for receiving the user call logs, user network access logs, and mobile terminal measurement reports, and splicing them according to preset association rules between data to obtain complete user call records after splicing.

[0017] The index calculation unit is used for extracting index fields in the complete user call records, and generating new fields and corresponding values according to calculation rules to obtain communication index information.

[0018] In one embodiment, the data analysis query system comprises: a regional hotspot analysis module, a regional fault statistics module, a user fault complaint module and a call detail query module; the regional hotspot analysis module and the regional fault statistics module are connected with the data analysis query module, and the user fault complaint module and the call detail query module are connected with the real-time positioning query module.

[0019] In one embodiment, the service management system comprises: a service request and processing module, a notification and processing module and a configuration management module.

[0020] The service request and processing module is used for synchronization with the notification and processing module and the configuration management module, and is responsible for service request receiving, service request processing and service request response.

[0021] The synchronization notification and processing module is responsible for receiving and processing synchronization notifications from other service processes.

[0022] The configuration management module reads system configuration files, and the configuration is divided into static configuration content and dynamic configuration content.

[0023] In one embodiment, the user call log records relevant information of a mobile terminal user in a call process, and the data is recorded and output by a communication transit station and a communication base station through which the call passes, the user network access log is data transmitted through a wireless network when a user uses wireless network data service, and the mobile terminal measurement report is wireless measurement information measured by a mobile terminal and output by a communication transit station.

[0024] In one embodiment, the distributed file storage system uses a distributed file system as an underlying file system, uses Parquet as a file storage format, and uses a Thrift format for network communication data transmission.

[0025] In one embodiment, the distributed file storage further comprises an index table generated by an HBase database.

[0026] In one embodiment, the user call log comprises information generated by a user through a mobile terminal in a call process, and the information is recorded and output by a communication transit station and a communication base station through which the call passes.

[0027] In one embodiment, the user network access log comprises data transmitted through a wireless network when a user uses wireless network data service.

[0028] The data analysis system provided by the embodiment of the application is used for the characteristics of high data throughput, complex structure, time sequence confusion, high real-time of query service and operation-intensive in communication data processing, adopts a communication data management platform to generate communication index information according to received user call logs, user network access logs and mobile terminal measurement reports, stores the communication index information in a distributed manner, and can realize functions such as regional hotspot analysis, regional fault statistical analysis, user fault complaint analysis, call detail query and the like through a data analysis query system, can realize service request receiving, service request processing and service request response, system configuration file reading and synchronous notification of other service processes through a service management system, and proposes a high-efficiency communication data management platform based on current distributed technology according to the characteristics of high data throughput, complex structure, time sequence confusion, high real-time of query service and operation-intensive in communication data processing, supports efficient data storage, near real-time query processing and dynamic business model generation, and can improve the performance of real-time query based on the combination of distributed platform, distributed computing mode and memory technology. BRIEF DESCRIPTION OF DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the application or prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0030] Figure 1 It is a structural schematic diagram of the data analysis system.

[0031] Figure 2 It is a principle schematic diagram of the distributed file storage system provided by the embodiment of the application.

[0032] Figure 3 It is a principle block diagram of the service management system provided by the embodiment of the application. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solutions and advantages of the application more clear, the technical solutions in the application will be described clearly and completely in the following with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are some embodiments of the application, not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the application.

[0034] Figure 1 It is a structural schematic diagram of the data analysis system. Refer to Figure 1The embodiment of the application provides a data analysis system, comprising a communication data management platform 11, a data analysis query system 12 and a service management system 13; the communication data management platform 11 and the data analysis query system 12 are in communication connection, and the service management system 13 is connected with the data analysis query system 12 and the communication data management platform 11 respectively;

[0035] The communication data management platform 11 is used for generating communication index information according to the received user call log, user network access log and mobile terminal measurement report, and performing distributed storage on the communication index information.

[0036] The data analysis query system 12 is used for performing regional hotspot analysis, regional fault statistical analysis, user fault complaint analysis and call detail query.

[0037] The service management system 13 is used for realizing service request receiving, service request processing and service request response, system configuration file reading and other service process synchronous notification.

[0038] In an optional embodiment, the user call log records relevant information of a mobile terminal user in a call process, data is recorded and output by a communication transfer station and a communication base station in a call process, the user network access log is data transmitted through a wireless network when a user uses wireless network data service, the mobile terminal measurement report is wireless measurement information measured by a mobile terminal and output by a communication transfer station, a communication base station monitors user call, short message or network access behavior and outputs a user call log, in the process of outputting the call log, if the call is network access, the communication base station generates different user network access logs along with the change of an access website, whether the call is or is not, a mobile terminal sends a terminal measurement report at regular time, in the processing process, each user call log may be related to multiple network access logs and multiple mobile terminal measurement reports on a time axis, in the data collection stage, it is required to associate the three data sources according to time and a user identifier to form a complete user call record recording all measurement information.

[0039] In an optional embodiment, the communication data management platform comprises a data processing module, a distributed file storage system, a data analysis query module and a real-time positioning query module; the distributed file storage system is connected with the data processing module, the data analysis query module and the real-time positioning query module respectively; the data processing module, the data analysis query module and the real-time positioning query module are connected with the data analysis query system.

[0040] A data processing module, which receives input data from multiple data sources in real time, performs data association and splicing operation according to the association rules among the data sources, and performs index calculation;

[0041] More specifically, the data processing module includes a data splicing unit and an index calculation unit.

[0042] The data splicing unit is configured to receive three types of data and perform connection according to the association rules among the data.

[0043] The index calculation unit is configured to extract part of the index fields in the complete user call record after splicing, generate new fields and corresponding values according to the calculation rules.

[0044] In the embodiments of the present application, since a large number of parameter information in the data cannot be interpreted by the user for calculation, the index calculation unit can convert the parameter information into index information that can be understood by the user, the data query module receives a query submitted by an upper user and converts the query into a logical query task for the currently stored data;

[0045] A distributed file storage system, which is connected with the data processing module, stores the processed data in a unified data model in the distributed file storage system by the data processing module, and completes data storage; the data analysis query module is connected with the distributed file storage system, and is configured to perform column data operation, i.e., complex logical operations such as data grouping, summarizing, sorting, etc. according to multiple dimensions, and finally returns a data analysis result; the real-time positioning query module is connected with the distributed file storage system, and is configured to quickly locate the information meeting the conditions in the massive data according to the data filtering conditions submitted by the user, and return the information to the user.

[0046] The data analysis query module is connected with the distributed file storage system, and is configured to perform column data operation, i.e., complex logical operations such as data grouping, summarizing, sorting, etc. according to multiple dimensions, and finally returns a data analysis result.

[0047] The real-time positioning query module is connected with the distributed file storage system, and is configured to quickly locate the information meeting the conditions in the massive data according to the data filtering conditions submitted by the user, and return the information to the user.

[0048] Optionally, in the embodiments of the present application, the data analysis query system includes a regional hotspot analysis module, a regional fault statistics module, a user fault complaint module and a call detail query module; the regional hotspot analysis module and the regional fault statistics module are connected with the data analysis query module, and the user fault complaint module and the call detail query module are connected with the real-time positioning query module.

[0049] The data analysis query system further comprises a user fault complaint module and a call detail query module, and is connected with the real-time positioning query module.

[0050] Preferably, the data processing module comprises a data splicing unit and an index calculation unit.

[0051] The data splicing unit is configured to splice the data according to the correlation rules between the data after receiving the user call log, the user network access log and the mobile terminal measurement report.

[0052] The index calculation unit is configured to extract part of the index fields in the complete user call record after splicing, generate new fields and corresponding values according to the calculation rules.

[0053] Preferably, the three kinds of data comprise the user call log, the user network access log and the mobile terminal measurement report.

[0054] Preferably, the user call log records the relevant information of the mobile terminal user in the call process, and the data is recorded and output by the communication transfer station and the communication base station during the call, the user network access log is the data transmitted through the wireless network when the user uses the wireless network data service, and the mobile terminal measurement report is the wireless measurement information measured by the mobile terminal and output by the communication transfer station.

[0055] Preferably, the index calculation unit calculates a large number of parameter information in the existing data which cannot be interpreted by the user, and converts the parameter information into index information which can be understood by the user, and the data query module receives the query submitted by the upper user and converts the query into a logical query task for the currently stored data.

[0056] Preferably, Figure 2 A principle diagram of a distributed file storage system provided by the embodiment of the application is shown in the figure. Figure 2 The distributed file storage system takes HDFS as the underlying file system, takes Parquet as the file storage format, and uses the Thrift format for the transmission of network communication data.

[0057] Preferably, the distributed file storage further comprises an index table generated by HBase.

[0058] Preferably, the communication data management platform takes Spark as the core distributed computing system, and introduces HBase to establish a query index for the data, thereby improving the query efficiency.

[0059] In some embodiments of the application, Figure 3 A principle block diagram of a service management system provided by the embodiment of the application is shown in the figure. Figure 3As shown, the service management system comprises: a service request and processing module, a notification and processing module and a configuration management module;

[0060] The service request and processing module is used for synchronously notifying the notification and processing module and the configuration management module, and is responsible for service request receiving, service request processing and service request response, wherein the service request processing further comprises updating a database server, updating cache data and synchronously notifying other service processes;

[0061] The synchronous notification and processing module is responsible for receiving and processing synchronous notifications from other service processes, and the processing only needs to update the cache data of the service process;

[0062] The configuration management module is implemented using a monostate mode, reads a system configuration file, and the configuration is divided into static configuration content and dynamic configuration content. For the dynamic configuration content, the management module re-reads at a regular time to ensure that the system configuration modification takes effect without restarting the service process. The system configuration content mainly includes name service allocation, allocation of a fixed logical name for each service process, and specification of the communication mode between service processes, thereby achieving communication self-adaptation.

[0063] In the embodiment of the present application, in view of the characteristics of high data throughput, complex structure, time sequence disorder, high real-time performance of query services, and computation-intensive in communication data processing, a communication data management platform is adopted to generate communication index information according to received user call logs, user network access logs and mobile terminal measurement reports, to store the communication index information in a distributed manner, and to realize regional hotspot analysis, regional fault statistical analysis, user fault complaint analysis, call detail query and other functions through a data analysis query system. Through a service management system, service request receiving, service request processing and service request response, system configuration file reading, and synchronous notification of other service processes can be realized. Through the characteristics of high data throughput, complex structure, time sequence disorder, high real-time performance of query services, and computation-intensive in communication data processing, a high-efficiency communication data management platform is designed and implemented based on the current distributed technology, to support efficient data storage, near-real-time query processing and dynamic business model generation. Based on the combination of the distributed platform and the distributed computing mode and the memory technology, large data processing is supported and the performance of real-time query is improved.

[0064] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed to multiple network units. Part or all of the modules can be selected to achieve the purposes of the embodiments according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0065] Through the description of the above embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software and the necessary general hardware platform, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of software products, and the computer software products can be stored in a computer readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and include a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0066] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A data analysis system, characterized in that, include: Communication data management platform, data analysis and query system, and service management system; The communication data management platform and the data analysis and query system are communicatively connected, and the service management system is connected to both the data analysis and query system and the communication data management platform. The communication data management platform is used to generate communication indicator information based on the received user call logs, user network access logs and mobile terminal measurement reports, and to store the communication indicator information in a distributed manner. The data analysis and query system is used for regional hotspot analysis, regional fault statistics analysis, user fault complaint analysis, and call detail query. The service management system is used to receive service requests, process service requests and respond to service requests, read system configuration files, and synchronize notifications to other service processes.

2. The data analysis system according to claim 1, characterized in that, The communication data management platform includes: a data processing module, a distributed file storage system, a data analysis and query module, and a real-time location query module; the distributed file storage system is connected to the data processing module, the data analysis and query module, and the real-time location query module respectively; the data processing module, the data analysis and query module, and the real-time location query module are all connected to the data analysis and query system. The data processing module is used to generate communication indicator information based on the received user call logs, user network access logs and mobile terminal measurement reports, and transmit the communication indicator information to the distributed file storage system for data storage. The data analysis and query module is used to perform complex logical operations such as data grouping, summarizing, and sorting on the data stored in the distributed file storage system from multiple dimensions, and to feed back the data analysis results to the distributed file storage system. The real-time location query module is used to locate target information from the data stored in the distributed file storage system according to data filtering conditions. The data analysis and query system is used for regional hotspot analysis, regional fault statistics, user fault complaint analysis, and call detail query.

3. The data analysis system according to claim 2, characterized in that, The data processing module includes a data splicing unit and an indicator calculation unit, wherein the indicator calculation unit is connected to the data splicing unit and the distributed file storage system, respectively. The data splicing unit is used to receive the user call log, user network access log and mobile terminal measurement report, and splice them according to the preset association rules between the data to obtain the complete spliced ​​user call record; The indicator calculation unit is used to extract indicator fields from the complete user call records and generate new fields and corresponding values ​​according to the calculation rules to obtain communication indicator information.

4. The data analysis system according to claim 2, characterized in that, The data analysis and query system includes: a regional hotspot analysis module, a regional fault statistics module, a user fault complaint module, and a call detail query module; the regional hotspot analysis module and the regional fault statistics module are connected to the data analysis and query module, and the user fault complaint module and the call detail query module are connected to the real-time location query module.

5. The data analysis system according to claim 1, characterized in that, The service management system includes: a service request and processing module, a notification and processing module, and a configuration management module; The service request and processing module is used to synchronize the notification and processing module and the configuration management module. The service request and processing module is responsible for receiving service requests, processing service requests, and responding to service requests. The notification and processing module is responsible for receiving and processing synchronous notifications from other service processes. The configuration management module reads the system configuration file, and the configuration is divided into static configuration content and dynamic configuration content.

6. The data analysis system according to claim 1, characterized in that, The user call log records relevant information of the mobile terminal user during the call process. The data is recorded and output by the communication relay station and communication base station through which the call passes. The user network access log is the data transmitted through the wireless network when the user uses wireless network data services. The mobile terminal measurement report is wireless measurement information measured by the mobile terminal and processed and output by the communication relay station.

7. The data analysis system according to claim 2, characterized in that, The distributed file storage system uses a distributed file system as the underlying file system, Parquet as the file storage format, and Thrift format for transmitting network communication data.

8. The data analysis system according to claim 2, characterized in that, The distributed file storage system also includes index tables generated by the HBase database.

9. The data analysis system according to claim 1, characterized in that, The user call log includes information generated by the user during a call via a mobile terminal. This information is recorded and output by the communication relay stations and communication base stations through which the call passes.

10. The data analysis system according to claim 1, characterized in that, The user network access log includes data transmitted through the wireless network when the user uses wireless network data services.

Images