Game log real-time processing system and method

Abstract

The present invention discloses a game log real-time processing system and method. The game log real-time processing system comprises: a data acquisition module, which is used for acquiring stream data at a row level from a distributed file system and storing the stream data into a memory; a data processing module, which is used for processing the stream data to obtain valid data; and an indicator counting module, which is used for performing indicator statistical calculation by using the valid data, so as to obtain an indicator counting result. According to the game log real-time processing system provided by the present invention, the real-time processing for mass data is ensured, which is beneficial to real-time evaluation of the game test effect.

Description

technical field [0001] The invention relates to a game log real-time processing system and method. Background technique [0002] As more and more games are connected to the data platform, the levels, levels, and attention of each game are different, so that there are different requirements for log interpretation, processing, statistics, analysis, and even presentation. Whenever a game for external testing is launched and the game log data is accessed, some key indicators, including operation, monitoring, billing, etc., require real-time performance. real time monitoring. figure 1 It is a work flow chart of using Hadoop to perform indicator statistics in the prior art. When a new game log is connected, the game log on HDFS (Hadoop Distributed File System, Hadoop Distributed File System) is processed and split in combination with Python scripts. . Finally, according to the configuration fields (the order of the fields is consistent with the order of the fields defined in th...

AI Technical Summary

Problems solved by technology

[0004] (2) The job scheduling of MapReduce is largely limited by the resource manager. When the system is busy, there will be a relatively large delay;

[0006] (4) The collaboration between multiple jobs is configured through dependencies, which is affected by the external scheduling system and is prone to blockage;

[0007] (5) The calculation of the data of new roles usually depends on the daily full data, and it is very troublesome to calculate the new users during the time period;

[0008] (6) Each MapReduce calculation process involves a large amount of IO, and it is difficult to significantly improve performance

[0011] (1) Storm needs to be deployed to the cluster, which brings certain installation and operation and maintenance costs;

[0012] (2) There will be a certain learning curve and cycle for the in-depth grasp of Strom components and the development process;

[0013] (3) It is necessary to invest in SA personnel for research and various tests, and the time will be limited by the system administrator;

[0014] (4) Involving big data roles or account duplication checks, batch cache checks are required. When tasks fail or abnormal conditions exit, in order not to lose information when the computing framework is automatically readjusted, it will greatly increase the difficulty of development or the difficulty of finding problems ;Because a node in Storm will automatically restart when there is an exception, the data cached in the memory will be lost; if it is not buffered, each data arrival will directly interact with HBase, which will bring a certain performance overhead ;

[0015] (5) Data is stored on HDFS files, and it is difficult to guarantee transactionality when reading fragments concurrently;

[0016] (6) It is difficult to quickly resume production when a problem occurs, and the controllability is not high;

[0018] To sum up, using the calculation method of MapReduce+Hive to count indicators cannot meet the requirements for timeliness, but there are still many unresolved problems when using Storm's technical solution. In addition, it also has high requirements for memory overhead. Therefore, it is necessary to independently develop a set of "game real-time log processing" system according to the specific business and data characteristics and the requirements of real-time index statistics, so as to complete the real-time data processing tasks at the minimum cost and make data expansion more flexible and convenience

Images