A game log real-time processing system and method
A real-time processing and log technology, applied in the direction of electrical digital data processing, instruments, hardware monitoring, etc., can solve the problems of new users, time-limited system administrators, and easy to be blocked
Active Publication Date: 2018-03-16
NETEASE (HANGZHOU) NETWORK CO LTD
View PDF3 Cites 0 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
[0004] (2) The job scheduling of MapReduce is largely limited by the resource manager. When the system is busy, there will be a relatively large delay;
[0006] (4) The collaboration between multiple jobs is configured through dependencies, which is affected by the external scheduling system and is prone to blockage;
[0007] (5) The calculation of the data of new roles usually depends on the daily full data, and it is very troublesome to calculate the new users during the time period;
[0008] (6) Each MapReduce calculation process involves a large amount of IO, and it is difficult to significantly improve performance
[0011] (1) Storm needs to be deployed to the cluster, which brings certain installation and operation and maintenance costs;
[0012] (2) There will be a certain learning curve and cycle for the in-depth grasp of Strom components and the development process;
[0013] (3) It is necessary to invest in SA personnel for research and various tests, and the time will be limited by the system administrator;
[0014] (4) Involving big data roles or account duplication checks, batch cache checks are required. When tasks fail or abnormal conditions exit, in order not to lose information when the computing framework is automatically readjusted, it will greatly increase the difficulty of development or the difficulty of finding problems ;Because a node in Storm will automatically restart when there is an exception, the data cached in the memory will be lost; if it is not buffered, each data arrival will directly interact with HBase, which will bring a certain performance overhead ;
[0015] (5) Data is stored on HDFS files, and it is difficult to guarantee transactionality when reading fragments concurrently;
[0016] (6) It is difficult to quickly resume production when a problem occurs, and the controllability is not high;
[0018] To sum up, using the calculation method of MapReduce+Hive to count indicators cannot meet the requirements for timeliness, but there are still many unresolved problems when using Storm's technical solution. In addition, it also has high requirements for memory overhead. Therefore, it is necessary to independently develop a set of "game real-time log processing" system according to the specific business and data characteristics and the requirements of real-time index statistics, so as to complete the real-time data processing tasks at the minimum cost and make data expansion more flexible and convenience
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0112] The present invention will be further described in detail below in combination with specific embodiments and with reference to the accompanying drawings. It should be emphasized that the following description is only exemplary and not intended to limit the scope of the invention and its application.
[0113] With reference to the following figures, non-limiting and non-exclusive embodiments will be described.
[0114] The present invention proposes a game log real-time processing system 1, see image 3 It is a structural frame of a game log real-time processing system provided by a specific embodiment of the present invention, including:
[0115] Data acquisition module 10, is used for obtaining stream data with row level from distributed file system (such as HDFS), and stream data is stored in internal memory, and distributed file system is used for storing game log files;
[0116] The data processing module 20 is used to filter and convert the flow data in the newly...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The present invention discloses a game log real-time processing system and method. The game log real-time processing system comprises: a data acquisition module, which is used for acquiring stream data at a row level from a distributed file system and storing the stream data into a memory; a data processing module, which is used for processing the stream data to obtain valid data; and an indicator counting module, which is used for performing indicator statistical calculation by using the valid data, so as to obtain an indicator counting result. According to the game log real-time processing system provided by the present invention, the real-time processing for mass data is ensured, which is beneficial to real-time evaluation of the game test effect.
Description
technical field [0001] The invention relates to a game log real-time processing system and method. Background technique [0002] As more and more games are connected to the data platform, the levels, levels, and attention of each game are different, so that there are different requirements for log interpretation, processing, statistics, analysis, and even presentation. Whenever a game for external testing is launched and the game log data is accessed, some key indicators, including operation, monitoring, billing, etc., require real-time performance. real time monitoring. figure 1 It is a work flow chart of using Hadoop to perform indicator statistics in the prior art. When a new game log is connected, the game log on HDFS (Hadoop Distributed File System, Hadoop Distributed File System) is processed in combination with Python scripts and split. Finally, according to the configuration fields (the order of the fields is consistent with the order of the fields defined in the ...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More Patent Type & AuthorityPatents(China)
IPC IPC(8): G06F11/34
CPCG06F11/3466
Inventor陈创生马海平
OwnerNETEASE (HANGZHOU) NETWORK CO LTD