Method and device for processing client log

A processing method and client-side technology, applied in the communication field, can solve the problems of limited number of logs, limited disk space, and low log security, and achieve the effect of ensuring security.

Active Publication Date: 2013-09-18
TAOBAO CHINA SOFTWARE
View PDF5 Cites 51 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0006] The embodiment of the present application provides a client log processing method and device, which are used to solve the problems in the prior art that the number of logs that can be processed is limited due to limited disk space, and the security of the saved logs is low

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Method and device for processing client log
  • Method and device for processing client log
  • Method and device for processing client log

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0027] Embodiment 1 of the present application provides a client log processing method, the steps of the method are as follows figure 2 shown, including:

[0028] Step 101, write the received log into the current buffer.

[0029] In this embodiment, two pieces of MMAP memory are set as the buffer, in this step, when receiving the log, write the log into the MMAP memory as the current buffer (this MMAP memory can be referred to as the first MMAP memory ).

[0030] Specifically, the data packet sent by the client through the POST mode of Hypertext Transfer Protocol (HTTP, HyperText Transfer Protocol) can be received, and the cyclic redundancy check (CRC) check is performed on the data packet, such as CRC16 check. When passing, an asynchronous thread can be started to write the log in the data packet into the first MMAP memory as the current buffer (that is, receiving the data packet and checking the data packet can be synchronous, that is, the same thread can be used for sequ...

Embodiment 2

[0041] Embodiment 2 of the present application provides a front-end processing method, the step flow of the method is as follows image 3 shown, including:

[0042] Step 201, receiving a data packet.

[0043] In this step, the mobile client can transmit the collected logs (in the form of data packets) to the server through the POST method of the HTTP protocol.

[0044] Step 202, perform verification.

[0045] In this step, after receiving the data packet, the server first performs a crc16 check on the integrity of the data in the data packet to determine whether it is complete. If it is determined to be complete, it can continue to perform steps 203 to 205. If it is determined to be incomplete, It indicates that the data packet is incomplete or the data packet has been tampered with, and the execution of step 206 can be skipped.

[0046] Step 203, obtaining logs.

[0047] In this step, according to the protocol, the data packet can be decompressed and assembled to obtain t...

Embodiment 3

[0055] Embodiment 3 of the present application provides a back-end double-buffering processing method. The steps of the method are as follows: Figure 4 shown, including:

[0056] Step 301. Determine the logs obtained after front-end processing.

[0057] After being processed by the front end, the logs sent by the client can be obtained. And the obtained logs can be used to asynchronously trigger back-end double-buffering processing.

[0058] Step 302, write the obtained log into the current buffer.

[0059] In this step, the obtained log may be written into the MMAP memory as the current buffer.

[0060] Step 303, judging whether there is an exception of insufficient space in the current buffer.

[0061] In this step, it is possible to determine whether the MMAP memory as the current buffer is full by judging whether the current buffer returns a prompt that the memory is full, and when receiving the prompt that the memory returned by the current buffer is full, determine ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The embodiment of the invention provides a method and a device for processing a client log. The method comprises the following steps of adopting a double buffering mechanism, previously setting two MMAP (map files or devices into memory) internal memories to be respectively used as a current buffering area and a standby buffering area, switching the standby buffering area to be the current buffering area when the current buffering area is filled, and transmitting a log stored in the filled buffering area to a distributed file system to be stored, so that the problem that the log processing capacity is limited because the quantity of the processed logs is limited by the limit space of the existing disk can be solved. Due to the characteristics of the MMAP technology, in the operation process, the data in the buffering area can be forcedly outputted to the disk by the system, so that even if a server is restarted or collapsed, the data stored in the buffering area can be restored through the disk without being lost, and the safety of the log stored in the buffering area can be guaranteed.

Description

technical field [0001] The present application relates to the field of communications, and in particular to a client log processing method and device. Background technique [0002] In the existing client log collection and processing scheme, when the client exits, it will compress the collected logs into data packets, for example, compress them into data packets in common formats such as RAR and zip, and send the data packets to to the server. The server verifies the validity of the received data packet according to the relevant protocol, decompresses the data packet after the verification is passed, stores the log obtained after decompression in the log file, and can transmit the log to the distributed server through the background program. On the distributed file system (such as HDFS, Hadoop Distributed File System) implemented by the system infrastructure (such as hadoop). That is, the collected logs are printed to log files. The log system implements a memory-based log...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F11/14G06F12/16
Inventor 张黎明
Owner TAOBAO CHINA SOFTWARE
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap