Log data processing method, system, equipment and medium

A data processing system and data processing technology, applied in the field of data processing, can solve problems such as inability to process large-scale data, and achieve the effect of reducing capacity, reducing operation and maintenance and management costs, and improving efficiency

Pending Publication Date: 2021-06-22
CTRIP TRAVEL NETWORK TECH SHANGHAI0
View PDF0 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0008] The technical problem to be solved by the present invention is to provide a log data processing method, system, equipment and medium in order to overcome the defect that the existing technology cannot handle large-scale data

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Log data processing method, system, equipment and medium
  • Log data processing method, system, equipment and medium
  • Log data processing method, system, equipment and medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0046] This embodiment provides a log data processing method, figure 1 A flowchart of this embodiment is shown. refer to figure 1 , the log data processing method of this embodiment is applied in a log storage system, and the log storage system includes a columnar database management system, and the log data processing method of this embodiment includes:

[0047] S101. Obtain target log data from a columnar database management system;

[0048] S102. Process the target log data in a preset management manner to obtain a log management result;

[0049] Among them, the default management mode corresponds to IDC management, NODE management, cluster management, database management, table structure management, data pipeline management, account management, log management, and log access management.

[0050] Specifically, this embodiment obtains log raw data from message queues such as kafka (a distributed publish-subscribe message system), and then performs log effective field extr...

Embodiment 2

[0054] figure 2 The flow chart of this embodiment is shown. The log data processing method of this embodiment is a further improvement on Embodiment 1. Specifically, the log storage system of this embodiment further includes a log data preprocessing system and a display system. The log data processing method of this embodiment also includes:

[0055] S103. Perform configuration processing based on the log data preprocessing series and the columnar database management system to obtain configuration results;

[0056]Among them, the configuration result corresponds to the lifecycle management of the target log data in the columnar database management system.

[0057] S104. Displaying the target log data based on the display system;

[0058] Specifically, the display system may use the Kibana system, which is an open-source visual chart display system.

[0059] By enabling Kibana to support Clickhouse, the user's usage habits remain unchanged, and the cost of user learning is ...

Embodiment 3

[0071] This embodiment provides a log data processing system, image 3 A schematic structural diagram of this embodiment is shown. refer to image 3 , the log data processing system of this embodiment is applied in a log storage system, and the log storage system includes a columnar database management system, and the log data processing system of this embodiment includes:

[0072] An acquisition module 1, configured to acquire target log data from the columnar database management system;

[0073] A processing module 2, configured to process the target log data in a preset management manner to obtain a log management result;

[0074] Wherein, the preset management mode corresponds to IDC management, NODE management, cluster management, database management, table structure management, data pipeline management, account management, log management, and log access management.

[0075] Specifically, this embodiment obtains log raw data from message queues such as kafka (a distrib...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a log data processing method, system and device and a medium, the log data processing method is applied to a log storage system, the log storage system comprises a column database management system, and the method comprises the steps of obtaining target log data from the column database management system; processing the target log data in a preset management mode to obtain a log management result, wherein the preset management mode corresponds to at least one of IDC management, NODE management, cluster management, database management, table structure management, data pipeline management, account management, log management and log access management. According to the invention, through mutual cooperation and mutual coordination among multiple preset management modes, self-service management of the full life cycle of the log is realized, processing of super-large-scale log data and management of the column database management system are realized, and the operation, maintenance and management costs of the column database management system are greatly reduced.

Description

technical field [0001] The present invention relates to the technical field of data processing, in particular to a log data processing method, system, device and medium. Background technique [0002] In order to improve the service quality of enterprises, Internet service providers need to record user access information, network operation status, etc. With the rapid development of the Internet and the continuous expansion of the company's business, it has brought an increasing number of Internet users, and the huge number of users will generate a large amount of log data in the process of recording information. How to manage and analyze these massive log data has become A difficult problem for Internet companies. [0003] Existing log analysis generally has the following problems: [0004] (1) The amount of data is large, both incremental data and stock data are very large, and the processing and storage costs are high, requiring a larger throughput of the log analysis sys...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F11/34G06F16/25G06F16/21
CPCG06F11/3476G06F16/21G06F16/254
Inventor 朱挺唐锐华陈汉徐利平李焜胤陈国杰
Owner CTRIP TRAVEL NETWORK TECH SHANGHAI0
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap