De-duplication storage method, device and equipment for massive log data and storage medium
A database and log technology, applied in the field of data processing, can solve the problems of insufficient disk capacity of a single computer and inability to expand infinitely, and achieve the effect of avoiding excessive demand for disk capacity
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
Method used
Image
Examples
Embodiment 1
[0027] figure 1 It is a flow chart of a method for deduplication and storage of massive log data provided by Embodiment 1 of the present invention. This embodiment can be applied to the situation where valuable information is stored after deduplication of massive log data, for example, to provide The deduplicated log data facilitates police investigation. The method can be performed by the device for deduplication and storage of massive log data provided by the embodiment of the present invention. The device can be implemented in the form of software and / or hardware, and can generally be integrated in computer equipment, such as figure 1 As shown, the method of this embodiment specifically includes:
[0028] S110. Obtain the massive log data to be stored in the first time interval.
[0029] The first time interval is a preset time interval, and the massive logs in this time interval are deduplicated and stored in the warehouse. Preferably, the first time interval is daily, t...
Embodiment 2
[0064] figure 2 Shown is a schematic structural diagram of a device for deduplication and storage of massive log data provided by Embodiment 2 of the present invention. This embodiment is applicable to situations where valuable information is stored after deduplication of massive log data, for example, for The police provide deduplicated log data to facilitate police investigation. The device can be implemented in the form of software and / or hardware, and generally can be integrated in computer equipment, such as figure 2 As shown, the device for deduplication and warehousing of massive log data specifically includes: a data acquisition module 210 for storage, a pre-deduplication result acquisition module 220 for storage, a full deduplication result acquisition module 230, and a database update module 240, wherein,
[0065] A data acquisition module 210 to be stored, configured to acquire a large amount of log data to be stored in the first time interval;
[0066] The pre-...
Embodiment 3
[0089] like image 3 As shown, it is a schematic diagram of the hardware structure of a computer device provided by Embodiment 3 of the present invention, as shown in image 3 As shown, the computer equipment includes:
[0090] one or more processors 310, image 3 Take a processor 310 as an example;
[0091] memory 320;
[0092] The computer device may further include: an input device 330 and an output device 340 .
[0093] The processor 310, the memory 320, the input device 330 and the output device 340 in the computer device can be connected by bus or other methods, image 3 Take connection via bus as an example.
[0094] The memory 320, as a non-transitory computer-readable storage medium, can be used to store software programs, computer-executable programs and modules, such as program instructions / modules (e.g., attached figure 2 Shown are the data to be loaded into the data acquisition module 210, the data to be stored in the pre-deduplication result acquisition ...
PUM
Login to View More Abstract
Description
Claims
Application Information
Login to View More