Method, device, equipment and storage medium for deduplication and storage of massive log data
A database and log technology, applied in the field of data processing, can solve the problems of insufficient disk capacity of a single computer and inability to expand infinitely, and achieve the effect of avoiding excessive demand for disk capacity
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
Method used
Image
Examples
Embodiment 1
[0027] figure 1 This is a flowchart of a method for deduplicating massive log data provided in the first embodiment of the present invention. This embodiment is applicable to the situation where valuable information is stored after deduplication of massive log data, for example, it is provided to the police The log data after deduplication provides convenience for police investigation. This method can be executed by the device for deduplication of massive log data provided by the embodiment of the present invention. The device can be implemented in software and / or hardware, and can generally be integrated in computer equipment, such as figure 1 As shown, the method of this embodiment specifically includes:
[0028] S110. Acquire massive log data to be stored in the first time interval.
[0029] The first time interval is a preset time interval, and the massive log data in the time interval is deduplicated. Preferably, the first time interval is daily, that is, the massive log data ...
Embodiment 2
[0064] figure 2 Shown is a schematic diagram of the structure of a device for deduplication of massive log data provided in the second embodiment of the present invention. This embodiment is applicable to a situation where valuable information is stored after deduplication of massive log data, for example, The police provide the log data after deduplication to facilitate police investigations. The device can be implemented in software and / or hardware, and generally can be integrated in computer equipment, such as figure 2 As shown, the device for deduplication of massive log data specifically includes: a data acquisition module 210 to be stored, a pre-deduplication result acquisition module 220 to be stored, a full deduplication result acquisition module 230, and a database update module 240, in which,
[0065] The to-be-stocked data acquisition module 210 is configured to acquire the massive log data to be-stocked in the first time interval;
[0066] The pre-deduplication result...
Embodiment 3
[0089] Such as image 3 Shown is a schematic diagram of the hardware structure of a computer device provided in the third embodiment of the present invention, such as image 3 As shown, the computer equipment includes:
[0090] One or more processors 310, image 3 Take one processor 310 as an example;
[0091] Memory 320;
[0092] The computer equipment may further include: an input device 330 and an output device 340.
[0093] The processor 310, the memory 320, the input device 330, and the output device 340 in the computer equipment may be connected by a bus or other means, image 3 Take the bus connection as an example.
[0094] As a non-transitory computer-readable storage medium, the memory 320 can be used to store software programs, computer-executable programs, and modules, such as program instructions / program instructions corresponding to a method for deduplication of massive log data in an embodiment of the present invention. Module (for example, attached figure 2 The dat...
PUM
Login to View More Abstract
Description
Claims
Application Information
Login to View More