A log collection system, method, apparatus and storage medium

Abstract

The present invention relates to the field of computer technology, and discloses a log collection system, method, device and storage medium. The system includes: a data collector is used to obtain collection information and file modification time corresponding to the file log to be collected, and collect the file log; a vector library container device is used to generate a current log collection vector according to the collection information, and generate the current corresponding relationship between the current log collection vector and the file modification time, and detect whether there is an existing corresponding relationship that matches the current corresponding relationship; a distributed file subsystem is used to store the file log according to the current log collection vector. The solution of this embodiment generates a log collection vector by adding a vector library container device, and performs duplicate file log detection based on the corresponding relationship between the log collection vector and the file modification time, so as to realize non-duplicate collection of file logs, avoid waste of system resources and reduce storage costs.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a log collection system, method, device and storage medium. Background Art

[0002] Log collection is an important step in log data processing. Achieving efficient and accurate log collection is of great significance to improving the processing effect of log data.

[0003] At present, the existing log collection method usually configures the collection log by calling the application programming interface. However, in the prior art, due to the use of regular matching during configuration, it is easy to cause repeated collection of file logs, which easily leads to waste of system resources and increased storage costs. In addition, the subsequent comparison and deletion of duplicate logs will further increase system performance consumption and increase the time spent on log collection. Summary of the invention

[0004] The present invention provides a log collection system, method, device and storage medium, which can realize non-repetitive collection of file logs, avoid waste of system resources and reduce storage costs, reduce system performance consumption and shorten the time spent on log collection.

[0005] According to one aspect of the present invention, there is provided a log collection system, comprising a data collector, a vector library container device and a distributed file subsystem;

[0006] The data collector is respectively connected to the vector library container device and the distributed file subsystem for communication, and is used to obtain the collection information and file modification time corresponding to the file log to be collected, and send the collection information and the file modification time to the vector library container device, and when receiving the verification success information fed back by the vector library container device, collect the file log and send the file log to the distributed file subsystem;

[0007] The vector library container device is communicatively connected with the distributed file subsystem, and is used to generate a current log collection vector according to the collection information, and generate a current correspondence between the current log collection vector and the file modification time, and when no existing correspondence matching the current correspondence is detected, send the verification success information to the data collector, send the current log collection vector to the distributed file subsystem, and store the current correspondence locally;

[0008] The distributed file subsystem is used to store the file log according to the current log collection vector.

[0009] According to another aspect of the present invention, a log collection method is provided, which is applied to the log collection system described in any embodiment of the present invention, comprising:

[0010] Obtaining collection information and file modification time corresponding to the file log to be collected, and generating a current log collection vector according to the collection information;

[0011] generating a current correspondence between the current log collection vector and the file modification time, and storing the current correspondence in a vector library container device when no stored correspondence matching the current correspondence is detected;

[0012] The file log is collected according to the collection information and the file modification time, and the file log is stored in a distributed file subsystem according to the current log collection vector.

[0013] According to another aspect of the present invention, a log collection device is provided, which is applied to the log collection system described in any embodiment of the present invention, including:

[0014] A log collection vector generation module is used to obtain the collection information and file modification time corresponding to the file log to be collected, and generate a current log collection vector according to the collection information;

[0015] A correspondence storage module, used for generating a current correspondence between the current log collection vector and the file modification time, and storing the current correspondence in a vector library container device when no stored correspondence matching the current correspondence is detected;

[0016] The file log storage module is used to collect the file log according to the collection information and the file modification time, and store the file log in the distributed file subsystem according to the current log collection vector.

[0017] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and the computer program is used to enable a processor to implement the log collection method described in any embodiment of the present invention when executed.

[0018] The technical solution of the embodiment of the present invention is to obtain the collection information and file modification time corresponding to the file log to be collected through a data collector, and send the collection information and file modification time to a vector library container device, and when receiving the verification success information fed back by the vector library container device, collect the file log and send the file log to a distributed file subsystem; generate a current log collection vector according to the collection information through the vector library container device, and generate a current corresponding relationship between the current log collection vector and the file modification time, and when no existing corresponding relationship matching the current corresponding relationship is detected, send the verification success information to the data collector, send the current log collection vector to the distributed file subsystem, and store the current corresponding relationship locally; store the file log according to the current log collection vector through the distributed file subsystem; generate a log collection vector according to the collection information of the file log through a newly added vector library container device, and perform duplicate file log detection based on the corresponding relationship between the log collection vector and the file modification time, so as to realize the non-duplicate collection of file logs, avoid the waste of system resources and reduce storage costs, reduce system performance consumption and shorten the time spent on log collection.

[0019] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 This is a structural diagram of a log collection system provided according to Embodiment 1 of the present invention;

[0022] Figure 2A is a flow chart of a log collection method provided according to Embodiment 2 of the present invention;

[0023] Figure 2B is a flow chart of another log collection method provided according to Embodiment 2 of the present invention;

[0024] Figure 3 It is a structural diagram of a log collection device provided according to Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0025] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0026] It should be noted that the terms "first", "second", "target", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0027] Embodiment 1

[0028] Figure 1 A structural diagram of a log collection system is provided for the first embodiment of the present invention. The log collection system 10 includes a data collector 11, a vector library container device 12 and a distributed file subsystem 13;

[0029] The data collector 11 is respectively connected to the vector library container device 12 and the distributed file subsystem 13 for communication, and is used for obtaining the collection information and file modification time corresponding to the file log to be collected, and sending the collection information and the file modification time to the vector library container device 12, and upon receiving the verification success information fed back by the vector library container device 12, collecting the file log and sending the file log to the distributed file subsystem 13.

[0030] The collection information may be parameter information required for collecting file logs, for example, file path, file name, etc. In this embodiment, the data collector 11 may parse the log collection task to obtain the collection information and file modification time corresponding to the file log to be collected, and may provide the collection information and file modification time to the vector library container device 12, so that the vector library container device 12 determines whether the log collection is repeated based on the collection information and file modification time. This embodiment does not specifically limit the type of file log.

[0031] Then, if the verification success information fed back by the vector library container device 12 is received, the data collector 11 can determine that the current file log is not a duplicate log, and can be collected. Based on the collection information and the file modification time, the file log can be collected from the corresponding log source device, and the file log can be sent to the distributed file subsystem 13 for storage.

[0032] In another case, after providing the collection information and file modification time to the vector library container device 12, if the data collector 11 receives verification failure information fed back by the vector library container device 12, it means that the current file log is a duplicate log and there is no need to collect it.

[0033] The vector library container device 12 is communicatively connected to the distributed file subsystem 13, and is used to generate a current log collection vector according to the collection information, and generate a current correspondence between the current log collection vector and the file modification time, and when no existing correspondence matching the current correspondence is detected, send the verification success information to the data collector 11, send the current log collection vector to the distributed file subsystem 13, and locally store the current correspondence.

[0034] The vector library container device 12 may be a collection of functions and data structures, allowing users to use vectors in programming languages, and some common operations that may be provided include creating and initializing vectors, accessing individual elements in vectors, adding or deleting elements from vectors, sorting and searching for elements in vectors, and performing mathematical operations such as addition, subtraction, multiplication, and division on vectors. Typically, the vector library container device 12 may include a standard template library in C++, a numpy library in Python, and a Vector class in Java.

[0035] In this embodiment, the vector library container device 12 can use a preset text conversion method, such as a vector space model, local sensitive hashing, word2vec, etc., to convert the collected information into the current log collection vector, and can establish the current corresponding relationship between the current log collection vector and the file modification time. Then, the vector library container device 12 can match the current corresponding relationship with all the existing corresponding relationships one by one. If an existing corresponding relationship that is exactly the same as the current corresponding relationship is detected, it means that the current file log is a duplicate log, and a verification failure message can be sent to the data collector 11. If it is detected that the current corresponding relationship is different from all the existing corresponding relationships (i.e., they do not match), it means that the current file log is not a duplicate log, and a verification success message can be sent to the data collector 11.

[0036] The current correspondence does not match the stored correspondence, which may be that the current log collection vector in the current correspondence is the same as the stored vector in the stored correspondence, but the file modification time is different; or the current log collection vector in the current correspondence is different from the stored vector in the stored correspondence, in which case the file modification time may be the same or different. The stored correspondence may be the correspondence between the log collection vector generated by the previous file log collection and the file modification time.

[0037] In this embodiment, after determining that no existing corresponding relationship matching the current corresponding relationship is detected, the vector library container device 12 may send the current log collection vector to the distributed file subsystem 13 and add the current corresponding relationship to the local storage.

[0038] The distributed file subsystem 13 is used to store the file log according to the current log collection vector. Typically, the distributed file subsystem 13 can be HDFS (Hadoop Distributed File System), which includes a master node and multiple data nodes. The master node is responsible for managing the metadata of the file system, such as the creation, deletion and modification of files and directories, and the data node is responsible for storing actual data. Fault tolerance can be achieved by storing multiple copies of data on multiple data nodes, and high throughput can be achieved through parallel processing.

[0039] In this embodiment, the distributed file subsystem 13 can use the current log collection vector as a classification directory and store the file log under the classification directory to achieve persistent storage of the file log. Specifically, if a classification directory corresponding to the current log collection vector already exists, the file log is directly stored under the classification directory; if a classification directory corresponding to the current log collection vector does not exist, the classification directory is first created, and then the file log is stored under the classification target.

[0040] The technical solution of the embodiment of the present invention is to obtain the collection information and file modification time corresponding to the file log to be collected through the data collector 11, and send the collection information and file modification time to the vector library container device 12, and when receiving the verification success information fed back by the vector library container device 12, collect the file log and send the file log to the distributed file subsystem 13; generate the current log collection vector according to the collection information through the vector library container device 12, and generate the current corresponding relationship between the current log collection vector and the file modification time, and when the existing corresponding relationship matching the current corresponding relationship is not detected, send the verification success information to the data collector 11, send the current log collection vector to the distributed file subsystem 13, and store the current corresponding relationship locally; store the file log according to the current log collection vector through the distributed file subsystem 13; generate the log collection vector according to the collection information of the file log through the newly added vector library container device 12, and perform duplicate file log detection based on the corresponding relationship between the log collection vector and the file modification time, so as to realize the non-duplicate collection of file logs, avoid the waste of system resources and reduce the storage cost, reduce the system performance consumption and shorten the time spent on log collection.

[0041] Embodiment 2

[0042] Figure 2A A flow chart of a log collection method is provided for the second embodiment of the present invention. This embodiment is applicable to the case of collecting file logs to remove duplicates. The method can be applied to the log collection system 10 described in the first embodiment and can be executed by a log collection device. The log collection device can be implemented in the form of hardware and / or software. Figure 2A As shown, the method includes:

[0043] S210: Acquire collection information and file modification time corresponding to the file log to be collected, and generate a current log collection vector according to the collection information.

[0044] Specifically, the log collection task can be parsed by the data collector 11 to obtain the collection information and file modification time corresponding to the file log to be collected; then, the collection information can be spliced, and based on the preset text conversion method, the current log collection vector corresponding to the spliced ​​collection information can be obtained. Among them, the log collection vector can be a one-dimensional array that can store multiple elements of the same data type.

[0045] The collected information may include a machine name, a file path, and a file name. Generating a current log collection vector according to the collected information may include:

[0046] Obtaining a first vector value corresponding to the machine name, a second vector value corresponding to the file path, and a third vector value corresponding to the file name;

[0047] The current log collection vector is generated according to the first vector value, the second vector value and the third vector value.

[0048] In a specific example, when generating a log collection vector, first, based on a preset text conversion method, vector values ​​corresponding to the machine name, file path, and file name can be obtained respectively; then, each vector value can be combined into a one-dimensional array to generate the current log collection vector. For example, the current log collection vector can be [0.23, 0.45, 0.56].

[0049] S220: Generate a current correspondence between the current log collection vector and the file modification time, and when no existing correspondence matching the current correspondence is detected, store the current correspondence in a vector library container device.

[0050] In this embodiment, it is possible to determine whether the current correspondence is the same as the existing correspondence, that is, the log collection vector and the file modification time are the same, to determine whether the two match; if no existing correspondence matching the current correspondence is detected, it means that the current file log is not a duplicate log, and the vector library container device 12 can feedback verification success information to the data collector 11 so that the data collector 11 can collect the file log, and at the same time, the current correspondence can be stored in the relationship set corresponding to the current log collection vector.

[0051] Optionally, after generating the current corresponding relationship between the current log collection vector and the file modification time, the method may further include: if it is detected that the current corresponding relationship matches an existing corresponding relationship, terminating the collection of the file log.

[0052] Specifically, if it is detected that the current correspondence matches an existing correspondence, it means that the current file log is a duplicate log, and the vector library container device 12 can feedback verification failure information to the data collector 11 so that the data collector 11 does not collect the file log.

[0053] S230: Collect the file log according to the collected information and the file modification time, and store the file log in a distributed file subsystem according to the current log collection vector.

[0054] Specifically, after the existing corresponding relationship matching the current corresponding relationship is not detected, the data collector 11 can collect the file log based on the collected information and the file modification time, and send the file log to the distributed file subsystem 13. After receiving the file log, the distributed file subsystem 13 can store the file log in the classification directory corresponding to the current log collection vector.

[0055] The technical solution of the embodiment of the present invention obtains the collection information and file modification time corresponding to the file log to be collected, and generates a current log collection vector according to the collection information; then, generates the current correspondence between the current log collection vector and the file modification time, and when no existing correspondence matching the current correspondence is detected, stores the current correspondence in a vector library container device 12; finally, collects the file log according to the collection information and the file modification time, and stores the file log in a distributed file subsystem 13 according to the current log collection vector; generates a log collection vector according to the collection information of the file log by adding a vector library container device 12, and performs duplicate file log detection based on the correspondence between the log collection vector and the file modification time, thereby achieving non-duplicate collection of file logs, avoiding waste of system resources and reducing storage costs, reducing system performance consumption and shortening log collection time.

[0056] In an optional implementation of this embodiment, when no stored corresponding relationship matching the current corresponding relationship is detected, storing the current corresponding relationship in the vector library container device 12 may include:

[0057] Obtaining stored vectors corresponding to each stored relationship set, and determining whether there is a target stored vector matching the current log collection vector;

[0058] If it is determined that there is no target stored vector that matches the current log collection vector, a new relationship set corresponding to the current log collection vector is established, and the current corresponding relationship is stored in the new relationship set.

[0059] In this embodiment, in the vector library container device 12, the stored relationship set corresponds to the stored vector one by one, that is, the stored corresponding relationships corresponding to the same stored vector constitute an stored relationship set. Therefore, when performing the matching detection of the current corresponding relationship, the stored vectors corresponding to each stored relationship set are first obtained, and it is determined whether there is a target stored vector matching the current log collection vector in each stored vector; if it is determined that there is no target stored vector matching the current log collection vector, it means that the current log collection vector appears for the first time, and a new relationship set corresponding to the current log collection vector can be newly created, and the current corresponding relationship is stored in the new relationship set.

[0060] Among them, when judging whether the current log collection vector matches the stored vector, it can be determined whether the array composition of the two is the same. If they are the same, it can be determined that the two match; if they are different, it can be determined that the two do not match; or, based on a preset vector similarity detection method, for example, a cosine similarity method, the similarity between the two vectors can be calculated. If the similarity is greater than or equal to a preset similarity threshold, it can be determined that the two match; otherwise, it can be determined that the two do not match.

[0061] Optionally, after determining whether there is a target stored vector matching the current log collection vector, the following steps may also be performed:

[0062] If it is determined that there is a target stored vector that matches the current log collection vector, then determining whether the file modification time of each stored corresponding relationship in the target stored relationship set corresponding to the target stored vector is the same as the file modification time corresponding to the current log collection vector;

[0063] If it is determined that the file modification time of each existing corresponding relationship in the target existing relationship set is different from the file modification time corresponding to the current log collection vector, the current corresponding relationship is stored in the target existing relationship set.

[0064] In another case, if a target stored vector matching the current log collection vector is detected, the target stored relationship set corresponding to the target stored vector can be determined; then, it can be determined whether the file modification time corresponding to the current log collection vector is the same as the file modification time of each stored corresponding relationship in the target stored relationship set; if it is detected that they are all different, it can be determined that there is no stored corresponding relationship matching the current corresponding relationship, and the current corresponding relationship can be stored in the target stored relationship set.

[0065] Optionally, if the file modification time of an existing corresponding relationship detected in the target existing relationship set is the same as the file modification time corresponding to the current log collection vector, it can be determined that the detected existing corresponding relationship matches the current corresponding relationship, that is, the current file log is a duplicate log and no collection is required.

[0066] In an optional implementation of this embodiment, storing the file log to the distributed file subsystem 13 according to the current log collection vector may include:

[0067] Determine whether there is a classification directory matching the current log collection vector;

[0068] If so, the file log is stored in the classification directory matched by the current log collection vector.

[0069] Specifically, the distributed file subsystem 13 can use the log collection vector as a storage classification directory. Therefore, after obtaining the current log collection vector, it can first determine whether there is a classification directory matching the current log collection vector; if it is determined that there is, the received file log can be directly stored in the classification directory.

[0070] Optionally, after determining whether there is a classification directory matching the current log collection vector, the following may also be included:

[0071] If not, a new category directory corresponding to the current log collection vector is established, and the file log is stored in the new category directory.

[0072] In another case, if it is determined that there is no classification directory matching the current log collection vector, it means that the current log collection vector appears for the first time, and a new classification directory corresponding to the current log collection vector can be created, and the file log is stored in the new classification directory.

[0073] In a specific implementation of this embodiment, the process of the log collection method can be as follows: Figure 2B As shown. Specifically, the data collector 11 obtains the collection information and file modification time corresponding to the file log to be collected, and sends the collection information and file modification time to the vector library container device 12; the vector library container device 12 generates the current log collection vector according to the collection information, and determines whether the current log collection vector already exists. If so, it further determines whether the file modification time is the same. If so, it sends a verification failure message to the data collector 11. When the data collector 11 receives the verification failure message, it does not collect the file log; if the current log collection vector does not already exist, or the file modification time is not the same, the vector library container device 12 can send a verification success message to the data collector 11, and at the same time send the current log collection vector to the distributed file subsystem 13, and generate a corresponding relationship between the current log collection vector and the file modification time, and store it in the corresponding relationship set corresponding to the current log collection vector.

[0074] When receiving the verification success information, the data collector 11 can collect the file log and send the file log to the distributed file subsystem 13. The distributed file subsystem 13 can store the file log in the classification directory corresponding to the current log collection vector.

[0075] In this embodiment, by converting the collected information into a log collection vector, compared with directly storing the machine name, file name and file path, less storage is used, and the search is faster. Moreover, by searching the single-value vector, it is possible to more quickly determine whether the log has been collected. Secondly, by using a collection form to store the existing correspondences with the same existing vectors, and by comparing the generated vectors and the file modification time to determine whether the log to be collected is repeated, compared with directly comparing all log files, the system performance consumption can be reduced, and the time spent on determining whether the log is repeated can be shortened.

[0076] Embodiment 3

[0077] Figure 3 This is a schematic diagram of the structure of a log collection device provided by Embodiment 3 of the present invention. Figure 3 As shown, the device can be applied to the log collection system 10 described in the first embodiment, and can include: a log collection vector generation module 310, a corresponding relationship storage module 320 and a file log storage module 330; wherein,

[0078] The log collection vector generation module 310 is used to obtain the collection information and file modification time corresponding to the file log to be collected, and generate the current log collection vector according to the collection information;

[0079] A correspondence storage module 320, configured to generate a current correspondence between the current log collection vector and the file modification time, and store the current correspondence in the vector library container device 12 when no stored correspondence matching the current correspondence is detected;

[0080] The file log storage module 330 is used to collect the file log according to the collection information and the file modification time, and store the file log in the distributed file subsystem 13 according to the current log collection vector.

[0081] The technical solution of the embodiment of the present invention obtains the collection information and file modification time corresponding to the file log to be collected, and generates a current log collection vector according to the collection information; then, generates the current correspondence between the current log collection vector and the file modification time, and when no existing correspondence matching the current correspondence is detected, stores the current correspondence in a vector library container device 12; finally, collects the file log according to the collection information and the file modification time, and stores the file log in a distributed file subsystem 13 according to the current log collection vector; generates a log collection vector according to the collection information of the file log by adding a vector library container device 12, and performs duplicate file log detection based on the correspondence between the log collection vector and the file modification time, thereby achieving non-duplicate collection of file logs, avoiding waste of system resources and reducing storage costs, reducing system performance consumption and shortening log collection time.

[0082] Optionally, the collected information includes a machine name, a file path, and a file name;

[0083] The log collection vector generation module 310 is specifically used to obtain a first vector value corresponding to the machine name, a second vector value corresponding to the file path, and a third vector value corresponding to the file name;

[0084] The current log collection vector is generated according to the first vector value, the second vector value and the third vector value.

[0085] Optionally, the corresponding relationship storage module 320 is specifically used to obtain the stored vectors corresponding to each stored relationship set, and determine whether there is a target stored vector matching the current log collection vector;

[0086] If it is determined that there is no target stored vector that matches the current log collection vector, a new relationship set corresponding to the current log collection vector is established, and the current corresponding relationship is stored in the new relationship set.

[0087] Optionally, the corresponding relationship storage module 320 is further used to determine whether the file modification time of each stored corresponding relationship in the target stored relationship set corresponding to the target stored vector is the same as the file modification time corresponding to the current log collection vector if it is determined that there is a target stored vector matching the current log collection vector;

[0088] If it is determined that the file modification time of each existing corresponding relationship in the target existing relationship set is different from the file modification time corresponding to the current log collection vector, the current corresponding relationship is stored in the target existing relationship set.

[0089] Optionally, the log collection device further includes:

[0090] The log collection termination module is used to terminate the collection of the file log if it is detected that the current corresponding relationship matches the existing corresponding relationship.

[0091] Optionally, the file log storage module 330 is specifically used to determine whether there is a classification directory matching the current log collection vector;

[0092] If so, the file log is stored in the classification directory matched by the current log collection vector.

[0093] Optionally, the file log storage module 330 is further configured to, if not, establish a new category directory corresponding to the current log collection vector, and store the file log in the new category directory.

[0094] The log collection device provided in the embodiment of the present invention can execute the log collection method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0095] Embodiment 4

[0096] In the present embodiment, the log collection method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium. In some embodiments, part or all of the computer program may be loaded and / or installed on an electronic device via a ROM and / or a communication unit. When the computer program is loaded into RAM and executed by a processor, one or more steps of the log collection method described above may be performed. Alternatively, in other embodiments, the processor may be configured to perform the log collection method by any other suitable means (e.g., by means of firmware).

[0097] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0098] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0099] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0100] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0101] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0102] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0103] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0104] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A log collection system, It is characterized in that It includes a data collector, a vector library container device and a distributed file subsystem; The data collector is respectively connected to the vector library container device and the distributed file subsystem for communication, and is used to obtain the collection information and file modification time corresponding to the file log to be collected, and send the collection information and the file modification time to the vector library container device, and when receiving the verification success information fed back by the vector library container device, collect the file log and send the file log to the distributed file subsystem; The vector library container device is communicatively connected with the distributed file subsystem, and is used for generating a current log collection vector according to the collection information, and generating a current correspondence between the current log collection vector and the file modification time, and when no existing correspondence matching the current correspondence is detected, sending the verification success information to the data collector, and sending the current log collection vector to the distributed file subsystem, the distributed file subsystem determines whether there is a classification directory matching the current log collection vector, and locally stores the current correspondence according to the determination result; The distributed file subsystem is used to determine whether there is a classification directory matching the current log collection vector according to the current log collection vector, and store the file log according to the determination result.

2. A log collection method, It is characterized in that The log collection system as claimed in claim 1 comprises: Obtaining collection information and file modification time corresponding to the file log to be collected, and generating a current log collection vector according to the collection information; generating a current correspondence between the current log collection vector and the file modification time, and storing the current correspondence in a vector library container device when no stored correspondence matching the current correspondence is detected; Collecting and obtaining the file log according to the collected information and the file modification time; Determine whether there is a classification directory matching the current log collection vector; If so, the file log is stored in the classification directory matched by the current log collection vector.

3. The method according to claim 2, It is characterized in that The collection information includes a machine name, a file path, and a file name. Generating a current log collection vector according to the collection information includes: Obtaining a first vector value corresponding to the machine name, a second vector value corresponding to the file path, and a third vector value corresponding to the file name; The current log collection vector is generated according to the first vector value, the second vector value and the third vector value.

4. The method according to claim 2, It is characterized in that When no stored corresponding relationship matching the current corresponding relationship is detected, storing the current corresponding relationship in a vector library container device includes: Obtaining stored vectors corresponding to each stored relationship set, and determining whether there is a target stored vector matching the current log collection vector; If it is determined that there is no target stored vector that matches the current log collection vector, a new relationship set corresponding to the current log collection vector is established, and the current corresponding relationship is stored in the new relationship set.

5. The method according to claim 4, It is characterized in that After determining whether there is a target stored vector matching the current log collection vector, the method further includes: If it is determined that there is a target stored vector that matches the current log collection vector, then determining whether the file modification time of each stored corresponding relationship in the target stored relationship set corresponding to the target stored vector is the same as the file modification time corresponding to the current log collection vector; If it is determined that the file modification time of each existing corresponding relationship in the target existing relationship set is different from the file modification time corresponding to the current log collection vector, the current corresponding relationship is stored in the target existing relationship set.

6. The method according to claim 2, It is characterized in that After generating the current corresponding relationship between the current log collection vector and the file modification time, the method further includes: If it is detected that the current corresponding relationship matches the existing corresponding relationship, the collection of the file log is terminated.

7. The method according to claim 2, It is characterized in that After determining whether there is a classification directory matching the current log collection vector, the method further includes: If not, a new category directory corresponding to the current log collection vector is established, and the file log is stored in the new category directory.

8. A log collection device, It is characterized in that The log collection system as claimed in claim 1 comprises: A log collection vector generation module is used to obtain the collection information and file modification time corresponding to the file log to be collected, and generate a current log collection vector according to the collection information; A correspondence storage module, used for generating a current correspondence between the current log collection vector and the file modification time, and storing the current correspondence in a vector library container device when no stored correspondence matching the current correspondence is detected; A file log collection module, used for collecting and obtaining the file log according to the collection information and the file modification time; A classification directory judgment module is used to judge whether there is a classification directory matching the current log collection vector; The file log storage module is used to store the file log in the classification directory matching the current log collection vector if there is a classification directory matching the current log collection vector.

9. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, and the computer program is used to enable a processor to implement the log collection method according to any one of claims 2 to 7 when executed.

Images