Log data collection method and apparatus, computer device, and readable storage medium

Abstract

The application relates to a log data collection method and device, computer equipment and a readable storage medium. The method comprises the following steps: determining the log source type of log data to be collected, wherein the log source type comprises a port data source or an addressing data source; if the log source type is the port data source, matching the log type of the log data to be collected with a first preset log type, and collecting first log data matched successfully; if the log source type is the addressing data source, determining preset position information of the log data to be collected, matching the log type of the log data to be collected with a second preset log type, and collecting second log data matched successfully based on the preset position information. The log data collection method provided by the application improves the availability of collected log data, reduces the data volume of collected log data, releases the operation resources of a big data processing system, and improves the data processing efficiency as a whole.

Description

Technical Field

[0001] The present application relates to the field of data collection technology, and in particular to a log data collection method, apparatus, computer equipment, and readable storage medium. Background Art

[0002] Log data is data or data sets that record system operational events and express them according to certain specifications. By collecting and analyzing log data, system behavior can be analyzed, and performance can be optimized or adjusted. In the field of computer security, log data is one of the primary sources for security audits. By collecting log data, network security behavior can be audited and documented.

[0003] In traditional technology, log data is mainly collected passively. By setting up a passive log data collector in the device, the collected log data is uploaded to the big data processing system without distinction and in full. After being stored in the file system, it is processed by the big data processing system.

[0004] However, as the amount of data that needs to be collected increases, the passive log data collection method will transmit a large amount of useless data to the big data processing system, which undoubtedly increases the burden on the big data processing system, occupies the computing resources of the processing system, and affects the efficiency of data processing. Summary of the Invention

[0005] Based on this, it is necessary to provide a log data collection method, device, computer equipment and readable storage medium that can improve the availability of log data in response to the above technical problems.

[0006] In a first aspect, the present application provides a log data collection method. The method comprises:

[0007] Determine the log source type of the log data to be collected, where the log source type includes a port data source or an addressing data source;

[0008] If the log source type is a port data source, matching the log type of the log data to be collected with a first preset log type, and collecting first log data that successfully matches;

[0009] If the log source type is an addressable data source, the preset location information of the log data to be collected is determined, and the log type of the log data to be collected is matched with a second preset log type, and the second log data that is successfully matched is collected based on the preset location information.

[0010] In one embodiment, determining the preset location information of the log data to be collected includes:

[0011] Determine preset storage information of the log data to be collected in a database, and / or determine preset path information of the log data to be collected in a shared file.

[0012] In one embodiment, determining the preset storage information of the log data to be collected in the database includes:

[0013] Determine the preset name information, preset address information, and preset login information of the database where the log data to be collected is located, and determine the preset name information, preset device information, and preset coding information of the log table where the log data to be collected is located.

[0014] In one embodiment, determining the preset path information of the log data to be collected in the shared file includes:

[0015] Determine the preset FTP address information and preset FTP login information of the shared file where the log data to be collected is located, and determine the preset path information, preset write type information, preset device information, and preset encoding information of the FTP where the log data to be collected is located.

[0016] In one embodiment, before matching the log type with the first preset log type, the process further includes:

[0017] Determine the preset device information, at least one preset port information and encoding information of each preset port of the log data to be collected.

[0018] In one embodiment, collecting the first log data that successfully matches includes:

[0019] The first log data that matches successfully is collected based on the preset device information, the at least one preset port information and the encoding information of each preset port.

[0020] In one embodiment, the log data collection method further includes:

[0021] caching the first log data and / or the second log data as data to be parsed;

[0022] The data to be parsed is parsed according to preset parsing rules and sent to the data processing system.

[0023] In a second aspect, the present application further provides a log data collection device. The device comprises:

[0024] A data type determination module is used to determine the log source type of the log data to be collected, wherein the log source type includes a port data source or an addressing data source;

[0025] A first log collection module is configured to match the log type of the log data to be collected with a first preset log type, and collect first log data that successfully matches, wherein the log source type is a port data source;

[0026] The second log collection module is used to determine the preset location information of the log data to be collected, and match the log type of the log data to be collected with a second preset log type, and collect the second log data that successfully matches based on the preset location information, wherein the log source type is an addressable data source.

[0027] In a third aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any one of the methods described in the first aspect when executing the computer program.

[0028] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any one of the methods described in the first aspect.

[0029] The above-mentioned log data collection method, device, computer equipment and readable storage medium, by determining the log source type of the log data to be collected, the log source type includes a port data source or an addressing data source; if the log source type is a port data source, the log type of the log data to be collected is matched with a first preset log type, and the first log data that successfully matches is collected; if the log source type is an addressing data source, the preset location information of the log data to be collected is determined, and the log type of the log data to be collected is matched with a second preset log type, and the second log data that successfully matches is collected based on the preset location information. The log data collection method provided in the embodiment of the present application can determine the log data to be collected according to actual needs, distinguish the log data to be collected according to the log source type, and then collect the log data based on the preset log type and / or location information according to the result of the distinction. Log data can be collected purposefully, effectively improving the availability of the collected log data, reducing the amount of collected log data, sharing the data processing work that should have been performed by the big data processing system, releasing the computing resources of the big data processing system, and improving the efficiency of data processing as a whole.

[0030] The details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0032] Figure 1 A diagram showing an application environment of a log data collection method in one embodiment;

[0033] Figure 2 This is a flow chart of a log data collection method in traditional technology;

[0034] Figure 3 Schematic diagram of a flow chart of a log data collection method in one embodiment;

[0035] Figure 4 A flowchart of a method for determining preset storage information of log data to be collected in a database in a specific embodiment;

[0036] Figure 5 Schematic diagram of a flow chart of a method for determining preset path information of log data to be collected in a shared file in a specific embodiment;

[0037] Figure 6 A schematic flow chart of a method for collecting first log data in a specific embodiment;

[0038] Figure 7 Schematic diagram of a flow chart of a method for collecting and processing log data in one embodiment;

[0039] Figure 8 A flow chart of a method for collecting and processing log data in another embodiment;

[0040] Figure 9 FIG. 4 is a structural block diagram of a log data collection device in an embodiment. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0042] Unless otherwise defined, the technical terms or scientific terms involved in this application should have the general meaning understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "an", "a", "the", "these" and the like in this application do not indicate quantitative restrictions, and they can be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "plurality" involved in this application refers to two or more. "And / or" describes the relationship between related objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, and B exists alone. Generally, the character " / " indicates that the related objects are in an "or" relationship. The terms "first," "second," "third," etc. used in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0043] The terms "module", "unit", etc. used below refer to a combination of software and / or hardware that can implement a predetermined function. Although the devices described in the following embodiments are preferably implemented in hardware, implementation using software or a combination of software and hardware is also possible and contemplated.

[0044] The log data collection method provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The terminal 102 or the server 104 determines the log source type of the log data to be collected, and the log source type includes a port data source or an addressing data source. If it is determined that the log source type is a port data source, the terminal 102 or the server 104 matches the log type of the log data to be collected with the first preset log type, and collects the first log data that successfully matches; if the log source type is an addressing data source, the terminal 102 or the server 104 determines the preset location information of the log data to be collected, and matches the log type of the log data to be collected with the second preset log type, and collects the second log data that successfully matches based on the preset location information. Among them, the server 104 can be implemented as an independent server or a server cluster consisting of multiple servers.

[0045] Log collection and data processing methods in traditional technologies are as follows: Figure 2 As shown, a collector passively collects log data from devices and caches all of it in Kafka before sending it to a big data processing system. Kafka is a high-throughput, persistent distributed message queue used to transmit the log data collected by the collector. After receiving large amounts of log data, the big data system filters, analyzes, or processes it. Traditionally, big data systems have to select and store large amounts of log data in a database. This data screening and filtering process increases the computing pressure of the big data system. When the data volume is large, this not only affects the data processing efficiency of the big data system but can also lead to data processing errors. Furthermore, the data transmitted by the passive collector to the big data processing system via Kafka may contain a large amount of low-availability or useless data. This filtering and screening of this data also forces the big data processing system to devote more resources to less efficient tasks, reducing overall data processing efficiency.

[0046] Based on this, in one embodiment, Figure 3 As shown, a log data collection method is provided, which is applied to Figure 1 The application environment in the example is used to illustrate the following steps:

[0047] S201: Determine the log source type of the log data to be collected, where the log source type includes a port data source or an addressing data source.

[0048] In embodiments of the present application, the log data may include system-generated procedural event records. For example, a user performing a specific operation on a device at a certain time will be recorded and saved as log data. Therefore, collecting and analyzing log data can reveal device or system information, the timing of historical events, event process information, or operation information, which is crucial for analyzing device or system usage, user behavior, or network security incidents.

[0049] In an embodiment of the present application, the log source is the source of log data, which can be a method for collecting log data. Based on different log collection methods, the log source type includes a port data source or an addressing data source. Among them, the port is the entrance and exit for the device where the log data to be collected is located to generate data communication with the outside world. When data flows into or out of the device, it will pass through the port, and then the port data source may include a data source that flows through the port of the device where the log data to be collected is located. It can be understood that the log data flowing through the port can be collected by monitoring the port of the device. On the other hand, if the preset location information of the log data to be collected can be determined, the log data can also be collected by addressing. The addressing data source includes a data source whose storage location of the log data to be collected can be determined based on the preset location information, wherein the preset location information can include the preset storage information of the log data to be collected in the database or the preset path information in the shared file.

[0050] In an embodiment of the present application, the method of determining the log source type of the log data to be collected may include judging whether the log source type of the log data to be collected is a port data source, and if not, determining that the log source type is an addressing data source. The method of determining the log source type of the log data to be collected may also include judging whether the log source type of the log data to be collected is an addressing data source, and if not, determining that the log source type is a port data source. Of course, it may also include directly judging whether the log source type of the log data to be collected is a port data source or an addressing data source. The present application does not limit the method of determining the log source type of the log data to be collected.

[0051] S203: If the log source type is a port data source, the log type of the log data to be collected is matched with a first preset log type, and first log data that is successfully matched is collected.

[0052] In an embodiment of the present application, based on the description of the log data in the above step S201, the log type may include user information data, behavioral event data, system event data, etc. The first preset log type is the target log type that needs to be collected, and can be set accordingly according to actual needs. If the log source type of the log data to be collected is a port data source, then the log data to be collected is the log data flowing into and / or out of the port, and the log type of the data to be collected is matched with the first preset log type. The log data to be collected that is successfully matched is collected as the first log data, and the log data to be collected that is not successfully matched is filtered out and not collected. It can be understood that for the device where the log data to be collected is located, there may be one or more ports that generate data communication with the outside world. When collecting log data, a collector can be set for any one or more ports to collect log data.

[0053] In an embodiment of the present application, if the log source type is determined to be a port data source, a first preset log type is set to collect log data that matches the first preset log type, and non-target log data that does not successfully match is filtered out. On the one hand, the target log data required can be collected according to actual needs. On the other hand, the volume of collected log data can be effectively reduced, and the usability of the collected log data can be increased. This can alleviate the computing pressure on the big data processing system after data transmission, and overall improve the processing efficiency of log data.

[0054] S205: If the log source type of the log data to be collected is an addressable data source, determine the preset location information of the log data to be collected, match the log type of the log data to be collected with a second preset log type, and collect the second log data that is successfully matched based on the preset location information.

[0055] In an embodiment of the present application, if it is determined that the log source type is an addressable data source, the log data to be collected may include log data stored in a log database or log data shared via FTP (File Transfer Protocol), and it is necessary to determine the preset location information of the log data to be collected. The determination of the preset location information of the log data to be collected includes determining the preset storage information of the log data to be collected in the database, and / or determining the preset path information of the log data to be collected in the shared file. The log type of the log data to be collected is matched with the second preset log type, and the second log data that is successfully matched is collected based on the preset location information. It can be understood that the second preset log type and the first preset log type can be the same or different, and the second preset log type and the first preset log type can be set accordingly based on actual needs, and the present application does not impose any restrictions on this.

[0056] In an embodiment of the present application, based on the second log data that is successfully collected and matched based on the preset location information, the required target log data can be actively obtained through addressing, and the log type of the log data to be collected can be matched with the second preset log type. It can also further filter out low-utilization or non-target log data, making the collection of log data more accurate and efficient, and further improving the availability of the collected log data and the processing efficiency of the log data.

[0057] The log data collection method provided in the embodiment of the present application can determine the log data to be collected according to actual needs, distinguish the log data to be collected according to the log source type, and then collect the log data based on the preset log type and / or location information according to the result of the distinction. Log data can be collected purposefully, which effectively improves the availability of the collected log data, reduces the amount of collected log data, shares the data processing work that should have been performed by the big data processing system, releases the computing resources of the big data processing system, and improves the efficiency of data processing as a whole.

[0058] In the embodiment of the present application, in step S203, determining the preset location information of the log data to be collected includes:

[0059] S301: Determine preset storage information of the log data to be collected in a database, and / or determine preset path information of the log data to be collected in a shared file.

[0060] In some embodiments, determining the preset storage information of the log data to be collected in the database in step S301 includes:

[0061] S3011: Determine the preset name information, preset address information, and preset login information of the database where the log data to be collected is located, and determine the preset name information, preset device information, and preset coding information of the log table where the log data to be collected is located.

[0062] In an embodiment of the present application, the database includes a database for storing log data. The preset name information may include the name information and / or database type information of the database where the log data to be collected is located, the preset address information may include the IP address information of the database, and the preset login information may include the login account information and login password information of the database. Furthermore, it is necessary to determine the information of the log table in the database where the log data to be collected is located. The preset name information of the log table where the log data to be collected is located may include the log table's display information and primary key name information, the preset device information may include the device type information, device manufacturer information, and device model information set for one or more log tables where the log data is located, and the preset encoding information may include encoding method information and encoding type information set for one or more log tables. It is understandable that in an embodiment of the present application, the log data to be collected may come from one or more databases, or from one or more log tables in a database. Therefore, the preset storage information may be one or more, and the present application does not specifically limit the number of preset storage information. In other embodiments, determining the preset storage information of the log data to be collected in the database may also include determining a first preset port number of the device where the log data to be collected is located, and determining the first active collector used to collect the log data and the collector's address information. Among them, the first collector can include log data collectors such as Flume and Chukwa.

[0063] In the embodiment of this application, Figure 4 The method for determining the preset storage information of the log data to be collected in a database in a specific embodiment is shown. After determining that the log source type is an addressable data source, the database name, database type, database IP address, database login account and password, and other information to be monitored for active addressing collection are set. The port number to be monitored for active addressing collection is set. The collector and its IP address information are selected. The table name and primary key name of the log table where the log data to be collected are located are set. The device type, manufacturer, and device model of the log data are set for each log table. The encoding method is set for each log table. A preset log type is set for each log table. The log type of the log data to be collected is matched with the preset log type. The log data that successfully matches is collected, and the log data that does not successfully match is filtered out.

[0064] In some embodiments, determining the preset path information of the log data to be collected in the shared file in step S301 includes:

[0065] S3012: Determine the preset FTP address information and preset FTP login information of the shared file where the log data to be collected is located, and determine the preset path information, preset write type information, preset device information, and preset encoding information of the FTP where the log data to be collected is located.

[0066] In an embodiment of the present application, the preset FTP login information may include FTP login account information and login password information of the log data to be collected, the FTP preset path information may include storage path information of the log data to be collected in a shared file, the preset device information may include device type information, device manufacturer information and device model information of the log data set for one or more preset path information, and the preset coding information may include coding method information and coding type information set for one or more preset path information.

[0067] It is understandable that the log data to be collected in the embodiment of the present application may come from one or more log files shared via FTP, so one or more preset path information may be set, and the present application does not impose any specific restrictions on the number of preset path information. In other embodiments, the determination of the preset path information of the log data to be collected in the shared file may also include determining a second preset port number of the device where the log data to be collected is located, and determining a second active collector for collecting log data and the address information of the collector. The selection of the second active collector may be the same as or different from the above-mentioned first active collector, and the present application does not impose any restrictions on this.

[0068] In the embodiment of this application, Figure 5 FIG. 1 is a method for determining the preset path information of the log data to be collected in a shared file in a specific embodiment.

[0069] After determining that the log source type is an addressable data source, set up active addressing to collect the FTP address and port number of the shared file log source to be monitored. Set up active addressing to collect the FTP username and password of the shared file to be monitored. Set up the active collector and its IP address. Set up active addressing to collect and monitor multiple preset paths, write type, and encoding methods for FTP. Set the encoding method for each preset path. Set the device type, manufacturer, and model of the log data for each preset path. Set a preset log type for each preset path. Match the log type of the log data to be collected with the preset log type. Collect the log data that matches successfully and filter out the log data that does not match successfully.

[0070] In an embodiment of the present application, if the log source type is an addressable data source, by determining the preset location information of the log data to be collected, and collecting the second log data that successfully matches the second preset log type based on the preset location information, the target log data can be accurately collected from the database and shared files. By setting the second preset log type, low-utilization or non-target log data is filtered out, thereby further improving the availability of the collected log data.

[0071] In an embodiment of the present application, if the log source type is a port data source, before matching the log type with the first preset log type, the method further includes:

[0072] S401: Determine preset device information, at least one preset port information, and encoding information of each preset port for the log data to be collected.

[0073] In the embodiment of the present application, after step S401, the first log data successfully collected and matched includes:

[0074] S501: Collecting successfully matched first log data based on the preset device information, at least one preset port information and encoding information of each preset port.

[0075] In an embodiment of the present application, the preset device information of the log data to be collected may include the address information of the device where the log data to be collected is located, the device type information, and the device manufacturer and model information. The at least one preset port information may include the port information of the log data flowing in or out of the device where the log data to be collected is located. The first log data that is successfully matched based on the preset device information, the at least one preset port information and the coding information of each preset port may include matching the type of log data to be collected from the preset port on the preset device with the preset coding information with the first preset log type, and collecting the first log data that is successfully matched.

[0076] In the embodiment of this application, Figure 6 The method for collecting successfully matched first log data in a specific embodiment is shown. After determining that the log source type of the data to be collected is a port data source, the IP address, device type, manufacturer, and model of the device to be monitored, where the log data to be collected resides, are set. A passive collector and its IP address are selected, multiple preset port information is set, and a corresponding encoding method is set for each monitored port. A preset log type is set for each preset port information, and the log type of the log data to be collected is matched with the preset log type. Successfully matched log data is collected, and unmatched log data is filtered out.

[0077] In order to further release the computing resources of the big data processing system, the embodiment of the present application also provides a method for parsing and caching collected log data. In the embodiment of the present application, the log data collection method further includes:

[0078] S601: Cache the first log data and / or the second log data as data to be parsed.

[0079] S603: Parse the data to be parsed according to preset parsing rules and send the data to a data processing system.

[0080] In an embodiment of the present application, after collecting the first log data and / or the second log data, the collected log data is cached as data to be parsed. The data to be parsed is parsed according to preset parsing rules and sent to a data processing system. The preset parsing rules may include at least one of a preset format rule, a preset data type rule, and a preset data structure rule. In an embodiment of the present application, the caching and parsing functions of log data can be implemented through data caching applications such as redis and memcached.

[0081] In an embodiment of the present application, the first log data and / or the second log data are cached and parsed according to preset parsing rules before being sent to the big data processing system. On the one hand, this alleviates the data cache pressure of the big data processing system. On the other hand, the task of parsing log data is separated from the big data processing system, further alleviating the computing pressure of the big data processing system, releasing computing resources, and further improving the overall efficiency of data processing.

[0082] In a specific embodiment of the present application, Figure 7The diagram provides a flow chart of log data collection and processing. A passive collector monitors the log data to be collected in devices 1 and 2, collecting first log data of a port data source type that meets a first preset log type. An active collector monitors the log data to be collected in the database of device 3 and the shared file of device 4, and collects second log data of an addressable data source type that meets a second preset log type. The collected first and / or second log data are sent to a cache application, Redis. Redis caches the collected log data and parses it according to preset parsing rules. The available log data is then uploaded to a big data cluster, Storm, and HDFS via Kafka. Storm is a fault-tolerant, distributed, and horizontally scalable big data framework that can perform computations on the collected log data. HDFS is a distributed file system for Hadoop (Hadoop Distributed File System), a highly fault-tolerant, high-throughput, distributed file system that can store processed log data. The big data cluster can calculate traffic data, user behavior analysis, security event analysis, and other data based on the parsed log data, and store the calculated data in a database (DB).

[0083] The following is a specific example to illustrate the process of log data collection and log data processing in this application. Figure 8 As shown, it is determined whether the data source type of the log data to be collected is a port data source (i.e., whether it is passive collection). If so, the first log data matching the first preset log type is collected by the passive collector and then reported; if not, it is further determined whether the log source is a database log source. If so, the database log data is collected. If not, the FTP shared file log data is collected, and the collected second log data matching the second preset log type is reported. The collected log data is stored in the cache application redis, and it is determined whether the data to be parsed needs to be parsed according to the preset parsing rules. If not, the process is terminated; if necessary, the log data is parsed according to the preset parsing rules, and the parsed log data is transmitted to the storm+HDFS big data cluster through kafka. The big data cluster calculates the log data and stores the calculation results in the database and then terminates the process.

[0084] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0085] Based on the same inventive concept, the present application also provides a log data collection device 700 for implementing the aforementioned log data collection method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations of one or more log data collection device embodiments provided below can be found in the above-mentioned limitations of the log data collection method and will not be repeated here.

[0086] In one embodiment, Figure 9 As shown, a log data collection device 700 is provided, comprising:

[0087] The data type determination module 701 is used to determine the log source type of the log data to be collected, wherein the log source type includes a port data source or an addressing data source;

[0088] A first log collection module 702 is configured to match the log type of the log data to be collected with a first preset log type, and collect first log data that successfully matches, wherein the log source type is a port data source;

[0089] The second log collection module 703 is used to determine the preset location information of the log data to be collected, and match the log type of the log data to be collected with the second preset log type, and collect the second log data that successfully matches based on the preset location information, wherein the log source type is an addressable data source.

[0090] Each module in the log data collection device 700 can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0091] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps of any of the above-mentioned log data collection methods when executing the computer program.

[0092] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned log data collection methods are implemented.

[0093] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0094] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, and the like.

[0095] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0096] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A log data collection method, characterized in that: The method comprises: Determine the log source type of the log data to be collected, where the log source type includes a port data source or an addressing data source; If the log source type is a port data source, matching the log type of the log data to be collected with a first preset log type, and collecting first log data that successfully matches; If the log source type is an addressable data source, determining the preset location information of the log data to be collected, matching the log type of the log data to be collected with a second preset log type, and collecting the second log data that successfully matches based on the preset location information; Determining the preset location information of the log data to be collected includes: Determine preset storage information of the log data to be collected in a database, and / or determine preset path information of the log data to be collected in a shared file.

2. The method according to claim 1, characterized in that The step of determining the preset storage information of the log data to be collected in the database includes: Determine the preset name information, preset address information, and preset login information of the database where the log data to be collected is located, and determine the preset name information, preset device information, and preset coding information of the log table where the log data to be collected is located.

3. The method according to claim 1, characterized in that Determining the preset path information of the log data to be collected in the shared file includes: Determine the preset FTP address information and preset FTP login information of the shared file where the log data to be collected is located, and determine the preset path information, preset write type information, preset device information, and preset encoding information of the FTP where the log data to be collected is located.

4. The method according to claim 1, wherein Before matching the log type with the first preset log type, the method further includes: Determine the preset device information, at least one preset port information and encoding information of each preset port of the log data to be collected.

5. The method according to claim 4, characterized in that The first log data successfully collected and matched includes: The first log data that matches successfully is collected based on the preset device information, the at least one preset port information and the encoding information of each preset port.

6. The method according to any one of claims 1 to 5, characterized in that Also includes: caching the first log data and / or the second log data as data to be parsed; The data to be parsed is parsed according to preset parsing rules and sent to the data processing system.

7. A log data collection device, characterized in that: The device comprises: A data type determination module is used to determine the log source type of the log data to be collected, wherein the log source type includes a port data source or an addressing data source; A first log collection module is configured to match the log type of the log data to be collected with a first preset log type, and collect first log data that successfully matches, wherein the log source type is a port data source; A second log collection module is configured to determine preset location information of the log data to be collected, match the log type of the log data to be collected with a second preset log type, and collect the second log data that successfully matches the preset location information, wherein the log source type is an addressable data source; Determining the preset location information of the log data to be collected includes: Determine preset storage information of the log data to be collected in a database, and / or determine preset path information of the log data to be collected in a shared file.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images