Log file uploading method and device and electronic equipment

Abstract

The embodiment of the invention discloses a log file uploading method and device and electronic equipment, and the method comprises the steps: uploading a target traffic log file to at least one target log storage node of a local service platform and / or a remote service platform according to the target uploading type of the target traffic log file when the target traffic log file is monitored; when it is detected that the target log storage node feeds back a confirmation file, progress record information corresponding to the target flow log file in an uploading progress record table is updated; and when the progress record information corresponding to the target flow log file in the uploading progress record table meets the preset condition, the uploading process of the target flow log file is completed, and the flow log file can be automatically uploaded to one or more corresponding log storage nodes according to the uploading type of the flow log file, so that the user experience is improved. The situations of missing transmission and retransmission of the log files are avoided, and the real-time performance and the uploading efficiency when different types of flow log files are uploaded to different platform servers are improved on the whole.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of computer information processing, and in particular to a log file uploading method, device and electronic device. Background Art

[0002] The use of the Internet is becoming increasingly popular. When users carry out various life, work and entertainment activities based on the Internet, a large amount of mobile network data will be generated, especially 5G mobile network data. In order to better manage these mobile network data, it is necessary to parse these mobile network data and generate traffic log files. These traffic log files need to be uploaded to multiple servers on multiple different platforms for backup storage, and different log files need to be uploaded to different platforms.

[0003] At present, log files are usually manually uploaded to a designated server through existing file transfer and network communication tools. When uploading different types of traffic log files to different platform servers, there are technical problems such as low upload efficiency, poor real-time performance, and missed transmission and retransmission. Summary of the invention

[0004] The present invention provides a log file uploading method, device and electronic device, which automatically upload traffic log files to corresponding one or more log storage nodes according to their upload types, and avoid the situation where log files are missed or retransmitted by confirming the progress record information in the file and upload progress record table, thereby improving the real-time performance and upload efficiency when uploading different types of traffic log files to different platform servers as a whole.

[0005] In a first aspect, an embodiment of the present invention provides a log file uploading method, the method comprising:

[0006] When a new target traffic log file is monitored in the file monitoring directory, the target traffic log file is uploaded to at least one target log storage node of at least one target service platform according to the target upload type corresponding to the target traffic log file; wherein the at least one target service platform includes at least one of a local service platform and a remote service platform;

[0007] When it is detected that the at least one target log storage node feeds back a confirmation file corresponding to the target traffic log file, updating the progress record information corresponding to the target traffic log file in the upload progress record table;

[0008] When the progress record information corresponding to the target traffic log file in the upload progress record table meets a preset condition, the upload process of the target traffic log file is completed.

[0009] In a second aspect, an embodiment of the present invention further provides a log file uploading device, the device comprising:

[0010] A log file uploading module, configured to upload a target traffic log file to at least one target log storage node of at least one target service platform according to a target upload type corresponding to the target traffic log file when a target traffic log file is newly added to the file monitoring directory; wherein the at least one target service platform includes at least one of a local service platform and a remote service platform;

[0011] A record information updating module, configured to update the progress record information corresponding to the target traffic log file in the upload progress record table when it is detected that the at least one target log storage node feeds back a confirmation file corresponding to the target traffic log file;

[0012] The upload completion determination module is used to complete the upload process of the target traffic log file when the progress record information corresponding to the target traffic log file in the upload progress record table meets a preset condition.

[0013] In a third aspect, an embodiment of the present invention further provides an electronic device, the electronic device comprising:

[0014] one or more processors;

[0015] A storage device is used to store one or more programs. When the one or more programs are executed by one or more processors, the one or more processors implement a log file uploading method as described in any of the embodiments of the present invention.

[0016] In a fourth aspect, an embodiment of the present invention further provides a storage medium comprising computer executable instructions, which, when executed by a computer processor, are used to execute a log file uploading method as described in any one of the embodiments of the present invention.

[0017] The technical solution of the embodiment of the present invention is that when a new target traffic log file is monitored in the file monitoring directory, the target traffic log file is uploaded to at least one target log storage node of at least one target service platform according to the target upload type corresponding to the target traffic log file; wherein the at least one target service platform includes at least one of a local service platform and a remote service platform; when it is detected that at least one target log storage node feeds back a confirmation file corresponding to the target traffic log file, the progress record information corresponding to the target traffic log file in the upload progress record table is updated; when the progress record information corresponding to the target traffic log file in the upload progress record table meets the preset conditions, the upload process of the target traffic log file is completed. The technical solution of this embodiment can monitor the traffic log file in real time, and when a new traffic log file is monitored, the upload task can be automatically completed according to the upload type of the traffic log file, and it can be uploaded to the corresponding one or more log storage nodes, and the situation of log file leakage and retransmission is avoided by confirming the file and the progress record information in the upload progress record table, which improves the real-time performance and upload efficiency when uploading different types of traffic log files to different platform servers as a whole. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the following briefly introduces the drawings required to describe the embodiments. Obviously, the drawings introduced are only drawings of a part of the embodiments to be described in the present invention, rather than all the drawings. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 A schematic diagram of the upload flow of a traffic log file involved in an embodiment of the present invention;

[0020] Figure 2 A flow chart of a log file uploading method provided by an embodiment of the present invention;

[0021] Figure 3 A file transfer flow diagram of a log retention system according to an embodiment of the present invention;

[0022] Figure 4 A flow chart of a log file uploading method provided by an embodiment of the present invention;

[0023] Figure 5 A schematic diagram of a priority uploading method according to an embodiment of the present invention;

[0024] Figure 6 A flow chart of a log file uploading method provided by an embodiment of the present invention;

[0025] Figure 7 A flow chart of a log file uploading method provided by an embodiment of the present invention;

[0026] Figure 8 A flow chart of a log file uploading method provided by an embodiment of the present invention;

[0027] Fig. 9 A structural schematic diagram of a log file uploading device provided by an embodiment of the present invention;

[0028] Fig.10 The present invention is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0030] Before introducing the technical solution, the application scenario can be described first. The technical solution provided by the embodiment of the present invention can be applied to any scenario where different types of traffic log files need to be uploaded to different platform servers. For example, the upload flow diagram of the traffic log file can be seen in Figure 1 ,like Figure 1 As shown, for a large number of traffic log files generated daily, a part of the traffic log files (for example, traffic log file 1) only needs to be uploaded to any server end in the first platform, a part of the traffic log files only needs to be uploaded to any server end in the second platform (for example, traffic log file 2), and another part of the traffic log files (for example, traffic log file 3) need to be uploaded to any server in the first platform and also need to be uploaded to any server in the second platform. With respect to the uploading process of the above-mentioned log files, the current method of manually uploading the log files to the designated server through the existing file transfer and network communication tools has the technical problems of low log uploading efficiency and poor real-time performance. With respect to the above-mentioned problems, the embodiments of the present invention aim to provide a method for automatically uploading the traffic log files to the corresponding server end in real time, so as to improve the real-time performance and efficiency of uploading the log files.

[0031] Embodiment 1

[0032] Figure 2A flow chart of a log file uploading method provided in an embodiment of the present invention. This embodiment can be applied to any situation where different types of traffic log files need to be uploaded to different platform servers. The method can be executed by a log file uploading device, which can be implemented in the form of software and / or hardware. The hardware can be an electronic device, such as a mobile terminal, a PC or a server.

[0033] like Figure 2 As shown, the log file upload method includes:

[0034] S110. When a new target traffic log file is detected in the file monitoring directory, the target traffic log file is uploaded to at least one target log storage node of at least one target service platform according to the target upload type corresponding to the target traffic log file.

[0035] In this embodiment, the traffic log file needs to be uploaded to at least one log storage node of at least one service platform, and the log storage node refers to the server side belonging to the service platform. At least one target service platform includes at least one of a local service platform and a remote service platform. The local service platform refers to a service platform that is close to the device space distance from which the traffic log file is obtained. The remote service platform refers to a service platform that is far away from the device space from which the traffic log file is obtained. In practical applications, among the large number of traffic log files generated daily, a part of the traffic log files need to be uploaded to the log storage node of the local service platform, a part of the traffic log files need to be uploaded to the log storage node of the remote service platform, and another part of the traffic log files need to be uploaded to the log storage nodes of the local service platform and the remote service platform. Based on this, a plurality of upload types are pre-set, and the plurality of upload types include a single platform local upload type, a single platform remote upload type, or a multi-platform mixed upload type. The target upload type is one of the pre-set multiple upload types, and different upload types correspond to different upload destinations. The target upload type corresponding to any traffic log file can be pre-configured.

[0036] Among them, the file monitoring directory refers to a folder used to receive and monitor newly generated traffic log files. For example, the file monitoring directory can be a folder that uses a file system event monitoring mechanism. The file monitoring directory can monitor changes in the file system in real time, and when a specific event occurs in a file or directory, the downstream program can be notified immediately. For example, when a file is created, modified, deleted, moved, or its attributes are changed, the file monitoring directory can trigger a corresponding notification at the moment the event occurs, allowing downstream programs to respond in a timely manner. The file monitoring directory can be used to achieve real-time monitoring of traffic log files, discover and upload files in real time, and reduce system data transmission delays.

[0037] In this embodiment, a file monitoring directory can be pre-built. When building the file monitoring directory, if a certain type of traffic log file needs to be monitored, a monitoring event for monitoring such traffic log file can be added to the file monitoring directory. For example, a first monitoring event for monitoring a single-platform local upload type traffic log file, a second monitoring event for monitoring a single-platform remote upload type traffic log file, and a third monitoring event for monitoring a multi-platform mixed upload type traffic log file can be pre-configured for a certain file monitoring directory. It can be understood that the user can also delete any monitoring event according to actual needs. In addition, multiple file monitoring directories can be pre-built, for example, a first file monitoring directory for monitoring a first monitoring event can be built, wherein the first monitoring event corresponds to a single-platform local upload type traffic log file; a second file monitoring directory for monitoring a second monitoring event can be built, wherein the second monitoring event corresponds to a single-platform remote upload type traffic log file; and a third file monitoring directory for monitoring a third monitoring event can be built, wherein the third monitoring event corresponds to a multi-platform mixed upload type traffic log file.

[0038] The target traffic log file is the log file to be uploaded for processing. The target service platform refers to the service platform to which the target traffic log file is ultimately uploaded. Specifically, the target service platform may be a local service platform, the target service platform may be a remote service platform, or the target service platform may be both a local service platform and a remote service platform. The target log storage node refers to the log storage node to which the target traffic log file is ultimately uploaded.

[0039] In this embodiment, the method of generating the target traffic log file can be described. Optionally, when a network traffic data packet is received, the network traffic data packet is parsed and associated data is backfilled to obtain the target traffic log file; the target traffic log file is added to the file monitoring directory.

[0040] Specifically, the technical solution provided by the embodiment of the present invention can be applied to the log retention system. The file transmission flow diagram of the log retention system is shown in Figure 3 ,like Figure 3As shown, the log retention system can be a 5G log retention system, which includes multiple deep packet inspection devices (Deep Packet Inspection, DPI), multiple synthetic devices and multiple server ends on multiple different platforms. Among them, the DPI device is the underlying device in the 5G log retention system, and its function is to parse a certain 5G network traffic data packet and output a data record file (eXtensible Data Recording, XDR call single file); the function of the synthetic device is to associate and backfill, such as backfilling user information and location information to the XDR call single file, and then a traffic log file corresponding to the 5G network traffic data packet can be generated, that is, a target traffic log file. The 5G log retention system includes multiple DPI devices and multiple synthetic devices. The relationship between the DPI device and the synthetic device is: the DPI device needs to load-balance the data record file it generates to the synthetic device, so that the synthetic device can associate and backfill the data record file to obtain the corresponding target traffic log file. Thus, the target traffic log file can be added to the pre-configured file monitoring directory.

[0041] On the basis of the above embodiment, after the target traffic log file is added to the pre-configured file monitoring directory, the file monitoring directory will monitor the target traffic log file. The upload strategies corresponding to different upload types can be pre-set. Based on this, based on the monitoring of the target traffic log file, the target traffic log file is determined to which upload type it belongs according to the additional information carried in the target traffic log file, that is, the target upload type, so that the target upload strategy corresponding to the target upload type can be determined. More specifically, if the target upload type is a single platform local upload type, it indicates that the target traffic log file needs to be uploaded to any local log storage node in the local service platform, and therefore, the target traffic log file can be uploaded to at least one target local log storage node of the local service platform. If the target upload type is a single platform remote upload type, it indicates that the target traffic log file needs to be uploaded to any remote log storage node in the remote service platform, and therefore, the target traffic log file can be uploaded to at least one target remote log storage node of the remote service platform. If the target upload type is a multi-platform mixed upload type, it means that the target traffic log file needs to be uploaded to any remote log storage node in the remote service platform, and needs to be uploaded to any remote log storage node in the remote service platform. Therefore, the target traffic log file can be uploaded to at least one target local log storage node of the local service platform, and the target traffic log file can be uploaded to at least one target remote log storage node of the remote service platform. Optionally, when the log upload task of each log storage node is completed, the suffix of the target traffic log file uploaded to the log storage node can also be automatically modified.

[0042] Optionally, in the process of uploading the target traffic log file to the target log storage node, the log file uploading task can be implemented through at least one data transmission method among File Transfer Protocol (FTP), File Transfer Protocol over SSL / TLS (FTPS), Hypertext Transfer Protocol (HTTP) and Hypertext Transfer Protocol Secure (HTTPS).

[0043] S120: When it is detected that at least one target log storage node feeds back a confirmation file corresponding to the target traffic log file, the progress record information corresponding to the target traffic log file in the upload progress record table is updated.

[0044] Among them, the confirmation file refers to the file that indicates that the target traffic log file has been successfully received and processed by the log storage node. The purpose of the target log storage node to feedback the confirmation file is to tell the other end that the file has been uploaded to avoid the other end processing an incomplete file. The upload progress record table is a tool or data table used to track and record the status of each stage in the file upload process. The progress record information refers to the data used to track and record the current completion status during the entire upload task of the target traffic log file.

[0045] In this embodiment, when a new target traffic log file is detected in the file monitoring directory, the identification code of the target traffic log file can be synchronized in the upload progress record table, and the initial progress information of the target traffic log file can be recorded in the upload progress record table. In the process of executing the corresponding log upload task according to the target upload type of the target traffic log file, the target traffic log file may be sent to multiple target log storage nodes. Based on this, whenever a target log storage node is detected to feedback a confirmation file corresponding to the target traffic log file, the progress record information corresponding to the target traffic log file in the upload progress record table can be updated.

[0046] Exemplarily, the upload progress record table can be recorded in the form of key-value pairs. If the target upload type corresponding to the target traffic log file A is a single-platform local upload type, the initial progress information corresponding to the target traffic log file A in the upload progress record table can be expressed as: {Key: target traffic log file A; Value: 0, ~}, where the first element position of Value corresponds to the local service platform, and the second element position of Value corresponds to the remote service platform. When it is detected that any target local log storage node feeds back a confirmation file corresponding to the target traffic log file A, the progress record information in the upload progress record table can be expressed as: {Key: target traffic log file A; Value: 1, ~}.

[0047] If the target upload type corresponding to the target traffic log file B is a single-platform remote upload type, the initial progress information corresponding to the target traffic log file B in the upload progress record table can be expressed as: {Key: target traffic log file B; Value: ~, 0}. When it is detected that any target remote log storage node feeds back a confirmation file corresponding to the target traffic log file B, the progress record information in the upload progress record table can be expressed as: {Key: target traffic log file B; Value: ~, 1}.

[0048] If the target upload type corresponding to the target traffic log file C is a multi-platform mixed upload type, the initial progress information corresponding to the target traffic log file C in the upload progress record table can be expressed as: {Key: target traffic log file C; Value: 0, 0}. When it is detected that any target local log storage node feeds back a confirmation file corresponding to the target traffic log file C, the progress record information in the upload progress record table is updated to: {Key: target traffic log file C; Value: 1, 0}; When it is detected that any target remote log storage node feeds back a confirmation file corresponding to the target traffic log file C, the progress record information in the upload progress record table is updated to: {Key: target traffic log file C; Value: 1, 1};

[0049] S130. When the progress record information corresponding to the target traffic log file in the upload progress record table meets the preset conditions, the upload process of the target traffic log file is completed.

[0050] In this embodiment, the preset condition is a pre-set rule for determining whether the target traffic log file has completed the entire upload process. For example, the preset condition may be: the value of each flag bit corresponding to the target upload type in the progress record information is converted from a first value to a second value.

[0051] On the basis of the above examples, if the target upload type corresponding to the target traffic log file A is a single-platform local upload type, the preset condition may be: the value on the flag corresponding to the local service platform in the progress record information is converted from 0 to 1, which indicates that the target traffic log file A has completed the log upload process. If the target upload type corresponding to the target traffic log file B is a single-platform remote upload type, the preset condition may be: the value on the flag corresponding to the remote service platform in the progress record information is converted from 0 to 1, which indicates that the target traffic log file B has completed the log upload process. If the target upload type corresponding to the target traffic log file C is a multi-platform mixed upload type, the preset condition may be: the values ​​on the flags corresponding to the local service platform and the remote service platform in the progress record information are both converted from 0 to 1, which indicates that the target traffic log file C has completed the log upload process.

[0052] It should be noted that the acquisition, storage, use, and processing of data in the technical solution of this application are in compliance with the relevant provisions of national laws and regulations. For example, network requests are intercepted and processed with the authorization of the user.

[0053] The technical solution of the embodiment of the present invention is that when a new target traffic log file is monitored in the file monitoring directory, the target traffic log file is uploaded to at least one target log storage node of at least one target service platform according to the target upload type corresponding to the target traffic log file; wherein the at least one target service platform includes at least one of a local service platform and a remote service platform; when it is detected that at least one target log storage node feeds back a confirmation file corresponding to the target traffic log file, the progress record information corresponding to the target traffic log file in the upload progress record table is updated; when the progress record information corresponding to the target traffic log file in the upload progress record table meets the preset conditions, the upload process of the target traffic log file is completed. The technical solution of this embodiment can monitor the traffic log file in real time, and when a new traffic log file is monitored, the upload task can be automatically completed according to the upload type of the traffic log file, and it can be uploaded to the corresponding one or more log storage nodes, and the situation of log file leakage and retransmission is avoided by confirming the file and the progress record information in the upload progress record table, which improves the real-time performance and upload efficiency when uploading different types of traffic log files to different platform servers as a whole.

[0054] Embodiment 2

[0055] Figure 4This is a schematic diagram of a log file upload method provided by an embodiment of the present invention. Based on the above embodiment, when the target upload type of the target traffic log file is a single platform local upload type, a detailed description is given on how to upload the target traffic log file. The specific implementation method can refer to the technical solution of this embodiment. Among them, the technical terms that are the same or corresponding to the above embodiment are not repeated here.

[0056] like Figure 4 As shown, the method specifically comprises the following steps:

[0057] S210. When a new target traffic log file is monitored in the file monitoring directory, if the target upload type corresponding to the target traffic log file is a single platform local upload type, the target traffic log file is uploaded to at least one target local log storage node of the local service platform based on the first network.

[0058] The single platform local upload type refers to the upload type in which the target traffic log file needs to be uploaded to any local log storage node in the local service platform. The target local log storage node is at least one log storage node belonging to the local service platform. The first network refers to a private network for short-range communication.

[0059] In this embodiment, if the target upload type is a single-platform local upload type, the target traffic log file can be uploaded to any log storage node corresponding to the local service platform through the first network, and this log storage node is the target local log storage node.

[0060] Based on the above embodiment, optionally, uploading the target traffic log file to at least one target local log storage node of the local service platform based on the first network may specifically include:

[0061] If the log storage node corresponding to the local service platform has upload priority information, uploading the target traffic log file to at least one target local log storage node of the local service platform through the first network based on the priority upload method;

[0062] The upload priority information is the priority of uploading the target traffic log file to each log storage node. The priority upload method refers to an upload method for realizing file upload according to the upload priority information of each log storage node.

[0063] In this embodiment, if the log storage node corresponding to the local service platform has upload priority information, it can be determined in sequence according to the order of each upload priority information to which log storage node in the local service platform the target traffic log file is uploaded, thereby uploading the target traffic log file to the log storage node through the first network. For example, see the schematic diagram of the priority upload method Figure 5 ,like Figure 5 As shown, the upload priority information corresponding to the log storage node A in the local service platform is the first priority, the upload priority information corresponding to the log storage node B is the second priority, and the upload priority information corresponding to the log storage node C is the third priority. The target traffic log file is uploaded to the log storage node A first. If the upload fails, the target traffic log file will be uploaded to the log storage node B again.

[0064] If the log storage node corresponding to the local service platform does not have upload priority information, the target traffic log file is uploaded to at least one target local log storage node of the local service platform through the first network based on the load balancing upload method.

[0065] The load balancing upload method refers to an upload method for realizing file upload according to the current working status information of each log storage node.

[0066] In this embodiment, if the log storage node corresponding to the local service platform does not have upload priority information, it can be determined which log storage node is in an idle state based on the current working status information of each log storage node, and then the target traffic log file will be uploaded to which log storage node in the local service platform through the first network in priority.

[0067] S220: When it is detected that at least one target log storage node feeds back a confirmation file corresponding to the target traffic log file, the progress record information corresponding to the target traffic log file in the upload progress record table is updated.

[0068] S230. When the progress record information corresponding to the target traffic log file in the upload progress record table meets the preset conditions, the upload process of the target traffic log file is completed.

[0069] The technical solution of the embodiment of the present invention, if the target upload type corresponding to the target traffic log file is a single-platform local upload type, then the target traffic log file is uploaded to at least one target local log storage node of the local service platform based on the first network, specifically including: if the log storage node corresponding to the local service platform has upload priority information, then based on the priority upload method, the target traffic log file is uploaded to at least one target local log storage node of the local service platform through the first network; if the log storage node corresponding to the local service platform does not have upload priority information, then based on the load balancing upload method, the target traffic log file is uploaded to at least one target local log storage node of the local service platform through the first network. The technical solution provided by this embodiment, if the target upload type corresponding to the target traffic log file is a single-platform local upload type, the target traffic log file can be uploaded to at least one target local log storage node of the local service platform through the first network using a priority upload method or a load balancing upload method, providing a variety of upload methods to further improve the efficiency of log upload.

[0070] Embodiment 3

[0071] Figure 6 This is a schematic diagram of a log file uploading method provided by an embodiment of the present invention. Based on the above embodiment, when the target upload type of the target traffic log file is a single platform remote upload type, a detailed description is given on how to upload the target traffic log file. The specific implementation method can refer to the technical solution of this embodiment. Among them, the technical terms that are the same or corresponding to the above embodiment are not repeated here.

[0072] like Figure 6 As shown, the method specifically comprises the following steps:

[0073] S310. When a new target traffic log file is monitored in the file monitoring directory, if the target upload type corresponding to the target traffic log file is a single platform remote upload type, the target traffic log file is uploaded to at least one target remote log storage node of the remote service platform based on the second network.

[0074] The single platform remote upload type refers to the upload type in which the target traffic log file needs to be uploaded to any remote log storage node in the remote service platform. The target remote log storage node is at least one log storage node belonging to the remote service platform. The second network refers to a public network for long-distance communication.

[0075] In this embodiment, if the target upload type is a single platform remote upload type, the target traffic log file can be uploaded to any log storage node corresponding to the remote service platform through the second network. This log storage node is the target remote log storage node.

[0076] Based on the above embodiment, optionally, uploading the target traffic log file to at least one target remote log storage node of the remote service platform based on the second network may specifically include: if the target upload type is a single platform remote upload type, compressing and encrypting the target traffic log file to obtain a first traffic log file; uploading the first traffic log file to at least one target remote log storage node of the remote service platform based on the second network.

[0077] The first traffic log file refers to a log file obtained by encrypting and compressing the target traffic log file.

[0078] In this embodiment, since the second network may have problems of insufficient bandwidth and insufficient security, in order to efficiently and safely implement the log file upload task, if the target upload type is a single platform remote upload type, the target traffic log file can be compressed and encrypted first to obtain the encrypted and compressed first traffic log file, so as to save bandwidth and improve the security of the log file. Then, the first traffic log file is uploaded to at least one target remote log storage node of the remote service platform through the second network. More specifically, in the specific implementation process of uploading the first traffic log file to at least one target remote log storage node of the remote service platform through the second network, if the log storage node corresponding to the remote service platform has upload priority information, then based on the priority upload method, the first traffic log file is uploaded to at least one target remote log storage node of the remote service platform through the second network; if the log storage node corresponding to the remote service platform does not have upload priority information, then based on the load balancing upload method, the first traffic log file is uploaded to at least one target remote log storage node of the remote service platform through the second network.

[0079] S320: When it is detected that at least one target log storage node feeds back a confirmation file corresponding to the target traffic log file, the progress record information corresponding to the target traffic log file in the upload progress record table is updated.

[0080] S330: When the progress record information corresponding to the target traffic log file in the upload progress record table meets the preset conditions, the upload process of the target traffic log file is completed.

[0081] The technical solution of the embodiment of the present invention, if the target upload type corresponding to the target traffic log file is a single platform remote upload type, then uploading the target traffic log file to at least one target remote log storage node of the remote service platform based on the second network, specifically includes: compressing and encrypting the target traffic log file to obtain a first traffic log file; uploading the first traffic log file to at least one target remote log storage node of the remote service platform based on the second network. The technical solution provided by this embodiment saves bandwidth and improves the security of the log file by compressing and encrypting the target traffic log file before uploading it remotely, thereby improving the security of the log file and the upload efficiency.

[0082] Embodiment 4

[0083] Figure 7 A schematic diagram of a log file uploading method provided by an embodiment of the present invention, based on the above embodiment, in the case where the target upload type of the target traffic log file is a multi-platform mixed upload type, a detailed description of how to upload the target traffic log file is provided, and its specific implementation method can refer to the technical solution of this embodiment. Among them, the technical terms that are the same or corresponding to the above embodiment are not repeated here.

[0084] like Figure 7 As shown, the method specifically comprises the following steps:

[0085] S410. When a new target traffic log file is detected in the file monitoring directory, if the target upload type corresponding to the target traffic log file is a multi-platform mixed upload type, the target traffic log file is compressed and encrypted to obtain a first traffic log file.

[0086] Among them, the multi-platform mixed upload type refers to the upload type in which the target traffic log file needs to be uploaded to any remote log storage node in the remote service platform and any remote log storage node in the local service platform.

[0087] In this embodiment, if the target upload type is a multi-platform mixed upload type, the target traffic log file may be first compressed and encrypted to obtain an encrypted and compressed first traffic log file.

[0088] S420: Upload the target traffic log file to at least one target local log storage node of the local service platform based on the first network, and upload the first traffic log file to at least one remote log storage node of the remote service platform based on the second network.

[0089] In this embodiment, on the basis of obtaining the first traffic log file, the target traffic log file can be uploaded to any log storage node corresponding to the local service platform through the first network, and this log storage node is the target local log storage node. Similarly, in the process of uploading the target traffic log file to any log storage node corresponding to the local service platform, if the log storage node corresponding to the local service platform has upload priority information, then based on the priority upload method, the first traffic log file can be uploaded to at least one target local log storage node of the local service platform through the first network; if the log storage node corresponding to the local service platform does not have upload priority information, then based on the load balancing upload method, the first traffic log file can be uploaded to at least one target local log storage node of the local service platform through the first network. And the first traffic log file can be uploaded to any log storage node corresponding to the remote service platform through the second network, and this log storage node is the target remote log storage node. Similarly, in the process of uploading the target traffic log file to any log storage node corresponding to the remote service platform, if the log storage node corresponding to the remote service platform has upload priority information, then based on the priority upload method, the first traffic log file can be uploaded to at least one target remote log storage node of the remote service platform through the second network; if the log storage node corresponding to the remote service platform does not have upload priority information, then based on the load balancing upload method, the first traffic log file can be uploaded to at least one target remote log storage node of the remote service platform through the second network.

[0090] S430: When it is detected that at least one target log storage node feeds back a confirmation file corresponding to the target traffic log file, the progress record information corresponding to the target traffic log file in the upload progress record table is updated.

[0091] S440: When the progress record information corresponding to the target traffic log file in the upload progress record table meets the preset conditions, the upload process of the target traffic log file is completed.

[0092] The technical solution of the embodiment of the present invention is that if the target upload type corresponding to the target traffic log file is a multi-platform mixed upload type, the target traffic log file is compressed and encrypted to obtain a first traffic log file, and the target traffic log file is uploaded to at least one target local log storage node of the local service platform based on the first network, and the first traffic log file is uploaded to at least one remote log storage node of the remote service platform based on the second network. The technical solution provided by this embodiment achieves the effect of multi-platform copy and upload of log files, and by compressing and encrypting the target traffic log file before uploading it remotely, the security and upload efficiency of the log file are improved.

[0093] Embodiment 5

[0094] Figure 8 A schematic diagram of a log file uploading method provided by an embodiment of the present invention, based on the above-mentioned embodiment, when the upload process of the target traffic log file is completed, the progress record information corresponding to the target traffic log file can be deleted from the upload progress record table, so that, in response to the device restart event, the upload progress record table is scanned and processed to determine the existing record data in the upload progress record table, and the breakpoint-resume transmission processing is performed for each traffic log file to be retransmitted contained in the existing record data according to the progress record information of the traffic log file to be retransmitted. Among them, the technical terms that are the same as or corresponding to the above-mentioned embodiment are not repeated here.

[0095] like Figure 8 As shown, the method specifically comprises the following steps:

[0096] S510. When a new target traffic log file is detected in the file monitoring directory, the target traffic log file is uploaded to at least one target log storage node of at least one target service platform according to the target upload type corresponding to the target traffic log file.

[0097] Wherein, at least one target service platform includes at least one of a local service platform and a remote service platform.

[0098] S520: When it is detected that at least one target log storage node feeds back a confirmation file corresponding to the target traffic log file, the progress record information corresponding to the target traffic log file in the upload progress record table is updated.

[0099] S530. When the progress record information corresponding to the target traffic log file in the upload progress record table meets the preset conditions, the upload process of the target traffic log file is completed, and the progress record information corresponding to the target traffic log file is deleted from the upload progress record table.

[0100] In this embodiment, when the upload process of the target traffic log file is completed, the progress record information corresponding to the target traffic log file can be deleted from the upload progress record table, or transferred and stored in other preset storage spaces.

[0101] S540: In response to the device restart event, scan the upload progress record table to determine the stock record data in the upload progress record table.

[0102] The device restart event refers to the event that the device that performs the log upload task fails and needs to be restarted when encountering an emergency. The stock record data refers to the static data that has been stored in the upload progress record table and has not been processed or cleared.

[0103] In this embodiment, when the device is operating normally, after the log file task of a certain traffic log file is uploaded, the corresponding progress record information in the upload progress record table will be cleared, so the traffic log file involved in the stock record data indicates that it has not completed the upload task and needs to be uploaded again. Based on this, if the device executing the log upload task fails and restarts, the upload progress record table can be scanned at this time to obtain the stock record data in the upload progress record table.

[0104] S550: For each flow log file to be retransmitted contained in the stock record data, a breakpoint-resume transmission process is performed according to the progress record information of the flow log file to be retransmitted.

[0105] The traffic log files to be retransmitted refer to the traffic log files included in the stock record data of the upload progress record table.

[0106] In this embodiment, for each traffic log file to be retransmitted contained in the stock record data, it can be determined based on the progress record information which log storage nodes have completed the upload task of the traffic log file to be retransmitted, and which log storage nodes have not completed the upload task, so that the traffic log file to be retransmitted can be re-uploaded to the log storage node that has not completed the upload task. In this way, even if a sudden equipment failure occurs, the log file that has not completed the upload task can be re-uploaded based on the stock progress record information in the upload progress record table, without having to repeat the log upload process of the log storage node that has completed the upload task, and the log file upload process can be efficiently and accurately realized.

[0107] The technical solution of the embodiment of the present invention is to delete the progress record information corresponding to the traffic log file from the upload progress record table when the upload process of the traffic log file is completed. Thus, in response to the device restart event, the upload progress record table can be scanned and processed to determine the target stock record data in the upload progress record table; for each traffic log file to be retransmitted contained in the stock record data, breakpoint-resume processing is performed according to the progress record information of the traffic log file to be retransmitted. The technical solution provided by this embodiment can perform breakpoint-resume transmission for log files that have not completed the upload task according to the stock progress record information in the upload progress record table when encountering a sudden equipment failure. There is no need to repeatedly upload the traffic log file to the log storage node that has completed the upload task, and the log file upload process can be efficiently and accurately implemented.

[0108] Embodiment 6

[0109] Fig. 9This is a structural diagram of a log file uploading device provided by an embodiment of the present invention. The device includes: a log file uploading module 610, a record information updating module 620 and an upload completion determination module 630.

[0110] The log file upload module 610 is used to upload the target traffic log file to at least one target log storage node of at least one target service platform according to the target upload type corresponding to the target traffic log file when a new target traffic log file is monitored in the file monitoring directory; wherein the at least one target service platform includes at least one of a local service platform and a remote service platform;

[0111] The record information updating module 620 is used to update the progress record information corresponding to the target traffic log file in the upload progress record table when it is detected that the at least one target log storage node feeds back a confirmation file corresponding to the target traffic log file;

[0112] The upload completion determination module 630 is used to complete the upload process of the target traffic log file when the progress record information corresponding to the target traffic log file in the upload progress record table meets a preset condition.

[0113] The technical solution of the embodiment of the present invention is that when a new target traffic log file is monitored in the file monitoring directory, the target traffic log file is uploaded to at least one target log storage node of at least one target service platform according to the target upload type corresponding to the target traffic log file; wherein the at least one target service platform includes at least one of a local service platform and a remote service platform; when it is detected that at least one target log storage node feeds back a confirmation file corresponding to the target traffic log file, the progress record information corresponding to the target traffic log file in the upload progress record table is updated; when the progress record information corresponding to the target traffic log file in the upload progress record table meets the preset conditions, the upload process of the target traffic log file is completed. The technical solution of this embodiment can monitor the traffic log file in real time, and when a new traffic log file is monitored, the upload task can be automatically completed according to the upload type of the traffic log file, and it can be uploaded to the corresponding one or more log storage nodes, and the situation of log file leakage and retransmission is avoided by confirming the file and the progress record information in the upload progress record table, which improves the real-time performance and upload efficiency when uploading different types of traffic log files to different platform servers as a whole.

[0114] Based on the above device, optionally, the target upload type includes a single-platform local upload type, a single-platform remote upload type, or a multi-platform mixed upload type, and the log file upload module 610 includes:

[0115] A log local uploading unit, configured to upload the target traffic log file to at least one target local log storage node of the local service platform based on the first network if the target upload type corresponding to the target traffic log file is a single platform local upload type;

[0116] A log remote uploading unit, configured to upload the target traffic log file to at least one target remote log storage node of the remote service platform based on a second network if the target upload type corresponding to the target traffic log file is a single platform remote upload type;

[0117] A log mixed upload unit is used to upload the target traffic log file to at least one target local log storage node of the local service platform based on the first network if the target upload type corresponding to the target traffic log file is a multi-platform mixed upload type, and to at least one target remote log storage node of the remote service platform based on the second network.

[0118] On the basis of the above-mentioned device, optionally, a local log uploading unit is specifically used to upload the target traffic log file to at least one target local log storage node of the local service platform through the first network based on a priority upload method if the log storage node corresponding to the local service platform has upload priority information; if the log storage node corresponding to the local service platform does not have upload priority information, then based on a load balancing upload method, the target traffic log file is uploaded to at least one target local log storage node of the local service platform through the first network.

[0119] Based on the above-mentioned device, optionally, a log remote upload unit is specifically used to compress and encrypt the target traffic log file to obtain a first traffic log file if the target upload type is a single-platform remote upload type; and upload the first traffic log file to at least one target remote log storage node of the remote service platform based on the second network.

[0120] Based on the above-mentioned device, optionally, a log mixed upload unit is specifically used to compress and encrypt the target traffic log file to obtain a first traffic log file if the target upload type corresponding to the target traffic log file is a multi-platform mixed upload type; upload the target traffic log file to at least one target local log storage node of the local service platform based on the first network, and upload the first traffic log file to at least one target remote log storage node of the remote service platform based on the second network.

[0121] Based on the above-mentioned device, optionally, the log file uploading device also includes: a file adding module, which is used to perform data parsing and associated data backfilling processing on the network traffic data packet when the network traffic data packet is received, to obtain the target traffic log file; and add the target traffic log file to the file monitoring directory.

[0122] Based on the above device, optionally, the log file uploading device also includes: a progress record clearing module, which is used to delete the progress record information corresponding to the target traffic log file from the upload progress record table when the upload process of the target traffic log file is completed.

[0123] Based on the above-mentioned device, optionally, the log file uploading device also includes: a breakpoint resume processing module, which is used to scan and process the upload progress record table in response to a device restart event, and determine the existing record data in the upload progress record table; for each traffic log file to be retransmitted contained in the existing record data, breakpoint resume processing is performed according to the progress record information of the traffic log file to be retransmitted.

[0124] The log file uploading device provided in the embodiment of the present invention can execute the log file uploading method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0125] It is worth noting that the various units and modules included in the above system are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present invention.

[0126] Embodiment 7

[0127] Fig.10 The present invention is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. Fig.10 A block diagram of an exemplary electronic device 70 suitable for use in implementing exemplary embodiments of the present invention is shown. Fig.10 The electronic device 70 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0128] like Fig.10 As shown, the electronic device 70 is in the form of a general-purpose computing device. The components of the electronic device 70 may include, but are not limited to: one or more processors or processing units 701, a system memory 702, and a bus 703 connecting different system components (including the system memory 702 and the processing unit 701).

[0129] Bus 703 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus structures. For example, these architectures include but are not limited to Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus and Peripheral Component Interconnect (PCI) bus.

[0130] The electronic device 70 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device 70, including volatile and non-volatile media, removable and non-removable media.

[0131] The system memory 702 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 704 and / or cache memory 705. The electronic device 70 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 706 may be used to read and write non-removable, non-volatile magnetic media ( Fig.10 not shown, usually called a "hard drive"). Although Fig.10 Not shown in the figure, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 703 via one or more data medium interfaces. The memory 702 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present invention.

[0132] A program / utility 707 having a set (at least one) of program modules 707 may be stored, for example, in the memory 702, such program modules 707 including but not limited to an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules 707 generally perform the functions and / or methods of the embodiments described herein.

[0133] The electronic device 70 may also communicate with one or more external devices 709 (e.g., keyboards, pointing devices, displays 710, etc.), may communicate with one or more devices that enable a user to interact with the electronic device 70, and / or communicate with any device that enables the electronic device 70 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface 711. Furthermore, the electronic device 70 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 712. As shown, the network adapter 712 communicates with other modules of the electronic device 70 via the bus 703. It should be understood that although Fig.10 Not shown, other hardware and / or software modules may be used in conjunction with the electronic device 70, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0134] The processing unit 701 executes various functional applications and page processing by running the programs stored in the system memory 702, such as implementing the log file uploading method provided in the embodiment of the present invention.

[0135] Embodiment 8

[0136] An embodiment of the present invention further provides a storage medium including computer executable instructions, which are used to execute a log file uploading method when executed by a computer processor. The method includes:

[0137] When a new target traffic log file is monitored in the file monitoring directory, the target traffic log file is uploaded to at least one target log storage node of at least one target service platform according to the target upload type corresponding to the target traffic log file; wherein the at least one target service platform includes at least one of a local service platform and a remote service platform;

[0138] When it is detected that the at least one target log storage node feeds back a confirmation file corresponding to the target traffic log file, updating the progress record information corresponding to the target traffic log file in the upload progress record table;

[0139] When the progress record information corresponding to the target traffic log file in the upload progress record table meets a preset condition, the upload process of the target traffic log file is completed.

[0140] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0141] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0142] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0143] Computer program code for performing the operation of embodiments of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0144] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A log file uploading method, characterized in that: include: When a new target traffic log file is monitored in the file monitoring directory, the target traffic log file is uploaded to at least one target log storage node of at least one target service platform according to the target upload type corresponding to the target traffic log file; wherein the at least one target service platform includes at least one of a local service platform and a remote service platform; When it is detected that the at least one target log storage node feeds back a confirmation file corresponding to the target traffic log file, updating the progress record information corresponding to the target traffic log file in the upload progress record table; When the progress record information corresponding to the target traffic log file in the upload progress record table meets a preset condition, the upload process of the target traffic log file is completed.

2. The method according to claim 1, characterized in that The target upload type includes a single-platform local upload type, a single-platform remote upload type, or a multi-platform mixed upload type, and uploading the target traffic log file to at least one target log storage node of at least one target service platform according to the target upload type corresponding to the target traffic log file includes: If the target upload type corresponding to the target traffic log file is a single-platform local upload type, uploading the target traffic log file to at least one target local log storage node of the local service platform based on the first network; If the target upload type corresponding to the target traffic log file is a single platform remote upload type, uploading the target traffic log file to at least one target remote log storage node of the remote service platform based on the second network; If the target upload type corresponding to the target traffic log file is a multi-platform mixed upload type, the target traffic log file is uploaded to at least one target local log storage node of the local service platform based on the first network, and the target traffic log file is uploaded to at least one target remote log storage node of the remote service platform based on the second network.

3. The method according to claim 2, characterized in that The uploading of the target traffic log file to at least one target local log storage node of the local service platform based on the first network includes: If the log storage node corresponding to the local service platform has upload priority information, uploading the target traffic log file to at least one target local log storage node of the local service platform through the first network based on the priority upload method; If the log storage node corresponding to the local service platform does not have upload priority information, the target traffic log file is uploaded to at least one target local log storage node of the local service platform through the first network based on a load balancing upload method.

4. The method according to claim 2, characterized in that: If the target upload type corresponding to the target traffic log file is a single platform remote upload type, uploading the target traffic log file to at least one target remote log storage node of the remote service platform based on the second network includes: If the target upload type is a single-platform remote upload type, compressing and encrypting the target traffic log file to obtain a first traffic log file; The first traffic log file is uploaded to at least one target remote log storage node of the remote service platform based on the second network.

5. The method according to claim 2, characterized in that: If the target upload type corresponding to the target traffic log file is a multi-platform mixed upload type, uploading the target traffic log file to at least one target local log storage node of the local service platform based on the first network, and uploading the target traffic log file to at least one target remote log storage node of the remote service platform based on the second network, including: If the target upload type corresponding to the target traffic log file is a multi-platform mixed upload type, compressing and encrypting the target traffic log file to obtain a first traffic log file; The target traffic log file is uploaded to at least one target local log storage node of the local service platform based on the first network, and the first traffic log file is uploaded to at least one target remote log storage node of the remote service platform based on the second network.

6. The method according to claim 1, characterized in that The method further comprises: When a network traffic data packet is received, data parsing and associated data backfilling processing are performed on the network traffic data packet to obtain a target traffic log file; Add the target traffic log file to the file monitoring directory.

7. The method according to claim 1, characterized in that The method further comprises: When the upload process of the target traffic log file is completed, the progress record information corresponding to the target traffic log file is deleted from the upload progress record table.

8. The method according to claim 7, characterized in that The method further comprises: In response to a device restart event, scanning the upload progress record table to determine the stock record data in the upload progress record table; For each flow log file to be retransmitted contained in the stock record data, breakpoint resume transmission is performed according to the progress record information of the flow log file to be retransmitted.

9. A log file uploading device, characterized in that: include: A log file uploading module, configured to upload a target traffic log file to at least one target log storage node of at least one target service platform according to a target upload type corresponding to the target traffic log file when a target traffic log file is newly added to the file monitoring directory; wherein the at least one target service platform includes at least one of a local service platform and a remote service platform; A record information updating module, configured to update the progress record information corresponding to the target traffic log file in the upload progress record table when it is detected that the at least one target log storage node feeds back a confirmation file corresponding to the target traffic log file; The upload completion determination module is used to complete the upload process of the target traffic log file when the progress record information corresponding to the target traffic log file in the upload progress record table meets a preset condition.

10. An electronic device, characterized in that: The electronic device comprises: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the log file uploading method as described in any one of claims 1-8.

Images