Data transmission method and device, electronic equipment and storage medium

Abstract

The invention discloses a data transmission method and device, electronic equipment and a storage medium. The method comprises the steps that event log data of each distributed device are acquired, the distributed devices are used for monitoring flow data of a target system, the event log data comprise a plurality of preliminary event logs, and the preliminary event logs are generated based on the flow data; aiming at the event log data of each distributed device, performing duplicate removal processing on the plurality of preliminary event logs to obtain at least one target event log, compressing the at least one target event log to obtain a compressed file, and encrypting the compressed file to obtain a target encrypted file; and transmitting the target encrypted file to the target data receiving end through the quickly opened transmission control protocol. Based on the technical scheme of the invention, after data of a plurality of distributed nodes are acquired at the same time, effective data (namely data reserved after de-duplication) can be transmitted to a data receiving end at high speed and high security.

Description

Technical Field

[0001] The present invention relates to the field of computer application technology, and in particular to a data transmission method, device, electronic equipment and storage medium. Background Art

[0002] With the rapid development of network security, the data in the event logs obtained through network attack monitoring equipment is becoming more and more complex. It is well known that analyzing event logs can effectively discover security threats in the system monitored by network attack monitoring equipment so as to solve them in time. In general, it is necessary to obtain event logs first and then analyze them.

[0003] Normally, event logs are obtained directly through the transmission control protocol for further data analysis, but there are many problems, such as slow data transmission speed, difficulty in ensuring security, data loss, and too much invalid data obtained from transmission, which is not conducive to subsequent data analysis. Summary of the invention

[0004] The present invention provides a data transmission method, device, electronic device and storage medium to solve the current lack of a high-efficiency, high-security, high-integrity and high-simple high-quality data transmission method.

[0005] According to one aspect of the present invention, a data transmission method is provided, the method comprising:

[0006] Acquire event log data of each distributed device, wherein the distributed device is used to monitor flow data of the target system, the event log data includes a plurality of preliminary event logs, and the preliminary event logs are generated based on the flow data;

[0007] For the event log data of each of the distributed devices, deduplication processing is performed on the multiple preliminary event logs to obtain at least one target event log, the at least one target event log is compressed to obtain a compressed file, and the compressed file is encrypted to obtain a target encrypted file;

[0008] The target encrypted file is transmitted to the target data receiving end via the fast-open transmission control protocol.

[0009] According to another aspect of the present invention, there is provided a data transmission device, the device comprising:

[0010] A data acquisition module, configured to acquire event log data of each distributed device, wherein the distributed device is configured to monitor flow data of a target system, the event log data comprising a plurality of preliminary event logs, and the preliminary event logs are generated based on the flow data;

[0011] A data processing module, configured to perform deduplication processing on the event log data of each of the distributed devices on the plurality of preliminary event logs to obtain at least one target event log, compress the at least one target event log to obtain a compressed file, and encrypt the compressed file to obtain a target encrypted file;

[0012] The data transmission module is used to transmit the target encrypted file to the target data receiving end through the fast open transmission control protocol.

[0013] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0014] at least one processor; and

[0015] a memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the data transmission method described in any embodiment of the present invention.

[0017] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data transmission method described in any embodiment of the present invention when executed.

[0018] The technical solution of the embodiment of the present invention obtains event log data of each distributed device, wherein the distributed device is used to monitor the flow data of the target system, and the event log data includes multiple preliminary event logs, which are generated based on the flow data, thereby achieving the effect of simultaneously obtaining event log data of multiple distributed nodes (i.e., the distributed devices); for the event log data of each of the distributed devices, the multiple preliminary event logs are deduplicated to obtain at least one target event log, at least one target event log is compressed to obtain a compressed file, and the compressed file is encrypted to obtain a target encrypted file, thereby achieving data deduplication, i.e., only transmitting the effective data after deduplication, and achieving data compression and data encryption, i.e., improving the speed and security of subsequent data transmission; the target encrypted file is transmitted to the target data receiving end through a fast-open transmission control protocol, and data transmission is performed through a fast-open transmission control protocol, which can effectively improve the speed of data transmission. In summary, based on the technical solution of the present invention, after simultaneously acquiring data from multiple distributed nodes, the valid data (i.e., the data retained after deduplication) can be transmitted to the data receiving end with high speed and high security, thereby meeting the needs of network security practitioners at the data receiving end for deduplication and false alarm removal of the received data, as well as the needs of security practitioners for efficient, secure and complete acquisition of transmitted data.

[0019] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 is a flowchart of a data transmission method provided according to Embodiment 1 of the present invention;

[0022] Figure 2 is a flowchart of a data transmission method provided according to Embodiment 2 of the present invention;

[0023] Figure 3 is an overall flow chart of a data transmission method provided according to an embodiment of the present invention;

[0024] Figure 4is a structural schematic diagram of a data transmission device provided according to Embodiment 3 of the present invention;

[0025] Figure 5 It is a structural schematic diagram of an electronic device for implementing the data transmission method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0028] Embodiment 1

[0029] Figure 1 A flowchart of a data transmission method is provided for the first embodiment of the present invention. This embodiment is applicable to the case of transmitting event log data. The method can be executed by a data transmission device. The data transmission device can be implemented in the form of hardware and / or software. The data transmission device can be configured in a computer. Figure 1 As shown, the method includes:

[0030] S110. Obtain event log data of each distributed device.

[0031] The distributed device is used to monitor the flow data of the target system, and the event log data includes a plurality of preliminary event logs, which are generated based on the flow data.

[0032] Among them, the distributed device can be understood as a device distributed at different deployment nodes. In an embodiment of the present invention, the distributed device can be used to monitor the flow data of the target system. Different distributed devices can be used to monitor the flow data of different target systems. The target system can be understood as the operating system of the terminal. The target system may include multiple operating systems, that is, one distributed device can be used to monitor the flow data of multiple operating systems. The target system can be preset according to the scenario requirements, and is not specifically limited here. Exemplarily, the target system can be A banking system, B audit system or C education system, etc. The flow data can be understood as data generated based on the request response of the target system. The flow data may include the request creation time, source port, target port, source Internet Protocol (IP) address, target IP address and uniform resource locator, etc.

[0033] The event log data can be understood as data of the event log. The event log can be used to record data related to the occurrence of events in the target system. In an embodiment of the present invention, the preliminary event log can be understood as a preliminary event log. The preliminary event log can be used to record traffic data of the occurrence of events in the target system. Wherein, the above-mentioned occurrence of events is associated with the request response of the system.

[0034] More specifically, the obtaining of the event log data of each distributed device includes: respectively obtaining the event log data output by each distributed device.

[0035] S120. For the event log data of each of the distributed devices, deduplicate the multiple preliminary event logs to obtain at least one target event log, compress the at least one target event log to obtain a compressed file, and encrypt the compressed file to obtain a target encrypted file.

[0036] The target event log may be understood as the event log retained after deduplication. In layman's terms, there is no duplicate event log between at least one target event log.

[0037] The compressed file can be understood as a file obtained by compressing the target time log.

[0038] Optionally, compressing at least one of the target event logs to obtain a compressed file includes:

[0039] At least one of the target event logs is compressed using a lossless compression algorithm to obtain a compressed file.

[0040] The lossless compression algorithm (Lempel-Ziv-Welch, LZW) can be understood as a data compression algorithm that allows data to be completely and losslessly restored during compression and decompression.

[0041] In the embodiment of the present invention, the step of obtaining event log data of each distributed device includes:

[0042] The event log data of each distributed device is acquired based on a preset data acquisition cycle.

[0043] Optionally, compressing at least one of the target event logs by using a lossless compression algorithm to obtain a compressed file includes:

[0044] Determine the preset compression cycle;

[0045] Compressing at least one of the target event logs corresponding to each compression period by using the lossless compression algorithm to obtain a sub-file corresponding to each compression period;

[0046] A compressed file corresponding to the event log data acquired based on a data acquisition period is determined based on at least one of the sub-files.

[0047] Among them, the data acquisition cycle can be understood as the cycle for acquiring the event log data of the distributed device. The compression cycle can be understood as the cycle for compressing the target event log. In an embodiment of the present invention, the data acquisition cycle and the compression cycle can be preset according to the scenario, respectively, and are not specifically limited here. The data acquisition cycle can be greater than the compression cycle. Exemplarily, the data acquisition cycle can be 1 minute, 2 minutes or 5 minutes, etc. The compression cycle can be 1 second, 3 seconds, 10 seconds, etc.

[0048] The target encrypted file may be understood as an encrypted file obtained by encrypting the compressed file.

[0049] Based on the above embodiment scheme, lossless compression of the target event log can be achieved, the data size can be reduced, the efficiency of subsequent data transmission is improved, and lossless recovery of subsequent data can be guaranteed, avoiding data loss during data transmission.

[0050] Optionally, encrypting the compressed file to obtain a target encrypted file includes:

[0051] Encrypting the compressed file by a symmetric encryption algorithm to obtain a transmission encrypted file;

[0052] Performing hash calculation on the transmission encrypted file by using a hash algorithm to obtain a target hash value;

[0053] A target encrypted file corresponding to the compressed file is determined based on the transmission encrypted file and the target hash value.

[0054] The symmetric encryption algorithm can be understood as a data encryption algorithm that uses the same key for both encryption and decryption. In an embodiment of the present invention, the symmetric encryption algorithm can include a Blowfish algorithm. The Blowfish algorithm can be understood as a symmetric key encryption algorithm that uses a block encryption method to encrypt data.

[0055] The transmission encrypted file can be understood as an encrypted file to be transmitted through a transmission protocol, and can also be understood as an encrypted file obtained by symmetrically encrypting the compressed file.

[0056] The hash algorithm can be understood as an algorithm that maps an input of any length to an output of a fixed length. In an embodiment of the present invention, the data in the transmission encrypted file can be mapped to a target hash value through the hash algorithm.

[0057] S130, transmitting the target encrypted file to a target data receiving end through a fast-open transmission control protocol.

[0058] The target data receiving end may be understood as a data management end, through which the received data may be managed in a unified manner.

[0059] The Transmission Control Protocol (TCP) can be used for data transmission. The Fast Open Transmission Control Protocol can be understood as a Transmission Control Protocol that establishes a connection based on the TCP Fast Open (TFO) technology.

[0060] Optionally, the target encrypted file includes the transmission encrypted file and the target hash value corresponding to the transmission encrypted file, and after the target encrypted file is transmitted to the target data receiving end through the transmission control protocol of the fast open, the method further includes:

[0061] The target data receiving end verifies the target hash value to obtain a verification result, and determines the target transmission data according to the verification result and the transmission encrypted file.

[0062] In the embodiment of the present invention, the target encrypted file includes the transmission encrypted file and the target hash value corresponding to the transmission encrypted file.

[0063] The verification result can be understood as a result indicating whether the target hash value has been tampered with. Optionally, the verification result can include that the hash value has not changed or that the hash value has changed.

[0064] Specifically, when the verification result is that the hash value has not changed, the transmission encrypted file is directly decrypted through the symmetric encryption to obtain the data in the transmission encrypted file (that is, the target transmission data), wherein the target transmission data can be all of the traffic data corresponding to the target system monitored by the distributed device, or part of the traffic data.

[0065] Based on the above embodiment scheme, the hash value verification method is adopted to achieve the effect of enabling the target data receiving end to obtain the transmission data with high accuracy that has not been tampered with.

[0066] The technical solution of the embodiment of the present invention obtains event log data of each distributed device, wherein the distributed device is used to monitor the flow data of the target system, and the event log data includes multiple preliminary event logs, which are generated based on the flow data, thereby achieving the effect of simultaneously obtaining event log data of multiple distributed nodes (i.e., the distributed devices); for the event log data of each of the distributed devices, the multiple preliminary event logs are deduplicated to obtain at least one target event log, at least one target event log is compressed to obtain a compressed file, and the compressed file is encrypted to obtain a target encrypted file, thereby achieving data deduplication, i.e., only transmitting the effective data after deduplication, and achieving data compression and data encryption, i.e., improving the speed and security of subsequent data transmission; the target encrypted file is transmitted to the target data receiving end through a fast-open transmission control protocol, and data transmission is performed through a fast-open transmission control protocol, which can effectively improve the speed of data transmission. In summary, based on the technical solution of the present invention, after simultaneously acquiring data from multiple distributed nodes, the valid data (i.e., the data retained after deduplication) can be transmitted to the data receiving end with high speed and high security, thereby meeting the needs of network security practitioners at the data receiving end for deduplication and false alarm removal of the received data, as well as the needs of security practitioners for efficient, secure and complete acquisition of transmitted data.

[0067] Embodiment 2

[0068] Figure 2 This is a flowchart of a data transmission method provided in Embodiment 2 of the present invention. This embodiment is to perform deduplication processing on multiple preliminary event logs as described in the above embodiment to obtain at least one target event log for refinement. Figure 2 As shown, the method includes:

[0069] S210. Obtain event log data of each distributed device.

[0070] S220. For the event log data of each of the distributed devices, determine the total data volume of the traffic data in the plurality of preliminary event logs.

[0071] The total data volume may represent the total data volume of the traffic data in a plurality of the preliminary event logs.

[0072] S230: When the total data volume exceeds a preset data volume threshold, similarity recognition is performed on every two preliminary event logs using a pre-trained data recognition model to obtain a recognition result.

[0073] The data volume threshold may be understood as a preset threshold related to the data volume. In the embodiment of the present invention, the data threshold may be preset according to the scenario requirements and is not specifically limited here.

[0074] The data recognition model can be understood as a model with a data similarity recognition function. Optionally, the data recognition model can be obtained by training a deep learning model based on training samples.

[0075] Optionally, the data recognition model obtained through pre-training performs similarity recognition on every two of the preliminary event logs to obtain a recognition result, including:

[0076] Determining target prompt data, wherein the target prompt data at least includes description data of description content related to the similarity recognition threshold;

[0077] The target prompt data and the plurality of preliminary event logs are input into the data recognition model to obtain the recognition result, wherein the recognition result is a result corresponding to the description content related to the similarity recognition threshold.

[0078] The target prompt data can be used to prompt the data similarity recognition of the data recognition model, so that the data recognition model outputs data associated with the target prompt data. In an embodiment of the present invention, the target prompt data can be pre-set or input in real time based on the operator's operation. In the scenario of real-time input, the input method of the target prompt data is not specifically limited, and can be text input or voice input, etc.

[0079] The similarity recognition threshold can be understood as a threshold related to similarity. In the embodiment of the present invention, the similarity recognition threshold can be related to the scene requirements, and is not specifically limited here. Exemplarily, the similarity recognition threshold can be 80%, 90% or 95%, etc.

[0080] For example, the target prompt data may be "please identify event logs with a similarity of more than 90%" or "please identify event logs with a similarity between 90% and 100%", etc. Correspondingly, the description content related to the similarity recognition threshold in the target prompt data may be "identify event logs with a similarity of more than 90%" or "identify event logs with a similarity between 90% and 100%".

[0081] Based on the above-mentioned embodiment scheme, when the amount of data is large, it is achieved that the similarity between every two event logs is identified through an intelligent model, so as to further deduplicate event logs with higher similarity (in the embodiment of the present invention, event logs with higher similarity are considered to be duplicate event logs, that is, false alarm event logs), thereby avoiding the transmission of false alarm event logs.

[0082] S240: Perform deduplication processing on the plurality of preliminary event logs based on the identification result to obtain at least one event log to be transmitted.

[0083] Specifically, the deduplication processing of the multiple preliminary event logs based on the recognition result may include: for the multiple preliminary event logs whose similarity exceeds the similarity recognition threshold, randomly retaining one of the preliminary event logs, and deleting the other preliminary event logs except the retained preliminary event log; and using the retained preliminary event log as the event log to be transmitted.

[0084] The event log to be transmitted may be understood as an event log to be transmitted to the target data receiving end.

[0085] Optionally, after determining the total data volume of the traffic data in the plurality of preliminary event logs, the method further includes:

[0086] When the total data volume does not exceed the preset data volume threshold, comparing the flow data in every two preliminary event logs field by field to obtain a data comparison result;

[0087] Based on the data comparison result, deduplication processing is performed on the multiple preliminary event logs to obtain at least one event log to be transmitted.

[0088] The data comparison result may represent the consistency of the flow data in each two preliminary event logs.

[0089] In the embodiment of the present invention, the specific method for deduplicating the multiple preliminary event logs based on the data comparison result is not specifically limited. Optionally, when the data comparison result indicates that the fields of the traffic data in the two preliminary event logs are completely consistent, one of the preliminary event logs is randomly deleted.

[0090] S250: For each of the event logs to be transmitted, deduplicate the traffic data in the event logs to be transmitted to obtain the target event log corresponding to the event log to be transmitted.

[0091] The target event log may be understood as an event log obtained after deduplication of traffic data.

[0092] Optionally, the performing deduplication processing on the traffic data in the transmission event log includes:

[0093] Dividing the traffic data in the event log to be transmitted into fields to obtain a plurality of divided field data;

[0094] Compare every two adjacent divided field data to obtain a field comparison result;

[0095] The traffic data is deduplicated based on the field comparison result.

[0096] Optionally, said dividing the flow data in the event log to be transmitted into fields may include: dividing the flow data in the event log to be transmitted into fields based on a preset field length. In the embodiment of the present invention, the field length may be preset according to scenario requirements and is not specifically limited here.

[0097] The field comparison result can represent the consistency degree of each two divided field data. The divided field data can be understood as the field data obtained by division.

[0098] In the embodiment of the present invention, the specific manner of performing deduplication processing on the traffic data based on the field comparison result is not specifically limited. Optionally, when the field comparison result indicates that the two divided field data are completely consistent, one of the field data is randomly deleted.

[0099] Based on the above embodiment scheme, the effect of deduplication of fields in the data flow in each event log is achieved.

[0100] S260: compress at least one of the target event logs to obtain a compressed file, and encrypt the compressed file to obtain a target encrypted file.

[0101] S270, transmitting the target encrypted file to the target data receiving end through the fast open transmission control protocol.

[0102] The technical solution of the embodiment of the present invention is to determine the total data volume of the traffic data in a plurality of the preliminary event logs; when the total data volume exceeds a preset data volume threshold, perform similarity recognition on every two of the preliminary event logs through a pre-trained data recognition model to obtain a recognition result; perform deduplication processing on the plurality of preliminary event logs based on the recognition result to obtain at least one event log to be transmitted; for each of the event logs to be transmitted, perform deduplication processing on the traffic data in the event log to be transmitted to obtain the target event log corresponding to the event log to be transmitted. Based on the above technical solution, the effects of deduplication of falsely reported event logs and deduplication of falsely reported traffic data in event logs are achieved, the data conciseness of the target event log that is subsequently transmitted through the transmission protocol is determined, and the data transmission efficiency is improved.

[0103] Figure 3 1 is an overall flow chart of a data transmission method provided according to an embodiment of the present invention. Figure 3 The overall process of the data transmission method is further explained:

[0104] 1. Obtain the event log output by the network attack detection device.

[0105] 2. Remove duplicate false positive data through intelligent algorithms. Identify and mark event logs or traffic data with excessive similarity, and deduplicate the marked data, that is, do not transmit the marked false positive data. The above intelligent algorithms include algorithms for calculating data volume, similarity recognition models, and algorithms for field comparison of data.

[0106] 3. Compress data using the LZW algorithm to reduce file size.

[0107] 4. Encrypt data using the Blowfish encryption algorithm to ensure data security.

[0108] 5. Data transmission is carried out through TFO to increase data transmission speed.

[0109] 6. Calculate the hash value through the Secure Hash Algorithm (HASH) algorithm, and verify the integrity of the data during transmission by verifying the hash value.

[0110] The technical solution of the present invention meets the needs of network security practitioners at the data receiving end for deduplication and false alarm removal of received data, and meets the needs of network security practitioners for efficient, secure and complete acquisition of transmitted data.

[0111] Embodiment 3

[0112] Figure 4 This is a schematic diagram of the structure of a data transmission device provided in Embodiment 3 of the present invention. Figure 4 As shown, the device includes: a data acquisition module 310, a data processing module 320 and a data transmission module 330.

[0113] Among them, the data acquisition module 310 is used to obtain the event log data of each distributed device, wherein the distributed device is used to monitor the flow data of the target system, and the event log data includes multiple preliminary event logs, and the preliminary event logs are generated based on the flow data; the data processing module 320 is used to deduplicate the multiple preliminary event logs for the event log data of each of the distributed devices to obtain at least one target event log, compress at least one of the target event logs to obtain a compressed file, and encrypt the compressed file to obtain a target encrypted file; the data transmission module 330 is used to transmit the target encrypted file to the target data receiving end via the fast open transmission control protocol.

[0114] The technical solution of the embodiment of the present invention obtains event log data of each distributed device, wherein the distributed device is used to monitor the flow data of the target system, and the event log data includes multiple preliminary event logs, which are generated based on the flow data, thereby achieving the effect of simultaneously obtaining event log data of multiple distributed nodes (i.e., the distributed devices); for the event log data of each of the distributed devices, the multiple preliminary event logs are deduplicated to obtain at least one target event log, at least one target event log is compressed to obtain a compressed file, and the compressed file is encrypted to obtain a target encrypted file, thereby achieving data deduplication, i.e., only transmitting the effective data after deduplication, and achieving data compression and data encryption, i.e., improving the speed and security of subsequent data transmission; the target encrypted file is transmitted to the target data receiving end through a fast-open transmission control protocol, and data transmission is performed through a fast-open transmission control protocol, which can effectively improve the speed of data transmission. In summary, based on the technical solution of the present invention, after simultaneously acquiring data from multiple distributed nodes, the valid data (i.e., the data retained after deduplication) can be transmitted to the data receiving end with high speed and high security, thereby meeting the needs of network security practitioners at the data receiving end for deduplication and false alarm removal of the received data, as well as the needs of security practitioners for efficient, secure and complete acquisition of transmitted data.

[0115] Optionally, the preliminary event log includes the traffic data; the data processing module 320 includes: a data volume determination unit, a similarity identification unit, a log deduplication unit and a traffic data deduplication unit;

[0116] Wherein, the data volume determination unit is used to determine the total data volume of the traffic data in a plurality of the preliminary event logs;

[0117] The similarity recognition unit is used to perform similarity recognition on every two preliminary event logs by using a pre-trained data recognition model to obtain a recognition result when the total data volume exceeds a preset data volume threshold;

[0118] The first log deduplication unit is used to perform deduplication processing on the plurality of preliminary event logs based on the identification result to obtain at least one event log to be transmitted;

[0119] The traffic data deduplication unit is used to perform deduplication processing on the traffic data in each of the event logs to be transmitted, so as to obtain the target event log corresponding to the event log to be transmitted.

[0120] Optionally, the similarity recognition unit is specifically used to:

[0121] Determining target prompt data, wherein the target prompt data at least includes description data of description content related to the similarity recognition threshold;

[0122] The target prompt data and the plurality of preliminary event logs are input into the data recognition model to obtain the recognition result, wherein the recognition result is a result corresponding to the description content related to the similarity recognition threshold.

[0123] Optionally, the data processing module 320 further includes: a data comparison unit and a second log deduplication unit;

[0124] The data comparison unit is used to compare the traffic data in each two preliminary event logs field by field after determining the total data volume of the traffic data in the plurality of preliminary event logs, if the total data volume does not exceed the preset data volume threshold, to obtain a data comparison result;

[0125] The second log deduplication unit is used to perform deduplication processing on the multiple preliminary event logs based on the data comparison result to obtain at least one event log to be transmitted.

[0126] Optionally, the traffic data deduplication unit is specifically used for:

[0127] Dividing the traffic data in the event log to be transmitted into fields to obtain a plurality of divided field data;

[0128] Compare every two adjacent divided field data to obtain a field comparison result;

[0129] The traffic data is deduplicated based on the field comparison result.

[0130] Optionally, the data processing module 320 includes: a symmetric encryption algorithm unit, a hash calculation unit, and an encrypted file determination unit;

[0131] The symmetric encryption algorithm unit is used to encrypt the compressed file by using a symmetric encryption algorithm to obtain a transmission encrypted file;

[0132] The hash calculation unit is used to perform hash calculation on the target encrypted file through a hash algorithm to obtain a target hash value;

[0133] The encrypted file determining unit is used to determine the target encrypted file corresponding to the compressed file based on the transmission encrypted file and the target hash value.

[0134] Optionally, the target encrypted file includes the transmission encrypted file and the target hash value corresponding to the transmission encrypted file; the data transmission device further includes:

[0135] A transmission verification module is used to enable the target data receiving end to verify the target hash value after the target encrypted file is transmitted to the target data receiving end through the fast-open transmission control protocol, obtain a verification result, and determine the target transmission data based on the verification result and the transmission encrypted file.

[0136] Optionally, the data processing module 320 includes:

[0137] The lossless compression unit is used to compress at least one of the target event logs using a lossless compression algorithm to obtain a compressed file.

[0138] The data transmission device provided in the embodiment of the present invention can execute the data transmission method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0139] Embodiment 4

[0140] Figure 5A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0141] like Figure 5 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0142] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0143] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs the various methods and processes described above, such as a data transmission method.

[0144] In some embodiments, the data transmission method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the data transmission method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the data transmission method in any other appropriate manner (e.g., by means of firmware).

[0145] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0146] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0147] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0148] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0149] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0150] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0151] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0152] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A data transmission method, characterized in that: include: Acquire event log data of each distributed device, wherein the distributed device is used to monitor flow data of the target system, the event log data includes a plurality of preliminary event logs, and the preliminary event logs are generated based on the flow data; For the event log data of each of the distributed devices, deduplication processing is performed on the multiple preliminary event logs to obtain at least one target event log, the at least one target event log is compressed to obtain a compressed file, and the compressed file is encrypted to obtain a target encrypted file; The target encrypted file is transmitted to the target data receiving end via the fast-open transmission control protocol.

2. The method according to claim 1, characterized in that The preliminary event log includes the traffic data; the deduplication processing of the plurality of preliminary event logs to obtain at least one target event log includes: determining a total data volume of the traffic data in a plurality of the preliminary event logs; When the total data volume exceeds a preset data volume threshold, similarity recognition is performed on every two preliminary event logs using a pre-trained data recognition model to obtain a recognition result; Based on the identification result, deduplication processing is performed on the plurality of preliminary event logs to obtain at least one event log to be transmitted; For each of the event logs to be transmitted, deduplication processing is performed on the traffic data in the event logs to be transmitted to obtain the target event log corresponding to the event log to be transmitted.

3. The method according to claim 2, characterized in that The data recognition model obtained by pre-training performs similarity recognition on every two of the preliminary event logs to obtain a recognition result, including: Determining target prompt data, wherein the target prompt data at least includes description data of description content related to the similarity recognition threshold; The target prompt data and the plurality of preliminary event logs are input into the data recognition model to obtain the recognition result, wherein the recognition result is a result corresponding to the description content related to the similarity recognition threshold.

4. The method according to claim 2, characterized in that: After determining the total data volume of the traffic data in the plurality of preliminary event logs, the method further includes: When the total data volume does not exceed the preset data volume threshold, comparing the flow data in every two preliminary event logs field by field to obtain a data comparison result; Based on the data comparison result, deduplication processing is performed on the multiple preliminary event logs to obtain at least one event log to be transmitted.

5. The method according to claim 2, characterized in that: The deduplication processing of the traffic data in the transmission event log includes: Dividing the traffic data in the event log to be transmitted into fields to obtain a plurality of divided field data; Compare every two adjacent divided field data to obtain a field comparison result; The traffic data is deduplicated based on the field comparison result.

6. The method according to claim 1, characterized in that The step of encrypting the compressed file to obtain a target encrypted file comprises: Encrypting the compressed file by a symmetric encryption algorithm to obtain a transmission encrypted file; Performing hash calculation on the target encrypted file by using a hash algorithm to obtain a target hash value; A target encrypted file corresponding to the compressed file is determined based on the transmission encrypted file and the target hash value.

7. The method according to claim 6, characterized in that The target encrypted file includes the transmission encrypted file and the target hash value corresponding to the transmission encrypted file; After transmitting the target encrypted file to the target data receiving end through the fast-open transmission control protocol, the method further includes: The target data receiving end verifies the target hash value to obtain a verification result, and determines the target transmission data according to the verification result and the transmission encrypted file.

8. The method according to claim 1, characterized in that The step of compressing at least one of the target event logs to obtain a compressed file includes: At least one of the target event logs is compressed using a lossless compression algorithm to obtain a compressed file.

9. A data transmission device, characterized in that: include: A data acquisition module, configured to acquire event log data of each distributed device, wherein the distributed device is configured to monitor flow data of a target system, the event log data comprising a plurality of preliminary event logs, and the preliminary event logs are generated based on the flow data; A data processing module, configured to perform deduplication processing on the event log data of each of the distributed devices on the plurality of preliminary event logs to obtain at least one target event log, compress the at least one target event log to obtain a compressed file, and encrypt the compressed file to obtain a target encrypted file; The data transmission module is used to transmit the target encrypted file to the target data receiving end through the fast open transmission control protocol.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data transmission method according to any one of claims 1 to 8 when executed.

Images