Traffic log capturing method and device, electronic equipment and computer program product
By combining the original traffic data flow and crawling instruction data flow of network equipment, the target connection data flow is obtained, and the problem of high traffic log crawling cost in the prior art is solved, and low-cost traffic log data crawling is achieved.
Patent Information
- Application Number
- CN202411795770.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-06
- Publication Date
- 2025-04-22
AI Technical Summary
The existing traffic log crawling method is costly and requires different crawling processes to be set up for network devices with different IP addresses.
By obtaining the original traffic data stream and the crawling instruction data stream of each network device of the target user, the data convergence process is performed, the target connection data stream is obtained, and the traffic log data of the key IP address is crawled according to the data stream.
The cost of crawling traffic logs is reduced, different crawling processes are avoided for each network device, and the original traffic data of each network device is centrally processed to obtain traffic data of all key IP addresses.
Smart Images
Figure CN119865334A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of network security technology, and in particular, relates to a method, device, electronic device and computer program product for capturing traffic logs. Background Art
[0002] In order to improve network security, it is usually necessary to obtain traffic logs of key IP addresses, so as to identify the risks of key IP addresses by analyzing the traffic logs.
[0003] In order to obtain the traffic logs of the required key IP addresses, the current common method is to set up different traffic log capture processes for network devices with different IP addresses. However, setting up different traffic log capture processes requires a lot of development and operation and maintenance costs, which increases the cost of capturing traffic logs. Summary of the invention
[0004] In view of this, the embodiments of the present application provide a method, device, electronic device and computer program for capturing traffic logs to solve the technical problem of high cost in existing traffic log capture.
[0005] In a first aspect, an embodiment of the present application provides a method for capturing a traffic log, including:
[0006] Obtaining the original traffic data stream and the capture instruction data stream of each network device of the target user, wherein the capture instruction data stream is used to capture the traffic data of the key IP address;
[0007] Performing data confluence processing on each of the original traffic data streams and the capture instruction data stream to obtain a target connection data stream;
[0008] According to the target connection data flow, the traffic log data of the key IP address is captured.
[0009] Optionally, the performing data confluence processing on each of the original traffic data streams and the capture instruction data stream to obtain a target connection data stream includes:
[0010] Performing data confluence processing on each of the original traffic data streams to obtain an initial connection data stream;
[0011] Performing data confluence processing on the initial connection data stream and the capture instruction data stream to obtain a target connection data stream;
[0012] Optionally, performing data confluence processing on each of the original traffic data streams to obtain an initial connection data stream includes:
[0013] Extracting preset fields from each of the original traffic data streams, and standardizing the field values of each extracted field to obtain a standard traffic data stream for each of the original traffic data streams;
[0014] Data confluence processing is performed on each of the standard traffic data flows to obtain the initial connection data flow.
[0015] Optionally, capturing the traffic log data of the key IP address according to the target connection data flow includes:
[0016] According to the target connection data stream, obtaining a mapping state element corresponding to the capture instruction data stream;
[0017] Determining the first IP field value of each of the standard traffic data flows respectively according to the target connection data flow;
[0018] For each of the standard traffic data streams, determine whether the mapping status element meets the capture conditions for the standard traffic data stream based on the mapping status element and the first IP field value of the standard traffic data stream. If the capture conditions are met, determine the capture result of the traffic log data based on the mapping status element and the standard traffic data stream.
[0019] Optionally, judging whether the mapping state element satisfies a capture condition for the standard traffic data flow according to the mapping state element and the first IP field value of the standard traffic data flow includes:
[0020] If the second IP field value in the mapping status element is equal to the first IP field value of the standard traffic data flow, and the time difference between the mapping status element and the standard traffic data is less than the time field value in the mapping status element, then it is determined that the mapping status element meets the capture conditions for the standard traffic data flow.
[0021] Optionally, capturing the traffic log data of the key IP address according to the target connection data flow includes:
[0022] In the process of capturing the traffic log data, the mapping state element corresponding to the capture instruction data flow in the target connection data flow is updated.
[0023] Optionally, updating the mapping state element corresponding to the capture instruction data stream in the target connection data stream includes:
[0024] Determine the issuing time and validity period of the crawling instruction data stream;
[0025] Determine the instruction update time according to the issuance time and the validity period;
[0026] When the instruction update time is reached, the mapping state element is updated.
[0027] In a second aspect, an embodiment of the present application provides a traffic log capture device, including:
[0028] A data flow acquisition unit, used to acquire the original traffic data flow and the capture instruction data flow of each network device of the target user, wherein the capture instruction data flow is used to capture the traffic data of the key IP address;
[0029] A data confluence unit, used for performing data confluence processing on each of the original traffic data streams and the capture instruction data stream to obtain a target connection data stream;
[0030] The traffic log capture unit is used to capture the traffic log data of the key IP address according to the target connection data flow.
[0031] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the processor implements each step of the method for capturing traffic logs as described in any one of the first aspects above.
[0032] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements each step of the method for capturing a traffic log as described in any one of the first aspects above.
[0033] In a fifth aspect, an embodiment of the present application provides a computer program product. When the computer program product is run on an electronic device, the electronic device executes each step of the method for capturing a traffic log as described in any one of the first aspects above.
[0034] The traffic log capture method, device, electronic device, and computer program product provided by the embodiments of the present application have the following beneficial effects:
[0035] In the method for capturing traffic logs provided in the embodiment of the present application, the original traffic data stream and the capture instruction data stream of each network device of the target user are first obtained, and then the original traffic data stream and the capture instruction data stream are processed for data confluence to obtain the target connection data stream, and finally the traffic log data of the key IP address is captured according to the target connection data stream. Through the method for capturing traffic logs of the present application, the original traffic data stream and the capture instruction data stream of each network device can be first merged to obtain the target connection data stream, and then the traffic log data of the key IP address can be captured according to the target connection data stream. Therefore, the original traffic data streams of each network device can be centrally processed to obtain the traffic data of all key IP addresses, without setting a different traffic log capture process for each network device, thereby reducing the capture cost of the traffic log. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0037] Figure 1 A flow chart of a method for capturing traffic logs provided in an embodiment of the present application;
[0038] Figure 2 An example diagram of a mapping state element corresponding to an instruction data stream provided in an embodiment of the present application;
[0039] Figure 3 An example diagram of a standard traffic data flow provided in an embodiment of the present application;
[0040] Figure 4 An example diagram of a result of capturing traffic log data provided in an embodiment of the present application;
[0041] Figure 5 A schematic diagram of an original traffic data flow provided in an embodiment of the present application;
[0042] Figure 6 A schematic diagram of another original traffic data flow provided in an embodiment of the present application;
[0043] Figure 7 A schematic diagram of the structure of a traffic log capture device provided in an embodiment of the present application;
[0044] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0045] It should be noted that the terms used in the embodiments of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application. In the description of the embodiments of the present application, unless otherwise specified, "multiple" refers to two or more than two, and "at least one", "one or more" refers to one, two or more. The terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, it is defined that the "first" and "second" features can explicitly or implicitly include one or more of the features.
[0046] References to "one embodiment" or "some embodiments" etc. described in this specification mean that a particular feature, structure or characteristic described in conjunction with the embodiment is included in one or more embodiments of the present application. Thus, the phrases "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear at different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.
[0047] The execution subject of the method for capturing traffic logs provided in the embodiment of the present application may be an electronic device. Exemplarily, the electronic device may include but is not limited to a back-end device of a network security platform. The electronic device may execute each step of the method for capturing traffic logs provided in the embodiment of the present application. Specifically, the electronic device may include but is not limited to mobile phones, tablet computers, laptop computers, and desktop computers.
[0048] The method for capturing traffic logs provided in the embodiments of the present application can be applied to various scenarios where traffic log data of key IP addresses need to be captured. For example, when an electronic device needs to capture traffic log data of a key IP address, the electronic device can execute the various steps of the method for capturing traffic logs provided in the embodiments of the present application, thereby being able to capture traffic log data of the key IP address at a low cost.
[0049] See also Figure 1 , Figure 1 A flow chart of a method for capturing a traffic log provided in an embodiment of the present application is provided. The method for capturing a traffic log may include S101 to S103, which are described in detail as follows:
[0050] In S101, the original traffic data stream and the capture instruction data stream of each network device of the target user are obtained.
[0051] In an embodiment of the present application, when it is necessary to capture the traffic log data of a key IP address, the electronic device may first obtain the original traffic data stream of each network device belonging to the target user, and obtain the capture instruction data stream.
[0052] Among them, the capture instruction data flow is used to capture the traffic data of key IP addresses.
[0053] Exemplarily, the electronic device can obtain the key IP address traffic capture instructions of the front-end page of the network security platform through a preset connector, and can generate a capture instruction data stream through the obtained key IP address traffic capture instructions.
[0054] In S102, data confluence processing is performed on each original traffic data stream and the capture instruction data stream to obtain a target connection data stream.
[0055] In an embodiment of the present application, after acquiring each original traffic data stream and the capture instruction data stream, the electronic device can perform data confluence processing on each original traffic data stream and the capture instruction data stream to obtain a target connection data stream.
[0056] In a possible implementation, the electronic device can implement data confluence processing on each original traffic data stream and the capture instruction data stream through step a and step b, as described in detail as follows:
[0057] In step a, data confluence processing is performed on each original traffic data stream to obtain an initial connection data stream.
[0058] In this implementation, when the electronic device needs to perform data confluence processing on each original traffic data stream and the capture instruction data stream, it can first perform data confluence processing on each original traffic data stream to obtain an initial connection data stream.
[0059] Specifically, the electronic device may extract preset fields from each original traffic data stream, and standardize the field values of each extracted field to obtain a standard traffic data stream for each original traffic data stream.
[0060] After obtaining the standard traffic data streams of each original traffic data stream, the electronic device can perform data confluence processing on each standard traffic data stream to obtain an initial connection data stream. The specific method of performing data confluence processing on each standard traffic data stream can be set according to actual needs and is not limited here.
[0061] In step b, the initial connection data stream and the capture instruction data stream are merged to obtain a target connection data stream.
[0062] In this implementation, after obtaining the initial connection data stream, the electronic device can perform data confluence processing on the initial connection data stream and the capture instruction data stream to obtain the target connection data stream. The specific method of performing data confluence processing on the initial connection data stream and the capture instruction data stream can be set according to actual needs and is not limited here.
[0063] In S103, traffic log data of key IP addresses is captured according to the target connection data flow.
[0064] In an embodiment of the present application, after performing data merging processing on each original traffic data stream and the capture instruction data stream to obtain the target connection data stream, the electronic device can capture the traffic log data of the key IP address according to the target connection data stream.
[0065] In the implementation corresponding to the above step a and step b, the electronic device can capture the traffic log data of the key IP address according to the target connection data flow.
[0066] In a possible implementation, the electronic device can capture the traffic log data of the key IP address according to the target connection data flow through steps c to e, as described in detail as follows:
[0067] In step c, a mapping state element corresponding to the crawl instruction data stream is obtained according to the target connection data stream.
[0068] In this implementation, the electronic device can obtain the mapping state element corresponding to the capture instruction data stream according to the target connection data stream.
[0069] For example, the mapping state element corresponding to the crawl instruction data stream may be "stateJson", see Figure 2 , Figure 2 An example diagram of a mapping state element corresponding to an instruction data stream provided in an embodiment of the present application.
[0070] exist Figure 2 In the described example, the "duration" field indicates the duration of the captured traffic in seconds; the "action" field defines the type of the capture operation, where 0 represents stop capturing and 1 represents start capturing; the "ipType" field is used to specify the matching method with the IP type in the traffic data logJson, 0 represents matching with "srcIp" and 1 represents matching with "dstIp".
[0071] In step d, the first IP field value of each standard traffic data flow is determined respectively according to the target connection data flow.
[0072] In this implementation, the electronic device can determine the first IP field value of each standard traffic data flow according to the target connection data flow.
[0073] For example, see Figure 3 , Figure 3 An example diagram of a standard traffic data flow provided in an embodiment of the present application is shown in Figure 3 The schematic diagram of FIG. 1 shows a sample of a standard traffic data stream logJson after the extraction of the preset fields in step a and the standardization of the field values of each field. Figure 3 In the schematic diagram, the first IP field value of each standard traffic data stream logJson can be "srcIp".
[0074] In step e, for each standard traffic data stream, determine whether the mapping status element meets the capture conditions for the standard traffic data stream based on the mapping status element and the first IP field value of the standard traffic data stream. If the capture conditions are met, determine the capture result of the traffic log data based on the mapping status element and the standard traffic data stream.
[0075] In this implementation, after obtaining the mapping status element corresponding to the capture instruction data stream and the first IP field value of each standard traffic data stream, the electronic device can, for each standard traffic data stream, determine whether the mapping status element meets the capture conditions for the standard traffic data stream based on the mapping status element and the first IP field value of the standard traffic data stream. If it is determined that the mapping status element meets the capture conditions for the standard traffic data stream, the electronic device can determine the capture result of the traffic log data based on the mapping status element and the standard traffic data stream; if it is determined that the mapping status element does not meet the capture conditions for the standard traffic data stream, the electronic device may not determine the capture result of the traffic log data.
[0076] In one possible implementation, the judgment conditions may include whether the second IP field value in the mapping status element is equal to the first IP field value of the standard traffic data flow, and whether the time difference between the mapping status element and the standard traffic data flow is less than the time field value in the mapping status element.
[0077] Based on this, the electronic device can determine that the mapping status element meets the capture conditions for the standard traffic data stream when it determines that the second IP field value in the mapping status element is equal to the first IP field value of the standard traffic data stream, and determines that the time difference between the mapping status element and the standard traffic data stream is less than the time field value in the mapping status element.
[0078] The electronic device can determine that the mapping status element does not meet the capture conditions for the standard traffic data stream when it determines that the second IP field value in the mapping status element is not equal to the first IP field value of the standard traffic data stream, or that the time difference between the mapping status element and the standard traffic data stream is not less than the time field value in the mapping status element.
[0079] exist Figure 2 and Figure 3 In the corresponding example, the second IP field value in the mapping state element is "10.10.10.221", and the first IP field value of the standard traffic data flow is "10.10.10.221". Figure 2 and Figure 3 In the corresponding example, the second IP field value in the mapping state element is equal to the first IP field value of the standard traffic data flow.
[0080] exist Figure 2 and Figure 3 In the corresponding example, the time difference between the mapping state element and the standard traffic data flow is the difference between the timestamp field of stateJson and the logTimestamp field of logJson. Therefore, the time difference between the mapping state element and the standard traffic data flow can be calculated as
[0081] logJson.logTimestamp–stateJson.timestamp=1727662845000–1727662641953=203047 (milliseconds)=203.047 (seconds). In addition, Figure 2 In the example, the time field value "duration" in the mapping status element is 300 (seconds), so the time difference between the mapping status element and the standard traffic data flow is less than the time field value in the mapping status element.
[0082] It can be seen that in Figure 2 and Figure 3 In the corresponding example, the electronic device can determine that the mapping state element satisfies the capture condition for the standard traffic data stream, and thus can determine the capture result of the traffic log data according to the mapping state element and the standard traffic data stream.
[0083] In the embodiment of the present application, the specific method of determining the capture result of the traffic log data according to the mapping state element and the standard traffic data flow can be set according to actual needs and is not limited here.
[0084] For example, see Figure 4 , Figure 4 This is an example diagram of a result of capturing traffic log data provided by an embodiment of the present application. Figure 4The electronic device is shown in Figure 2 The mapping state elements shown and Figure 3 The standard traffic data stream shown is an example of the captured result of the traffic log data constructed.
[0085] After determining the capture result of the traffic log data according to the mapping state element and the standard traffic data flow, the electronic device can send the capture result of the traffic log data to the network security platform.
[0086] In a possible implementation, the electronic device may also update a mapping state element corresponding to the capture instruction data stream in the target connection data stream while capturing the traffic log data of the key IP address according to the target connection data stream.
[0087] Specifically, the electronic device can determine the issuing time and effective time limit of the grab instruction data stream. After determining the issuing time and effective time limit of the grab instruction data stream, it can determine the instruction update time according to the issuing time and effective time limit of the grab instruction data stream, and when the instruction update time is reached, the mapping status element corresponding to the grab instruction data stream is mapped.
[0088] Among them, the field value "action" in the mapping status element corresponding to the grab instruction data stream can be used to determine the type of the grab instruction. For example, when the field value "action" in the mapping status element corresponding to the grab instruction data stream is 0, the type of the grab instruction can be stop grabbing; when the field value "action" in the mapping status element corresponding to the grab instruction data stream is 1, the type of the grab instruction can be start grabbing.
[0089] In this implementation, the mapping state element corresponding to the grab instruction data stream can be updated by setting a timer with a duration equal to the effective time limit of the grab instruction data stream. Exemplarily, the effective time limit of the grab instruction data stream can be 300 seconds. Based on this, the electronic device can set a timer with a duration of 300 seconds. When the timer determines that the sending time of the grab instruction data stream has reached 300 seconds, the electronic device can update the mapping state element corresponding to the grab instruction data stream and delete the timer.
[0090] By using a timer to update the mapping state element corresponding to the capture instruction data stream, it is possible to ensure that the traffic log capture actions of the front-end and back-end of the network security platform are consistent.
[0091] The following Figure 2 Taking the mapping state element corresponding to the grab instruction data flow shown as an example, the specific steps of updating the mapping state element are exemplarily described:
[0092] First, we can add new elements to the map state element. Figure 2 For example, when adding a new element to the mapping state element, taskID="375479fd-8495-4f84-8372-ec4649edf3a7" can be used as the key and ipJson itself can be used as the value to add it.
[0093] In addition, you can also register a timer, specifically, Figure 2 For example, timestamp = 1727662641953, duration = 300. Figure 2 For ipJson in, you can register a timestamp+
[0094] The timer is triggered at duration*1000=1727662941953 (this is a 13-bit timestamp, converted to GMT+8 time, which is 2024-09-30 10:22:21).
[0095] Afterwards, the mapping state element corresponding to the capture instruction data stream can be updated through the timer. Specifically, at the time when the timer is triggered (1727662941953), the electronic device can traverse the element curstateJson in the mapping state element corresponding to the capture instruction data stream. If 1727662941953>=curstateJson.timestamp+curstateJson.duration*1000, curstateJson can be deleted from the mapping state element.
[0096] Through the above process, the mapping state element corresponding to the capture instruction data stream can be updated by setting a timer.
[0097] The following is an example of a method for capturing traffic logs provided in an embodiment of the present application.
[0098] The network devices of the target user may be network device 1 and network device 2, the original traffic data stream corresponding to network device 1 may be logJson_1, and the original traffic data stream corresponding to network device 2 may be logJson_2.
[0099] See also Figure 5 , Figure 5 A schematic diagram of an original traffic data flow provided in an embodiment of the present application, Figure 5 The raw traffic data stream shown can be logJson_1; see Figure 6 , Figure 6 A schematic diagram of another original traffic data flow provided in an embodiment of the present application, Figure 6The raw traffic data stream shown may be logJson_2.
[0100] In addition, the grab instruction data stream obtained by the electronic device can be as follows Figure 2 Based on Figure 2 The fetch instruction data flow shown, Figure 5 The original traffic data flow logJson_1 and Figure 6 The original traffic data stream logJson_2 shown in the figure can be determined according to the relevant description of step e in the embodiment of the present application: Figure 2 The mapping state element corresponding to the capture instruction data flow shown meets the capture condition for logJson_1, and Figure 2 The mapping state element corresponding to the capture instruction data flow shown does not meet the capture condition for logJson_2, so the electronic device can Figure 2 The mapping state elements shown and Figure 5 The standard traffic data stream corresponding to the original traffic data stream shown determines the capture result of the traffic log data.
[0101] exist Figure 2 The crawling instruction data flow in the start time is issued, and the crawling countdown of the front-end page of the network security platform ends after 300 seconds. Figure 2 From the "duration" field of the crawling instruction data stream, we can know that the effective time limit of the crawling instruction is also 300 seconds. Therefore, 300 seconds after the crawling instruction data stream is issued, the crawling instruction data stream will be deleted according to the timer mechanism, thereby realizing the unification of the front-end and back-end traffic log crawling actions of the network security platform.
[0102] From the above, it can be seen that in the method for capturing traffic logs provided in the embodiment of the present application, the original traffic data stream and the capture instruction data stream of each network device of the target user are first obtained, and then the original traffic data stream and the capture instruction data stream are processed for data confluence to obtain the connection data stream, and finally the traffic log data of the key IP address is captured according to the connection data stream. Through the method for capturing traffic logs of the present application, the original traffic data stream and the capture instruction data stream of each network device can be first merged to obtain the connection data stream, and then the traffic log data of the key IP address can be captured according to the connection data stream. Therefore, the original traffic data streams of each network device can be centrally processed to obtain the traffic data of all key IP addresses, without setting a different traffic log capture process for each network device, thereby reducing the capture cost of the traffic log.
[0103] Based on the method for capturing traffic logs provided in the above embodiment, the present application further provides a device for capturing traffic logs to implement the above method embodiment. Figure 7 , Figure 7 A schematic diagram of the structure of a traffic log capture device provided in an embodiment of the present application. Figure 7 As shown, the traffic log capture device 70 may include: a data stream acquisition unit 71, a data converging unit 72 and a traffic log capture unit 73. Among them:
[0104] The data flow acquisition unit 71 is used to acquire the original traffic data flow and the capture instruction data flow of each network device of the target user. The capture instruction data flow is used to capture the traffic data of the key IP address.
[0105] The data confluence unit 72 is used to perform data confluence processing on each original traffic data stream and the capture instruction data stream to obtain a target connection data stream.
[0106] The traffic log capturing unit 73 is used to capture the traffic log data of the key IP address according to the target connection data flow.
[0107] Optionally, the data merging unit 72 is specifically configured to:
[0108] Perform data confluence processing on each original traffic data stream to obtain an initial connection data stream;
[0109] Performing data confluence processing on the initial connection data stream and the capture instruction data stream to obtain the target connection data stream;
[0110] Optionally, the traffic log capturing unit 73 is specifically used for:
[0111] Based on the target connection data flow, capture the traffic log data of key IP addresses.
[0112] Optionally, the data merging unit 72 is specifically configured to:
[0113] Extracting preset fields from each original traffic data stream, and standardizing the field values of each extracted field to obtain a standard traffic data stream for each original traffic data stream;
[0114] Perform data confluence processing on each standard traffic data stream to obtain an initial connection data stream.
[0115] Optionally, the traffic log capturing unit 73 is specifically used for:
[0116] According to the target connection data stream, obtain the mapping state element corresponding to the crawl instruction data stream;
[0117] Determine the first IP field value of each standard traffic data flow according to the target connection data flow;
[0118] For each standard traffic data stream, determine whether the mapping status element meets the capture conditions for the standard traffic data stream based on the mapping status element and the first IP field value of the standard traffic data stream. If the capture conditions are met, determine the capture results of the traffic log data based on the mapping status element and the standard traffic data stream.
[0119] Optionally, the traffic log capturing unit 73 is specifically used for:
[0120] If the second IP field value in the mapping status element is equal to the first IP field value of the standard traffic data stream, and the time difference between the mapping status element and the standard traffic data is less than the time field value in the mapping status element, then it is determined that the mapping status element meets the capture conditions for the standard traffic data stream.
[0121] Optionally, the traffic log capturing unit 73 is specifically used for:
[0122] In the process of capturing traffic log data, the mapping state element corresponding to the capture instruction data flow in the target connection data flow is updated.
[0123] Optionally, the traffic log capturing unit 73 is specifically used for:
[0124] Determine the time and validity period of the capture instruction data stream;
[0125] Determine the instruction update time based on the issuance time and validity period;
[0126] When the instruction update time is reached, the map state element is updated.
[0127] See also Figure 8 , Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 8 As shown, the electronic device 8 provided in this embodiment may include: a processor 80, a memory 81, and a computer program 82 stored in the memory 81 and executable on the processor 80, such as a program corresponding to the method for capturing traffic logs. When the processor 80 executes the computer program 82, the steps in the embodiment of the method for capturing traffic logs are implemented, such as Figure 1 Alternatively, when the processor 80 executes the computer program 82, the functions of each module / unit in the above-mentioned traffic log capture device embodiment are realized, for example Figure 7 The functions of the units 71 to 73 are shown.
[0128] Exemplarily, the computer program 82 may be divided into one or more modules / units, one or more modules / units are stored in the memory 81 and executed by the processor 80 to complete the present application. One or more modules / units may be a series of computer program instruction segments that can complete specific functions, and the instruction segments are used to describe the execution process of the computer program 82 in the electronic device 8. For example, the computer program 82 may be divided into a data stream acquisition unit, a data confluence unit, and a traffic log capture unit. For the specific functions of each unit, please refer to Figure 7 The relevant descriptions in the corresponding embodiments are not repeated here.
[0129] Those skilled in the art will understand that Figure 8 The electronic device 8 is merely an example and does not limit the electronic device 8 , and may include more or less components than those shown in the figure, or may combine certain components, or may include different components.
[0130] The processor 80 may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.
[0131] The memory 81 may be an internal storage unit of the electronic device 8, such as a hard disk or memory of the electronic device 8. The memory 81 may also be an external storage device of the electronic device 8, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, or a flash card, etc., equipped on the electronic device 8. Further, the memory 81 may also include both an internal storage unit of the electronic device 8 and an external storage device. The memory 81 is used to store computer programs and other programs and data required by the electronic device. The memory 81 may also be used to temporarily store data that has been output or is to be output.
[0132] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units is used as an example for illustration. In actual applications, the above-mentioned functions can be assigned to different functional units as needed, that is, the internal structure of the traffic log capture device can be divided into different functional units to complete all or part of the functions described above. The functional units in the embodiment can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.
[0133] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.
[0134] An embodiment of the present application provides a computer program product. When the computer program product is executed on a terminal device, the terminal device implements the steps in the above-mentioned various method embodiments.
[0135] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0136] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0137] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.
Claims
1. A method for capturing traffic logs, characterized in that: include: Obtaining the original traffic data stream and the capture instruction data stream of each network device of the target user, wherein the capture instruction data stream is used to capture the traffic data of the key IP address; Performing data confluence processing on each of the original traffic data streams and the capture instruction data stream to obtain a target connection data stream; According to the target connection data flow, the traffic log data of the key IP address is captured.
2. The method according to claim 1, characterized in that The step of performing data confluence processing on each of the original traffic data streams and the capture instruction data stream to obtain a target connection data stream includes: Performing data confluence processing on each of the original traffic data streams to obtain an initial connection data stream; The initial connection data stream and the capture instruction data stream are subjected to data confluence processing to obtain the target connection data stream.
3. The method according to claim 2, characterized in that The step of performing data confluence processing on each of the original traffic data streams to obtain an initial connection data stream includes: Extracting preset fields from each of the original traffic data streams, and standardizing the field values of each extracted field to obtain a standard traffic data stream for each of the original traffic data streams; Data confluence processing is performed on each of the standard traffic data flows to obtain the initial connection data flow.
4. The method according to claim 3, characterized in that The capturing of the traffic log data of the key IP address according to the target connection data flow includes: According to the target connection data stream, obtaining a mapping state element corresponding to the capture instruction data stream; Determining the first IP field value of each of the standard traffic data flows respectively according to the target connection data flow; For each of the standard traffic data streams, determine whether the mapping status element meets the capture conditions for the standard traffic data stream based on the mapping status element and the first IP field value of the standard traffic data stream. If the capture conditions are met, determine the capture result of the traffic log data based on the mapping status element and the standard traffic data stream.
5. The method according to claim 4, characterized in that The determining, according to the mapping state element and the first IP field value of the standard traffic data flow, whether the mapping state element satisfies a capture condition for the standard traffic data flow comprises: If the second IP field value in the mapping status element is equal to the first IP field value of the standard traffic data flow, and the time difference between the mapping status element and the standard traffic data is less than the time field value in the mapping status element, then it is determined that the mapping status element meets the capture conditions for the standard traffic data flow.
6. The method according to claim 2, characterized in that The capturing of the traffic log data of the key IP address according to the target connection data flow includes: In the process of capturing the traffic log data, the mapping state element corresponding to the capture instruction data flow in the target connection data flow is updated.
7. The method according to claim 6, characterized in that The updating of the mapping state element corresponding to the capture instruction data stream in the target connection data stream includes: Determine the issuing time and validity period of the crawling instruction data stream; Determine the instruction update time according to the issuance time and the validity period; When the instruction update time is reached, the mapping state element is updated.
8. A device for capturing traffic logs, characterized in that: include: A data flow acquisition unit, used to acquire the original traffic data flow and the capture instruction data flow of each network device of the target user, wherein the capture instruction data flow is used to capture the traffic data of the key IP address; A data confluence unit, used for performing data confluence processing on each of the original traffic data streams and the capture instruction data stream to obtain a target connection data stream; The traffic log capture unit is used to capture the traffic log data of the key IP address according to the target connection data flow.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, each step of the method for capturing traffic logs as described in any one of claims 1 to 7 is implemented.
10. A computer program product, characterized in that When the computer program product is executed by a processor, each step of the method for capturing traffic logs as claimed in any one of claims 1 to 7 is implemented.