Data processing method and device, electronic equipment, storage medium and program product

Abstract

The embodiment of the invention discloses a data processing method and device, electronic equipment, a storage medium and a program product. The method comprises the steps of receiving a to-be-processed data packet in a data stream in a process of detecting the data stream; under the condition that the to-be-processed data packet meets the cache triggering condition, determining a segment sequence type of the to-be-processed data packet according to the attribute information of the to-be-processed data packet, and performing cache recombination on the to-be-processed data packet according to the segment sequence type and a cache flow structure corresponding to the data flow; and under the condition that the continuous data buffer area meets the rule detection triggering condition, determining an abnormal rule hit result of the data stream to which the service data belongs according to the cached service data in the continuous data buffer area, so as to process the data stream based on the abnormal rule hit result. According to the technical scheme, under the condition that flow detection is carried out in a single packet processing mode, the received data packets are cached and recombined in order, so that rule detection is carried out on the recombined service data.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a data processing method, apparatus, electronic device, storage medium, and program product. Background Technology

[0002] In network security applications such as metropolitan area network (MAN) egress and network traffic inspection, Deep Packet Inspection (DPI) technology is widely used in network security devices such as firewalls and intrusion detection systems as a core means of identifying network traffic and detecting malicious data. To adapt to the demands of high-concurrency, high-volume network data processing, existing DPI systems mostly adopt a single-packet inspection architecture. However, this single-packet inspection architecture exhibits significant technical shortcomings when processing TCP-based encrypted application layer traffic: when a client sends a large protocol packet that is segmented by the TCP protocol, and key detection fields such as SNI do not appear in the first segment, single-packet inspection cannot extract the complete fields due to the lack of flow-level reassembly capabilities, thus failing to hit the security detection rules.

[0003] In related technologies, when using a single-packet inspection architecture to process TCP-based encrypted application layer traffic, the common methods are complete TCP protocol stack reassembly or deep caching of stream data. However, these methods may lead to a surge in resource consumption, a significant decrease in system concurrency, and system memory pressure. Summary of the Invention

[0004] This invention provides a data processing method, apparatus, electronic device, storage medium, and program product to achieve the effect of orderly caching and reassembling of received data packets in the case of traffic detection using a single-packet processing mode, so as to perform rule detection on the reassembled business data.

[0005] According to one aspect of the present invention, a data processing method is provided, wherein the flow detection device performs flow detection using a single-packet processing mode, the method comprising: During the detection of the data stream, a data packet to be processed is received from the data stream; wherein the data packet to be processed carries the service data to be detected. When the data packet to be processed meets the buffer triggering condition, the segment order type of the data packet to be processed during transmission is determined according to the attribute information of the data packet to be processed, and the data packet to be processed is buffered and reassembled according to the segment order type and the buffer stream structure corresponding to the data stream; wherein, the segment order type includes sequential type and out-of-order type; the buffer stream structure includes a continuous data buffer and an out-of-order segmented linked list; When the continuous data buffer meets the rule detection triggering condition, the abnormal rule hit result of the data stream to which it belongs is determined based on the business data already cached in the continuous data buffer, and the data stream is processed based on the abnormal rule hit result.

[0006] According to another aspect of the present invention, a data processing apparatus is provided, configured in a flow detection device, the flow detection device performing flow detection in a single-packet processing mode, the apparatus comprising: The data packet receiving module is used to receive data packets to be processed in the data stream during the detection of the data stream; wherein the data packets to be processed carry service data to be detected. A data packet caching module is used to determine the segment order type of the data packet during transmission based on the attribute information of the data packet when the data packet to be processed meets the caching trigger condition, and to perform cache reassembly on the data packet according to the segment order type and the cache stream structure corresponding to the data stream; wherein, the segment order type includes sequential type and out-of-order type; the cache stream structure includes a continuous data buffer and an out-of-order segmented linked list; The data stream matching module is used to determine the abnormal rule hit result of the data stream to which the continuous data buffer belongs, based on the business data already cached in the continuous data buffer, when the rule detection trigger condition is met, so as to process the data stream based on the abnormal rule hit result.

[0007] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: One or more processors; Storage device for storing one or more programs. When one or more programs are executed by one or more processors, the one or more processors implement any of the data processing methods described in the embodiments of this disclosure.

[0008] According to another aspect of the present invention, a computer-readable storage medium is provided, which stores computer instructions for causing a processor to execute and implement any of the data processing methods of the present invention.

[0009] According to another aspect of the present disclosure, a computer program product is provided, which, when executed by a processor, implements any of the data processing methods described in the embodiments of the present disclosure.

[0010] The technical solution of this disclosure, during the detection of a data stream, receives data packets to be processed within the data stream and acquires traffic data in real time, providing a data source for subsequent cache reassembly and anomaly detection, ensuring the normal startup of the detection process. Furthermore, when a data packet to be processed meets the cache triggering condition, the segment sequence type of the data packet during transmission is determined based on its attribute information. Then, based on the segment sequence type and the cache stream structure corresponding to the data stream, the data packet to be processed is cache reassembly performed, distinguishing between ordered and out-of-order packets, and storing them respectively in a continuous data buffer and an out-of-order segmented linked list, achieving ordered data packet splicing and ensuring the integrity and usability of business data. Furthermore, when the continuous data buffer meets the rule detection triggering condition, the anomaly rule hit result of the data stream to which it belongs is determined based on the cached business data in the continuous data buffer. The data stream is processed based on the anomaly rule hit result, and rule matching is performed based on the cached business data, ensuring the accuracy and completeness of anomaly detection, avoiding misjudgments or omissions due to missing or out-of-order data, thereby achieving precise and reliable security control of the data stream. The technical solution provided in this disclosure solves the problems of increased resource consumption, significant decrease in system concurrent processing capability, and system memory pressure in related technologies. It achieves the effect of orderly caching and reassembling of received data packets in traffic detection using a single packet processing mode, so as to perform rule detection on the reassembled business data. This ensures the accuracy of abnormal rule detection while significantly reducing system resource consumption, improving device concurrent processing capability, and alleviating system memory pressure.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 A schematic flowchart illustrating a data processing method provided in an embodiment of this disclosure; Figure 2 This is a schematic diagram of the structure of a data processing apparatus provided in an embodiment of the present disclosure; Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. Detailed Implementation

[0014] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0015] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0016] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0017] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0018] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0019] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0020] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0021] This paper first provides an illustrative example of the application scenarios of this technical solution: This technical solution is suitable for high-concurrency, low-latency traffic detection scenarios in the field of network security, especially for TCP protocol traffic detection needs in core business scenarios such as anti-fraud and network anomaly behavior detection. Specifically, it can be deployed in traffic detection devices such as operator backbone network exits, enterprise border gateways, and anti-fraud detection probes. The traffic detection device can use a single-packet processing mode to perform real-time detection of massive network data streams, solving the problems of high resource consumption, large detection latency, and long-term memory occupation of invalid streams in traditional TCP segmentation and caching reassembly schemes. It can accurately identify TCP segmented data packets of application layer protocols such as HTTP / TLS, achieve efficient reassembly of ordered / out-of-order segmented data through a lightweight cached stream structure, and complete anomaly rule detection based on multi-dimensional trigger conditions. At the same time, it relies on the flow timeout management mechanism to achieve fallback control of cached resources. Under the premise of ensuring the processing capacity of millions of data packets per second, it accurately extracts complete business data fields for anomaly rule matching, effectively identifying and handling abnormal network data streams.

[0022] Figure 1 This is a flowchart illustrating a data processing method provided in an embodiment of this disclosure. This embodiment is applicable to situations involving security detection of network traffic. The method can be executed by a data processing device in a traffic detection device. This data processing device can be implemented in hardware and / or software and can be configured in electronic devices such as computers or servers. Figure 1 As shown, the method in this embodiment includes: S110. During the detection of the data stream, receive the data packets to be processed in the data stream; wherein the data packets to be processed carry the business data to be detected.

[0023] Traffic detection devices can be understood as network devices deployed at network transmission nodes to collect, parse, identify, and perform security checks on data streams transmitted in the network in real time. Simply put, a traffic detection device can receive all passing network data packets, parse the information within the network data packets, determine whether the network data packets contain abnormal traffic, and determine the appropriate handling method—allowing or blocking—based on the abnormal traffic judgment result. Optionally, traffic detection devices include, but are not limited to, gateway devices, deep packet inspection devices, network anomaly detection devices, and traffic auditing devices. In this embodiment, the traffic detection device uses a single-packet processing mode for traffic detection. The single-packet processing mode is an efficient working mode for traffic detection devices to process packets: the traffic detection device can use a single data packet as the smallest processing unit, independently performing parsing, identification, and detection processes on each arriving data packet, without maintaining a complete and complex transport layer connection state machine. It only establishes a simplified temporary flow structure when necessary, and releases resources promptly after processing, thereby achieving high concurrency, low latency, and low resource consumption in traffic detection. A data stream can refer to a collection of related data packets generated by the same network transmission session, sharing the same source / destination IP address, source / destination port number, and transport layer protocol. In this embodiment, the data stream can primarily refer to a service stream transmitted via the TCP protocol, which may exhibit packet fragmentation and out-of-order delivery. A data packet to be processed can be understood as a single network data packet currently received by the traffic detection device that requires cache reassembly and rule detection; it is the smallest processing unit of the traffic detection device. That is, during the process of detecting the data stream through the traffic detection device, each data packet received by the traffic detection device and about to enter the cache reassembly process can be called a data packet to be processed. A data packet to be processed typically consists of two parts: protocol header information and service data. The data packet to be processed carries the service data to be detected. Service data can refer to the payload data in the data packet to be processed, excluding protocol header information such as IP headers and TCP headers, such as HTTP request content, TLS handshake messages, domain name information, etc., and is the actual object for anomaly detection.

[0024] In one implementation, during the real-time detection of data streams in the network by the traffic detection device, the device continuously receives and acquires the data packets to be processed within the data stream. These data packets, as the smallest unit of data processing, carry valid application layer information during network transmission. The device parses these data packets to obtain the business data used for subsequent anomaly rule matching and traffic detection, providing a data foundation for subsequent cache trigger judgment, segment type determination, and cache reorganization processing.

[0025] S120. When the data packet to be processed meets the buffer triggering condition, determine the segment sequence type of the data packet to be processed during the transmission process according to the attribute information of the data packet to be processed, and perform buffer reassembly on the data packet to be processed according to the segment sequence type and the buffer stream structure corresponding to the data stream.

[0026] The cache trigger condition can be understood as a pre-set judgment condition used to determine whether the currently processed data packet needs to enter the cache reassembly process. Generally, the subsequent segment type judgment and cache reassembly operation will only be performed on the data packet if the data packet meets the cache trigger condition. In this embodiment, the data packet that needs to enter the cache reassembly process is usually at least a data packet that does not contain complete protocol fields. Attribute information can be understood as relevant parameters that characterize the transmission characteristics of the data packet, parsed from the protocol header of the data packet. These mainly include the TCP sequence number indicating the transmission position and order of the data packet in the data stream, and flag bits used to distinguish data packet types. Segment type can be understood as classification information indicating the order and continuity of the data packet relative to the cached service data during transmission within its data stream. Optionally, segment type includes two categories: sequential and out-of-order. Sequential type can be understood as the sequence number of the data packet being processed being continuously connected to the cached service data, and can be directly appended after existing cached data, or the data packet being processed being the first data packet in its data stream. Out-of-order data can be understood as data whose sequence number is discontinuous with the cached business data, exhibiting lead, delay, or overlap, making direct concatenation impossible. In contrast, if the data to be processed and the cached business data are consecutive in sequence number, they can be directly concatenated; or if the data to be processed is the first data packet in its data stream, its segment type can be determined as sequential. If the data to be processed and the cached business data are discontinuous in sequence number, leading or lagging, their segment type can be determined as out-of-order. The buffered stream structure can be understood as a lightweight temporary storage structure allocated by the traffic detection device in single-packet processing mode for the currently processed data stream. It is used for temporary storage and ordered reassembly of segmented and out-of-order business data to recover complete and detectable business data without maintaining complex transmission connection states. The buffered stream structure includes at least two parts: a continuous data buffer and an out-of-order segmented linked list. The continuous data buffer can be a cache area in the buffered stream structure used to store sequential, directly concatenable business data. The data is stored continuously according to the transmission order and can be directly used for rule detection. Out-of-order segmented linked lists can be linked list structures in buffered stream structures used to temporarily store out-of-order data packets. They are used to store data packets that cannot be concatenated temporarily, and then concatenate them after subsequent data packets arrive.

[0027] During network data transmission, due to factors such as link transmission characteristics, network congestion, and the fragmentation mechanism of the protocol itself, data packets in the data stream often experience transmission fragmentation, out-of-order delivery, and incomplete packets when passing through traffic detection devices. To ensure high-concurrency processing capabilities, traffic detection devices typically employ a single-packet processing mode, processing only individual data packets independently without maintaining complex transmission connection states. This makes it difficult to directly extract complete business data from fragmented and out-of-order packets for effective detection. Existing technologies typically employ a full-stream caching mechanism to reassemble fragmented and out-of-order packets. This mechanism maintains a complete transport layer connection state for each data stream, uniformly caching, sorting, and reassembling all arriving data packets. However, this approach has significant drawbacks: firstly, it consumes enormous system resources, easily leading to excessive memory usage and increased processing latency in high-concurrency, high-traffic scenarios; secondly, it conflicts with the lightweight design philosophy of the single-packet processing mode, requiring the maintenance of complex session states and timeout mechanisms, resulting in a decline in overall device processing performance and making it difficult to meet the needs of large-scale real-time traffic detection.

[0028] To address the above issues, in this embodiment, when performing traffic detection using a single-packet processing mode through a traffic detection device, upon receiving the data packet to be processed, the segmentation type of the data packet during transmission can be determined first. Furthermore, based on the segmentation type of the data packet, it can be stored in a continuous data buffer or an out-of-order segmented list within the buffer stream structure allocated to that data stream. This allows for ordered caching and dynamic reassembly of segmented and out-of-order service data. Consequently, continuous and complete service data can be recovered without maintaining a complete transmission connection state or consuming significant system resources. This approach maintains the advantages of high concurrency, low latency, and lightweight design of the single-packet processing mode while effectively addressing the inability to directly detect segmented and out-of-order packets, thus improving the accuracy and applicability of traffic detection.

[0029] In this embodiment, the data packet to be processed meeting the cache triggering condition can include at least one of the following detection methods: Based on the length and message characteristics of the data packet to be processed, determine whether it is a starting message or a fragmented message with a length less than a preset threshold; if so, the cache triggering condition is met. When the transport layer protocol of the data packet to be processed is a first protocol type, the application layer protocol is a second protocol type, and the service data does not contain complete protocol fields, the cache triggering condition is determined to be met. The following will provide a detailed explanation of each of these at least one cache triggering condition.

[0030] Optionally, the data packet to be processed meets the cache triggering condition, including: determining whether the length of the data packet to be processed is less than a preset message length threshold based on the length information of the data packet to be processed; determining whether the data packet to be processed belongs to a fragmented message or a start message of an application layer protocol based on the protocol type and message characteristics of the data packet to be processed; if the length of the data packet to be processed is less than the preset threshold and belongs to a fragmented message or a start message of an application layer protocol, determining that the data packet to be processed meets the cache triggering condition.

[0031] The length information can be understood as the overall byte length of the data packet to be processed. It is a message size parameter obtained from parsing the packet header and is used to determine whether the data packet is a segmented or incomplete fragment. The message length threshold can be understood as a pre-set critical value for the data packet length, used to distinguish between normal complete messages and short fragmented messages that may need to be reassembled. This value can be configured according to the application layer protocol characteristics. The protocol type can be understood as the type of transport layer or application layer protocol used by the data packet to be processed, such as TCP, HTTP, TLS, etc., used to determine whether the data packet needs to be cached and reassembled. Message characteristics can be understood as the identifying content characteristics in the data packet payload, such as protocol header tags, handshake information, request start fields, etc., used to identify whether it is a fragmented or start message. A fragmented message of an application layer protocol can be understood as a partial message formed by splitting a complete application layer message by the network layer or transport layer, which cannot constitute complete and detectable business data on its own. The start message of an application layer protocol can be understood as the first message sent in an application layer session or request, which usually contains key fields that need to be fully assembled to be identified.

[0032] In one implementation, during the data stream detection process, the traffic detection device first acquires the length information of the data packet to be processed and compares the length with a preset message length threshold to determine whether the data packet length is less than the threshold. At the same time, based on the protocol type and message characteristics in the payload of the data packet to be processed, it identifies whether the data packet is a fragmented message or a start message of an application layer protocol. When the data packet length is less than the preset threshold and it is confirmed to be a fragmented message or a start message of an application layer protocol, it is determined that the data packet to be processed meets the cache triggering condition and enters the subsequent cache reassembly processing flow.

[0033] Optionally, the data packet to be processed satisfies the cache triggering condition, including: if the transport layer protocol in the data packet to be processed belongs to a first protocol type, obtaining the application layer protocol information corresponding to the data packet to be processed; wherein, the application layer protocol information includes at least one of port number and payload characteristics; if it is determined from the application layer protocol information that the application layer protocol in the data packet to be processed belongs to a second protocol type, obtaining the service data in the data packet to be processed; if it is determined from the service data that the data packet to be processed does not contain complete protocol fields, determining that the data packet to be processed satisfies the cache triggering condition.

[0034] The transport layer protocol can be understood as the transport layer control protocol used by data packets during network transmission. It identifies the data transmission method, such as TCP and UDP, and is the basis for determining whether cache reassembly is needed. The first protocol type can be a pre-defined transport layer protocol type that requires cache reassembly. In this embodiment, the first protocol type usually refers to the TCP protocol because it has scenarios such as packet fragmentation and out-of-order delivery. The application layer protocol information can be understood as relevant information parsed from the data packet to identify the application layer service type. It mainly includes at least one of the port number and payload characteristics, used to determine which application layer protocol the data packet belongs to. The port number can be understood as the number in the data packet header used to identify the application layer service, such as 80 or 443, which can be used to initially identify application layer protocols such as HTTP and TLS. The payload characteristics can be understood as the identifying content characteristics in the data packet payload, such as protocol headers, request fields, handshake markers, etc., used to accurately identify the application layer protocol type. The second protocol type can be understood as a pre-defined application layer protocol type that requires complete field extraction for security testing, such as HTTP, TLS, and DNS. Business data can be understood as the payload data in a data packet after removing protocol control information such as IP headers and TCP headers; it is the actual object used for anomaly detection and rule matching. A complete protocol field can be understood as a complete piece of information in an application-layer message that can be used for traffic identification and security detection. For example, a complete protocol field may include the complete domain name, URL, and request headers.

[0035] In one implementation, when the traffic detection device receives a data packet to be processed, it can parse the header of the data packet, extract the transport layer protocol type from the header, and determine whether the extracted transport layer protocol type is a first protocol type. Further, if it is determined to belong to the first protocol type, it can continue to obtain the port number from the header and extract payload features from the service data of the data packet to be processed, identifying the extracted port number and payload features as application layer protocol information corresponding to the data packet to be processed. Further, the port number can be compared with a pre-set preset port number belonging to a second protocol type (e.g., port 80 for HTTP, port 443 for HTTPS or TLS). If the port number matches the preset port number, the payload features can be identified to determine whether the payload features contain features matching the second protocol type, such as HTTP request lines or TLS handshake messages. Further, if the payload features contain features matching the second protocol type, it can be determined that the application layer protocol of the data packet to be processed belongs to the preset second protocol type. Further, the service data in the data packet to be processed is parsed to detect whether the service data contains complete key fields that can be directly used for traffic identification. If the business data is incomplete and valid and complete protocol fields cannot be extracted, it can be determined that the data packet to be processed meets the cache triggering conditions, and the subsequent segment type judgment and cache reorganization process can be entered.

[0036] In this embodiment, the segment order type of the data packet to be processed during transmission is determined based on its attribute information, including at least one of the following determination methods: The byte offset of the data packet to be processed is calculated based on the message sequence number and the initial sequence number. If the data packet is the first data packet in the data stream, or its byte offset is consistent with the total length of the data already cached in the continuous data buffer, it is determined to be sequential; otherwise, it is determined to be out of order. The sequence number of the data packet to be processed is compared with the maximum sequence number of the data packets already cached in the continuous data buffer: if the sequence number of the data packet to be processed is equal to the maximum sequence number plus one, it is determined to be sequential; otherwise, it is determined to be out of order. The following uses one of these determination methods as an example to specifically illustrate the segment order type determination process.

[0037] Optionally, the attribute information includes the message sequence number and the initial sequence number of the data stream to which the data packet to be processed belongs. Based on the attribute information corresponding to the data packet to be processed, the segment order type of the data packet to be processed during transmission is determined, including: determining the byte offset of the data packet to be processed based on the message sequence number and the initial sequence number; determining the segment order type of the data packet to be processed during transmission as sequential if the byte offset determines that the data packet to be processed is the first data packet in its data stream; determining the total data length of the cached service data if there is cached service data in the continuous data buffer; determining the segment order type of the data packet to be processed as sequential if the total data length and byte offset determine that the data packet to be processed is continuous with the cached service data; and determining the segment order type of the data packet to be processed as out of order if the total data length and byte offset determine that the data packet to be processed is not continuous with the cached service data.

[0038] The sequence number can be understood as a number carried in the TCP header, used to identify the position of the first byte of the current data packet in the data stream, and is used to calculate the offset and determine whether the data is continuous. The initial sequence number can be the starting number determined when a data stream is established, and it is the reference for calculating the byte offset of all data packets within that data stream. The byte offset can be understood as the relative starting position of the first byte of a data packet in the entire data stream. Generally, the byte offset can be determined by the difference between the sequence number and the initial sequence number. For example, suppose the sequence number of a data packet is 2050, and the initial sequence number of the data stream to which this data packet belongs is 2000. Further, the byte offset of this data packet can be: 2050 - 2000 = 50, that is, the byte offset is 50, indicating that the first byte of this data starts from the 50th byte position in the entire data stream.

[0039] The cached business data can be valid application layer data that is stored in a continuous and sequential order within the continuous data buffer. In other words, it refers to the complete and continuous business data within the data stream to which the received data packet to be processed belongs. That is, if a data packet to be processed is received and cached business data exists in the continuous data buffer, the data packet to be processed and the cached business data belong to the same data stream. The total data length can be understood as the total number of bytes of cached business data in the continuous data buffer, indicating the end position of the cached business data within the continuous data buffer.

[0040] In one implementation, after receiving the data packet to be processed, the data packet can be parsed to obtain its sequence number and the initial sequence number of its data stream. The difference between the sequence number and the initial sequence number is determined, and this difference is used as the byte offset of the data packet to be processed. Further, if the byte offset is a preset value, the data packet to be processed can be determined as the first data packet in its data stream, and its segment type during transmission can be determined as sequential. Further, if service data is already cached in the continuous buffer, the total length of the cached service data can be obtained. Further, if the byte offset matches the total data length (i.e., the start position of the byte in the data packet is equal to the end position of the cached data), it indicates that the data packet to be processed and the cached service data are continuous, and the segment type of the data packet to be processed is determined as sequential; if the byte offset does not match the total data length (i.e., the start position of the byte in the data packet is not equal to the end position of the cached data), it indicates that the data packet to be processed and the cached service data are not continuous, and the segment type of the data packet to be processed is determined as out-of-order. Furthermore, the data packets to be processed can be cached and reassembled based on the segment sequence type of the data packets to be processed and the cache stream structure corresponding to the data stream.

[0041] For example, if the total length of the cached business data in the continuous data buffer is 100 bytes, it means that the end position of the current cached data is the 100th byte. If the byte offset of the data packet to be processed is 100, it means that the start position of the data packet to be processed matches the end position of the cached data perfectly, and the two can be directly connected continuously. Therefore, the segment type of the data packet to be processed is determined to be sequential. If the byte offset of the data packet to be processed is 150 or 80, which is a value not equal to 100, it means that the data packet to be processed and the cached data cannot be directly and continuously concatenated. Therefore, the segment type is determined to be out of order.

[0042] Optionally, based on the segment order type and the cache stream structure corresponding to the data stream, the data packets to be processed are cached and reassembled, including: if the segment order type of the data packets to be processed is sequential, copying the business data in the data packets to be processed to a continuous data buffer in the cache stream structure corresponding to the data stream, and updating the total data length of the cached business data in the continuous data buffer based on the byte offset of the data packets to be processed; if the segment order type of the data packets to be processed is out of order, storing the data packets to be processed into an out-of-order segmented linked list in the cache stream structure corresponding to the data stream.

[0043] In one implementation, after determining the segment order type of the data packet to be processed, if the segment order type is sequential, the service data in the data packet to be processed can be copied to a continuous data buffer in the cache stream structure corresponding to the data stream to which it belongs, and the byte offset of the data packet to be processed is added to the total data length of the cached service data in the continuous data buffer. The data length obtained after the addition is determined as the updated total data length. If the segment order type is out of order, the data packet to be processed is temporarily stored in the out-of-order segment linked list in the cache stream structure corresponding to the data stream to which it belongs.

[0044] It should be noted that, in order to achieve timely reassembly of out-of-order data packets and ensure the continuity of business data, the data processing method may also include: scanning the out-of-order segmented linked list; if there is a target data packet in the out-of-order segmented linked list that is continuous with the business data already cached in the continuous data buffer, then the target data packet is removed from the out-of-order segmented linked list and the business data in the target data packet is copied to the continuous data buffer.

[0045] In this context, the target data packet can be understood as an out-of-order data packet to be processed that can be directly connected to the currently cached data in the continuous data buffer, within the out-of-order segmented list. Removal means deleting the target data packet that meets the continuity condition from the out-of-order segmented list, thus ceasing further caching.

[0046] In one implementation, after the received data packets to be processed are cached and reassembled, the out-of-order segmented list is traversed and scanned to determine whether there is a target data packet that can be continuously connected with the service data already cached in the continuous data buffer. If so, the target data packet is removed from the out-of-order segmented list, and the service data it contains is copied to the continuous data buffer. Further, the steps of receiving data packets to be processed, caching and reassembling the data packets to be processed, and scanning the out-of-order segmented list are repeated until the continuous data buffer meets the rule detection trigger condition, triggering the rule matching process.

[0047] S130. When the continuous data buffer meets the rule detection triggering condition, determine the abnormal rule hit result of the data stream to which it belongs based on the business data already cached in the continuous data buffer, and process the data stream based on the abnormal rule hit result.

[0048] In this embodiment, during the detection of the data stream, the above-mentioned method can be used to cache and reassemble the multiple data packets to be processed. Furthermore, when the continuous data buffer corresponding to the data stream meets the rule detection triggering condition, abnormal rule detection can be performed on the cached business data in the continuous data buffer.

[0049] The rule detection trigger condition can be understood as a pre-set condition used to determine whether to initiate abnormal rule matching. The continuous data buffer meeting the rule detection trigger condition can be understood as the data within the continuous data buffer reaching a state where security detection, protocol parsing, or rule matching can be performed. Optionally, the rule detection trigger condition may include at least one of the following: the amount of cached business data in the continuous data buffer reaches a first threshold; the cached business data in the continuous data buffer contains complete protocol fields. The amount of cached business data can be understood as the total byte length of the continuous, concatenated, and uninterrupted business data in the continuous data buffer. The first threshold can be understood as a pre-configured data length threshold, such as 512 bytes, 1024 bytes, and 2048 bytes, etc., triggering rule detection when the amount of cached business data reaches this value. Complete protocol fields can be understood as the business data containing complete key information required for protocol parsing, such as the HTTP request line, request headers, domain name, URL, SNI, etc., sufficient for rule matching and anomaly identification.

[0050] The anomaly rule hit result can be understood as the detection result obtained by matching cached business data with preset anomaly rules. Optionally, the anomaly rule hit result includes hitting an anomaly rule, not hitting an anomaly, and no result. Processing the data stream can include performing control operations such as allowing, blocking, alarming, rate limiting, and logging on the data stream.

[0051] In one implementation, after reassembling the data packets to be processed and performing a scan operation on the out-of-order segmented linked list, data volume detection and / or field detection can be performed on the cached business data in the continuous data buffer. Furthermore, if the amount of cached business data in the continuous data buffer is detected to reach a first threshold, it can be determined that the continuous data buffer meets the rule detection triggering condition, and the abnormal rule hit result of its data stream can be determined based on the cached business data in the continuous data buffer.

[0052] Alternatively, if the cached business data in the continuous data buffer contains complete protocol fields, it can be determined that the continuous data buffer meets the rule detection triggering condition, and the abnormal rule hit result of the data stream to which it belongs can be determined based on the cached business data in the continuous data buffer.

[0053] Alternatively, if the amount of business data cached in the continuous data buffer reaches the first threshold and the business data cached in the continuous data buffer contains complete protocol fields, it is determined that the continuous data buffer meets the rule detection triggering condition, and the abnormal rule hit result of the data stream to which it belongs is determined based on the business data cached in the continuous data buffer.

[0054] It should be noted that, even if the rule detection trigger condition is only that the amount of cached business data reaches the first threshold, there may be cases where the cached business data in the continuous data buffer does not contain complete protocol fields. In this case, rule matching cannot be performed on the data stream, and therefore, the corresponding abnormal rule hit result is no result, and the data stream will be discarded.

[0055] Optionally, based on the business data already cached in the continuous data buffer, determine the abnormal rule hit result of the data stream to which it belongs, including: if the complete protocol field is extracted from the business data already cached in the continuous data buffer, match the complete protocol field with the predefined abnormal rules to obtain the abnormal rule hit result of the data stream to which it belongs.

[0056] The predefined anomaly rules can be rules pre-configured by the traffic detection device to detect abnormal data flows. Optionally, the anomaly rules include at least one of the following: attack signature rules, malicious behavior rules, violation policy rules, and blacklist rules. Anomaly rule hit results include at least one of the following: hit an attack signature rule; hit a malicious behavior rule; hit a violation policy rule; hit a blacklist rule; no anomaly rule hit; no result.

[0057] In one implementation, when the continuous data buffer meets the rule detection triggering condition, fields can be extracted from the cached business data within the continuous data buffer. Furthermore, if a complete protocol field is extracted from the cached business data in the continuous data buffer, the complete protocol field can be matched against at least one predefined anomaly rule to determine whether the complete protocol field conforms to the attack characteristics, violation characteristics, or abnormal behavior characteristics defined in the anomaly rule. Based on the field matching result, it is determined whether the data stream to which the complete protocol field belongs hits the anomaly rule, ultimately obtaining the corresponding anomaly rule hit result.

[0058] It should be noted that, in order to avoid long-term occupation of cache resources and improve the processing efficiency of traffic detection devices, the data processing method may also include: releasing resources within the cache stream structure when the resource release conditions are met.

[0059] The resource release condition can be understood as a pre-set trigger condition used to determine whether cached resources can be released. Optionally, the resource release condition includes at least one of the following: the number of received pending data packets reaches a first threshold; the idle time corresponding to the cache stream structure reaches a preset timeout threshold; the preset timeout threshold is determined based on the data stream status of the last pending data packet stored in the cache stream structure; the rule detection trigger condition is met and the rule detection is completed. The number of data packets can be understood as the total number of pending data packets that the traffic detection device has received and processed in the current data stream. The first threshold can be understood as a pre-configured upper limit for the number of data packets. When the total number of received pending data packets reaches this value, resource release is triggered. The idle time corresponding to the cache stream structure can be understood as the duration from the last operation on the cache stream structure (such as storing data packets, merging data, scanning the linked list, etc.) to the current moment, during which no new data is accessed. The preset timeout threshold can be understood as a pre-set timeout threshold. When the idle time reaches this threshold, it is determined that the data stream has not interacted for a long time, triggering resource release. The last data packet to be processed and stored in the buffer stream structure can be the most recently written data packet to the continuous data buffer or the out-of-order segmented linked list. The data stream status can be understood as the current data stream's transmission status, connection status, or protocol status, such as connection established, data transmission in progress, connection closed, or half-closed. It can be understood that the preset timeout threshold can be dynamically set or adjusted based on the data stream connection status reflected by the most recent data packet. Rule detection completion can be understood as having completed protocol field extraction and anomaly rule matching for the cached business data in the continuous data buffer, and obtaining the corresponding anomaly rule hit results.

[0060] The resources within the cache stream structure can include the memory space occupied by the cache stream structure, linked list nodes, cache data, state information, and other related hardware and software resources.

[0061] In one implementation, when the cache stream structure does not receive any new data packets after the last operation, its idle time gradually increases. If the idle time reaches a preset timeout threshold, the resource release condition is determined to be met. Furthermore, the memory resources occupied by the continuous data buffer and the out-of-order segmented linked list within the cache stream structure are reclaimed, and the cached business data and data packet information are cleared, thus releasing the resources within the cache stream structure, avoiding long-term resource occupation, and improving the resource utilization of the traffic detection device.

[0062] The technical solution of this disclosure, during the detection of a data stream, receives data packets to be processed within the data stream and acquires traffic data in real time, providing a data source for subsequent cache reassembly and anomaly detection, ensuring the normal startup of the detection process. Furthermore, when a data packet to be processed meets the cache triggering condition, the segment sequence type of the data packet during transmission is determined based on its attribute information. Then, based on the segment sequence type and the cache stream structure corresponding to the data stream, the data packet to be processed is cache reassembly performed, distinguishing between ordered and out-of-order packets, and storing them respectively in a continuous data buffer and an out-of-order segmented linked list, achieving ordered data packet splicing and ensuring the integrity and usability of business data. Furthermore, when the continuous data buffer meets the rule detection triggering condition, the anomaly rule hit result of the data stream to which it belongs is determined based on the cached business data in the continuous data buffer. The data stream is processed based on the anomaly rule hit result, and rule matching is performed based on the cached business data, ensuring the accuracy and completeness of anomaly detection, avoiding misjudgments or omissions due to missing or out-of-order data, thereby achieving precise and reliable security control of the data stream. The technical solution provided in this disclosure solves the problems of increased resource consumption, significant decrease in system concurrent processing capability, and system memory pressure in related technologies. It achieves the effect of orderly caching and reassembling of received data packets in traffic detection using a single packet processing mode, so as to perform rule detection on the reassembled business data. This ensures the accuracy of abnormal rule detection while significantly reducing system resource consumption, improving device concurrent processing capability, and alleviating system memory pressure.

[0063] Figure 2 This is a schematic diagram of the structure of a data processing apparatus provided in an embodiment of this disclosure. Figure 2As shown, the data processing device is configured in the traffic detection device, which uses a single packet processing mode for traffic detection. The device includes: a data packet receiving module 210, a data packet buffering module 220, and a data stream matching module 230. The data packet receiving module 210 is used to receive data packets to be processed in the data stream during the detection of the data stream; wherein the data packets to be processed carry service data to be detected; the data packet caching module 220 is used to determine the segment sequence type of the data packets to be processed during transmission based on the attribute information of the data packets to be processed when the data packets to be processed meet the caching trigger condition, and to perform cache reassembly of the data packets to be processed based on the segment sequence type and the cache stream structure corresponding to the data stream; wherein the segment sequence type includes sequential type and out-of-order type; the cache stream structure includes a continuous data buffer and an out-of-order segmented linked list; the data stream matching module 230 is used to determine the abnormal rule hit result of the data stream to which the continuous data buffer belongs based on the service data already cached in the continuous data buffer when the continuous data buffer meets the rule detection trigger condition, so as to process the data stream based on the abnormal rule hit result.

[0064] The technical solution of this disclosure, during the detection of a data stream, receives data packets to be processed within the data stream and acquires traffic data in real time, providing a data source for subsequent cache reassembly and anomaly detection, ensuring the normal startup of the detection process. Furthermore, when a data packet to be processed meets the cache triggering condition, the segment sequence type of the data packet during transmission is determined based on its attribute information. Then, based on the segment sequence type and the cache stream structure corresponding to the data stream, the data packet to be processed is cache reassembly performed, distinguishing between ordered and out-of-order packets, and storing them respectively in a continuous data buffer and an out-of-order segmented linked list, achieving ordered data packet splicing and ensuring the integrity and usability of business data. Furthermore, when the continuous data buffer meets the rule detection triggering condition, the anomaly rule hit result of the data stream to which it belongs is determined based on the cached business data in the continuous data buffer. The data stream is processed based on the anomaly rule hit result, and rule matching is performed based on the cached business data, ensuring the accuracy and completeness of anomaly detection, avoiding misjudgments or omissions due to missing or out-of-order data, thereby achieving precise and reliable security control of the data stream. The technical solution provided in this disclosure solves the problems of increased resource consumption, significant decrease in system concurrent processing capability, and system memory pressure in related technologies. It achieves the effect of orderly caching and reassembling of received data packets in traffic detection using a single packet processing mode, so as to perform rule detection on the reassembled business data. This ensures the accuracy of abnormal rule detection while significantly reducing system resource consumption, improving device concurrent processing capability, and alleviating system memory pressure.

[0065] In some embodiments of this disclosure, optionally, the data packet caching module 220 includes: a condition detection unit, configured to: when the transport layer protocol in the data packet to be processed belongs to a first protocol type, obtain application layer protocol information corresponding to the data packet to be processed; wherein the application layer protocol information includes at least one of port number and payload characteristics; when it is determined based on the application layer protocol information that the application layer protocol in the data packet to be processed belongs to a second protocol type, obtain service data in the data packet to be processed; and when it is determined based on the service data that the data packet to be processed does not contain complete protocol fields, determine that the data packet to be processed meets the caching trigger condition.

[0066] In some embodiments of this disclosure, optionally, the attribute information includes a message sequence number and an initial sequence number of the data stream to which the data packet to be processed belongs; the data packet caching module 220 includes: a segment order type determination unit, configured to determine the byte offset corresponding to the data packet to be processed based on the message sequence number and the initial sequence number; if the data packet to be processed is determined to be the first data packet in its data stream based on the byte offset, determine that the segment order type of the data packet to be processed during transmission is sequential; if cached service data exists in the continuous data buffer, determine the total data length of the cached service data; if the data packet to be processed is determined to be continuous with the cached service data based on the total data length and the byte offset, determine that the segment order type of the data packet to be processed during transmission is sequential; if the data packet to be processed is determined to be non-continuous with the cached service data based on the total data length and the byte offset, determine that the segment order type of the data packet to be processed during transmission is out of order.

[0067] In some embodiments of this disclosure, optionally, the data packet caching module 220 includes: a data packet caching unit, configured to, when the segment order type of the data packet to be processed is sequential, copy the service data in the data packet to be processed to a continuous data buffer in a cache stream structure corresponding to the data stream, and update the total data length of the cached service data in the continuous data buffer based on the byte offset of the data packet to be processed; and, when the segment order type of the data packet to be processed is out of order, store the data packet to be processed into an out-of-order segmented linked list in a cache stream structure corresponding to the data stream.

[0068] In some embodiments of this disclosure, optionally, the apparatus further includes: a segmented linked list scanning module, used to scan the out-of-order segmented linked list; if there is a target data packet in the out-of-order segmented linked list that is continuous with the service data already cached in the continuous data buffer, then the target data packet is removed from the out-of-order segmented linked list and the service data in the target data packet is copied to the continuous data buffer.

[0069] In some embodiments of this disclosure, optionally, the rule detection triggering condition includes at least one of the following: the amount of service data cached in the continuous data buffer reaches a first threshold; the service data cached in the continuous data buffer contains complete protocol fields.

[0070] In some embodiments of this disclosure, optionally, the data stream matching module 230 is specifically used to match the complete protocol field with predefined exception rules when the complete protocol field is extracted from the business data cached in the continuous data buffer, so as to obtain the exception rule hit result of the data stream to which it belongs.

[0071] In some embodiments of this disclosure, the apparatus may optionally further include a resource release module, configured to release resources within the cache stream structure when resource release conditions are met.

[0072] In some embodiments of this disclosure, optionally, the resource release condition includes at least one of the following: the number of received pending data packets reaches a second threshold; the idle time corresponding to the cache stream structure reaches a preset timeout threshold; the preset timeout threshold is determined based on the data stream status of the last pending data packet stored in the cache stream structure; and the rule detection trigger condition is met and the rule detection is completed.

[0073] The data processing apparatus provided in this disclosure can execute the data processing method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the method.

[0074] It is worth noting that the various units and modules included in the above-mentioned data processing device are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the protection scope of the embodiments of this disclosure.

[0075] Figure 3This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. The electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0076] like Figure 3 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0077] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0078] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as data processing methods.

[0079] In some embodiments, the data processing method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or mounted on electronic device 10 via read-only memory (ROM) 12 and / or communication unit 19. When the computer program is loaded into random access memory (RAM) 13 and executed by processor 11, one or more steps of the data processing method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the data processing method by any other suitable means (e.g., by means of firmware).

[0080] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0081] Computer programs used to implement the data processing methods of this disclosure may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0082] This disclosure provides a computer-readable storage medium storing computer instructions for causing a processor to execute a data processing method, comprising: during the detection of a data stream, receiving a data packet to be processed in the data stream; wherein the data packet to be processed carries service data to be detected; when the data packet to be processed meets a cache triggering condition, determining the segment order type of the data packet to be processed during transmission based on the attribute information of the data packet to be processed, and performing cache reassembly on the data packet to be processed based on the segment order type and a cache stream structure corresponding to the data stream; wherein the segment order type includes sequential and out-of-order types; the cache stream structure includes a continuous data buffer and an out-of-order segmented linked list; when the continuous data buffer meets a rule detection triggering condition, determining the abnormal rule hit result of the data stream to which the service data to which it belongs based on the service data already cached in the continuous data buffer, and processing the data stream based on the abnormal rule hit result.

[0083] In the context of this disclosure, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0084] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0085] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0086] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0087] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication unit 19, or installed from storage unit 18, or installed from ROM 12. When the computer program is executed by processor 11, it performs the functions defined in the methods of embodiments of this disclosure.

[0088] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements a data processing method according to any embodiment of this disclosure.

[0089] In implementing a computer program product, computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0090] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this disclosure can be achieved, and this is not limited herein.

[0091] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A data processing method, characterized in that, The method, applied to a flow detection device that uses a single-packet processing mode for flow detection, includes: During the detection of the data stream, a data packet to be processed is received from the data stream; wherein the data packet to be processed carries the service data to be detected. When the data packet to be processed meets the buffer triggering condition, the segment order type of the data packet to be processed during transmission is determined according to the attribute information of the data packet to be processed, and the data packet to be processed is buffered and reassembled according to the segment order type and the buffer stream structure corresponding to the data stream; wherein, the segment order type includes sequential type and out-of-order type; the buffer stream structure includes a continuous data buffer and an out-of-order segmented linked list; When the continuous data buffer meets the rule detection triggering condition, the abnormal rule hit result of the data stream to which it belongs is determined based on the business data already cached in the continuous data buffer, and the data stream is processed based on the abnormal rule hit result.

2. The data processing method according to claim 1, characterized in that, The data packet to be processed satisfies the cache triggering conditions, including: If the transport layer protocol in the data packet to be processed belongs to a first protocol type, obtain the application layer protocol information corresponding to the data packet to be processed; wherein, the application layer protocol information includes at least one of port number and payload characteristics; If it is determined that the application layer protocol in the data packet to be processed belongs to the second protocol type based on the application layer protocol information, the service data in the data packet to be processed is obtained. If, based on the business data, it is determined that the data packet to be processed does not contain complete protocol fields, then the data packet to be processed is determined to meet the cache triggering condition.

3. The data processing method according to claim 1, characterized in that, The attribute information includes the message sequence number and the initial sequence number of the data stream to which the data packet to be processed belongs; determining the segment sequence type of the data packet to be processed during transmission based on the attribute information of the data packet to be processed includes: The byte offset corresponding to the data packet to be processed is determined based on the message sequence number and the initial sequence number; If it is determined from the byte offset that the data packet to be processed is the first data packet in its data stream, then the segment order type of the data packet to be processed during transmission is determined to be sequential. If cached business data exists in the continuous data buffer, determine the total length of the cached business data. If it is determined that the data packet to be processed is continuous with the cached service data based on the total data length and the byte offset, the segment order type of the data packet to be processed during transmission is determined to be sequential. If, based on the total data length and the byte offset, it is determined that the data packet to be processed is not continuous with the cached service data, then the segment order type of the data packet to be processed during transmission is determined to be out of order.

4. The data processing method according to claim 1, characterized in that, The step of buffering and reassembling the data packets to be processed according to the segment type and the buffer stream structure corresponding to the data stream includes: When the segment order type of the data packet to be processed is sequential, the service data in the data packet to be processed is copied to the continuous data buffer in the cache stream structure corresponding to the data stream, and the total data length of the cached service data in the continuous data buffer is updated based on the byte offset of the data packet to be processed. If the segment order type of the data packet to be processed is out of order, the data packet to be processed is stored in the out-of-order segment linked list in the buffer stream structure corresponding to the data stream.

5. The data processing method according to claim 1, characterized in that, Also includes; The out-of-order segmented linked list is scanned. If a target data packet that is contiguous with the service data already cached in the continuous data buffer exists in the out-of-order segmented linked list, the target data packet is removed from the out-of-order segmented linked list, and the service data in the target data packet is copied to the continuous data buffer.

6. The data processing method according to claim 1, characterized in that, The rule detection triggering condition includes at least one of the following: The amount of business data cached in the continuous data buffer has reached a first threshold. The cached business data in the continuous data buffer contains complete protocol fields.

7. The data processing method according to claim 1, characterized in that, The step of determining the anomaly rule hit result of the data stream to which the service data belongs based on the cached service data in the continuous data buffer includes: When the complete protocol field is extracted from the business data already cached in the continuous data buffer, the complete protocol field is matched with predefined exception rules to obtain the exception rule hit result of the data stream to which it belongs.

8. The data processing method according to claim 1, characterized in that, Also includes: If the resource release conditions are met, the resources within the cache stream structure are released.

9. The data processing method according to claim 8, characterized in that, The resource release conditions include at least one of the following: The number of received pending data packets has reached the second threshold; The idle time corresponding to the cache stream structure reaches a preset timeout threshold; the preset timeout threshold is determined based on the data stream status of the last pending data packet stored in the cache stream structure. The rule detection triggering conditions are met and the rule detection is completed.

10. A data processing apparatus, characterized in that, Configured in a traffic detection device, the traffic detection device performs traffic detection using a single-packet processing mode, the device comprising: The data packet receiving module is used to receive data packets to be processed in the data stream during the detection of the data stream; wherein the data packets to be processed carry service data to be detected. A data packet caching module is used to determine the segment order type of the data packet during transmission based on the attribute information of the data packet when the data packet to be processed meets the caching trigger condition, and to perform cache reassembly on the data packet according to the segment order type and the cache stream structure corresponding to the data stream; wherein, the segment order type includes sequential type and out-of-order type; the cache stream structure includes a continuous data buffer and an out-of-order segmented linked list; The data stream matching module is used to determine the abnormal rule hit result of the data stream to which the continuous data buffer belongs, based on the business data already cached in the continuous data buffer, when the rule detection trigger condition is met, so as to process the data stream based on the abnormal rule hit result.

Images