Dynamic processing device, method, electronic device, medium and product of data stream

Abstract

The application relates to a dynamic processing device of a data stream, which comprises an optical bypasser, a convergence and distribution single board, a DPI processing single board and a switching processing module, the optical bypasser is connected with the switching processing module through an RL1 / RL2 serial interface, the convergence and distribution single board is connected with the switching processing module through an R1 / R2 interface, the DPI processing single board is connected with the switching processing module through a network input port GR1 / GR2, and the optical bypasser, the convergence and distribution single board and the DPI processing single board are connected in parallel. The optical bypasser, the convergence and distribution single board and the DPI processing single board are integrated on a backboard, and the data stream of the convergence and distribution single board and the DPI processing single board is effectively distributed through the switching processing module, so that external optical fiber connection is not needed and the propagation path is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of communication equipment, and specifically relates to a dynamic processing apparatus, method, electronic device, medium and product for data streams. Background Art

[0002] Optical bypass devices and convergence shunt devices are commonly used devices in communication networks, and DPI servers are usually deployed as customized systems based on general-purpose servers. An information security management system needs to be equipped with an optical bypass device, a convergence shunt device and a DPI server device. The independent operation of each device requires external optical fiber connections, which leads to long data transmission paths, high latency, and requires manual synchronization for policy configuration, making real-time linkage impossible. Moreover, their deployment and interconnection not only occupy a large amount of equipment room space, but also require supporting corresponding systems to operate normally. Summary of the Invention

[0003] The purpose of the present invention is to provide a dynamic processing method, electronic device, medium and product for data streams, so as to solve the problems raised in the above background art.

[0004] In a first aspect of the present invention, a dynamic processing apparatus for data streams is provided, comprising an optical bypass unit, a convergence shunt single board, a DPI processing single board and a switching processing module, wherein the optical bypass unit is connected to the switching processing module through RL1 / RL2 serial interfaces, the convergence shunt single board is connected to the switching processing module through R1 / R2 interfaces, the DPI processing single board is connected to the switching processing module through network input ports GR1 / GR2, and the optical bypass unit, the convergence shunt single board and the DPI processing single board are connected in parallel.

[0005] In a possible implementation, a Retimer chip is respectively arranged between the optical bypass unit and the switching processing module, between the convergence shunt single board and the switching processing module, and between the DPI processing single board and the switching processing module.

[0006] In a second aspect of the present invention, a dynamic processing method for data streams is provided, comprising: S1. Detect the working mode of the optical bypass unit in real time, and determine that the optical bypass unit is in a bypass state or an access state; S2. When in the bypass state, perform full mirror replication on core network traffic through the built-in optical splitter of the optical bypass unit, and transmit the mirrored traffic to the convergence shunt single board through the backplane direct connection channel; S3. Load a first-level rule matching policy in the convergence shunt single board, perform protocol filtering, traffic label marking and load balancing processing on the mirrored traffic, and output labeled traffic to the DPI processing single board; S4. When in access state, control the optical bypass to forward all traffic to the aggregation and distribution board, and activate the second-level rule matching strategy. The second-level rule matching strategy includes the dynamic priority determination logic of port effective rules and switching processing module rules. S5. Monitor the service load of the DPI processing board in real time through the switching processing module, and dynamically adjust the weight and traffic allocation ratio of each rule in the second-level rule matching strategy of the aggregation and distribution board. In one possible implementation, after step S5, the method further includes; S6. Update the configuration instructions of the backplane control channel according to the adjusted strategy parameters to achieve coordinated execution of strategies among multiple boards.

[0007] In one possible implementation, the mirrored traffic in step S2 is re-formed by a Retimer chip in the backplane, and an identification field containing a timestamp and source port number is added during transmission.

[0008] In one possible implementation, the first-level rule matching strategy includes: Pre-classify mirrored traffic based on 5-tuple features to generate metadata tags containing service type and traffic size; A dynamic hash algorithm is used to allocate traffic of different service types to the GR1 / GR2 interfaces of the DPI processing board, and the allocation weight is dynamically calculated based on the real-time throughput of the interface.

[0009] In one possible implementation, the dynamic priority determination logic includes: When the rule matching success rate of port R1 / R2 is detected to be lower than the preset matching threshold, the rule matching right is automatically switched to the switching processing module. During the switching process, the matching results of the port rules are cached and cross-validated with the matching results of the switching processing module.

[0010] In one possible implementation, step S5 includes: The decryption chip utilization, GPU memory usage, and message queue depth of the DPI processing board are collected via the backplane management bus. When any indicator exceeds the dynamic warning threshold, the aggregation and distribution board is triggered to enable the traffic sampling compression mode, and the sampling rate is negatively correlated with the degree of overload.

[0011] In one possible implementation, the policy coordination execution in step S6 includes: When switching from bypass mode to access mode, a link switching command is sent to the optical bypass via the backplane control channel, and the traffic buffer queue of the aggregation and splitter board is cleared simultaneously. After the state switch is completed, the switching processing module injects simulated test traffic into the DPI processing board to verify the integrity of the processing link.

[0012] A third aspect of the present invention provides an electronic device, the device comprising: One or more processors; and a memory storing computer program instructions that, when executed, cause the processors to perform the aforementioned method.

[0013] A fourth aspect of the present invention provides a computer-readable medium having stored thereon computer program instructions that can be executed by a processor to implement the aforementioned method.

[0014] A fifth aspect of the present invention provides a computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method described above.

[0015] Compared with the prior art, the beneficial effects of the present invention are: 1. By integrating an optical bypass, aggregation and splitter board, and DPI processing board on a single backplane, and effectively distributing the data streams of the aggregation and splitter board and the DPI processing board through a switching and processing module, no external fiber optic connection is required, reducing the propagation path. 2. By applying a two-level rule matching strategy, we can effectively achieve coordinated execution of strategies among multiple boards, reasonably manage the load of each board, and maximize the resource utilization of each board. 3. By adopting dynamic priority determination logic and real-time load feedback, data flow can be more rationally distributed among various boards. Attached Figure Description

[0016] Figure 1 An example schematic diagram of a dynamic processing device for a data stream provided for some embodiments of this application; Figure 2 A schematic diagram of a backplane connection method for a dynamic data stream processing device provided for some embodiments of this application; Figure 3 A flowchart illustrating a dynamic data stream processing method provided for some embodiments of this application; Figure 4 This is a schematic diagram of an exemplary structure of a processor and a memory according to this application; Figure 5 This is an exemplary structural diagram of an electronic device according to this application. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. Based on the embodiments of this invention, those skilled in the art will understand... All other embodiments obtained through creative labor are within the scope of protection of this invention.

[0018] It should be noted that the serial numbers assigned to the components in the embodiments of the present invention, such as "first" and "second", are only used to distinguish the described objects and have no sequential or technical meaning.

[0019] The following is combined Figure 1 and 2 The structure of the dynamic data stream processing device is described. In the diagram, RX represents download, TX represents upload, and the dashed lines represent network interface connections.

[0020] The dynamic data stream processing device of the present invention includes an optical bypass, a convergence and splitter board, a DPI processing board, and a switching processing module. The optical bypass is connected to the switching processing module via an RL1 / RL2 serial interface. The convergence and splitter board is connected to the switching processing module via an R1 / R2 interface. The DPI processing board is connected to the switching processing module via a network input port GR1 / GR2. The optical bypass, convergence and splitter board, and DPI processing board are connected in parallel. A Retimer chip is respectively provided between the optical bypass and the switching processing module, between the convergence and splitter board and the switching processing module, and between the DPI processing board and the switching processing module.

[0021] In this invention, the optical bypass, aggregation and splitter board, and DPI processing board are connected in parallel, and their respective operating states are adjusted and allocated through the switching processing module. Data requiring processing (as shown in the core NE1 and NE2) is first transmitted to the optical bypass, which can switch between bypass and access states according to instructions from the switching processing module. Different paths are used to transmit data to core NE1 and NE2 depending on the state. When the optical bypass is in bypass mode, all data is directly mirrored and transmitted to the aggregation and splitter board. When the optical bypass is in access mode, the switching processing module allocates the data flow between the aggregation and splitter board and the DPI board based on factors such as the DPI board's load. The retimer chip in this invention maintains data flow stability, facilitating real-time processing by the switching processing module. The switching processing module can use various commercially available ARM architecture processors.

[0022] In this invention, the optical bypass, aggregation and splitter boards, DPI processing boards, and switching processing modules are all integrated onto a single backplane. The backplane has eight service slots. The optical bypass board supports line protection for two links; the aggregation and splitter board supports multiple input / output ports; and the DPI processing board supports four 100G interfaces (two external interfaces and two internal interfaces, with the internal interfaces interconnected with the switching processing module) and two 10G interfaces. The optical bypass board, aggregation and splitter board, and DPI processing board are connected to the backplane via physical interfaces. The backplane is connected to the switching processing module, and the entire system operates through the backplane interfaces.

[0023] This invention also provides a method for performing dynamic data stream processing using the aforementioned dynamic data stream processing device, which is described below in conjunction with... Figure 3 The method will be described in detail below: A dynamic data stream processing method, comprising: S1. Real-time detection of the optical bypass's operating mode to determine whether the optical bypass is in bypass or access mode; The operating status of an optical bypass can be determined by checking the indicator lights on the optical bypass, or by performing data stream detection on the optical bypass. If a bidirectional data stream is generated inside the optical bypass, it is in the access state; otherwise, it is in the bypass state.

[0024] S2. When in bypass mode, the core network traffic is fully mirrored by the built-in splitter of the optical bypass device, and the mirrored traffic is transmitted to the aggregation and splitter board via the backplane direct connection channel; the physical isolation design between the mirrored traffic in bypass mode and the direct connection traffic in access mode effectively avoids the spread of single point of failure.

[0025] The mirrored traffic undergoes signal reshaping via a Retimer chip in the backplane, and an identification field containing a timestamp and source port number is added during transmission. Adding these identification fields facilitates effective identification of the data source during subsequent data transmission and processing. The Retimer chip in this application supports the PCIe 4.0 protocol and is suitable for interface signal reshaping at speeds of 100Gbps and above.

[0026] S3. Load the first-level rule matching strategy in the aggregation and distribution board, perform protocol filtering, traffic labeling and load balancing on the mirrored traffic, and output the labeled traffic to the DPI processing board; The first-level rule matching strategy includes: Mirrored traffic is pre-classified based on five-tuple characteristics to generate metadata tags containing service type and traffic size. A five-tuple is a set consisting of the source IP address, source port, destination IP address, destination port, and transport layer protocol. Service types can be classified according to priority into urgent, priority, and normal levels. During data stream transmission, urgent data is transmitted first, followed by priority data, and finally normal data.

[0027] A dynamic hash algorithm is used to allocate traffic of different service types to the GR1 / GR2 interfaces of the DPI processing board, and the allocation weight is dynamically calculated based on the real-time throughput of the interface.

[0028] Specifically, for each traffic data packet, the following features are extracted to generate a composite hash key K. This hash value is generated using a CRC32 checksum, and then a weighted dynamic mapping algorithm is used. First, the total weight is calculated: W. 总 =W1+W2, where W1 is the weight of the GR1 interface and W2 is the weight of the GR2 interface; then calculate the interface intervals to which the GR1 and GR2 interfaces belong respectively, with the GR1 interface interval being [0, W1 / W2]. 总 *2 64 The GR2 interface range is (W2 / W) 总 *2 64 ,2 64 ), perform 2 on the hash value K 64 The calculation is performed, and the output interface is rotated based on the calculation result.

[0029] S4. When in access state, control the optical bypass to forward all traffic to the aggregation and distribution board, and activate the second-level rule matching strategy. The second-level rule matching strategy includes the dynamic priority determination logic of port effective rules and switching processing module rules. The second-level rule matching strategy is used for matching R1 / R2 ports. When matching R1 / R2 ports, the matching can be performed using the GR1 / GR2 interface method described above. Furthermore, this strategy may also include dynamic priority determination logic for port activation rules and switching processing module rules, where the dynamic priority determination logic includes: When the success rate of rule matching on port R1 / R2 is detected to be lower than the preset matching threshold, the rule matching right is automatically switched to the switching processing module. The switching processing module can effectively allocate the use of ports, which can effectively improve the efficiency of port allocation on the aggregation and distribution board and reduce the workload of the main processor.

[0030] During the switchover process, the matching results of port rules are cached and cross-validated with the matching results from the switch processing module. Cross-validation involves retrieving matching results for the same traffic from both the Port Cache and SwitchCache based on the 5-tuple hash value and a timestamp window (±1ms). The rule IDs and action commands of the two caches are then compared for consistency.

[0031] S5. Monitor the service load of the DPI processing board in real time through the switching processing module, and dynamically adjust the weight and traffic allocation ratio of each rule in the second-level rule matching strategy of the aggregation and distribution board. Specifically, the following method can be used: collect the decryption chip utilization rate, GPU memory usage rate and message queue depth of the DPI processing board through the backplane management bus; When any indicator exceeds the dynamic warning threshold, the aggregation and distribution board is triggered to activate the traffic sampling and compression mode. The sampling rate is negatively correlated with the overload level. In overload scenarios, the dynamic activation of traffic sampling and compression improves the measured resource utilization and ensures business continuity under high throughput.

[0032] S6. Update the configuration instructions of the backplane control channel according to the adjusted strategy parameters to achieve coordinated execution of strategies among multiple boards.

[0033] When switching from bypass mode to access mode, a link switching command is sent to the optical bypass via the backplane control channel, and the traffic buffer queue of the aggregation and splitter board is cleared simultaneously. After the state switch is completed, the switching processing module injects simulated test traffic into the DPI processing board to verify the integrity of the processing link.

[0034] Furthermore, some embodiments of this application also provide an electronic device. The electronic device can be various forms of digital computer, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, etc. The electronic device can also be various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices.

[0035] The electronic device includes: one or more processors; and a memory storing computer program instructions that, when executed, cause the processor to perform the steps of the methods provided in any one or more of the above embodiments. Figure 4 An exemplary structural diagram of the electronic device is disclosed. For example... Figure 4 As shown, the electronic device includes: One or more processors 11, memory 12, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components are interconnected using different buses and can be mounted on a common motherboard or otherwise as required. The processors can process instructions that execute within the electronic device, including instructions stored in or on memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to the interface). In some other embodiments, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple electronic devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). The components, their connections and relationships, and their functions shown herein are merely examples and are not intended to limit the implementation of the present application described and / or claimed herein.

[0036] The electronic device may further include an input device 13 and an output device 14. The processor 11, memory 12, input device 13, and output device 14 may be connected via a bus or other means. Figure 5 Taking the example of a connection between China and Israel via a bus.

[0037] Input device 13 can receive input numerical or character information, and generate key signal inputs related to user settings and function control of the electronic device, such as a touch screen, keypad, mouse, trackpad, touchpad, joystick, one or more mouse buttons, trackball, joystick, etc. Output device 14 may include a display device, auxiliary lighting device (e.g., LED), and haptic feedback device (e.g., vibration motor). The display device may include, but is not limited to, a liquid crystal display (LCD), a light-emitting diode (LED) display, and a plasma display. In some embodiments, the display device may be a touch screen.

[0038] In this embodiment, a computer-readable medium stores a computer program / instructions that, when executed by a processor, implement the steps of the methods provided in any one or more of the above embodiments. This computer-readable medium may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into that device. The aforementioned computer-readable medium carries one or more computer-readable instructions.

[0039] The memory 12 can serve as a non-transitory computer-readable storage medium, used to store non-transitory software programs, non-transitory computer-executable programs, and modules. The processor 11 executes various functional applications and data processing of the server by running the non-transitory software programs, instructions, and modules stored in the memory 12, thereby implementing the program instructions / modules corresponding to the methods provided in any one or more of the embodiments described above in this application.

[0040] The computer program product provided in this application includes one or more computer programs / instructions. When executed by a processor, these computer programs / instructions generate, in whole or in part, the processes or functions described in this application. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive (SSD)).

[0041] It will be apparent to those skilled in the art that this application is not limited to the details of the exemplary embodiments described above, and that this application can be implemented in other specific forms without departing from the spirit or essential characteristics of this application. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of this application is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within this application. No reference numerals in the claims should be construed as limiting the scope of the claims. Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in the apparatus claims may also be implemented by a single unit or device in software or hardware. The terms "first," "second," etc., are used to indicate names and do not indicate any particular order.

Claims

1. A method for dynamic processing of data streams, characterized in that, include: S1. Real-time detection of the optical bypass's operating mode to determine whether the optical bypass is in bypass or access mode; S2. When in bypass mode, the core network traffic is fully mirrored by the built-in splitter of the optical bypass and the mirrored traffic is transmitted to the aggregation and splitter board via the backplane direct connection channel. S3. Load the first-level rule matching strategy in the aggregation and distribution board, perform protocol filtering, traffic labeling and load balancing on the mirrored traffic, and output the labeled traffic to the DPI processing board; The first-level rule matching strategy includes: Pre-classify mirrored traffic based on 5-tuple features to generate metadata tags containing service type and traffic size; A dynamic hash algorithm is used to allocate traffic of different service types to the GR1 / GR2 interfaces of the DPI processing board, and the allocation weight is dynamically calculated based on the real-time throughput of the interface. S4. When in access state, control the optical bypass to forward all traffic to the aggregation and splitter board, activate the second-level rule matching strategy, which includes dynamic priority determination logic for port activation rules and switching processing module rules; The dynamic priority determination logic includes: When the rule matching success rate of port R1 / R2 is detected to be lower than the preset matching threshold, the rule matching right is automatically switched to the switching processing module. During the switching process, the matching results of the port rules are cached and cross-validated with the matching results of the switching processing module; S5. Monitor the service load of the DPI processing board in real time through the switching processing module, and dynamically adjust the weight and traffic allocation ratio of each rule in the second-level rule matching policy of the aggregation and distribution board.

2. The method according to claim 1, characterized in that, In step S2, the mirrored traffic is re-organized by the Retimer chip in the backplane, and an identification field containing a timestamp and source port number is added during transmission.

3. The method according to claim 1, characterized in that: Step S5 includes: The decryption chip utilization, GPU memory usage, and message queue depth of the DPI processing board are collected via the backplane management bus. When any indicator exceeds the dynamic warning threshold, the aggregation and distribution board is triggered to enable the traffic sampling compression mode, and the sampling rate is negatively correlated with the degree of overload.

4. An electronic device, characterized in that, The electronic device includes: One or more processors; and a memory storing computer program instructions that, when executed, cause the processor to perform the method as described in any one of claims 1 to 3.

5. A computer-readable medium having stored thereon computer program instructions that can be executed by a processor to implement the method as claimed in any one of claims 1 to 3.

6. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 3.

Images