Service processing method based on efficient analysis
By implementing traffic identification, echo response control, analysis management and investigation and transmission processing in the Internet service processing system, the efficient and low-latency response problems in the processing of large-scale Internet data in the existing technology are solved, efficient and accurate data transmission and analysis processing are achieved, and the quality of system operation service is improved.
Patent Information
- Application Number
- CN202510244161.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-06
- Publication Date
- 2025-05-30
AI Technical Summary
When processing large-scale Internet data, the prior art is difficult to meet the needs of efficient processing and low-latency response at the same time, resulting in serious computing pressure in data centers and affecting Internet data processing services.
By implementing traffic identification and split conversion processing, echo response control, analysis control management and investigation and transmission processing in the Internet service processing system, it includes using a state machine to control data stream transmission, designing UDP custom data segment format, configuring the internal DDR4 space and PCIE channel memory access method of FPGA, and collecting and transmitting user opinion information.
It realizes the efficiency and accuracy of data transmission service processing, improves the system's data analysis and processing efficiency, avoids the occurrence of analysis and processing errors, and meets the needs of high-speed data reading, and improves the quality of system operation service.
Smart Images
Figure CN120075151A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of service processing, and particularly to a service processing method based on efficient analysis. Background Art
[0002] With the increasing development of Internet technology, the amount of data generated by individuals and enterprises is constantly increasing, and the difficulty of data processing services is also increasing. It not only requires the ability to process large-scale data, but also needs to minimize data response latency. In order to improve the ability to process large batches of data flow services and improve data transmission efficiency, stream processing technology has emerged. Both stream processing and batch processing are common means of processing data. Compared with stream processing, batch processing usually processes non-online data, cold data, and a large amount of single-process data, and the processing speed is slower than that of stream processing. Stream processing targets on-line and real-time generated data. The amount of data processed at one time is very small, but the processing latency is lower. The large amount of data brought by stream processing technology also brings severe computing pressure to the data center. The data center often cannot generate the required calculation results in time, thus seriously affecting Internet data processing services. Therefore, it is necessary to design an intelligent control management and an efficient service processing method based on efficient analysis. Summary of the Invention
[0003] The purpose of the present invention is to provide a service processing method based on efficient analysis to solve the problems raised in the above background art.
[0004] To solve the above technical problems, the present invention provides the following technical solutions: A service processing method based on efficient analysis, including: Performing traffic identification and splitting / transformation processing on the Internet service processing system; Performing echo response control on the Internet service processing system; Performing analysis control management on the Internet service processing system; Performing investigation transmission processing on the Internet service processing system; The performing traffic identification and splitting / transformation processing on the Internet service processing system includes: Splitting traffic from the Ethernet data packet according to the data header part. After stripping the data to complete the data traffic identification processing, the transmission processing of the data stream is controlled by a state machine. The three states of the state machine are header processing, data forwarding, and data discarding. The data in the header processing state includes Ethernet header information. When processing, the type field in the Ethernet header is first obtained. If the type field is ARP, the tdest signal is set to 0. If the type field is IPV4 and the IP version number field is 4, the protocol type is judged, and the tdest signal of the ICMP type is set to 1, the tdest of the TCP type is set to 2, and UDP is set to 3; During the traffic recognition process, after entering the header processing state, first determine whether the packet type is ARP. Since ARP requires the MAC address information in the Ethernet header during processing, packets in ARP format need to retain the Ethernet header for forwarding and directly enter the data forwarding state. Packets of other types need to remove the Ethernet header. When processing, directly store the data after the Ethernet header into the forwarding variable and consider it as data with a set length offset. At the end of the header processing state, determine whether the end of the packet has been reached through the last signal of the packet. If the end of the packet has not been reached, the next state is the data elimination state; otherwise, re-enter the header processing state.
[0005] According to the above technical solution, the echo response control of the Internet service processing system includes: When reading packets in the Internet service processing system, read the destination IP address from the IP field, read the ICMP type field and ICMP code field from the ICMP header, analyze and obtain the ICMP packet checksum, and at the same time add an 8-bit reverse rearrangement operation for each group of the IP header.
[0006] According to the above technical solution, the analysis control management of the Internet service processing system includes: Store the data table of the Internet service processing system inside the FPGA, allocate internal DDR4 space for the data table, and further configure the XDMA memory access mode. Read the data in the internal DDR4 space of the FPGA through the PCIE channel, and establish a translation lookaside buffer (TLB) in the FPGA. Use the last 5 bits of the port number to construct a data table with a length of 32 to store the table ID information of the Socket table. Therefore, after configuration, the Socket table entry can be directly located through the port number, eliminating the loop traversal operation.
[0007] According to the above technical solution, the analysis control management of the Internet service processing system further includes: Design a custom data segment format within UDP. In addition to the original UDP header, further add fields such as packet sequence number, number of sub-packets, current sub-packet sequence number, and timestamp. When sending data, for packets exceeding 9000 bytes, send them in sub-packets. When sending sub-packets, set the packet sequence number to the same value, and the sub-packet sequence number increases sequentially. Further complete multi-packet caching by using a cache table; The control channel interacts with the control information channel of the NodeManager structure of the node host from the master node to complete the information transmission between the master and slave nodes in the system.
[0008] According to the above technical solution, the investigation transmission processing of the Internet service processing system includes: During the process of users using the Internet service processing system, a questionnaire is issued to users to collect and survey user opinion information; The user-collected survey information is transmitted to the workbench to make the subsequent optimization of the system more in line with user needs.
[0009] According to the above technical solution, a service processing system based on efficient analysis includes: An identification processing module for performing identification processing on the Internet service processing system; An analysis and management module for performing analysis, optimization and management of the Internet service processing system; A survey transmission module for performing survey transmission of user opinion information.
[0010] According to the above technical solution, the identification processing module includes: A traffic identification module for performing identification processing on the number of users and traffic of the Internet service processing system; A splitting and conversion module for controlling the splitting and conversion of data packets; An echo response module for managing the echo response of data packet reading.
[0011] According to the above technical solution, the analysis and management module includes: A configuration processing module for performing FPGA configuration processing on the Internet service processing system; A multi-packet caching module for controlling multi-packet caching of the Internet service processing system; A transmission control module for managing the transmission control of data information between nodes.
[0012] According to the above technical solution, the survey transmission module includes: A service survey module for conducting service surveys on user opinion information; A transmission processing module for performing transmission control processing of data information.
[0013] Compared with the prior art, the beneficial effects achieved by the present invention are as follows: By setting an identification processing module, an analysis and management module, and a survey transmission module, the present invention makes data transmission service processing more efficient, effectively improves the accuracy and stability of data service processing, and at the same time can improve the efficiency of system data analysis and processing, avoid the generation of analysis and processing errors, and can be configured in the form of external memory access through PCIE while meeting the high-speed data reading requirements of the system, effectively improving the operation service quality of the system. Description of the Drawings
[0014] The drawings are used to provide a further understanding of the present invention, and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention, and do not constitute a limitation to the present invention. In the drawings: Figure 1 It is a flowchart of a service processing method based on efficient analysis provided by Embodiment 1 of the present invention; Figure 2 It is a module composition diagram of a service processing system based on efficient analysis provided by Embodiment 2 of the present invention. Specific embodiments
[0015] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.
[0016] Embodiment 1: Figure 1 It is a flowchart of a service processing method based on efficient analysis provided by Embodiment 1 of the present invention. This embodiment can be applied to an Internet service processing system. This method can be executed by a service processing system based on efficient analysis provided by the embodiments of the present invention. The system consists of multiple software and hardware modules, such as Figure 1 As shown, the method specifically includes the following steps: S101. Perform traffic identification and split conversion processing on the Internet service processing system; Exemplarily, in an embodiment of the present invention, the traffic is segmented from the Ethernet data packet according to the data packet header. After stripping the data and completing the data traffic identification process, the transmission process of the data stream is controlled by a state machine. The three states of the state machine are header processing, data forwarding, and data discarding. The data in the header processing state includes Ethernet header information. When processing, the type field in the Ethernet header is first obtained. If the type field is ARP, the tdest signal is set to 0. If the type field is IPV4 and the IP version number field is 4, the protocol type is judged, the tdest signal of the ICMP type is set to 1, the tdest of the TCP type is set to 2, and UDP is set to 3; in this step, tdest represents data in the ARP, ICMP, TCP, and UDP formats from 0 to 3 respectively. Except for the cases where the tdest signal is set in this step, data packets in other cases are not forwarded. Here, a boolean variable, called forwarding judgment, is used for control. At the end of the header processing state, the conversion to the next state is performed. Through the forwarding judgment variable, if the value of the variable meets the set requirements, it means that this type of data packet needs to be further processed and will enter the data forwarding state, otherwise it will enter the data discarding state. In the data forwarding state, the subsequent data will be stored in the forwarding variable, including the byte information in the data packet, the keep field, and the tdest state obtained in the header processing state. The data forwarding state will repeatedly obtain and forward the data until the last signal of the data packet is obtained. After the data forwarding state ends, it will re-enter the header processing state. In the data discarding state, a null operation will be performed until the last signal of the data packet is obtained. After this state ends, it will enter the header processing state.
[0017] During the traffic identification and processing, after entering the header processing state, first determine whether the packet type is ARP. Since ARP requires the MAC address information in the Ethernet header during processing, packets in ARP format need to retain the Ethernet header for forwarding and directly enter the data forwarding state. Packets of other types need to remove the Ethernet header. During processing, the data after the Ethernet header is directly stored in the forwarding variable and is regarded as having been offset by a set length. At the end of the header processing state, determine whether the end of the packet has been reached through the last signal of the packet. If the end of the packet has not been reached, the next state is the data elimination state; otherwise, enter the header processing state again. In this step, in the data elimination state, since the Ethernet header is removed, it is equivalent to a storage data offset. Therefore, a part of the data in each group of transmitted data needs to be retained for the next transmission group. Therefore, the fields sent from the data elimination state include the second half of the data in the previous group and the first half of the data in this group. Repeat this process until the end of the packet, that is, the last signal is true. After the last group of data arrives, it is necessary to determine whether the second half of the data in this group has been transmitted. If it has not been transmitted, enter the extra data state and send the remaining data to make the data transmission service processing more efficient and effectively improve the accuracy and stability of the data service processing.
[0018] S102. Perform echo response control on the Internet service processing system; Exemplarily, in the embodiment of the present invention, when reading a packet in the Internet service processing system, the destination IP address is read from the IP field, the ICMP type field and the ICMP code field are read from the ICMP header, and the ICMP packet checksum is analyzed. At the same time, an 8-bit reverse rearrangement operation of the IP header is added. In this step, after the packet reading is completed, it will enter the state evaluation stage. First, an echo judgment is performed. The judgment conditions include that the previously obtained IP address field is equal to the FPGA-side IP address, the ICMP type field is an echo request and the ICMP code field is 0, and the checksum calculation is 0. If the judgment conditions are met, enter the header sending stage; otherwise, enter the next judgment. If the end of the packet is reached, that is, the last field is true, enter the packet acquisition stage; if the end of the packet has not been reached, discard the data. Through this step, the system data analysis and processing efficiency is effectively improved, and the generation of analysis and processing errors is avoided.
[0019] S103. Perform analysis control management on the Internet service processing system; Exemplarily, in an embodiment of the present invention, the data table of the Internet service processing system is stored inside the FPGA, an internal DDR4 space is allocated for the data table, and the XDMA memory access mode is further configured. The data in the internal DDR4 space of the FPGA is read through the PCIE channel, and a fast table is established in the FPGA. A data table with a length of 32 is constructed using the last 5 bits of the port number to store the table ID information of the Socket table. Therefore, after the configuration is completed, the Socket table entry can be directly located through the port number, eliminating the loop traversal operation; in this step, the DDR4 is the internal high-speed storage space of the FPGA, which has a high data reading rate and operating frequency, and can be accessed by various functions of the system. It can be configured as an external memory access form through the PCIE while meeting the high-speed data reading requirements of the system, effectively improving the operation service quality of the system.
[0020] Design a custom data segment format within UDP. In addition to the original UDP header, fields such as packet sequence number, number of sub-packets, current sub-packet sequence number, and timestamp are further added. When sending data, for data packets exceeding 9000 bytes, they are sent in sub-packets. When sending sub-packets, the packet sequence number is set to the same value, and the sub-packet sequence number increases sequentially. A multi-packet cache is further completed by using a cache table; in this step, the cache table includes fields such as sequence number, timestamp, bitmap structure, and data area. The size of the cache table structure is 30, and the sequence number is set to the modulo operation of the packet sequence number by 30. Since a data packet can contain at most 5 sub-packets, the size of the cache data area is also set to 5. The bitmap structure uses the exclusive OR operation between the current bitmap state and the data packet bitmap value to determine whether the caching is completed. For example, if a data packet has three sub-packets, the data packet bitmap value is 11100, and the first and third data packets are currently cached, then the bitmap state is 10100. The value of the exclusive OR operation of the two values is not 0, indicating that the current caching is not completed. Through this step, the data caching processing of the system can be made more efficient and accurate, avoiding the out-of-order phenomenon that traditional UDP data packets have when being sent, and the need to rearrange the data packets according to the packet sequence number, which may lead to a situation of low service processing efficiency and chaos.
[0021] The control channel is the interaction channel between the NodeManager structure of the slave node host and the control information of the master node, completing the information transmission between the master and slave nodes in the system; in this step, the transmitted control information includes the load degree of the FPGA computing resources, calculation task arrival and completion notifications, heartbeat, and fault tolerance mechanisms, etc., while the data channel is the transmission channel for computing data.
[0022] S104. Perform the investigation transmission processing of the Internet service processing system; Exemplarily, in an embodiment of the present invention, during the user's use of the Internet service processing system, a questionnaire is issued to the user to collect and investigate the user's opinion information; Transmit the user - collected survey information to the workbench, making the subsequent optimization of the system more in line with user needs.
[0023] Embodiment 2: Embodiment 2 of the present invention provides a service processing system based on efficient analysis. Figure 2 It is a schematic diagram of the module composition of a service processing system based on efficient analysis provided for this Embodiment 2, as Figure 2 shown. This system includes: An identification processing module, used for the identification processing of the Internet service processing system. An analysis and management module, used for the analysis, optimization, and management of the Internet service processing system. A survey transmission module, used for the survey and transmission of user opinion information.
[0024] In some embodiments of the present invention, the identification processing module includes: A traffic identification module, used for the identification processing of the number of users and traffic of the Internet service processing system. A splitting and conversion module, used for the splitting and conversion control of data packets. An echo response module, used for the echo response management of data packet reading.
[0025] In some embodiments of the present invention, the analysis and management module includes: A configuration processing module, used for the FPGA configuration processing of the Internet service processing system. A multi - packet cache module, used for the multi - packet cache control of the Internet service processing system. A transmission control module, used for the transmission control and management of data information between nodes.
[0026] In some embodiments of the present invention, the survey transmission module includes: A service survey module, used for the service survey of user opinion information. A transmission processing module, used for the transmission control processing of data information.
[0027] It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "including", "comprising" or any other variant thereof is intended to cover non - exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or further includes elements inherent to such process, method, article or device.
[0028] Finally, it should be noted that the above are only preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or perform equivalent replacements for some of the technical features. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.
Claims
1. A service processing method based on efficient analysis, characterized in that: include: Carry out traffic identification, splitting and conversion processing of Internet service processing system; Conduct echo response control on the Internet service processing system; Conduct analysis, control and management of Internet service processing systems; Conduct survey transmission processing of Internet service processing systems; The traffic identification and split conversion process of the Internet service processing system includes: From the Ethernet data packet, the traffic is segmented according to the data packet header, and after stripping the data to complete the data traffic identification process, the transmission process of the data stream is controlled by the state machine. The three states of the state machine are header processing, data forwarding and data discarding. The data in the header processing state contains the Ethernet header information. When processing, the type field in the Ethernet header is first obtained. If the type field is ARP, the tdest signal is set to 0. If the type field is IPV4 and the IP version number field is 4, the protocol type is judged, and the tdest signal of the ICMP type is set to 1, the tdest of the TCP type is set to 2, and the UDP type is set to 3; Except for setting the tdest signal in several cases in this step, the data packets in other cases are not forwarded. Here, a Boolean type variable, called forwarding judgment, is used for control. At the end of the header processing state, the next state transition will be performed. Through the forwarding judgment variable, if the variable value meets the set requirements, it means that this type of data packet needs to be further processed and it will enter the data forwarding state, otherwise it will enter the data discarding state. In the data forwarding state, subsequent data will be stored in the forwarding variable, including the byte information in the data packet, the keep field and the tdest state obtained in the header processing state. The data forwarding state will repeatedly obtain and forward data until the last signal of the data packet is obtained. After the data forwarding state ends, it will re-enter the header processing state. In the data discarding state, no operation will be performed until the last signal of the data packet is obtained. After this state ends, it will enter the header processing state. During the traffic identification process, after entering the header processing state, first determine whether the data packet type is ARP. Since ARP requires the MAC address information in the Ethernet header during processing, the data packet in ARP format needs to retain the Ethernet header for forwarding and directly enter the data forwarding state. Other types of data packets need to remove the Ethernet header. During processing, the data after the Ethernet header is directly stored in the forwarding variable and is regarded as having an offset of the set length. At the end of the header processing state, the last signal of the data packet is used to determine whether the end of the data packet has been reached. If the end of the data packet has not been reached, the next state is the data discarding state, otherwise the header processing state is re-entered. In the data elimination state, a part of each group of transmitted data needs to be retained in the next transmission group. The fields sent out from the data elimination state include the second half of the data in the previous group and the first half of the data in this group. This process is repeated until the end of the data packet is reached, that is, the last signal is true. After the last group of data arrives, it is necessary to determine whether the second half of this group of data has been transmitted. If not, enter the extra data state and send the remaining data.
2. The service processing method based on efficient analysis according to claim 1, characterized in that: The echo response control of the Internet service processing system includes: When reading data packets in the Internet service processing system, the destination IP address is read from the IP field, the ICMP type field and the ICMP code field are read from the ICMP header, and the ICMP message checksum is obtained by analysis, and a reverse reordering operation is added to a group of 8 bits of the IP header.
3. The service processing method based on efficient analysis according to claim 1, characterized in that: The analysis, control and management of the Internet service processing system includes: The data table of the Internet service processing system is stored inside the FPGA, and internal DDR4 space is allocated for the data table. The XDMA memory access method is further configured, and a fast table is established in the FPGA. The data table is constructed using the last 5 bits of the port number to store the table ID information of the Socket table so that the Socket table item can be located by the port number.
4. The service processing method based on efficient analysis according to claim 1, characterized in that: The analysis, control and management of the Internet service processing system further includes: Design a custom data segment format in UDP. In addition to the original UDP header, add the packet number, sub-packet number, current sub-packet number and timestamp fields. When sending data, packets larger than 9000 bytes are sent in separate packets. When sending sub-packets, the packet number is set to the same value, and the sub-packet number is incremented in sequence. Multi-packet caching is further accomplished by using a cache table. The control channel is a control information exchange channel between the NodeManager structure of the slave node host and the master node, completing the information transmission between the master and slave nodes in the system.
5. The service processing method based on efficient analysis according to claim 1, characterized in that: The investigation transmission process of the Internet service processing system includes: During the process of users using the Internet service processing system, questionnaires are issued to users to collect user opinion information; The user-collected survey information is transferred to the workbench, so that the subsequent optimization of the system can better meet the user's needs.