Network message processing method and apparatus, and computer device and storage medium

Abstract

The present application belongs to the technical field of data processing and relates to a network message processing method and apparatus, and a computer device and a storage medium. The method comprises: on the basis of a received target network message, acquiring a target memory block from a pre-constructed shared memory pool, so as to generate a message memory; on the basis of a network-interface-card driver, performing first identification processing on the target network message, so as to obtain a target network message descriptor; on the basis of the target network message descriptor, performing second identification processing on the message memory, so as to obtain a first socket buffer; and performing parsing processing on the first socket buffer by means of a network protocol stack, and on the basis of a parsing processing result, reading network message data from a virtual memory corresponding to the message memory, so as to complete the processing of the network message. In the present application, on the basis of a constructed shared memory pool, zero-copy network packet reception can be realized during network message processing, thereby avoiding performance loss caused by the memory copy of messages.

Description

Network message processing method, device, computer equipment and storage medium

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to the Chinese patent application filed with the China Patent Office on February 4, 2024, with application number 202410157537.2, and application name “Network Message Processing Method, Device, Computer Equipment and Storage Medium”, all contents of which are incorporated by reference into this application. Technical Field

[0003] The present application relates to the field of data processing technology, and in particular to a network message processing method, apparatus, computer equipment, and storage medium. Background Art

[0004] In the existing packet sending and receiving process, when the message volume is large, memory copying and context switching are the main reasons for performance loss. As a result, many technologies to avoid message copying have emerged. The industry collectively refers to them as zero-copy technologies. Existing network zero-copy technologies can be divided into two categories according to their characteristics: kernel bypass technology and kernel optimization technology. Kernel bypass technology bypasses the kernel and requires the use of a user-mode network protocol stack to connect with it. However, the industry currently does not have software that has the maturity and universal performance to reach the level of the kernel protocol stack. Moreover, how to adapt a large number of existing APPs (Applications) to the user-mode protocol stack while ensuring performance gains is also a challenge. Important work: Kernel optimization technology only supports zero-copy packet sending. If it is reversed and used for zero-copy packet receiving, the following problems will occur: in the packet receiving direction, because the address of the message memory is not mapped to the APP process page table, the APP cannot access the memory allocated by the network card driver. If the memory allocated by the network card driver is mapped to the APP, the APP will be able to read and write the kernel memory. When an error occurs in the APP, the entire operating system may crash. To avoid the above problems, the range of the APP reading and writing kernel memory can be controlled, and only the message memory allocated by the network card driver can be mapped to the APP. However, there are still security vulnerabilities. Malicious APPs may monitor or tamper with the messages of other APPs.

[0005] Therefore, there is an urgent need to propose a network message processing method, device, computer equipment and storage medium suitable for general scenarios. Summary of the Invention

[0006] On the one hand, a network message processing method is provided, the method comprising:

[0007] Receive the target network message, and based on the target network message, obtain the target memory block from the pre-built shared memory pool to generate the message memory, and map the memory block in the shared memory pool to the virtual memory of the application process one by one;

[0008] In response to detecting that the message memory is successfully generated, based on the network card driver, performing a first identification process on the target network message to obtain a target network message descriptor;

[0009] Based on the target network message descriptor, performing a second identification process on the message memory to obtain a first socket buffer; and

[0010] The first socket buffer is parsed through the network protocol stack, and according to the parsing result, the network message data is read from the virtual memory corresponding to the message memory to complete the processing of the network message.

[0011] Optionally, the shared memory pool construction method includes:

[0012] Based on the first system call, a target device file is constructed, and the target device file is stored in the operating system kernel;

[0013] Based on the second system call, setting target parameters corresponding to the target device file; and

[0014] Based on the target parameters, a shared memory pool is generated, where the shared memory pool includes at least a payload data unit and a control data unit.

[0015] Optionally, based on the second system call, setting target parameters corresponding to the target device file includes:

[0016] Based on the second system call, setting the attribute tag, memory block size, shared memory pool usage mode, and number of memory blocks corresponding to the shared memory pool; and

[0017] Define the size of a memory page as the default memory block size;

[0018] The target parameters include at least the attribute label corresponding to the shared memory pool, the memory block size, the shared memory pool usage mode, and the number of memory blocks.

[0019] Optionally, after generating the shared memory pool based on the target parameters, the method further includes:

[0020] Based on the third system call, a memory page corresponding to the payload data unit is mapped into the virtual memory of the application process, where the payload data unit includes at least one memory page composed of memory blocks.

[0021] Optionally, after the shared memory pool is built, the method further includes:

[0022] Based on the shared memory pool, an application programming interface is constructed. The application programming interface includes at least a memory block application interface, a memory block release interface, and a network card instance interface.

[0023] Based on the second system call and the network card instance interface, binding the shared memory pool to the network card instance, wherein one shared memory pool can be bound to multiple network card instances; and

[0024] Based on the memory block application interface and the memory block release interface, the shared memory pool is bound to other modules corresponding to the interface, and the other modules at least include a network card driver module and a socket module.

[0025] Optionally, the control data unit includes at least one lock-free circular queue. When setting the number of memory blocks corresponding to the shared memory pool, the method further includes:

[0026] In response to detecting a memory block number setting instruction corresponding to the shared memory pool, allocating a plurality of shared memory pages to the shared memory pool through the shared memory pool module; and

[0027] Based on the shared memory page, a target number of memory blocks are obtained, and the addresses of the divided memory blocks are pushed into a target lock-free circular queue, wherein the physical addresses in a memory block are continuous.

[0028] Optionally, after allocating multiple shared memory pages to the shared memory pool through the shared memory pool module, the method further includes:

[0029] Lock the shared memory page.

[0030] Optionally, before obtaining a target memory block from a pre-built shared memory pool based on the target network message and generating the message memory, the method further includes:

[0031] Building a socket module based on the first application and setting a zero-copy option for receiving packets. At the same time, the first application and the network protocol stack respectively initialize their corresponding transaction ID variables to target values; and

[0032] Building a shared memory pool based on the second application, and binding the shared memory pool to at least one network card instance;

[0033] The first application is equal to or not equal to the second application.

[0034] Optionally, based on the target network message, a target memory block is obtained from a pre-built shared memory pool to generate message memory, including:

[0035] In response to receiving a target network message, detecting whether the target network card instance is bound to a pre-built shared memory pool;

[0036] In response to detecting that the target network card instance is bound to the pre-built shared memory pool, obtaining the target memory block from the shared memory pool; and

[0037] Generate message memory based on the target memory block.

[0038] Optionally, in response to detecting that the message memory is successfully generated, performing a first identification process on the target network message based on the network card driver to obtain a target network message descriptor includes:

[0039] Based on the target network message, use the network card driver to build a private message descriptor; and

[0040] In response to detecting that the network card instance corresponding to the network card driver is bound to the shared memory pool, a packet receiving zero copy flag is set on the private message descriptor to generate a target network message descriptor.

[0041] Optionally, after performing a first identification process on the target network message based on the network card driver to obtain the target network message descriptor, the method further includes:

[0042] Enqueue the target network packet descriptor into the packet receiving queue of the network card hardware;

[0043] The network card hardware copies the relevant data of the target network message to the message memory through DMA operation and sends a packet receiving instruction to the network card driver.

[0044] Optionally, performing a second identification process on the message memory based on the target network message descriptor to obtain the first socket buffer includes:

[0045] In response to detecting a packet receiving instruction, building an initial socket buffer for the message memory based on the network card driver; and

[0046] In response to detecting that a packet receiving zero copy flag is set on the target network message descriptor, a shared memory pointer is set on the initial socket buffer to generate a first socket buffer.

[0047] Optionally, after performing a second identification process on the message memory based on the target network message descriptor to obtain the first socket buffer, the method further includes:

[0048] Based on the network protocol stack packet receiving function, the first socket buffer is sent to the network protocol stack.

[0049] Optionally, parsing the first socket buffer through the network protocol stack, and reading the network message data from the virtual memory corresponding to the message memory according to the parsing result includes:

[0050] In response to detecting the first socket buffer, performing upward layer-by-layer parsing processing on the first socket buffer using a network protocol stack;

[0051] Putting the first socket buffer after upward layer-by-layer parsing into the corresponding socket module packet receiving queue, and sending a packet receiving instruction to the application; and

[0052] Based on the packet receiving instruction, the network message data is read from the virtual memory corresponding to the message memory.

[0053] Optionally, based on the packet receiving instruction, reading the network packet data from the virtual memory corresponding to the packet memory includes:

[0054] In response to detecting the packet receiving instruction, detecting whether a shared memory pointer is set on the first socket buffer based on the first system call function; and

[0055] In response to detecting that a shared memory pointer is set on the first socket buffer, a virtual memory address of the message corresponding to the shared memory pointer in the application process is assigned to the target field in the structure array parameter to read the network message data.

[0056] Optionally, when reading network message data, the method further includes:

[0057] Record the current value of the transaction ID in the first socket buffer and increment the transaction ID by 1 to obtain the second socket buffer; and

[0058] The second socket buffer is transmitted to the socket buffer to be recycled queue in the socket module.

[0059] Optionally, after reading the network message data, the method further includes:

[0060] Checks whether the current field in the structure array parameter is consistent with the target field; and

[0061] In response to detecting that the current field is inconsistent with the target field, the association between the current value of the transaction ID and the message memory is recorded, and the transaction ID is incremented by 1.

[0062] Optionally, the method further includes:

[0063] Analyze and process network message data;

[0064] In response to detecting that the parsing process is complete, queuing the transaction ID corresponding to the message memory into an event queue of the socket module based on the second system call function;

[0065] Based on the transaction ID, determine the second socket buffer from the socket buffer to be recycled queue; and

[0066] Based on the second socket buffer, the corresponding first socket buffer and the memory block corresponding to the first socket buffer are released.

[0067] In another aspect, a network message processing device is provided, the device comprising:

[0068] A message memory generation module is used to receive a target network message and, based on the target network message, obtain a target memory block from a pre-built shared memory pool to generate a message memory. The memory blocks in the shared memory pool are mapped one-to-one with the virtual memory of the application process.

[0069] A first identification processing module is used to perform a first identification processing on the target network message based on the network card driver to obtain a target network message descriptor when detecting that the message memory is successfully generated;

[0070] a second identification processing module, configured to perform a second identification processing on the message memory based on the target network message descriptor to obtain a first socket buffer; and

[0071] The reading module is used to parse the first socket buffer through the network protocol stack, and read the network message data from the virtual memory corresponding to the message memory according to the parsing result to complete the processing of the network message.

[0072] In another aspect, a computer device is provided, comprising a memory, a processor, and computer-readable instructions stored in the memory and executable by the processor, wherein when the processor executes the computer-readable instructions, the following steps are implemented:

[0073] Receive the target network message, and based on the target network message, obtain the target memory block from the pre-built shared memory pool to generate the message memory, and map the memory block in the shared memory pool to the virtual memory of the application process one by one;

[0074] In response to detecting that the message memory is successfully generated, based on the network card driver, performing a first identification process on the target network message to obtain a target network message descriptor;

[0075] Based on the target network message descriptor, performing a second identification process on the message memory to obtain a first socket buffer; and

[0076] The first socket buffer is parsed through the network protocol stack, and according to the parsing result, the network message data is read from the virtual memory corresponding to the message memory to complete the processing of the network message.

[0077] In another aspect, one or more non-transitory computer-readable storage media storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the following steps:

[0078] Receive the target network message, and based on the target network message, obtain the target memory block from the pre-built shared memory pool to generate the message memory, and map the memory block in the shared memory pool to the virtual memory of the application process one by one;

[0079] In response to detecting that the message memory is successfully generated, based on the network card driver, performing a first identification process on the target network message to obtain a target network message descriptor;

[0080] Based on the target network message descriptor, performing a second identification process on the message memory to obtain a first socket buffer; and

[0081] The first socket buffer is parsed through the network protocol stack, and according to the parsing result, the network message data is read from the virtual memory corresponding to the message memory to complete the processing of the network message. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] FIG1 is a flow chart of a method for processing network messages in the prior art;

[0083] FIG2 is a schematic diagram of an application environment of a network message processing method in one or more embodiments;

[0084] FIG3 is a schematic flow chart of a method for processing network messages in one or more embodiments;

[0085] FIG4 is another schematic flow chart of a method for processing network messages in one or more embodiments;

[0086] FIG5 is a schematic diagram of the overall structure of a shared memory pool of a network message processing method in one or more embodiments;

[0087] FIG6 is a schematic diagram of the structure of a network message processing device in one or more embodiments;

[0088] FIG7 is a diagram illustrating the internal structure of a computer device in one or more embodiments. DETAILED DESCRIPTION

[0089] To make the purpose, technical solutions, and advantages of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0090] It should be understood that in the description of this application, unless the context clearly requires otherwise, words such as "include", "comprises", and the like throughout the specification should be interpreted as inclusive rather than exclusive or exhaustive; that is, as "including but not limited to".

[0091] It should also be understood that the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of this application, unless otherwise specified, the meaning of "plurality" is two or more.

[0092] It should be noted that the terms "S1", "S2", etc. are used only for the purpose of describing the steps and do not specifically refer to the order or sequence, nor are they used to limit this application. They are merely for the convenience of describing the method of this application and should not be understood as indicating the order of the steps. In addition, the technical solutions between the various embodiments can be combined with each other, but this must be based on the fact that ordinary technicians in this field can implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0093] According to the background technology, with the deployment of Unix-like operating systems (time-sharing operating systems) in most production environments, the kernel-integrated network protocol stack has also been widely used. In some scenarios with high network performance requirements, the performance of the kernel protocol stack has become a hot topic of concern. As shown in Figure 1, in the packet sending direction, the process of the message from the APP to the network card hardware is as follows: the APP calls the API (Application Programming Interface) such as sendmsg (a computer function that can only be used when the socket is in a connected state) to enter the kernel protocol stack, and the socket allocates skb (socket buffer, a general descriptor of network messages in the Linux kernel) and message memory, and copies the data to be sent from the APP memory to the skb message memory. Then, after the protocol stack is processed layer by layer downward, the skb is handed over to the network card driver; the network card driver associates the IO (input and output) address of the skb and the message memory with a private message descriptor, and then pushes the message descriptor into the packet sending queue, and then notifies the network card hardware to send the packet; after receiving the notification, the network card hardware transmits the packet through DMA (Direct Memory Access). The NIC driver then calls the callback function of the skb associated with the message descriptor to release the skb and the message memory. In the existing technology, the process of sending messages from the NIC hardware to the APP in the packet receiving direction is as follows: the NIC driver allocates message memory for the private message descriptor and pushes it into the packet receiving queue for future use by the NIC hardware; the NIC hardware copies the message data into the message memory through DMA operation, and then notifies the NIC driver to receive the packet; the NIC driver creates an skb for the message memory, calls the network protocol stack packet receiving function, and sends the skb into the protocol stack; after upward layer-by-layer processing, the protocol stack hangs the skb in the corresponding socket packet receiving queue, and then notifies the APP to receive the packet; the APP calls APIs such as recvmsg (system call function) to enter the kernel, and the socket copies the data in the message memory to the APP memory, and then releases the skb and the message memory. In the above two processes, when the message volume is large, memory copying and context switching are the main causes of performance loss. As a result, many technologies to avoid message copying have emerged, which the industry collectively refers to as zero-copy technology.

[0094] Among them, network zero-copy technology can be divided into two categories according to its characteristics: (1) Kernel bypass technology: The idea of ​​this type of technology is to bypass the kernel protocol stack in whole or in part, so that the APP and the network card driver can directly exchange messages. A typical representative is DPDK (Data Plane Development Kit, which is a data plane development kit provided by Intel). It is an open source project promoted by Intel. It uses the kernel's UIO (Userspace I / O, which is an I / O technology running in user space) framework to completely bypass the kernel from the network card driver, and realize zero copy of messages between the network card driver and the APP. Although this type of technology realizes zero copy and improves performance, because it bypasses the kernel, its disadvantages are also obvious. That is, it needs to use the user-mode network protocol stack to connect with it. However, the industry currently does not have software with the maturity and universal performance that reaches the level of the kernel protocol stack. Moreover, under the premise of ensuring performance benefits, how to adapt a large number of existing APPs to the user-mode protocol stack is also an important task.(2) Kernel optimization technology: This type of technology is the optimization of specific processes in the kernel, such as the sendfile (network file transfer function). It is an optimization technology generated to solve the specific scenario of "sending file contents to a remote end through the network". Although it only reduces the number of copies and does not completely achieve zero copy, it avoids data copying between the kernel and the APP, and also avoids frequent context switching between the kernel and the APP, thereby improving performance. However, because this function is developed for a specific scenario, it is not universal and cannot help improve the performance of other scenarios. In kernel optimization technology, a very important and highly universal technology is TX ZERO COPY (packet zero copy) supported since Linux 4.14. However, it only supports zero copy in the packet sending direction, but not in the packet receiving direction. The principle of TX ZERO COPY is as follows: Pre-process: When the APP creates a socket, it sets TX ZERO COPY. In the COPY option, the APP and the network protocol stack initialize their respective "transaction IDs" to 0. The transaction ID uses the socket as a carrier and is the unique identifier of a successful zero-copy call between the APP and the kernel. The kernel uses it to notify the APP that it will no longer use the corresponding APP memory. When the APP sends a packet, it first records the relationship between the current value of the transaction ID and the APP memory, and then calls sendmsg (system call function) to enter the kernel protocol stack. The socket allocates skb and uses the APP memory directly as the skb message memory (the original behavior when it is not zero copy: allocate skb message memory and copy the data in the APP memory to the skb message memory). The current value of the transaction ID is recorded in the skb, and then the transaction ID is increased by 1. Finally, after the protocol stack processes the skb layer by layer, it hands it over to the network card driver. ; When the APP calls sendmsg and returns successfully, the transaction ID is increased by 1; the network card driver associates the IO address of the skb and the message memory with a private message descriptor, and then pushes the descriptor into the hardware packet sending queue, and then notifies the network card hardware to send the packet; the network card hardware copies the data in the message memory to its own hardware register or line through DMA operation, and after completing the packet sending process, it notifies the network card driver that the message descriptor has been used up; the network card driver recycles the message descriptor, calls the callback function of skb, releases skb, and pushes the transaction ID of skb into the event queue of the socket; the APP reads the event queue of the socket, takes out the transaction ID value from it, and releases the APP memory including the transaction ID value and the previous one. At this point, in the packet sending direction, zero copy of the message is achieved from the APP to the network card driver.

[0095] The above method only supports zero-copy packet sending. If it is reversed to support zero-copy packet receiving, there are three problems:

[0096] (1) Feasibility: In the packet sending direction, after address translation, DMA can access the APP memory. However, in the packet receiving direction, because the address of the message memory is not mapped into the APP process page table, the APP cannot access the memory allocated by the network card driver.

[0097] (2) Robustness: In the packet receiving direction, in order to solve the feasibility problem, if the memory allocated by the network card driver is mapped to the APP, the APP will be able to read and write the kernel memory. When an error occurs in the APP, the entire operating system may crash.

[0098] (3) Security: To avoid robustness issues, the range of kernel memory read and written by the APP can be controlled, and only the message memory allocated by the network card driver can be mapped to the APP. However, there are still security vulnerabilities. Malicious APPs may monitor or tamper with the messages of other APPs.

[0099] Therefore, according to the background technology, in terms of zero-copy packet reception based on the kernel protocol stack, there is no available technology suitable for general scenarios.

[0100] To solve the above technical problems, the present application provides a network message processing method, apparatus, computer equipment and storage medium. Based on the constructed shared memory pool, zero copy of network packets can be achieved during network message processing, thereby avoiding the performance loss caused by memory copy of the message.

[0101] The network message processing method provided in this application can be applied in the application environment shown in Figure 2. The terminal 102 communicates with a data processing platform provided on a server 104 via a network. The terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablet computers, and portable wearable devices. The server 104 can be implemented as an independent server or a server cluster consisting of multiple servers.

[0102] In one embodiment, as shown in FIG3 to FIG5 , a network message processing method is provided. The method is described by taking the terminal in FIG2 as an example, and includes the following steps:

[0103] S1: Receive the target network message, and based on the target network message, obtain the target memory block from the pre-built shared memory pool to generate message memory. The memory block in the shared memory pool is mapped one-to-one with the virtual memory of the application process.

[0104] It should be noted that the target network message is a network message sent from the network card hardware to the application APP (i.e., the packet receiving direction). In addition, as shown in Figure 5, the shared memory pool is constructed, configured, and allocated and released by the shared memory pool module in the application APP. The shared memory pool module is a module running in the operating system kernel. It provides an interface for APP in the form of a device file. The directory where the device file is located is / dev / shmempool / . The shared memory pool includes a payload data unit and a control data unit. The payload data unit part includes multiple memory blocks for storing network message data, and the control data unit part includes at least one lock-free circular queue. When the memory block is successfully divided, its memory block address will be pushed into a lock-free circular queue. It will be dequeued from the lock-free circular queue when the user applies and enqueued when released.

[0105] In some specific implementations, the method for constructing a shared memory pool includes:

[0106] Based on the first system call, a target device file is constructed, where the target device file is stored in the operating system kernel, wherein the first system call may be an open system call (open is a function in multiple languages ​​and environments). In a LINUX system, the open system call is used to open and create a file, and the target device file may be / dev / shmempool / x;

[0107] Based on the second system call, target parameters corresponding to the target device file are set, wherein the second system call can be an ioctl (input / output control, a system call dedicated to device input / output operations) system call, and the target parameters include at least an attribute tag corresponding to the shared memory pool, a memory block size, a shared memory pool usage mode, and the number of memory blocks, wherein the attribute tag is the aforementioned x, and a default memory block size is defined as the size of a memory page, preferably 4KB. The shared memory pool usage mode can be single producer and single consumer, single producer and multiple consumers, multiple producers and single consumer, multiple producers and multiple consumers, etc., with the default being multiple producers and multiple consumers. The ioctl system call for defining the number of memory blocks in the shared memory pool x is located at the end of other ioctl system calls;

[0108] Based on the target parameters, a shared memory pool is generated. The shared memory pool includes at least a payload data unit and a control data unit. The payload data unit and the control data unit have been described above and will not be described again here.

[0109] In some specific implementations, after generating the shared memory pool based on the target parameters, the method further includes:

[0110] Based on the third system call, the memory page corresponding to the payload data unit is mapped into the virtual memory of the application process, where the payload data unit includes at least one memory page composed of memory blocks. The third system call can be an mmap system call, which is used to map a file or other object into the memory. That is, the application uses the mmap system call to map the memory page corresponding to the payload data unit into the virtual memory of its own process, so as to achieve zero copy when subsequently reading network messages.

[0111] In some specific implementations, after the shared memory pool is constructed, the method further includes:

[0112] Based on the shared memory pool, an application program interface is constructed. The application program interface includes at least a memory block application interface, a memory block release interface, and a network card instance interface. The network card instance interface is an API newly added by the network device for the application APP, which is used to bind the shared memory pool and the network card instance. The memory block application interface and the memory block release interface are APIs provided by the shared memory pool to other modules in the kernel. Users of the memory block application interface may include the network card driver module, and users of the memory block release interface may include the socket (socket module) and the network card driver module.

[0113] Based on the second system call and the network card instance interface, the shared memory pool is bound to the network card instance. One shared memory pool can be bound to multiple network card instances. That is, the app uses the ioctl system call to bind the shared memory pool to the network card instance. One shared memory pool can be used by multiple network card instances.

[0114] Based on the memory block application interface and the memory block release interface, the shared memory pool is bound to other modules corresponding to the interface, and the other modules at least include a network card driver module and a socket module.

[0115] In some specific implementations, the control data unit includes at least one lock-free circular queue, and when setting the number of memory blocks corresponding to the shared memory pool, the method further includes:

[0116] In response to detecting a memory block quantity setting instruction corresponding to the shared memory pool, allocating a plurality of shared memory pages to the shared memory pool through the shared memory pool module;

[0117] Based on the shared memory page, a target number of memory blocks are obtained, and the addresses of the divided memory blocks are pushed into a target lock-free circular queue, wherein the physical addresses in a memory block are continuous.

[0118] In some specific implementations, after allocating a plurality of shared memory pages to the shared memory pool by the shared memory pool module, the method further includes:

[0119] Lock the shared memory page.

[0120] Specifically, when the APP uses the ioctl system call to set the number n of memory blocks in the shared memory pool x, the shared memory pool module is triggered to allocate several shared memory pages for the shared memory pool x, and then these memory pages are divided into n memory blocks. The addresses of the divided memory blocks are pushed into a lock-free circular queue. When the user applies for them, they are dequeued from the lock-free circular queue and enqueued when they are released. In addition, in order to enable DMA operations (Direct Memory Access, group data transfer operations or direct memory operations) to use these memory blocks correctly, the physical addresses within a memory block are guaranteed to be continuous when the memory blocks are divided. In other words, a memory block will not contain memory from two memory pages with discontinuous physical addresses at the same time. In addition, the shared memory pages requested by the shared memory pool x must be locked to prevent these memory pages from being swapped out of the physical memory.

[0121] In the above implementation, the message memory is obtained by constructing a shared memory pool, and the virtual address of the message memory when receiving the packet is mapped to the APP process page table in advance, so that the APP process, kernel and hardware can operate the same physical memory in the entire process and use it as the packet receiving memory to solve the feasibility problem of zero copy of packet receiving, thereby avoiding the performance loss caused by the memory copy of the message. In addition, the shared memory pool provides an interface for the APP in the form of a device file, which can be set based on the operating permission of the operating system. As long as the malicious APP cannot obtain the corresponding permission, it cannot monitor or tamper with the messages of other APPs, thereby improving the security of subsequent message transmission.

[0122] S2: In response to detecting that the message memory is successfully generated, based on the network card driver, a first identification process is performed on the target network message to obtain a target network message descriptor.

[0123] It should be noted that the first identification processing is to set the packet receiving zero copy flag on the constructed private message descriptor to obtain the target network message descriptor, wherein the packet receiving zero copy flag is the RX ZERO COPY flag.

[0124] In some specific implementations, before obtaining a target memory block from a pre-built shared memory pool based on the target network message and generating the message memory, the method further includes:

[0125] A socket module is constructed based on the first application, and a zero-copy option for receiving packets is set. Meanwhile, the first application and the network protocol stack respectively initialize their corresponding transaction ID variables to target values.

[0126] Building a shared memory pool based on the second application, and binding the shared memory pool to at least one network card instance;

[0127] The first application is equal to or not equal to the second application, and the first application is APP1 and the second application is APP2 respectively.

[0128] Specifically, before performing the packet receiving process, the second application builds a shared memory pool through the method of step S1 and binds it to the corresponding network card instance. The first application creates a socket module, that is, a socket. When creating the socket module, the RX ZERO COPY option is set. The first application and the network protocol stack respectively initialize their respective "transaction ID" variables to 0. In the front-end process, the first application and the second application can be the same or different. The second application requires root permission when creating the shared memory pool. The second application can perform processes such as building the socket module without requiring root permission.

[0129] In the above embodiment, before processing the network message, two pre-processes are also included. In pre-process 2, a shared memory pool is built based on the second application and the network card instance is bound, which requires root permission to ensure that malicious APPs cannot obtain corresponding permissions and thus cannot monitor or tamper with messages of other APPs, so as to improve the security of subsequent message transmission; in pre-process 1, a socket module is created and the corresponding RX ZERO COPY option is set, and the transaction ID is initialized to 0, so as to be used for subsequent determination of whether the network message is a zero-copy message, and to determine the skb that needs to be released.

[0130] In some specific implementations, based on the target network message, obtaining a target memory block from a pre-built shared memory pool and generating a message memory include:

[0131] In response to receiving a target network message, detecting whether the target network card instance is bound to a pre-built shared memory pool;

[0132] In response to detecting that the target network card instance is bound to the pre-built shared memory pool, obtaining a target memory block from the shared memory pool;

[0133] Generate message memory based on the target memory block.

[0134] In some specific implementations, in response to detecting that the message memory is successfully generated, based on the network card driver, performing a first identification process on the target network message to obtain a target network message descriptor includes:

[0135] Based on the target network message, use the network card driver to build a private message descriptor;

[0136] In response to detecting that the network card instance corresponding to the network card driver is bound to the shared memory pool, a packet receiving zero copy flag is set on the private message descriptor to generate a target network message descriptor.

[0137] In some specific implementations, after performing a first identification process on the target network message based on the network card driver to obtain the target network message descriptor, the method further includes:

[0138] Enqueue the target network packet descriptor into the packet receiving queue of the network card hardware;

[0139] The network card hardware copies the relevant data of the target network message to the message memory through DMA operation and sends a packet receiving instruction to the network card driver.

[0140] Specifically, when the target network message is detected, message memory needs to be allocated. If the network card instance corresponding to the network card driver that needs to build a private message descriptor is bound to a shared memory pool, then a memory block is obtained from the shared memory pool, and message memory is generated based on the memory block. At the same time, the RX ZERO COPY flag is set on the private message descriptor, and the private message descriptor with the RX ZERO COPY flag set (that is, the target network message descriptor) is placed in the hardware packet receiving queue for future use by the network card hardware. Then the network card hardware copies the network message data to the message memory through DMA operation, and notifies the network card driver to receive the packet. If the memory block cannot be obtained from the shared memory pool, the existing method of applying for message memory is used, and there is no need to set the RX ZERO COPY flag.

[0141] In the above implementation, by constructing the target network packet descriptor, it is determined whether the skb needs to be identified when the skb is subsequently created, so as to achieve zero copy of the network packet.

[0142] S3: Based on the target network message descriptor, perform a second identification process on the message memory to obtain a first socket cache.

[0143] It should be noted that the second identification process is to set a shared memory pointer in the skb created based on the network card driver, thereby obtaining a first socket buffer to identify that the network message memory comes from the shared memory pool.

[0144] In some specific implementations, performing a second identification process on the message memory based on the target network message descriptor to obtain the first socket buffer includes:

[0145] In response to detecting a packet receiving instruction, an initial socket buffer is constructed for the message memory based on the network card driver, wherein the initial socket buffer is the initial skb;

[0146] In response to detecting that a packet receiving zero copy flag is set on the target network message descriptor, a shared memory pointer is set on the initial socket buffer to generate a first socket buffer.

[0147] In some specific implementations, after performing a second identification process on the message memory based on the target network message descriptor to obtain the first socket buffer, the method further includes:

[0148] Based on the network protocol stack packet receiving function, the first socket buffer is sent to the network protocol stack.

[0149] Specifically, the network card driver creates an skb for the above message memory. If the private message descriptor has the RX ZERO COPY flag, the shared memory pool pointer is set in the skb, and then the network protocol stack packet receiving function is called to send the skb to the network protocol stack.

[0150] In the above embodiment, based on the target network packet descriptor, when creating skb, the skb is identified to determine that the packet memory corresponding to the skb comes from the shared memory pool, thereby achieving zero copy of the network packet.

[0151] S4: parsing the first socket buffer through the network protocol stack, and reading the network message data from the virtual memory corresponding to the message memory according to the parsing result, thereby completing the processing of the network message.

[0152] It should be noted that, as shown in FIG4 , the network protocol stack includes a network layer, a transport layer, and a socket module.

[0153] In some specific implementations, parsing the first socket buffer through the network protocol stack, and reading the network message data from the virtual memory corresponding to the message memory according to the parsing result includes:

[0154] In response to detecting the first socket buffer, the first socket buffer is parsed upward layer by layer using the network protocol stack, wherein the method of parsing the skb upward layer by layer using each layer of the network protocol stack is a common method and will not be repeated here;

[0155] The first socket buffer after upward layer-by-layer parsing is queued into the corresponding socket module packet receiving queue, and a packet receiving instruction is sent to the application, that is, the skb is hung in the corresponding socket packet receiving queue, and then the APP is notified to receive the packet;

[0156] Based on the packet receiving instruction, the network message data is read from the virtual memory corresponding to the message memory, specifically including:

[0157] In response to detecting a packet receiving instruction, detecting whether a shared memory pointer is set on the first socket buffer based on a first system call function, wherein the first system call function may be recvmsg. Before calling the function, the application saves the value of an iov (structure array, each element of which indicates a buffer in a memory) parameter, and after saving the parameter value, calls recvmsg to enter the system kernel;

[0158] In response to detecting that a shared memory pointer is set on the first socket cache, the virtual memory address in the application process of the message corresponding to the shared memory pointer is assigned to the target field in the structure array parameter to read the network message data, that is, if a shared memory pool pointer is set in skb, then the virtual address in the APP process of the corresponding message memory is assigned to the iov_base field in the iov parameter to read the network message data.

[0159] In some specific implementations, when reading network message data, the method further includes:

[0160] Record the current value of the transaction ID in the first socket buffer and add 1 to the transaction ID to obtain the second socket buffer;

[0161] The second socket buffer is transmitted to the socket buffer to be recycled queue in the socket module.

[0162] Specifically, after assigning the virtual address, the current value of the transaction ID is recorded on the skb, and then the transaction ID is increased by 1. The skb is then placed in the socket's skb recycling queue for subsequent release of the skb.

[0163] Furthermore, if the shared memory pool pointer is not set in the skb, the socket copies the data in the message memory to the APP memory pointed to by iov_base, and then releases the skb; if the application APP does not set the RX ZERO COPY option when creating the socket module, but the shared memory pool pointer is set in the skb, the socket copies the data in the message memory to the APP memory pointed to by iov_base, and then releases the skb.

[0164] In some specific implementations, after the application reads the network message data, the method further includes:

[0165] Check whether the current field in the structure array parameter is consistent with the target field, that is, check whether the iov parameter part iov_base has changed;

[0166] In response to detecting that the current field is inconsistent with the target field, the association between the current value of the transaction ID and the message memory is recorded, and the transaction ID is incremented by 1. That is, when the iov parameter part iov_base changes, it means that this call has obtained a zero-copy message, then the association between the current value of the transaction ID and the message memory is recorded, and then the transaction ID is incremented by 1.

[0167] Furthermore, after the transaction ID is incremented by 1, the network message data is parsed and processed. The method for parsing the message data is a common technique and will not be described in detail here.

[0168] In response to detecting that the parsing process is completed, the transaction ID corresponding to the message memory is queued into the event queue of the socket module, that is, the event queue of the socket based on the second system call function, wherein the second system call function may be sendmsg;

[0169] Based on the transaction ID, the second socket cache is determined from the socket cache to be recycled queue. That is, the transaction ID in the socket cache to be recycled queue is matched according to the transaction ID in the event queue, so that the socket finds the corresponding skb in the skb to be recycled queue, that is, the second socket cache;

[0170] Based on the second socket buffer, release the corresponding first socket buffer and the memory block corresponding to the first socket buffer, that is, release all skb, and release the shared memory block when releasing skb;

[0171] In some specific implementations, if the APP processes network messages in sequence, the processing can be simplified: there is no need to record the association between the current value of the transaction ID and the message memory. It is only necessary to push the "current value of the transaction ID - 1" into the socket's event queue when the network message data is processed and the kernel is notified to release the message memory.

[0172] If all iov_bases in the iov array have not changed, it means that the message read by this call is a non-zero copy message. The processing flow is the same as the existing non-zero copy message processing flow and will not be repeated here.

[0173] In the above implementation, based on the virtual address mapped in the APP process page table, the corresponding network message data is read, and after the APP processes the network message data, the skb and shared memory block are released according to the corresponding rules, thereby avoiding the performance loss caused by the memory copy of the message.

[0174] In the above-mentioned network message processing method, the method includes: receiving a target network message, and based on the target network message, obtaining a target memory block from a pre-built shared memory pool to generate a message memory; in response to detecting that the message memory is successfully generated, based on the network card driver, performing a first identification processing on the target network message to obtain a target network message descriptor; based on the target network message descriptor, performing a second identification processing on the message memory to obtain a first socket cache; parsing the first socket cache through the network protocol stack, and reading the network message data from the virtual memory corresponding to the message memory according to the parsing result to complete the processing of the network message. This application is based on the constructed shared memory pool, and maps the virtual address of the message memory when receiving the packet in advance. It is mapped into the APP process page table, so that the APP process, kernel and hardware can all operate the same physical memory and use it as packet receiving memory, which solves the feasibility problem of zero-copy packet receiving, thereby avoiding the performance loss caused by memory copy of the message. At the same time, when implementing zero-copy packet receiving, the only memory shared between the APP and the kernel is the memory for loading message data. Even if an error occurs in the APP, other data structures of the kernel will not be damaged, avoiding operating system abnormalities. The shared memory pool function provides an interface for the APP in the form of a device file, which can be set based on the operating permission of the operating system. As long as a malicious APP cannot obtain the corresponding permission, it cannot monitor or tamper with the messages of other APPs, thereby improving the security of message transmission.

[0175] It should be understood that, although the various steps in the flow charts of Figures 3-5 are shown in sequence according to the indication of the arrows, these steps are not necessarily performed in sequence according to the order indicated by the arrows. Unless otherwise specified herein, the execution of these steps does not have strict order restrictions, and these steps can be performed in other orders. Moreover, at least a portion of the steps in Figures 3-5 may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these sub-steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of other steps or sub-steps or stages of other steps.

[0176] In one embodiment, as shown in FIG6 , a network message processing device is provided, comprising: a message memory generation module, a first identification processing module, a second identification processing module, and a reading module, wherein:

[0177] A message memory generation module is used to receive a target network message and, based on the target network message, obtain a target memory block from a pre-built shared memory pool to generate a message memory. The memory blocks in the shared memory pool are mapped one-to-one with the virtual memory of the application process.

[0178] A first identification processing module is used to perform a first identification processing on the target network message based on the network card driver to obtain a target network message descriptor when detecting that the message memory is successfully generated;

[0179] A second identification processing module, configured to perform a second identification processing on the message memory based on the target network message descriptor to obtain a first socket buffer;

[0180] The reading module is used to parse the first socket buffer through the network protocol stack, and read the network message data from the virtual memory corresponding to the message memory according to the parsing result to complete the processing of the network message.

[0181] As a preferred implementation, in an embodiment of the present invention, the message memory generation module is specifically used to:

[0182] Based on the first system call, a target device file is constructed, and the target device file is stored in the operating system kernel;

[0183] Based on the second system call, setting target parameters corresponding to the target device file;

[0184] Based on the target parameters, a shared memory pool is generated, where the shared memory pool includes at least a payload data unit and a control data unit.

[0185] As a preferred implementation, in an embodiment of the present invention, the message memory generation module is further configured to:

[0186] Based on the second system call, the attribute tag, memory block size, shared memory pool usage mode and number of memory blocks corresponding to the shared memory pool are set;

[0187] Defines the size of a memory page as the default memory block size.

[0188] As a preferred implementation, in an embodiment of the present invention, the apparatus further includes a mapping module, which is specifically configured to:

[0189] Based on the third system call, a memory page corresponding to the payload data unit is mapped into the virtual memory of the application process, where the payload data unit includes at least one memory page composed of memory blocks.

[0190] As a preferred implementation, in an embodiment of the present invention, the device further includes an interface construction module, and the mapping module is specifically configured to:

[0191] Based on the shared memory pool, an application programming interface is constructed. The application programming interface includes at least a memory block application interface, a memory block release interface, and a network card instance interface.

[0192] Based on the second system call and the network card instance interface, the shared memory pool is bound to the network card instance, wherein one shared memory pool can be bound to multiple network card instances;

[0193] Based on the memory block application interface and the memory block release interface, the shared memory pool is bound to other modules corresponding to the interface, and the other modules at least include a network card driver module and a socket module.

[0194] As a preferred implementation, in an embodiment of the present invention, the message memory generation module is further configured to:

[0195] In response to detecting a memory block quantity setting instruction corresponding to the shared memory pool, allocating a plurality of shared memory pages to the shared memory pool through the shared memory pool module;

[0196] Based on the shared memory page, a target number of memory blocks are obtained, and the addresses of the divided memory blocks are pushed into a target lock-free circular queue, wherein the physical addresses in a memory block are continuous.

[0197] As a preferred implementation, in an embodiment of the present invention, the message memory generation module is further configured to:

[0198] Lock the shared memory page.

[0199] As a preferred implementation, in an embodiment of the present invention, the apparatus further includes a precondition construction module, which is specifically configured to:

[0200] A socket module is constructed based on the first application, and a zero-copy option for receiving packets is set. Meanwhile, the first application and the network protocol stack respectively initialize their corresponding transaction ID variables to target values.

[0201] Building a shared memory pool based on the second application, and binding the shared memory pool to at least one network card instance;

[0202] As a preferred implementation, in an embodiment of the present invention, the message memory generation module is further configured to:

[0203] In response to receiving a target network message, detecting whether the target network card instance is bound to a pre-built shared memory pool;

[0204] In response to detecting that the target network card instance is bound to the pre-built shared memory pool, obtaining a target memory block from the shared memory pool;

[0205] Generate message memory based on the target memory block.

[0206] As a preferred implementation, in the embodiment of the present invention, the first identification processing module is specifically configured to:

[0207] Based on the target network message, use the network card driver to build a private message descriptor;

[0208] In response to detecting that the network card instance corresponding to the network card driver is bound to the shared memory pool, a packet receiving zero copy flag is set on the private message descriptor to generate a target network message descriptor.

[0209] As a preferred embodiment, in an embodiment of the present invention, the device further includes a data storage module, which is specifically used to:

[0210] Enqueue the target network packet descriptor into the packet receiving queue of the network card hardware;

[0211] The network card hardware copies the relevant data of the target network message to the message memory through DMA operation and sends a packet receiving instruction to the network card driver.

[0212] As a preferred implementation, in the embodiment of the present invention, the second identification processing module is specifically used to:

[0213] In response to detecting a packet receiving instruction, building an initial socket buffer for the message memory based on the network card driver;

[0214] In response to detecting that a packet receiving zero copy flag is set on the target network message descriptor, a shared memory pointer is set on the initial socket buffer to generate a first socket buffer.

[0215] As a preferred implementation, in an embodiment of the present invention, the device further includes a data storage and transmission module, and the data transmission module is specifically used to:

[0216] Based on the network protocol stack packet receiving function, the first socket buffer is sent to the network protocol stack.

[0217] As a preferred implementation, in the embodiment of the present invention, the reading module is specifically configured to:

[0218] In response to detecting the first socket buffer, performing upward layer-by-layer parsing processing on the first socket buffer using a network protocol stack;

[0219] Put the first socket buffer after upward layer-by-layer parsing into the corresponding socket module packet receiving queue, and send a packet receiving instruction to the application;

[0220] Based on the packet receiving instruction, the network message data is read from the virtual memory corresponding to the message memory.

[0221] As a preferred implementation, in an embodiment of the present invention, the reading module is further configured to:

[0222] In response to detecting the packet receiving instruction, detecting whether a shared memory pointer is set on the first socket buffer based on the first system call function;

[0223] In response to detecting that a shared memory pointer is set on the first socket buffer, a virtual memory address of the message corresponding to the shared memory pointer in the application process is assigned to the target field in the structure array parameter to read the network message data.

[0224] As a preferred implementation, in an embodiment of the present invention, the reading module is further configured to:

[0225] Record the current value of the transaction ID in the first socket buffer and add 1 to the transaction ID to obtain the second socket buffer;

[0226] The second socket buffer is transmitted to the socket buffer to be recycled queue in the socket module.

[0227] As a preferred implementation, in an embodiment of the present invention, the device further includes a detection module, which is specifically configured to:

[0228] Check whether the current field in the structure array parameter is consistent with the target field;

[0229] In response to detecting that the current field is inconsistent with the target field, the relationship between the current value of the transaction ID and the message memory is recorded.

[0230] As a preferred implementation, in an embodiment of the present invention, the device further includes a release module, which is specifically configured to:

[0231] Analyze and process network message data;

[0232] In response to detecting that the parsing process is complete, queuing the transaction ID corresponding to the message memory into an event queue of the socket module based on the second system call function;

[0233] Based on the transaction ID, determine the second socket cache from the socket cache to be recycled queue;

[0234] Based on the second socket buffer, the corresponding first socket buffer and the memory block corresponding to the first socket buffer are released.

[0235] For the specific definition of the network message processing device, please refer to the definition of the network message processing method above and will not be repeated here. Each module in the above-mentioned network message processing device can be implemented in whole or in part through software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each of the above modules.

[0236] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as shown in Figure 7. The computer device includes a processor, memory, a network interface, a display screen, and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and computer-readable instructions. The internal memory provides an environment for the operation of the operating system and computer-readable instructions in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When executed by the processor, the computer-readable instructions implement a network message processing method. The display screen of the computer device may be a liquid crystal display or an electronic ink display screen. The input device of the computer device may be a touch layer covering the display screen, or may be a key, trackball, or touchpad provided on the computer device housing, or may be an external keyboard, touchpad, or mouse.

[0237] Those skilled in the art will understand that the structure shown in FIG7 is merely a block diagram of a portion of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0238] In one embodiment, a computer device is provided, comprising a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor. When the processor executes the computer-readable instructions, the following steps are implemented:

[0239] S1: Receive the target network message and, based on the target network message, obtain the target memory block from the pre-built shared memory pool to generate the message memory. The memory block in the shared memory pool is mapped one-to-one with the virtual memory of the application process.

[0240] S2: In response to detecting that the message memory is successfully generated, based on the network card driver, performing a first identification process on the target network message to obtain a target network message descriptor;

[0241] S3: Based on the target network message descriptor, perform a second identification process on the message memory to obtain a first socket buffer;

[0242] S4: parsing the first socket buffer through the network protocol stack, and reading the network message data from the virtual memory corresponding to the message memory according to the parsing result, thereby completing the processing of the network message.

[0243] In one embodiment, when the processor executes the computer-readable instructions, it further implements the following steps:

[0244] Based on the first system call, a target device file is constructed, and the target device file is stored in the operating system kernel;

[0245] Based on the second system call, setting target parameters corresponding to the target device file;

[0246] Based on the target parameters, a shared memory pool is generated, where the shared memory pool includes at least a payload data unit and a control data unit.

[0247] In one embodiment, when the processor executes the computer-readable instructions, the processor further implements the following steps:

[0248] Based on the second system call, the attribute tag, memory block size, shared memory pool usage mode and number of memory blocks corresponding to the shared memory pool are set;

[0249] Defines the size of a memory page as the default memory block size.

[0250] In one embodiment, when the processor executes the computer-readable instructions, the processor further implements the following steps:

[0251] Based on the third system call, a memory page corresponding to the payload data unit is mapped into the virtual memory of the application process, where the payload data unit includes at least one memory page composed of memory blocks.

[0252] In one embodiment, when the processor executes the computer-readable instructions, the processor further implements the following steps:

[0253] Based on the shared memory pool, an application programming interface is constructed. The application programming interface includes at least a memory block application interface, a memory block release interface, and a network card instance interface.

[0254] Based on the second system call and the network card instance interface, the shared memory pool is bound to the network card instance, wherein one shared memory pool can be bound to multiple network card instances;

[0255] Based on the memory block application interface and the memory block release interface, the shared memory pool is bound to other modules corresponding to the interface, and the other modules at least include a network card driver module and a socket module.

[0256] In one embodiment, when the processor executes the computer-readable instructions, the processor further implements the following steps:

[0257] In response to detecting a memory block quantity setting instruction corresponding to the shared memory pool, allocating a plurality of shared memory pages to the shared memory pool through the shared memory pool module;

[0258] Based on the shared memory page, a target number of memory blocks are obtained, and the addresses of the divided memory blocks are pushed into a target lock-free circular queue, wherein the physical addresses in a memory block are continuous.

[0259] In one embodiment, when the processor executes the computer-readable instructions, it further implements the following steps:

[0260] Lock the shared memory page.

[0261] In one embodiment, when the processor executes the computer-readable instructions, it further implements the following steps:

[0262] A socket module is constructed based on the first application, and a zero-copy option for receiving packets is set. Meanwhile, the first application and the network protocol stack respectively initialize their corresponding transaction ID variables to target values.

[0263] Building a shared memory pool based on the second application, and binding the shared memory pool to at least one network card instance;

[0264] The first application is equal to or not equal to the second application.

[0265] In one embodiment, when the processor executes the computer-readable instructions, it further implements the following steps:

[0266] In response to receiving a target network message, detecting whether the target network card instance is bound to a pre-built shared memory pool;

[0267] In response to detecting that the target network card instance is bound to the pre-built shared memory pool, obtaining a target memory block from the shared memory pool;

[0268] Generate message memory based on the target memory block.

[0269] In one embodiment, when the processor executes the computer-readable instructions, it further implements the following steps:

[0270] Based on the target network message, use the network card driver to build a private message descriptor;

[0271] In response to detecting that the network card instance corresponding to the network card driver is bound to the shared memory pool, a packet receiving zero copy flag is set on the private message descriptor to generate a target network message descriptor.

[0272] In one embodiment, when the processor executes the computer-readable instructions, it further implements the following steps:

[0273] Enqueue the target network packet descriptor into the packet receiving queue of the network card hardware;

[0274] The network card hardware copies the relevant data of the target network message to the message memory through DMA operation and sends a packet receiving instruction to the network card driver.

[0275] In one embodiment, when the processor executes the computer-readable instructions, it further implements the following steps:

[0276] In response to detecting a packet receiving instruction, building an initial socket buffer for the message memory based on the network card driver;

[0277] In response to detecting that a packet receiving zero copy flag is set on the target network message descriptor, a shared memory pointer is set on the initial socket buffer to generate a first socket buffer.

[0278] In one embodiment, when the processor executes the computer-readable instructions, it further implements the following steps:

[0279] Based on the network protocol stack packet receiving function, the first socket buffer is sent to the network protocol stack.

[0280] In one embodiment, when the processor executes the computer-readable instructions, it further implements the following steps:

[0281] In response to detecting the first socket buffer, performing upward layer-by-layer parsing processing on the first socket buffer using a network protocol stack;

[0282] Put the first socket buffer after upward layer-by-layer parsing into the corresponding socket module packet receiving queue, and send a packet receiving instruction to the application;

[0283] Based on the packet receiving instruction, the network message data is read from the virtual memory corresponding to the message memory.

[0284] In one embodiment, when the processor executes the computer-readable instructions, it further implements the following steps:

[0285] In response to detecting the packet receiving instruction, detecting whether a shared memory pointer is set on the first socket buffer based on the first system call function;

[0286] In response to detecting that a shared memory pointer is set on the first socket buffer, a virtual memory address of the message corresponding to the shared memory pointer in the application process is assigned to the target field in the structure array parameter to read the network message data.

[0287] In one embodiment, when the processor executes the computer-readable instructions, it further implements the following steps:

[0288] Record the current value of the transaction ID in the first socket buffer and add 1 to the transaction ID to obtain the second socket buffer;

[0289] The second socket buffer is transmitted to the socket buffer to be recycled queue in the socket module.

[0290] In one embodiment, when the processor executes the computer-readable instructions, the processor further implements the following steps:

[0291] Check whether the current field in the structure array parameter is consistent with the target field;

[0292] In response to detecting that the current field is inconsistent with the target field, the association between the current value of the transaction ID and the message memory is recorded, and the transaction ID is incremented by 1.

[0293] In one embodiment, when the processor executes the computer-readable instructions, the processor further implements the following steps:

[0294] Analyze and process network message data;

[0295] In response to detecting that the parsing process is complete, queuing the transaction ID corresponding to the message memory into an event queue of the socket module based on the second system call function;

[0296] Based on the transaction ID, determine the second socket cache from the socket cache to be recycled queue;

[0297] Based on the second socket buffer, the corresponding first socket buffer and the memory block corresponding to the first socket buffer are released.

[0298] In one embodiment, one or more non-volatile computer-readable storage media storing computer-readable instructions are provided, wherein the computer-readable instructions are stored thereon and, when executed by a processor, perform the following steps:

[0299] S1: Receive the target network message and, based on the target network message, obtain the target memory block from the pre-built shared memory pool to generate the message memory. The memory block in the shared memory pool is mapped one-to-one with the virtual memory of the application process.

[0300] S2: In response to detecting that the message memory is successfully generated, based on the network card driver, performing a first identification process on the target network message to obtain a target network message descriptor;

[0301] S3: Based on the target network message descriptor, perform a second identification process on the message memory to obtain a first socket buffer;

[0302] S4: parsing the first socket buffer through the network protocol stack, and reading the network message data from the virtual memory corresponding to the message memory according to the parsing result, thereby completing the processing of the network message.

[0303] In one embodiment, the computer readable instructions, when executed by a processor, further implement the following steps:

[0304] Based on the first system call, a target device file is constructed, and the target device file is stored in the operating system kernel;

[0305] Based on the second system call, setting target parameters corresponding to the target device file;

[0306] Based on the target parameters, a shared memory pool is generated, where the shared memory pool includes at least a payload data unit and a control data unit.

[0307] In one embodiment, the computer readable instructions, when executed by a processor, further implement the following steps:

[0308] Based on the second system call, the attribute tag, memory block size, shared memory pool usage mode and number of memory blocks corresponding to the shared memory pool are set;

[0309] Defines the size of a memory page as the default memory block size.

[0310] In one embodiment, the computer readable instructions, when executed by a processor, further implement the following steps:

[0311] Based on the third system call, a memory page corresponding to the payload data unit is mapped into the virtual memory of the application process, where the payload data unit includes at least one memory page composed of memory blocks.

[0312] In one embodiment, the computer readable instructions, when executed by a processor, further implement the following steps:

[0313] Based on the shared memory pool, an application programming interface is constructed. The application programming interface includes at least a memory block application interface, a memory block release interface, and a network card instance interface.

[0314] Based on the second system call and the network card instance interface, the shared memory pool is bound to the network card instance, wherein one shared memory pool can be bound to multiple network card instances;

[0315] Based on the memory block application interface and the memory block release interface, the shared memory pool is bound to other modules corresponding to the interface, and the other modules at least include a network card driver module and a socket module.

[0316] In one embodiment, the computer readable instructions, when executed by a processor, further implement the following steps:

[0317] In response to detecting a memory block quantity setting instruction corresponding to the shared memory pool, allocating a plurality of shared memory pages to the shared memory pool through the shared memory pool module;

[0318] Based on the shared memory page, a target number of memory blocks are obtained, and the addresses of the divided memory blocks are pushed into a target lock-free circular queue, wherein the physical addresses in a memory block are continuous.

[0319] In one embodiment, the computer readable instructions, when executed by a processor, further implement the following steps:

[0320] Lock the shared memory page.

[0321] In one embodiment, the computer readable instructions, when executed by a processor, further implement the following steps:

[0322] A socket module is constructed based on the first application, and a zero-copy option for receiving packets is set. Meanwhile, the first application and the network protocol stack respectively initialize their corresponding transaction ID variables to target values.

[0323] Building a shared memory pool based on the second application, and binding the shared memory pool to at least one network card instance;

[0324] The first application is equal to or not equal to the second application.

[0325] In one embodiment, the computer readable instructions, when executed by a processor, further implement the following steps:

[0326] In response to receiving a target network message, detecting whether the target network card instance is bound to a pre-built shared memory pool;

[0327] In response to detecting that the target network card instance is bound to the pre-built shared memory pool, obtaining a target memory block from the shared memory pool;

[0328] Generate message memory based on the target memory block.

[0329] In one embodiment, the computer readable instructions, when executed by a processor, further implement the following steps:

[0330] Based on the target network message, use the network card driver to build a private message descriptor;

[0331] In response to detecting that the network card instance corresponding to the network card driver is bound to the shared memory pool, a packet receiving zero copy flag is set on the private message descriptor to generate a target network message descriptor.

[0332] In one embodiment, the computer readable instructions, when executed by a processor, further implement the following steps:

[0333] Enqueue the target network packet descriptor into the packet receiving queue of the network card hardware;

[0334] The network card hardware copies the relevant data of the target network message to the message memory through DMA operation and sends a packet receiving instruction to the network card driver.

[0335] In one embodiment, the computer readable instructions, when executed by a processor, further implement the following steps:

[0336] In response to detecting a packet receiving instruction, building an initial socket buffer for the message memory based on the network card driver;

[0337] In response to detecting that a packet receiving zero copy flag is set on the target network message descriptor, a shared memory pointer is set on the initial socket buffer to generate a first socket buffer.

[0338] In one embodiment, the computer readable instructions, when executed by a processor, further implement the following steps:

[0339] Based on the network protocol stack packet receiving function, the first socket buffer is sent to the network protocol stack.

[0340] In one embodiment, the computer readable instructions, when executed by a processor, further implement the following steps:

[0341] In response to detecting the first socket buffer, performing upward layer-by-layer parsing processing on the first socket buffer using a network protocol stack;

[0342] Put the first socket buffer after upward layer-by-layer parsing into the corresponding socket module packet receiving queue, and send a packet receiving instruction to the application;

[0343] Based on the packet receiving instruction, the network message data is read from the virtual memory corresponding to the message memory.

[0344] In one embodiment, the computer readable instructions, when executed by a processor, further implement the following steps:

[0345] In response to detecting the packet receiving instruction, detecting whether a shared memory pointer is set on the first socket buffer based on the first system call function;

[0346] In response to detecting that a shared memory pointer is set on the first socket buffer, a virtual memory address of the message corresponding to the shared memory pointer in the application process is assigned to the target field in the structure array parameter to read the network message data.

[0347] In one embodiment, the computer readable instructions, when executed by a processor, further implement the following steps:

[0348] Record the current value of the transaction ID in the first socket buffer and add 1 to the transaction ID to obtain the second socket buffer;

[0349] The second socket buffer is transmitted to the socket buffer to be recycled queue in the socket module.

[0350] In one embodiment, the computer readable instructions, when executed by a processor, further implement the following steps:

[0351] Check whether the current field in the structure array parameter is consistent with the target field;

[0352] In response to detecting that the current field is inconsistent with the target field, the association between the current value of the transaction ID and the message memory is recorded, and the transaction ID is incremented by 1.

[0353] In one embodiment, when the computer readable instructions are executed by a processor, the following steps are further implemented:

[0354] Analyze and process network message data;

[0355] In response to detecting that the parsing process is complete, queuing the transaction ID corresponding to the message memory into an event queue of the socket module based on the second system call function;

[0356] Based on the transaction ID, determine the second socket cache from the socket cache to be recycled queue;

[0357] Based on the second socket buffer, the corresponding first socket buffer and the memory block corresponding to the first socket buffer are released.

[0358] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing related hardware through computer-readable instructions, and the computer-readable instructions can be stored in a non-volatile computer-readable storage medium. When the computer-readable instructions are executed, they may include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0359] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0360] The above embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A network message processing method, characterized in that: The method comprises: Receive a target network message, and based on the target network message, obtain a target memory block from a pre-built shared memory pool to generate a message memory, wherein the memory blocks in the shared memory pool are mapped one-to-one with the virtual memory of the application process; In response to detecting that the message memory is successfully generated, based on the network card driver, performing a first identification process on the target network message to obtain a target network message descriptor; Based on the target network message descriptor, performing a second identification process on the message memory to obtain a first socket buffer; and The first socket buffer is parsed through the network protocol stack, and according to the parsing result, the network message data is read from the virtual memory corresponding to the message memory to complete the processing of the network message.

2. The network message processing method according to claim 1, characterized in that: The method for constructing the shared memory pool includes: Based on the first system call, construct a target device file, wherein the target device file is stored in the operating system kernel; Based on the second system call, setting target parameters corresponding to the target device file; and Based on the target parameters, the shared memory pool is generated, where the shared memory pool at least includes a payload data unit and a control data unit.

3. The network message processing method according to claim 2, characterized in that: The step of setting the target parameters corresponding to the target device file based on the second system call includes: Based on the second system call, setting the attribute tag, memory block size, shared memory pool usage mode and number of memory blocks corresponding to the shared memory pool; and Defining the size of a memory page as the default memory block size; The target parameters include at least the attribute label corresponding to the shared memory pool, the memory block size, the shared memory pool usage mode and the number of memory blocks.

4. The network message processing method according to claim 2, characterized in that: After generating the shared memory pool based on the target parameters, the method further includes: Based on the third system call, a memory page corresponding to the payload data unit is mapped into the virtual memory of the application process, wherein the payload data unit includes at least one memory page composed of memory blocks.

5. The network message processing method according to claim 1, characterized in that: After the shared memory pool is constructed, the method further includes: Based on the shared memory pool, an application program interface is constructed, wherein the application program interface at least includes a memory block application interface, a memory block release interface, and a network card instance interface; Based on the second system call and the network card instance interface, binding the shared memory pool to the network card instance, wherein one shared memory pool can be bound to multiple network card instances; and Based on the memory block application interface and the memory block release interface, the shared memory pool is bound to other modules corresponding to the interfaces, and the other modules at least include a network card driver module and a socket module.

6. The network message processing method according to claim 2, characterized in that: The control data unit includes at least one lock-free circular queue. When setting the number of memory blocks corresponding to the shared memory pool, the method further includes: In response to detecting a memory block number setting instruction corresponding to the shared memory pool, allocating a plurality of shared memory pages to the shared memory pool by a shared memory pool module; and Based on the shared memory page, a target number of memory blocks are obtained by division, and the addresses of the divided memory blocks are pushed into a target lock-free circular queue, wherein the physical addresses in one memory block are continuous.

7. The network message processing method according to claim 6, characterized in that: After allocating a plurality of shared memory pages to the shared memory pool by the shared memory pool module, the method further includes: Locking the shared memory page.

8. The network message processing method according to claim 3, characterized in that: Before obtaining a target memory block from a pre-built shared memory pool based on the target network message and generating a message memory, the method further includes: Building a socket module based on the first application and setting a zero-copy option for receiving packets. At the same time, the first application and the network protocol stack respectively initialize their corresponding transaction ID variables to target values; and Building a shared memory pool based on the second application, and binding the shared memory pool to at least one network card instance; The first application is equal to or not equal to the second application.

9. The network message processing method according to claim 1, characterized in that: Based on the target network message, a target memory block is obtained from a pre-built shared memory pool, and message memory is generated, including: In response to receiving the target network message, detecting whether the target network card instance is bound to the pre-built shared memory pool; In response to detecting that the target network card instance is bound to a pre-built shared memory pool, obtaining a target memory block from the shared memory pool; and The message memory is generated based on the target memory block.

10. The network message processing method according to claim 1, characterized in that: In response to detecting that the message memory is successfully generated, based on the network card driver, the target network message is subjected to a first identification process to obtain a target network message descriptor including: Based on the target network message, using the network card driver, construct a private message descriptor; and In response to detecting that the network card instance corresponding to the network card driver is bound to the shared memory pool, a packet receiving zero copy flag is set on the private message descriptor to generate the target network message descriptor.

11. The network message processing method according to claim 10, characterized in that: After performing a first identification process on the target network message based on the network card driver to obtain a target network message descriptor, the method further includes: Enqueue the target network message descriptor into the packet receiving queue of the network card hardware; The network card hardware copies the relevant data of the target network message to the message memory through DMA operation, and sends a packet receiving instruction to the network card driver.

12. The network message processing method according to claim 1, characterized in that: Based on the target network message descriptor, performing a second identification process on the message memory to obtain a first socket buffer includes: In response to detecting a packet receiving instruction, constructing an initial socket buffer for the message memory based on a network card driver; and In response to detecting that a packet receiving zero copy flag is set on the target network message descriptor, a shared memory pointer is set on the initial socket buffer to generate the first socket buffer.

13. The network message processing method according to claim 1, characterized in that: After performing a second identification process on the message memory based on the target network message descriptor to obtain a first socket buffer, the method further includes: Based on a network protocol stack packet receiving function, the first socket buffer is sent to the network protocol stack.

14. The network message processing method according to claim 1, characterized in that: Parsing the first socket buffer through the network protocol stack, and reading the network message data from the virtual memory corresponding to the message memory according to the parsing result includes: In response to detecting the first socket buffer, performing upward layer-by-layer parsing processing on the first socket buffer using the network protocol stack; placing the first socket buffer after the upward layer-by-layer parsing processing into a corresponding socket module packet receiving queue, and sending a packet receiving instruction to the application; and Based on the packet receiving instruction, network message data is read from the virtual memory corresponding to the message memory.

15. The network message processing method according to claim 14, characterized in that: Reading the network message data from the virtual memory corresponding to the message memory based on the packet receiving instruction includes: In response to detecting the packet receiving instruction, detecting whether a shared memory pointer is set on the first socket buffer based on the first system call function; and In response to detecting that a shared memory pointer is set on the first socket buffer, a virtual memory address in the application process corresponding to the shared memory pointer in the message is assigned to the target field in the structure array parameter to read the network message data.

16. The network message processing method according to claim 14, characterized in that: When reading the network message data, the method further includes: Recording the current value of the transaction ID in the first socket buffer, and adding 1 to the transaction ID to obtain a second socket buffer; and The second socket buffer is transmitted to the socket buffer to-be-reclaimed queue in the socket module.

17. The network message processing method according to claim 15, characterized in that: After reading the network message data, the method further includes: Check whether the current field in the structure array parameter is consistent with the target field; and In response to detecting that the current field is inconsistent with the target field, the association between the current value of the transaction ID and the message memory is recorded, and an increment operation is performed on the transaction ID.

18. The network message processing method according to claim 1, characterized in that: The method further comprises: Parsing and processing the network message data; In response to detecting that the parsing process is completed, the transaction ID corresponding to the message memory is queued into the event queue of the socket module based on the second system call function; Based on the transaction ID, determining a second socket buffer from the socket buffer to be recycled queue; and Based on the second socket cache, the corresponding first socket cache and the memory block corresponding to the first socket cache are released.

19. A network message processing device, characterized in that: The device comprises: A message memory generation module is used to receive a target network message and, based on the target network message, obtain a target memory block from a pre-built shared memory pool to generate a message memory, wherein the memory blocks in the shared memory pool are mapped one-to-one with the virtual memory of the application process; A first identification processing module is configured to perform a first identification processing on the target network message based on a network card driver to obtain a target network message descriptor when detecting that the message memory is successfully generated; a second identification processing module, configured to perform a second identification processing on the message memory based on the target network message descriptor to obtain a first socket buffer; and The reading module is used to parse the first socket buffer through the network protocol stack, and read the network message data from the virtual memory corresponding to the message memory according to the parsing result to complete the processing of the network message.

20. A computer device comprising a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, wherein: When the processor executes the computer-readable instructions, the method of any one of claims 1 to 18 is implemented.

21. One or more non-volatile computer-readable storage media storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the steps of the method according to any one of claims 1 to 18.

Images