Network communication system based on peer-to-peer network communication middleware

Abstract

The invention belongs to the technical field of data communication processing, and particularly relates to a network communication system based on peer-to-peer network communication middleware, which comprises an initialization process and a data exchange process, and solves the problems that in the prior art, unified management of endpoints and a main memory is lacked, and flexible communication between the endpoints cannot be realized. According to the technical scheme of the invention, the PCIe space is managed in a unified manner, and then a convenient single-multicast communication manner is constructed in a queue virtualization manner, so that the technical effect of direct communication between the endpoints is realized.

Description

Technical Field

[0001] The invention belongs to the technical field of data communication processing, and in particular relates to a network communication system based on peer-to-peer network communication middleware. Background Art

[0002] PCIe is generally used for intra-board interconnection to solve the CPU fan-out problem. As a bus protocol, PCIe has the characteristics of high speed, high bandwidth and low latency. It is widely used for data exchange between CPU, NVME, FPGA, GPU and DPU. Since PCIe itself only supports tree networking and simple routing, it is very difficult for PCIe to build peer-to-peer communication between RC and EP, and between EP and EP. Currently, most of the market only uses PCIe for local interconnection, but with the development of GPU and the rise of AI computing, PCIe has become the de facto standard for interconnection between GPU, CPU and DPU. Among them, NVIDIA has its own standard but it is not open source. Due to the imperfect implementation of the Linux kernel, after multiple GPUs, CPUs, DUPs and FPGAs are interconnected through PCIe, it is not possible to easily initiate communication from any node like Ethernet. At this time, a middleware is needed to solve the problem of unified addressing and routing of devices. Traditional communication process: IO is generally moved to the main memory driver space through DMA; the main memory moves from the IO driver space to the general computing node or GPU driver space through memory copy; the general computing and GPU are then moved to the internal space through DMA. Therefore, one memory copy and two DMA operations are required to complete one communication.

[0003] In summary, the prior art has the problem of lack of unified management of endpoints and main memory and inability of flexible communication between endpoints. Summary of the invention

[0004] The present invention provides a network communication system based on peer-to-peer network communication middleware to solve the above problems existing in the above background technology. Specifically, the technical problem solved by the present invention is achieved by adopting the following technical solutions: A network communication system based on peer-to-peer network communication middleware, comprising: The root complex RC communicates directly with the endpoint EP through the PCIe middleware, including: Initialization process: RC side user layer: notifies the RC side service driver layer of RC side initialization and data source application; RC side business driver layer: in response to the upper layer RC side initialization request, sends the queue creation request to the PCIe middleware, and forwards the queue related parameters to the RC side user layer according to the queue related parameters returned by the lower layer. At the same time, in response to the upper layer data source application request, forwards the data source application request to the PCIe middleware, and returns the related status of the EP side hardware queue to the RC side user layer according to the successful saving status of the lower layer. PCIe middleware: In response to the upper layer's request to create a queue, create a hardware queue on the RC side, and modify the RC side queue initialization state to a sendable state according to the current operation state of the endpoint EP; in response to the upper layer's data source application request, send the RC side queue mapping to the endpoint EP, and according to the successful save state of the endpoint EP, the relevant state of the EP side hardware queue is returned to the RC side service driver layer; Data exchange process: The user layer on the RC side: starts the data transmission of this batch and notifies the service driver layer on the RC side that the data transmission has started, and determines the completion status of the data transmission of this layer based on the upper layer data transmission completion status feedback returned by the service driver layer on the RC side; RC side business driver layer: responds to the upper layer data transmission start notification, executes RC side data storage, and notifies the PCIe middleware of the data transmission start, and judges the data transmission completion status of this layer according to the RC side move request feedback returned by the PCIe middleware, and at the same time, feeds back the data transmission completion status of this layer to the RC side user layer; PCIe middleware: In response to the upper-layer data transfer start notification, the converted DMA-related operations are sent down to the endpoint EP, and based on the EP-side interrupt notification feedback returned by the endpoint EP, the completion status of the batch transfer on the RC side is determined. At the same time, the RC-side transfer request is fed back to the RC-side business driver layer.

[0005] Preferably, the initialization process includes: RC side user layer: sends the RC side initialization request to the RC side business driver layer, and at the same time, sends the data source application request to the RC side business driver layer; RC side business driver layer: responds to the upper layer RC side initialization request, sends the queue creation request to the PCIe middleware; forwards the queue number, queue base address, and maximum number of queues to the RC side user layer according to the relevant parameters of the queue returned by the lower layer; if the RC side initialization request fails, the reason for the RC side initialization failure is returned to the RC side user layer; at the same time, responds to the upper layer data source application request, forwards the data source application request to the PCIe middleware; according to the lower layer's successful saving status, if successful, returns the RC side queue address to the RC side business driver layer; if failed, returns the reason for the failure to the RC side user layer; PCIe middleware: in response to the upper layer's request to create a queue, create a hardware queue on the RC side, select the EP DMA engine of this endpoint, and create a queue mapping relationship for the hardware queue on the RC side; send the EP side hardware queue initialization request to the endpoint EP; according to the current operation status of the EP of this endpoint, modify the RC side queue initialization status to a sendable status, and at the same time, return the queue number, queue base address, and maximum number of queues to the RC side business driver layer. If the RC side initialization request fails, the reason for the RC side initialization failure is returned to the RC side business driver layer; in response to the upper layer data source application request, send the RC side queue mapping to the endpoint EP, and according to the successful saving status of the EP of this endpoint, if successful, return the RC side queue address to the RC side business driver layer, if failed, return the failure reason to the RC side business driver layer; The data exchange process includes: RC side user layer: used to send the RC read and write requests of this batch to the RC side business driver layer after starting the data transmission of this batch; at the same time, according to the upper layer data transmission completion status feedback returned by the RC side business driver layer, determine whether the RC side has completed the migration of this batch. If it is completed, send the subsequent operation notification to the RC side business driver layer; at the same time, determine whether the business logic is completed. If it is completed, start the next batch of data transmission; RC side business driver layer: used to receive the business data of this batch of RC read and write requests and put them into the write storage space according to this batch of RC read and write requests; construct this batch of RC read and write requests as PCIe read and write requests in a packet format compatible with the standard PCIe link layer protocol, wherein the PCIe read and write request format is [RC side business data address, data length, request ID]; send PCIe read and write requests and PCIe write notifications to the PCIe middleware; judge whether the RC side has completed the move of this batch according to the feedback of the RC side move request returned by the PCIe middleware, and if it has been completed, send the completion notification to the RC side user layer; judge whether the subsequent operations are completed, and send the business logic completion notification to the RC side user layer; PCIe middleware: used to determine the legality of PCIe read and write requests based on PCIe read and write requests. If legal, the PCIe read and write requests are converted into standard DMA write operations; and the DMA write operations are sent down to the endpoint EP; based on the EP-side interrupt notification feedback returned by the endpoint EP, it is determined whether the PCIe read and write tasks are completed, and the RC-side move request is returned to the RC-side business driver layer and it is determined whether the RC-side move of this batch is completed. If completed, wait for the next batch of data transmission.

[0006] Preferably, the network communication system includes a root complex RC, a PCIe middleware, and an endpoint EP; The root complex RC includes an RC side user mode area and an RC side kernel mode area, wherein the RC side user mode area includes an RC side user layer, and the RC side kernel mode area includes an RC side business driver layer, a PCIe domain resource management module of a PCIe middleware, and an RC side hardware driver layer.

[0007] At the same time, the present invention also provides a network communication system based on peer-to-peer network communication middleware, including: The endpoint EP communicates directly with the root complex RC through the PCIe middleware, including: Initialization process: EP-side service driver layer: converts the EP-side hardware queue initialization request sent by the PCIe middleware into an EP-side hardware initialization operation, and forwards the EP-side hardware queue initialization request to the EP-side application layer; The EP side application layer: performs the EP side queue initialization operation according to the EP side hardware queue initialization request forwarded by the EP side service driver layer; Data exchange process: EP-side service driver layer: according to the DMA write operation issued by the PCIe middleware, convert the DMA write operation into a DMA descriptor, and send the DMA descriptor to the EP-side application layer; convert the RC-side service data address of the DMA write operation into the EP-side service data address, and send the EP-side service data address to the EP-side application layer; EP side application layer: in response to the upper layer EP side service data address, saves the EP side service data based on the EP side service data address locally.

[0008] Preferably, the initialization process: EP-side business driver layer: according to the EP-side hardware queue initialization request sent by the PCIe middleware, it is converted into the EP-side hardware initialization operation, and the EP-side hardware queue initialization request is forwarded to the EP-side application layer; according to the EP-side queue initialization status returned by the EP-side application layer, the EP-side queue initialization status is forwarded to the PCIe middleware; according to the RC-side queue mapping sent by the PCIe middleware, the memory mapping of the application source data is saved, and the current operation status is returned to the PCIe middleware; The EP side application layer: performs the EP side queue initialization operation according to the EP side hardware queue initialization request forwarded by the EP side service driver layer, and returns the EP side queue initialization status to the EP side service driver layer; Data exchange process: EP-side service driver layer: according to the DMA write operation sent by the PCIe middleware, convert the DMA write operation into a DMA descriptor and send the DMA descriptor to the EP-side application layer; convert the RC-side service data address of the DMA write operation into the EP-side service data address, and send the EP-side service data address to the EP-side application layer, split the DMA write operation into various DMA sub-operations, and trigger each DMA sub-operation; in response to the lower-layer interrupt notification feedback, forward the EP-side interrupt notification to the PCIe middleware and determine whether the EP-side data polling is completed; EP side application layer: responds to the upper-layer DMA descriptor, forwards the DMA descriptor to the EP side base layer; responds to the upper-layer EP side service data address, saves the EP side service data based on the EP side service data address locally; returns the EP side interrupt notification to the EP side service driver layer and determines whether the EP side data polling is completed.

[0009] Preferably, the network communication system includes a root complex RC, a PCIe middleware, and an endpoint EP; The endpoint EP includes an EP side kernel state area and an EP side application state area, wherein the EP side kernel state area includes an EP side hardware driver layer and an EP node resource management module of the EP side base layer, and the EP side application state area includes an EP side application layer.

[0010] At the same time, the present invention also provides a network communication system based on peer-to-peer network communication middleware, including: The first endpoint EP directly communicates with the second endpoint EP through the PCIe middleware, which includes: Initialization process: The application layer on the first EP side: notifies the service driver layer on the first EP side of the initialization request and data source application on the first EP side; The first EP side service driver layer: in response to the upper layer first EP side initialization request, sends a queue creation request to the PCIe middleware, and forwards the queue related parameters to the first EP side application layer according to the queue related parameters returned by the lower layer, and at the same time, in response to the upper layer data source application request, forwards the data source application request to the PCIe middleware, and returns the related status of the second EP side hardware queue to the first EP side application layer according to the successful saving status of the lower layer; PCIe middleware: in response to the upper layer request to create a queue, create a hardware queue on the RC side, and modify the initialization state of the queue on the first EP side to a sendable state according to the current operation state of the second endpoint EP; in response to the upper layer data source application request, map the first EP queue to the second endpoint EP, and return the relevant state of the hardware queue on the first EP side to the service driver layer on the first EP side according to the successful save state of the second endpoint EP; Data exchange process: The application layer on the first EP side starts the data transmission of this batch and notifies the service driver layer on the first EP side to start the data transmission, and determines the completion status of the data transmission of this layer according to the upper layer data transmission completion status feedback returned by the service driver layer on the first EP side; The first EP side service driver layer: in response to the upper layer data transmission start notification, executes the first EP side data storage, notifies the PCIe middleware of the data transmission start, and determines the data transmission completion status of this layer according to the first EP move request feedback returned by the PCIe middleware, and at the same time, feeds back the data transmission completion status of this layer to the first EP side application layer; PCIe middleware: In response to the upper layer data transfer start notification, the converted DMA related operations are sent down to the second endpoint EP, and based on the second EP side interrupt notification feedback returned by the second endpoint EP, the completion status of the batch transfer on the first EP side is determined, and at the same time, the first EP side transfer request is fed back to the first EP side business driver layer.

[0011] Preferably, the initialization process includes: The user layer on the first EP side: sends the first EP side initialization request to the service driver layer on the first EP side, and at the same time, sends the data source application request to the service driver layer on the first EP side; The first EP side service driver layer: responds to the upper layer first EP side initialization request, sends the queue creation request to the PCIe middleware; forwards the queue number, queue base address, and maximum number of queues to the first EP side user layer according to the relevant parameters of the queue returned by the lower layer; if the first EP side initialization request fails, returns the first EP side initialization failure reason to the first EP side user layer; at the same time, responds to the upper layer data source application request, forwards the data source application request to the PCIe middleware; according to the lower layer save success status, if successful, returns the first EP side queue address to the first EP side service driver layer, if failed, returns the failure reason to the first EP side user layer; PCIe middleware: in response to the upper layer's request to create a queue, create a hardware queue on the RC side, select the second endpoint EP DMA engine, and create a queue mapping relationship for the hardware queue on the first EP side; send the hardware queue initialization request on the first EP side to the endpoint EP; according to the current operation status of the second endpoint EP, modify the queue initialization status on the first EP side to a sendable status, and at the same time, return the queue number, queue base address, and maximum number of queues to the first EP side business driver layer. If the initialization request on the first EP side fails, return the reason for the initialization failure on the first EP side to the business driver layer on the first EP side; in response to the upper layer's data source application request, send the first EP side queue mapping down to the second endpoint EP, and according to the successful saving status of the second endpoint EP, if successful, return the second EP side queue address to the first EP side business driver layer, and if failed, return the reason for the failure to the first EP side business driver layer; The data exchange process includes: The application layer on the first EP side is used to send the RC read and write request of this batch to the business driver layer on the first EP side after starting the data transmission of this batch; at the same time, according to the upper layer data transmission completion status feedback returned by the business driver layer on the first EP side, it is judged whether the migration of this batch on the first EP side is completed. If it is completed, the subsequent operation notification is sent to the business driver layer on the RC side; at the same time, it is judged whether the business logic is completed. If it is completed, the next batch of data transmission is started; The first EP side business driver layer: is used to receive the business data of the current batch of RC read and write requests and put them into the write storage space according to the current batch of RC read and write requests; construct the current batch of first EP read and write requests as PCIe read and write requests in a packet format compatible with the standard PCIe link layer protocol, wherein the PCIe read and write request format is [RC side business data address, data length, request ID]; send the PCIe read and write request and PCIe write notification to the PCIe middleware; according to the feedback of the first EP side move request returned by the PCIe middleware, determine whether the current batch move of the first EP side is completed, and if completed, send the completion notification to the first EP side application layer; determine whether the subsequent operation is completed, and send the business logic completion notification to the first EP side user layer; PCIe middleware: used to determine the legality of PCIe read / write requests according to PCIe read / write requests. If legal, convert the PCIe read / write requests into standard DMA write operations; and send the DMA write operations to the second endpoint EP; determine whether the PCIe read / write task is completed according to the EP-side interrupt notification feedback returned by the second endpoint EP, and return the first EP-side move request to the first EP-side business driver layer and determine whether the first EP-side move is completed. If completed, wait for the next batch of data transmission; The network communication system includes a first endpoint EP, a PCIe middleware, and a second endpoint EP; The first endpoint EP includes a first EP side user state area and a first EP side kernel state area, wherein the first EP side user state area includes a first EP side user layer, and the first EP side kernel state area includes a first EP side service driver layer, a PCIe domain resource management module of the PCIe middleware, and a first EP side hardware driver layer.

[0012] At the same time, the present invention also provides a network communication system based on peer-to-peer network communication middleware, including: The second endpoint EP directly communicates with the first endpoint EP through the PCIe middleware, including: Initialization process: The second EP side service driver layer: converts the second EP side hardware queue initialization request sent by the PCIe middleware into a second EP side hardware initialization operation, and forwards the second EP side hardware queue initialization request to the second EP side application layer; The second EP side application layer: performs the second EP side queue initialization operation according to the second EP side hardware queue initialization request forwarded by the second EP side service driver layer; Data exchange process: The second EP side service driver layer: according to the DMA write operation issued by the PCIe middleware, converts the DMA write operation into a DMA descriptor, and sends the DMA descriptor to the second EP side application layer; converts the second EP side service data address of the DMA write operation into the second EP side service data address, and sends the second EP side service data address to the second EP side application layer; The second EP side application layer: in response to the upper layer second EP side service data address, saves the second EP side service data based on the second EP side service data address locally.

[0013] Preferably, the initialization process: The second EP side service driver layer: according to the second EP side hardware queue initialization request issued by the PCIe middleware, convert it into the second EP side hardware initialization operation, and forward the second EP side hardware queue initialization request to the second EP side application layer; according to the second EP side queue initialization status returned by the second EP side application layer, forward the second EP side queue initialization status to the PCIe middleware; according to the second EP side queue mapping issued by the PCIe middleware, save the application source data memory mapping, and return the current operation status to the PCIe middleware; The second EP side application layer: performs the second EP side queue initialization operation according to the second EP side hardware queue initialization request forwarded by the second EP side service driver layer, and returns the second EP side queue initialization state to the second EP side service driver layer; Data exchange process: The second EP side service driver layer: according to the DMA write operation sent by the PCIe middleware, convert the DMA write operation into a DMA descriptor, and send the DMA descriptor to the second EP side application layer; convert the second EP side service data address of the DMA write operation into the second EP side service data address, and send the second EP side service data address to the second EP side application layer, split the DMA write operation into each DMA sub-operation, and trigger each DMA sub-operation; in response to the lower layer interrupt notification feedback, forward the second EP side interrupt notification to the PCIe middleware and determine whether the second EP side data polling is completed; The second EP side application layer: in response to the upper layer DMA descriptor, forwards the DMA descriptor to the second EP side base layer; in response to the upper layer second EP side service data address, saves the second EP side service data based on the second EP side service data address locally; returns the second EP side interrupt notification to the second EP side service driver layer and determines whether the second EP side data polling is completed; The network communication system includes a first endpoint EP, a PCIe middleware, and a second endpoint EP; The second endpoint EP includes a second EP side inner kernel state area and a second EP side application state area, wherein the second EP side inner kernel state area includes a second EP side hardware driver layer and a second EP node resource management module of the second EP side base layer, and the second EP side application state area includes a second EP side application layer.

[0014] Beneficial technical effects:

[0015] The present solution generally includes four modes, namely, (i) the root complex RC directly communicates with the endpoint EP through the PCIe middleware, (ii) the endpoint EP directly communicates with the root complex RC through the PCIe middleware, (iii) the first endpoint EP directly communicates with the second endpoint EP through the PCIe middleware, and (iv) the second endpoint EP directly communicates with the first endpoint EP through the PCIe middleware. The first and second modes achieve the purpose of direct communication between the root complex RC and the endpoint EP through the PCIe middleware, and the third and fourth modes achieve the purpose of direct communication between the first endpoint EP and the second endpoint EP through the PCIe middleware. The purpose of direct communication of e middleware is as follows: unified management of endpoints and main memory, that is, EP endpoint maps internal memory based on BAR and registers it to RC; RC uniformly manages endpoint mapped memory and main memory; endpoints can flexibly communicate in P2P: that is, between IO endpoints and general computing endpoints, or between IO endpoints and GPU, or between general computing and GPU, RC and endpoints; among them, one communication is completed with 0 memory copy and 1 DMA operation. This solution uniformly manages PCIe space, and then builds convenient single and multicast communication methods through queue virtualization to realize direct communication between endpoints. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a data exchange flow chart of the root complex RC of the present invention directly communicating with the endpoint EP through the PCIe middleware; Figure 2 It is a data exchange flow chart of the endpoint EP of the present invention directly communicating with the root complex RC through the PCIe middleware; Figure 3 It is a data exchange flow chart of the first endpoint EP of the present invention directly communicating with the second endpoint EP through the PCIe middleware; Figure 4 It is a data exchange flow chart of the second endpoint EP of the present invention directly communicating with the first endpoint EP through the PCIe middleware; Figure 5 It is a communication flow chart of the root complex RC and the endpoint EP of the present invention; Figure 6 is a communication flow chart between the first endpoint EP and the second endpoint EP of the present invention; Figure 7 It is a topological structure diagram of the network communication system of the present invention; Figure 8 It is a module structure diagram of the network communication system of the present invention; Fig. 9 It is a resource structure diagram of the network communication system of the present invention. DETAILED DESCRIPTION

[0017] The present invention is further described below in conjunction with the accompanying drawings: In the figure: S101-notify the RC side service driver layer of RC side initialization and data source application; S102-In response to the upper layer RC side initialization request, send a queue creation request to the PCIe middleware, and forward the queue related parameters to the RC side user layer according to the queue related parameters returned by the lower layer. At the same time, in response to the upper layer data source application request, forward the data source application request to the PCIe middleware, and return the relevant status of the EP side hardware queue to the RC side user layer according to the successful saving status of the lower layer; S103-In response to the upper layer's request to create a queue, create a hardware queue on the RC side, and modify the RC side queue initialization state to a sendable state according to the current operation state of the endpoint EP; in response to the upper layer's request to apply for a data source, send the RC side queue mapping to the endpoint EP, and according to the successful saving state of the endpoint EP, the relevant state of the EP side hardware queue is returned to the RC side service driver layer; S104-Start the data transmission of this batch and notify the service driver layer on the RC side to start the data transmission, and judge the completion status of the data transmission of this layer according to the upper layer data transmission completion status feedback returned by the service driver layer on the RC side; S105-In response to the upper layer data transmission start notification, execute RC side data storage, notify the PCIe middleware of the data transmission start, and judge the completion status of the data transmission at this layer according to the RC side move request feedback returned by the PCIe middleware, and at the same time, feedback the completion status of the data transmission at this layer to the RC side user layer; S106-In response to the upper layer data transfer start notification, the converted DMA related operations are sent to the endpoint EP, and according to the EP side interrupt notification feedback returned by the endpoint EP, the completion status of the batch transfer on the RC side is determined, and at the same time, the RC side transfer request is fed back to the RC side service driver layer; S201-Convert the EP side hardware queue initialization request sent by the PCIe middleware into an EP side hardware initialization operation, and forward the EP side hardware queue initialization request to the EP side application layer; S202-According to the EP side hardware queue initialization request forwarded by the EP side service driver layer, perform the EP side queue initialization operation; Data exchange process: S203-Convert the DMA write operation into a DMA descriptor according to the DMA write operation issued by the PCIe middleware, and send the DMA descriptor to the EP side application layer; convert the RC side service data address of the DMA write operation into the EP side service data address, and send the EP side service data address to the EP side application layer; S204 - In response to the upper-layer EP-side service data address, the EP-side service data based on the EP-side service data address is saved locally.

[0018] S301-notify the service driver layer on the first EP side of the initialization request and data source application on the first EP side; S302-In response to the upper layer first EP side initialization request, send a queue creation request to the PCIe middleware, and forward the queue related parameters to the first EP side application layer according to the queue related parameters returned by the lower layer. At the same time, in response to the upper layer data source application request, forward the data source application request to the PCIe middleware, and return the related status of the second EP side hardware queue to the first EP side application layer according to the lower layer save success status; S303-In response to the upper layer's request to create a queue, create a hardware queue on the RC side, and modify the initialization state of the first EP side queue to a sendable state according to the current operation state of the second endpoint EP; in response to the upper layer's request to apply for a data source, map the first EP queue to the second endpoint EP, and return the relevant state of the hardware queue on the first EP side to the service driver layer on the first EP side according to the successful save state of the second endpoint EP; S304-Start the data transmission of this batch and notify the service driver layer on the first EP side to start the data transmission, and judge the completion status of the data transmission of this layer according to the upper layer data transmission completion status feedback returned by the service driver layer on the first EP side; S305-In response to the upper layer data transmission start notification, execute the first EP side data storage, notify the PCIe middleware of the data transmission start, and judge the data transmission completion status of this layer according to the first EP move request feedback returned by the PCIe middleware, and at the same time, feedback the data transmission completion status of this layer to the first EP side application layer; S306-In response to the upper layer data transfer start notification, the converted DMA related operations are sent down to the second endpoint EP, and based on the second EP side interrupt notification feedback returned by the second endpoint EP, the completion status of the batch move on the first EP side is judged, and at the same time, the first EP side move request is fed back to the first EP side business driver layer.

[0019] S401-Convert the hardware queue initialization request on the second EP side issued by the PCIe middleware into a hardware initialization operation on the second EP side, and forward the hardware queue initialization request on the second EP side to the application layer on the second EP side; S402-According to the second EP side hardware queue initialization request forwarded by the second EP side service driver layer, perform the second EP side queue initialization operation; S403-Convert the DMA write operation into a DMA descriptor according to the DMA write operation sent by the PCIe middleware, and send the DMA descriptor to the second EP side application layer; convert the second EP side service data address of the DMA write operation into the second EP side service data address, and send the second EP side service data address to the second EP side application layer; S404 - In response to the upper-layer second EP side service data address, the second EP side service data based on the second EP side service data address is saved locally.

[0020] Embodiment 1:

[0021] like Figure 1 , 5 As shown, the network communication system based on peer-to-peer network communication middleware includes: The root complex RC communicates directly with the endpoint EP through the PCIe middleware, including: Initialization process: RC side user layer: notifies the RC side service driver layer of RC side initialization and data source application S101; RC side business driver layer: in response to the upper layer RC side initialization request, sends a queue creation request to the PCIe middleware, and forwards the queue related parameters to the RC side user layer according to the queue related parameters returned by the lower layer. At the same time, in response to the upper layer data source application request, forwards the data source application request to the PCIe middleware, and returns the related status of the EP side hardware queue to the RC side user layer according to the successful saving status of the lower layer S102; PCIe middleware: in response to the upper layer's request to create a queue, create a hardware queue on the RC side, and modify the RC side queue initialization state to a sendable state according to the current operation state of the endpoint EP; in response to the upper layer's request to apply for a data source, send the RC side queue mapping to the endpoint EP, and according to the successful saving state of the endpoint EP, return the relevant state of the EP side hardware queue to the RC side service driver layer S103; Data exchange process: The user layer on the RC side: starts the data transmission of this batch and notifies the service driver layer on the RC side that the data transmission has started, and determines the data transmission completion status of this layer according to the upper layer data transmission completion status feedback returned by the service driver layer on the RC side S104; RC side business driver layer: in response to the upper layer data transmission start notification, execute RC side data storage, notify the PCIe middleware data transmission start, and judge the data transmission completion status of this layer according to the RC side move request feedback returned by the PCIe middleware, and at the same time, feedback the data transmission completion status of this layer to the RC side user layer S105; PCIe middleware: In response to the upper-layer data transfer start notification, the converted DMA-related operations are sent down to the endpoint EP, and based on the EP-side interrupt notification feedback returned by the endpoint EP, the completion status of the batch transfer on the RC side is determined. At the same time, the RC-side transfer request is fed back to the RC-side business driver layer S106.

[0022] The initialization process includes: RC side user layer: sends the RC side initialization request to the RC side business driver layer, and at the same time, sends the data source application request to the RC side business driver layer; RC side business driver layer: responds to the upper layer RC side initialization request, sends the queue creation request to the PCIe middleware; forwards the queue number, queue base address, and maximum number of queues to the RC side user layer according to the relevant parameters of the queue returned by the lower layer; if the RC side initialization request fails, the reason for the RC side initialization failure is returned to the RC side user layer; at the same time, responds to the upper layer data source application request, forwards the data source application request to the PCIe middleware; according to the lower layer's successful saving status, if successful, returns the RC side queue address to the RC side business driver layer; if failed, returns the reason for the failure to the RC side user layer; PCIe middleware: in response to the upper layer's request to create a queue, create a hardware queue on the RC side, select the EP DMA engine of this endpoint, and create a queue mapping relationship for the hardware queue on the RC side; send the EP side hardware queue initialization request to the endpoint EP; according to the current operation status of the EP of this endpoint, modify the RC side queue initialization status to a sendable status, and at the same time, return the queue number, queue base address, and maximum number of queues to the RC side business driver layer. If the RC side initialization request fails, the reason for the RC side initialization failure is returned to the RC side business driver layer; in response to the upper layer data source application request, send the RC side queue mapping to the endpoint EP, and according to the successful saving status of the EP of this endpoint, if successful, return the RC side queue address to the RC side business driver layer, if failed, return the failure reason to the RC side business driver layer; The data exchange process includes: RC side user layer: used to send the RC read and write requests of this batch to the RC side business driver layer after starting the data transmission of this batch; at the same time, according to the upper layer data transmission completion status feedback returned by the RC side business driver layer, determine whether the RC side has completed the migration of this batch. If it is completed, send the subsequent operation notification to the RC side business driver layer; at the same time, determine whether the business logic is completed. If it is completed, start the next batch of data transmission; RC side business driver layer: used to receive the business data of this batch of RC read and write requests and put them into the write storage space according to this batch of RC read and write requests; construct this batch of RC read and write requests as PCIe read and write requests in a packet format compatible with the standard PCIe link layer protocol, wherein the PCIe read and write request format is [RC side business data address, data length, request ID]; send PCIe read and write requests and PCIe write notifications to the PCIe middleware; judge whether the RC side has completed the move of this batch according to the feedback of the RC side move request returned by the PCIe middleware, and if it has been completed, send the completion notification to the RC side user layer; judge whether the subsequent operations are completed, and send the business logic completion notification to the RC side user layer; PCIe middleware: used to determine the legality of PCIe read and write requests based on PCIe read and write requests. If legal, the PCIe read and write requests are converted into standard DMA write operations; and the DMA write operations are sent down to the endpoint EP; based on the EP-side interrupt notification feedback returned by the endpoint EP, it is determined whether the PCIe read and write tasks are completed, and the RC-side move request is returned to the RC-side business driver layer and it is determined whether the RC-side move of this batch is completed. If completed, wait for the next batch of data transmission.

[0023] The network communication system includes a root complex RC, a PCIe middleware, and an endpoint EP; The root complex RC includes an RC side user mode area and an RC side kernel mode area, wherein the RC side user mode area includes an RC side user layer, and the RC side kernel mode area includes an RC side business driver layer, a PCIe domain resource management module of a PCIe middleware, and an RC side hardware driver layer.

[0024] At the same time, the present invention also provides an endpoint EP to communicate directly with a root complex RC through a PCIe middleware, which includes: Initialization process: The EP side service driver layer: converts the EP side hardware queue initialization request sent by the PCIe middleware into an EP side hardware initialization operation, and forwards the EP side hardware queue initialization request to the EP side application layer S201; The EP side application layer: performs the EP side queue initialization operation S202 according to the EP side hardware queue initialization request forwarded by the EP side service driver layer; Data exchange process: The EP side service driver layer: according to the DMA write operation issued by the PCIe middleware, converts the DMA write operation into a DMA descriptor, and sends the DMA descriptor to the EP side application layer; converts the RC side service data address of the DMA write operation into the EP side service data address, and sends the EP side service data address to the EP side application layer S203; The EP side application layer: in response to the upper layer EP side service data address, saves the EP side service data based on the EP side service data address locally S204.

[0025] Initialization process: EP-side business driver layer: according to the EP-side hardware queue initialization request sent by the PCIe middleware, it is converted into the EP-side hardware initialization operation, and the EP-side hardware queue initialization request is forwarded to the EP-side application layer; according to the EP-side queue initialization status returned by the EP-side application layer, the EP-side queue initialization status is forwarded to the PCIe middleware; according to the RC-side queue mapping sent by the PCIe middleware, the memory mapping of the application source data is saved, and the current operation status is returned to the PCIe middleware; The EP side application layer: performs the EP side queue initialization operation according to the EP side hardware queue initialization request forwarded by the EP side service driver layer, and returns the EP side queue initialization status to the EP side service driver layer; Data exchange process: EP-side service driver layer: according to the DMA write operation sent by the PCIe middleware, convert the DMA write operation into a DMA descriptor and send the DMA descriptor to the EP-side application layer; convert the RC-side service data address of the DMA write operation into the EP-side service data address, and send the EP-side service data address to the EP-side application layer, split the DMA write operation into various DMA sub-operations, and trigger each DMA sub-operation; in response to the lower-layer interrupt notification feedback, forward the EP-side interrupt notification to the PCIe middleware and determine whether the EP-side data polling is completed; EP side application layer: responds to the upper-layer DMA descriptor, forwards the DMA descriptor to the EP side base layer; responds to the upper-layer EP side service data address, saves the EP side service data based on the EP side service data address locally; returns the EP side interrupt notification to the EP side service driver layer and determines whether the EP side data polling is completed.

[0026] The network communication system includes a root complex RC, a PCIe middleware, and an endpoint EP; The endpoint EP includes an EP side kernel state area and an EP side application state area, wherein the EP side kernel state area includes an EP side hardware driver layer and an EP node resource management module of the EP side base layer, and the EP side application state area includes an EP side application layer.

[0027] like Figure 1 , 5As shown, the data transmission part: based on the RC side user state area of ​​the root complex RC, in the RC side user layer in the RC side user state area: start this batch of data transmission; send this batch of RC read and write requests to the RC side business driver layer, in the RC side business driver layer in the RC side kernel state area: put the business data of the RC read and write requests into the write storage space; construct this batch of RC read and write requests as PCIe read and write requests in a packet format compatible with the standard PCIe link layer protocol, wherein the PCIe read and write request format is [RC side business data address, data length, request ID]; at the same time, send the PCIe read and write request and PCIe write notification to the PCIe middleware; PCIe middleware (PCIe domain resource management module and RC side hardware driver layer): judge the legality of the PCIe read and write request, if legal, convert the PCIe read and write request into a standard DMA write operation; and write the DMA The operation is sent to each endpoint EP; the above part is sent from the root complex RC to the endpoint EP through the PCIe middleware, and the following part is returned from the endpoint EP to the root complex RC through the PCIe middleware: PCIe middleware: determines whether the processing in response to the PCIe read and write request is completed, and returns the RC-side move request to the RC-side business driver layer and determines whether the RC-side move is completed; the RC-side business driver layer in the kernel state area of ​​the RC side: determines whether the RC-side move is completed, and if completed, sends the completion notification to the RC-side user layer; determines whether the subsequent operations are completed, and sends the business logic completion notification to the RC-side user layer; based on the RC-side user state area of ​​the root complex RC, in the RC-side user layer in the RC-side user state area: if the RC-side move is completed, the subsequent operation notification is sent to the RC-side business driver layer; if the business logic is completed, the next batch of data transmission is started.

[0028] like Figure 1 , 5As shown, the initialization part: root complex RC: RC side user state area: RC side user layer: send the RC side initialization request to the RC side business driver layer; send the application source data request to the RC side business driver layer; RC side kernel state area: RC side business driver layer: according to the RC side initialization request, send the queue creation request to the PCIe middleware; forward the queue number, queue base address, and maximum number of queues to the RC side user layer. If the RC side initialization request fails, the reason for the RC side initialization failure is returned to the RC side user layer; forward the application source data request to the RC side business driver layer; forward the application source data memory to the RC side business driver layer. If the save success status is failure, forward the save failure reason; PCIe middleware (PCIe Intra-domain resource management module and RC-side hardware driver layer): Create the RC-side hardware queue according to the queue creation request; and select the EPDMA engine of this endpoint; create the queue mapping relationship of the RC-side hardware queue; send the EP-side hardware queue initialization request to each endpoint EP; according to the EP-side queue initialization status, modify the RC-side queue initialization status to a sendable status; return the queue number, queue base address, and maximum number of queues. If the RC-side initialization request fails, the reason for the RC-side initialization failure is returned to the RC-side business driver layer; allocate the source data memory applied by this endpoint EP; send the source data memory mapping to each endpoint EP; return the source data memory applied to the RC-side business driver layer. If the save success status is failure, the reason for the save failure is returned.

[0029] like Figure 2 , 5 As shown, the EP side kernel state area of ​​the endpoint EP, the EP side base layer (EP side hardware driver layer and EP node resource management module) of the EP side kernel state area: EP side business driver layer: convert the DMA write operation into a DMA descriptor, and send the DMA descriptor to the EP side application layer; convert the RC side business data address of the DMA write operation into the EP side business data address, and send the EP side business data address to the EP side application layer; split the DMA write operation into each DMA sub-operation, and trigger each DMA sub-operation; the EP side application layer of the EP side application state area, forwards the DMA descriptor to the EP side base layer; saves the EP side business data based on the EP side business data address locally; returns the EP side interrupt notification to the EP side business driver layer and determines whether the EP side data polling is completed; the above is the receiving part of the endpoint EP, and the following is the sending part of the endpoint EP: EP side business driver layer: forwards the EP side interrupt notification to the PCIe middleware and determines whether the EP side data polling is completed.

[0030] like Figure 2 , 5As shown, the initialization part is as follows: endpoint EP: EP side kernel state area: EP side base layer (EP side hardware driver layer and EP node resource management module): EP side business driver layer: convert the EP side hardware queue initialization request into the EP side hardware initialization operation; forward the EP side hardware queue initialization request to the EP side application layer; forward the EP side queue initialization state to the PCIe middleware; save the application source data memory mapping, and return the current operation state to the PCIe middleware. RC side application state area: EP side application layer: perform the EP side queue initialization operation according to the EP side hardware queue initialization request; return the EP side queue initialization state to the EP side business driver layer.

[0031] At the same time, the present invention also provides a network communication system based on peer-to-peer network communication middleware, including: The first endpoint EP directly communicates with the second endpoint EP through the PCIe middleware, which includes: Initialization process: The first EP side application layer: notifies the first EP side service driver layer of the first EP side initialization request and data source application S301; The first EP side service driver layer: in response to the upper layer first EP side initialization request, sends a queue creation request to the PCIe middleware, and forwards the relevant parameters of the queue to the first EP side application layer according to the relevant parameters of the queue returned by the lower layer, and at the same time, in response to the upper layer data source application request, forwards the data source application request to the PCIe middleware, and returns the relevant status of the second EP side hardware queue to the first EP side application layer according to the successful saving status of the lower layer S302; PCIe middleware: in response to the upper layer request to create a queue, create a hardware queue on the RC side, and modify the initialization state of the queue on the first EP side to a sendable state according to the current operation state of the second endpoint EP; in response to the upper layer data source application request, map the first EP queue to the second endpoint EP, and return the relevant state of the hardware queue on the first EP side to the service driver layer on the first EP side S303 according to the successful saving state of the second endpoint EP; Data exchange process: The application layer on the first EP side starts the data transmission of this batch and notifies the service driver layer on the first EP side to start the data transmission, and determines the data transmission completion status of this layer according to the upper layer data transmission completion status feedback returned by the service driver layer on the first EP side S304; The first EP side service driver layer: in response to the upper layer data transmission start notification, executes the first EP side data storage, notifies the PCIe middleware of the data transmission start, and determines the data transmission completion status of this layer according to the first EP move request feedback returned by the PCIe middleware, and at the same time, feeds back the data transmission completion status of this layer to the first EP side application layer S305; PCIe middleware: In response to the upper layer data transfer start notification, the converted DMA related operations are sent down to the second endpoint EP, and based on the second EP side interrupt notification feedback returned by the second endpoint EP, the completion status of the batch transfer on the first EP side is determined, and at the same time, the first EP side transfer request is fed back to the first EP side business driver layer S306.

[0032] The initialization process includes: The user layer on the first EP side: sends the first EP side initialization request to the service driver layer on the first EP side, and at the same time, sends the data source application request to the service driver layer on the first EP side; The first EP side service driver layer: responds to the upper layer first EP side initialization request, sends the queue creation request to the PCIe middleware; forwards the queue number, queue base address, and maximum number of queues to the first EP side user layer according to the relevant parameters of the queue returned by the lower layer; if the first EP side initialization request fails, returns the first EP side initialization failure reason to the first EP side user layer; at the same time, responds to the upper layer data source application request, forwards the data source application request to the PCIe middleware; according to the lower layer save success status, if successful, returns the first EP side queue address to the first EP side service driver layer, if failed, returns the failure reason to the first EP side user layer; PCIe middleware: in response to the upper layer's request to create a queue, create a hardware queue on the RC side, select the second endpoint EP DMA engine, and create a queue mapping relationship for the hardware queue on the first EP side; send the hardware queue initialization request on the first EP side to the endpoint EP; according to the current operation status of the second endpoint EP, modify the queue initialization status on the first EP side to a sendable status, and at the same time, return the queue number, queue base address, and maximum number of queues to the first EP side business driver layer. If the initialization request on the first EP side fails, return the reason for the initialization failure on the first EP side to the business driver layer on the first EP side; in response to the upper layer's data source application request, send the first EP side queue mapping down to the second endpoint EP, and according to the successful saving status of the second endpoint EP, if successful, return the second EP side queue address to the first EP side business driver layer, and if failed, return the reason for the failure to the first EP side business driver layer; The data exchange process includes: The application layer on the first EP side is used to send the RC read and write request of this batch to the business driver layer on the first EP side after starting the data transmission of this batch; at the same time, according to the upper layer data transmission completion status feedback returned by the business driver layer on the first EP side, it is judged whether the migration of this batch on the first EP side is completed. If it is completed, the subsequent operation notification is sent to the business driver layer on the RC side; at the same time, it is judged whether the business logic is completed. If it is completed, the next batch of data transmission is started; The first EP side business driver layer: is used to receive the business data of the current batch of RC read and write requests and put them into the write storage space according to the current batch of RC read and write requests; construct the current batch of first EP read and write requests as PCIe read and write requests in a packet format compatible with the standard PCIe link layer protocol, wherein the PCIe read and write request format is [RC side business data address, data length, request ID]; send the PCIe read and write request and PCIe write notification to the PCIe middleware; according to the feedback of the first EP side move request returned by the PCIe middleware, determine whether the current batch move of the first EP side is completed, and if completed, send the completion notification to the first EP side application layer; determine whether the subsequent operation is completed, and send the business logic completion notification to the first EP side user layer; PCIe middleware: used to determine the legality of PCIe read / write requests according to PCIe read / write requests. If legal, convert the PCIe read / write requests into standard DMA write operations; and send the DMA write operations to the second endpoint EP; determine whether the PCIe read / write task is completed according to the EP-side interrupt notification feedback returned by the second endpoint EP, and return the first EP-side move request to the first EP-side business driver layer and determine whether the first EP-side move is completed. If completed, wait for the next batch of data transmission; The network communication system includes a first endpoint EP, a PCIe middleware, and a second endpoint EP; The first endpoint EP includes a first EP side user state area and a first EP side kernel state area, wherein the first EP side user state area includes a first EP side user layer, and the first EP side kernel state area includes a first EP side service driver layer, a PCIe domain resource management module of the PCIe middleware, and a first EP side hardware driver layer.

[0033] At the same time, the present invention also provides a network communication system based on peer-to-peer network communication middleware, including: The second endpoint EP directly communicates with the first endpoint EP through the PCIe middleware, including: Initialization process: The second EP side service driver layer: converts the second EP side hardware queue initialization request sent by the PCIe middleware into a second EP side hardware initialization operation, and forwards the second EP side hardware queue initialization request to the second EP side application layer S401; The second EP side application layer: performs the second EP side queue initialization operation S402 according to the second EP side hardware queue initialization request forwarded by the second EP side service driver layer; Data exchange process: The second EP side service driver layer: according to the DMA write operation issued by the PCIe middleware, converts the DMA write operation into a DMA descriptor, and sends the DMA descriptor to the second EP side application layer; converts the second EP side service data address of the DMA write operation into the second EP side service data address, and sends the second EP side service data address to the second EP side application layer S403; The second EP side application layer: in response to the upper layer second EP side service data address, saves the second EP side service data based on the second EP side service data address locally S404.

[0034] Initialization process: The second EP side service driver layer: according to the second EP side hardware queue initialization request issued by the PCIe middleware, convert it into the second EP side hardware initialization operation, and forward the second EP side hardware queue initialization request to the second EP side application layer; according to the second EP side queue initialization status returned by the second EP side application layer, forward the second EP side queue initialization status to the PCIe middleware; according to the second EP side queue mapping issued by the PCIe middleware, save the application source data memory mapping, and return the current operation status to the PCIe middleware; The second EP side application layer: performs the second EP side queue initialization operation according to the second EP side hardware queue initialization request forwarded by the second EP side service driver layer, and returns the second EP side queue initialization state to the second EP side service driver layer; Data exchange process: The second EP side service driver layer: according to the DMA write operation sent by the PCIe middleware, convert the DMA write operation into a DMA descriptor, and send the DMA descriptor to the second EP side application layer; convert the second EP side service data address of the DMA write operation into the second EP side service data address, and send the second EP side service data address to the second EP side application layer, split the DMA write operation into each DMA sub-operation, and trigger each DMA sub-operation; in response to the lower layer interrupt notification feedback, forward the second EP side interrupt notification to the PCIe middleware and determine whether the second EP side data polling is completed; The second EP side application layer: in response to the upper layer DMA descriptor, forwards the DMA descriptor to the second EP side base layer; in response to the upper layer second EP side service data address, saves the second EP side service data based on the second EP side service data address locally; returns the second EP side interrupt notification to the second EP side service driver layer and determines whether the second EP side data polling is completed; The network communication system includes a first endpoint EP, a PCIe middleware, and a second endpoint EP; The second endpoint EP includes a second EP side inner kernel state area and a second EP side application state area, wherein the second EP side inner kernel state area includes a second EP side hardware driver layer and a second EP node resource management module of the second EP side base layer, and the second EP side application state area includes a second EP side application layer.

[0035] like Figure 3 , 6 As shown, the data transmission part: source endpoint EP: user state area on the source EP side: user layer on the source EP side: send the internal trigger request to the service driver layer on the source EP side; start data transmission of this batch; send the RC read and write request of this batch to the service driver layer on the source EP side; send the data push notification to the destination endpoint EP2; kernel state area on the source EP side: service driver layer on the source EP side: put the service data of this batch into the write storage space; construct the RC read and write request of this batch into a PCIe read and write request in a packet format compatible with the standard PCIe link layer protocol, wherein the PCIe read and write request format is [service data address on the source EP side, data According to the length, request ID]; send PCIe read and write requests and PCIe write notifications to the PCIe middleware; determine whether the batch migration on the source EP side is completed, if it is completed, send the completion notification to the user layer on the source EP side; PCIe middleware (resource management module in the PCIe domain and hardware driver layer on the source EP side): determine the legality of the PCIe read and write requests, if it is legal, convert the PCIe read and write requests into standard DMA write operations; and send the DMA write operations to each endpoint EP; determine whether the processing in response to the PCIe read and write requests is completed, and return the source EP side migration request to the source EP side service driver Layer and determine whether the batch migration on the source EP side is completed; the above is the data sending part, and the following is the data receiving part: Destination endpoint EP1: Destination EP side 1 kernel state area: Destination EP side 1 base layer (destination EP side 1 hardware driver layer and EP node resource management module): Destination EP side 1 business driver layer: Convert the DMA write operation into a DMA descriptor, and send the DMA descriptor to the destination EP side 1 application layer; Convert the destination EP side 1 business data address of the DMA write operation into the destination EP side 1 business data address, and send the destination EP side 1 business data address to the destination EP side 1 application layer; Convert the DMA write operation into a DMA descriptor, and send the destination EP side 1 business data address to the destination EP side 1 application layer; The operation is split into various DMA sub-operations, and each DMA sub-operation is triggered; the destination EP side 1 interrupt notification is forwarded to the PCIe middleware and it is determined whether the destination EP side 1 data polling is completed; the destination EP side 1 application state area: the destination EP side 1 application layer: forward the DMA descriptor to the destination EP side 1 base layer; the destination EP side 1 business data of the destination EP side 1 business data address is saved locally; the destination EP side 1 interrupt notification is returned to the destination EP side 1 business driver layer and it is determined whether the destination EP side 1 data polling is completed; the destination EP side 1 business data of the destination EP side 1 business data address is pushed to the destination endpoint EP2; like Figure 4 , 6 As shown, the initialization part: root complex RC: RC side user state area: RC side user layer: send the RC side initialization request to the RC side business driver layer; send the source data application request to the RC side business driver layer; RC side kernel state area: RC side business driver layer: according to the RC side initialization request, send the queue creation request to the PCIe middleware; forward the queue number, queue base address, and maximum number of queues to the RC side user layer. If the RC side initialization request fails, the reason for the RC side initialization failure is returned to the RC side user layer; forward the source data application request to the RC side business driver layer; forward the source data application memory to the RC side business driver layer. If the save success status is failure, forward the save failure reason; PCIe middleware (PCIe domain resource management module and RC side hardware driver layer): create the RC side hardware queue according to the queue creation request; and select the next endpoint EP DMA engine; create queue mapping relationship of RC side hardware queue; send EP side hardware queue initialization request to each endpoint EP; according to the EP side queue initialization status, modify the RC side queue initialization status to sendable status; return queue number, queue base address, maximum number of queues, if the RC side initialization request fails, return the reason for RC side initialization failure to the RC side business driver layer; allocate the next endpoint EP to apply for source data memory; send the source data memory mapping to each endpoint EP; return the source data memory application to the RC side business driver layer, if the save success status is failure, return the save failure reason; endpoint EP1: EP side kernel state area: EP side base layer (EP side hardware driver layer and EP node resource management module): EP side business driver layer: according to the EP side hardware queue initialization request, convert it into EP side hardware initialization operation; forward the EP side hardware queue initialization request to the EP side application layer; forward the EP side queue initialization status to the PCIe middleware; save the source data memory mapping, and return the save success status to the PCIe middleware. RC side application state area: EP side application layer: perform EP side queue initialization operation according to EP side hardware queue initialization request; return EP side queue initialization state to EP side service driver layer; like Figure 7 , 8As shown in Figure 9, the RC side includes user state: PCIe business application, which is defined by the user; the second is kernel state, which is divided into the following modules. a) Business driver module: Based on functional abstraction, user layer applications uniformly call this module to implement business functions. Implement communication model interfaces such as unicast, multicast, and broadcast between RC and EP, shield complex communication details, and provide users with multi-model communication interfaces. Implement communication configuration interfaces such as unicast, multicast, and broadcast between EP and EP, and the P2P application scenario can be realized after the user calls. Implement user state interface API and provide standard queue reading and writing for the application layer; b) PCIe domain resource management module, uniformly abstract addressing for EP devices, and provide a unified device identifier for upper-layer functional diversified EP applications; PCIe domain resources include hardware queues, DMA channels, domain memory, and routing management. Provide resource application and release interfaces for the main driver module. Coordinate management and allocation of resources within the PCIe domain. Implement unicast, multicast, and broadcast queue planning. Implement topology analysis and path calculation in the enumeration stage. Implement functions such as resource recovery when the device is offline or the link is abnormal. c) Hardware driver module, connected to specific hardware devices: standard hardware driver; connected to the resource management platform, the resource management driver calls the interface provided by the resource platform in the PCIe domain to implement the application and release of PCIe resources. 3. Hardware layer: Specific hardware: EP side: 1. EP side hardware driver: logical interface for resource management with RC; responsible for communicating with RC resource management; 2. EP node resource management module: manages EP side hardware queues, memory and other resources. Contains multicast, unicast, and broadcast queues in P2P scenarios, used to implement P2P applications. 3. Business driver module: connected to the hardware on the RC side, endpoint side SOC hardware driver; 4. Business application layer: user business processing, loaded in as a kernel module, connected to the resource management platform, obtains memory and queues from the resource management platform, and implements its own initialization; Unicast: Unicast communication between RC and EP; the establishment of communication between RC and EP is simplified to the subscription of queue resources; the communication operation between RC and EP is simplified to the read and write operation of queue, supporting DMA operation; Unicast communication between EPs: Communication establishment between EPs is simplified to queue resource subscription; communication between EPs is simplified to queue reading and writing; P2P between EPs is supported; Multicast: Multicast establishment is simplified to multicast queue resource subscription; supports multicast message publishing from RC to EP; supports multicast message publishing between EPs; supports multicast message publishing from EP to RC; multicast exit is simplified to exit and release of multicast queue resources.

[0036] Utilizing the technical solution of the present invention, or those skilled in the art designing similar technical solutions inspired by the technical solution of the present invention, and achieving the above technical effects, all fall within the protection scope of the present invention. It should be pointed out that the specific implementations in the embodiments of the present invention can be at least partially combined with each other, and the combination and use of the above embodiments, or their different combinations, should also be regarded as falling within the protection scope of the present invention.

Claims

1. A network communication system based on peer-to-peer network communication middleware, characterized in that: include: The root complex RC communicates directly with the endpoint EP through the PCIe middleware, including: Initialization process: RC side user layer: notifies the RC side service driver layer of RC side initialization and data source application; RC side business driver layer: in response to the upper layer RC side initialization request, sends the queue creation request to the PCIe middleware, and forwards the queue related parameters to the RC side user layer according to the queue related parameters returned by the lower layer. At the same time, in response to the upper layer data source application request, forwards the data source application request to the PCIe middleware, and returns the related status of the EP side hardware queue to the RC side user layer according to the successful saving status of the lower layer. PCIe middleware: In response to the upper layer's request to create a queue, create a hardware queue on the RC side, and modify the RC side queue initialization state to a sendable state according to the current operation state of the endpoint EP; in response to the upper layer's data source application request, send the RC side queue mapping to the endpoint EP, and according to the successful save state of the endpoint EP, the relevant state of the EP side hardware queue is returned to the RC side service driver layer; Data exchange process: The user layer on the RC side: starts the data transmission of this batch and notifies the service driver layer on the RC side that the data transmission has started, and determines the completion status of the data transmission of this layer based on the upper layer data transmission completion status feedback returned by the service driver layer on the RC side; RC side business driver layer: responds to the upper layer data transmission start notification, executes RC side data storage, and notifies the PCIe middleware of the data transmission start, and judges the data transmission completion status of this layer according to the RC side move request feedback returned by the PCIe middleware, and at the same time, feeds back the data transmission completion status of this layer to the RC side user layer; PCIe middleware: In response to the upper-layer data transfer start notification, the converted DMA-related operations are sent down to the endpoint EP, and based on the EP-side interrupt notification feedback returned by the endpoint EP, the completion status of the batch transfer on the RC side is determined. At the same time, the RC-side transfer request is fed back to the RC-side business driver layer.

2. The network communication system according to claim 1, characterized in that: include: The initialization process includes: RC side user layer: sends the RC side initialization request to the RC side business driver layer, and at the same time, sends the data source application request to the RC side business driver layer; RC side business driver layer: responds to the upper layer RC side initialization request, sends the queue creation request to the PCIe middleware; forwards the queue number, queue base address, and maximum number of queues to the RC side user layer according to the relevant parameters of the queue returned by the lower layer; if the RC side initialization request fails, the reason for the RC side initialization failure is returned to the RC side user layer; at the same time, responds to the upper layer data source application request, forwards the data source application request to the PCIe middleware; according to the lower layer's successful saving status, if successful, returns the RC side queue address to the RC side business driver layer; if failed, returns the reason for the failure to the RC side user layer; PCIe middleware: in response to the upper layer's request to create a queue, create a hardware queue on the RC side, select the EP DMA engine of this endpoint, and create a queue mapping relationship for the hardware queue on the RC side; send the EP side hardware queue initialization request to the endpoint EP; according to the current operation status of the EP of this endpoint, modify the RC side queue initialization status to a sendable status, and at the same time, return the queue number, queue base address, and maximum number of queues to the RC side business driver layer. If the RC side initialization request fails, the reason for the RC side initialization failure is returned to the RC side business driver layer; in response to the upper layer data source application request, send the RC side queue mapping to the endpoint EP, and according to the successful saving status of the EP of this endpoint, if successful, return the RC side queue address to the RC side business driver layer, if failed, return the failure reason to the RC side business driver layer; The data exchange process includes: RC side user layer: used to send the RC read and write requests of this batch to the RC side business driver layer after starting the data transmission of this batch; at the same time, according to the upper layer data transmission completion status feedback returned by the RC side business driver layer, determine whether the RC side has completed the migration of this batch. If it is completed, send the subsequent operation notification to the RC side business driver layer; at the same time, determine whether the business logic is completed. If it is completed, start the next batch of data transmission; RC side business driver layer: used to receive the business data of this batch of RC read and write requests and put them into the write storage space according to this batch of RC read and write requests; construct this batch of RC read and write requests as PCIe read and write requests in a packet format compatible with the standard PCIe link layer protocol, wherein the PCIe read and write request format is [RC side business data address, data length, request ID]; send PCIe read and write requests and PCIe write notifications to the PCIe middleware; judge whether the RC side has completed the move of this batch according to the feedback of the RC side move request returned by the PCIe middleware, and if it has been completed, send the completion notification to the RC side user layer; judge whether the subsequent operations are completed, and send the business logic completion notification to the RC side user layer; PCIe middleware: used to determine the legality of PCIe read and write requests based on PCIe read and write requests. If legal, the PCIe read and write requests are converted into standard DMA write operations; and the DMA write operations are sent down to the endpoint EP; based on the EP-side interrupt notification feedback returned by the endpoint EP, it is determined whether the PCIe read and write tasks are completed, and the RC-side move request is returned to the RC-side business driver layer and it is determined whether the RC-side move of this batch is completed. If completed, wait for the next batch of data transmission.

3. The network communication system according to claim 1, characterized in that: include: The network communication system includes a root complex RC, a PCIe middleware, and an endpoint EP; The root complex RC includes an RC side user mode area and an RC side kernel mode area, wherein the RC side user mode area includes an RC side user layer, and the RC side kernel mode area includes an RC side business driver layer, a PCIe domain resource management module of a PCIe middleware, and an RC side hardware driver layer.

4. A network communication system based on peer-to-peer network communication middleware, characterized in that: include: The endpoint EP communicates directly with the root complex RC through the PCIe middleware, including: Initialization process: EP-side service driver layer: converts the EP-side hardware queue initialization request sent by the PCIe middleware into an EP-side hardware initialization operation, and forwards the EP-side hardware queue initialization request to the EP-side application layer; The EP side application layer: performs the EP side queue initialization operation according to the EP side hardware queue initialization request forwarded by the EP side service driver layer; Data exchange process: EP-side service driver layer: according to the DMA write operation issued by the PCIe middleware, convert the DMA write operation into a DMA descriptor, and send the DMA descriptor to the EP-side application layer; convert the RC-side service data address of the DMA write operation into the EP-side service data address, and send the EP-side service data address to the EP-side application layer; EP side application layer: in response to the upper layer EP side service data address, saves the EP side service data based on the EP side service data address locally.

5. The network communication system according to claim 4, characterized in that: include: Initialization process: EP-side business driver layer: according to the EP-side hardware queue initialization request sent by the PCIe middleware, it is converted into the EP-side hardware initialization operation, and the EP-side hardware queue initialization request is forwarded to the EP-side application layer; according to the EP-side queue initialization status returned by the EP-side application layer, the EP-side queue initialization status is forwarded to the PCIe middleware; according to the RC-side queue mapping sent by the PCIe middleware, the memory mapping of the application source data is saved, and the current operation status is returned to the PCIe middleware; The EP side application layer: performs the EP side queue initialization operation according to the EP side hardware queue initialization request forwarded by the EP side service driver layer, and returns the EP side queue initialization status to the EP side service driver layer; Data exchange process: EP-side service driver layer: according to the DMA write operation sent by the PCIe middleware, convert the DMA write operation into a DMA descriptor and send the DMA descriptor to the EP-side application layer; convert the RC-side service data address of the DMA write operation into the EP-side service data address, and send the EP-side service data address to the EP-side application layer, split the DMA write operation into various DMA sub-operations, and trigger each DMA sub-operation; in response to the lower-layer interrupt notification feedback, forward the EP-side interrupt notification to the PCIe middleware and determine whether the EP-side data polling is completed; EP side application layer: responds to the upper-layer DMA descriptor, forwards the DMA descriptor to the EP side base layer; responds to the upper-layer EP side service data address, saves the EP side service data based on the EP side service data address locally; returns the EP side interrupt notification to the EP side service driver layer and determines whether the EP side data polling is completed.

6. The network communication system according to claim 4, characterized in that: The network communication system includes a root complex RC, a PCIe middleware, and an endpoint EP; The endpoint EP includes an EP side kernel state area and an EP side application state area, wherein the EP side kernel state area includes an EP side hardware driver layer and an EP node resource management module of the EP side base layer, and the EP side application state area includes an EP side application layer.

7. A network communication system based on peer-to-peer network communication middleware, characterized in that: include: The first endpoint EP directly communicates with the second endpoint EP through the PCIe middleware, which includes: Initialization process: The application layer on the first EP side: notifies the service driver layer on the first EP side of the initialization request and data source application on the first EP side; The first EP side service driver layer: in response to the upper layer first EP side initialization request, sends a queue creation request to the PCIe middleware, and forwards the queue related parameters to the first EP side application layer according to the queue related parameters returned by the lower layer, and at the same time, in response to the upper layer data source application request, forwards the data source application request to the PCIe middleware, and returns the related status of the second EP side hardware queue to the first EP side application layer according to the successful saving status of the lower layer; PCIe middleware: in response to the upper layer request to create a queue, create a hardware queue on the RC side, and modify the initialization state of the queue on the first EP side to a sendable state according to the current operation state of the second endpoint EP; in response to the upper layer data source application request, map the first EP queue to the second endpoint EP, and return the relevant state of the hardware queue on the first EP side to the service driver layer on the first EP side according to the successful save state of the second endpoint EP; Data exchange process: The application layer on the first EP side starts the data transmission of this batch and notifies the service driver layer on the first EP side to start the data transmission, and determines the completion status of the data transmission of this layer according to the upper layer data transmission completion status feedback returned by the service driver layer on the first EP side; The first EP side service driver layer: in response to the upper layer data transmission start notification, executes the first EP side data storage, notifies the PCIe middleware of the data transmission start, and determines the data transmission completion status of this layer according to the first EP move request feedback returned by the PCIe middleware, and at the same time, feeds back the data transmission completion status of this layer to the first EP side application layer; PCIe middleware: In response to the upper layer data transfer start notification, the converted DMA related operations are sent down to the second endpoint EP, and based on the second EP side interrupt notification feedback returned by the second endpoint EP, the completion status of the batch transfer on the first EP side is determined, and at the same time, the first EP side transfer request is fed back to the first EP side business driver layer.

8. The network communication system according to claim 7, characterized in that: include: The initialization process includes: The user layer on the first EP side: sends the first EP side initialization request to the service driver layer on the first EP side, and at the same time, sends the data source application request to the service driver layer on the first EP side; The first EP side service driver layer: responds to the upper layer first EP side initialization request, sends the queue creation request to the PCIe middleware; forwards the queue number, queue base address, and maximum number of queues to the first EP side user layer according to the relevant parameters of the queue returned by the lower layer; if the first EP side initialization request fails, returns the first EP side initialization failure reason to the first EP side user layer; at the same time, responds to the upper layer data source application request, forwards the data source application request to the PCIe middleware; according to the lower layer save success status, if successful, returns the first EP side queue address to the first EP side service driver layer, if failed, returns the failure reason to the first EP side user layer; PCIe middleware: in response to the upper layer's request to create a queue, create a hardware queue on the RC side, select the second endpoint EP DMA engine, and create a queue mapping relationship for the hardware queue on the first EP side; send the hardware queue initialization request on the first EP side to the endpoint EP; according to the current operation status of the second endpoint EP, modify the queue initialization status on the first EP side to a sendable status, and at the same time, return the queue number, queue base address, and maximum number of queues to the first EP side business driver layer. If the initialization request on the first EP side fails, return the reason for the initialization failure on the first EP side to the business driver layer on the first EP side; in response to the upper layer's data source application request, send the first EP side queue mapping down to the second endpoint EP, and according to the successful saving status of the second endpoint EP, if successful, return the second EP side queue address to the first EP side business driver layer, and if failed, return the reason for the failure to the first EP side business driver layer; The data exchange process includes: The application layer on the first EP side is used to send the RC read and write request of this batch to the business driver layer on the first EP side after starting the data transmission of this batch; at the same time, according to the upper layer data transmission completion status feedback returned by the business driver layer on the first EP side, it is judged whether the migration of this batch on the first EP side is completed. If it is completed, the subsequent operation notification is sent to the business driver layer on the RC side; at the same time, it is judged whether the business logic is completed. If it is completed, the next batch of data transmission is started; The first EP side business driver layer: is used to receive the business data of the current batch of RC read and write requests and put them into the write storage space according to the current batch of RC read and write requests; construct the current batch of first EP read and write requests as PCIe read and write requests in a packet format compatible with the standard PCIe link layer protocol, wherein the PCIe read and write request format is [RC side business data address, data length, request ID]; send the PCIe read and write request and PCIe write notification to the PCIe middleware; according to the feedback of the first EP side move request returned by the PCIe middleware, determine whether the current batch move of the first EP side is completed, and if completed, send the completion notification to the first EP side application layer; determine whether the subsequent operation is completed, and send the business logic completion notification to the first EP side user layer; PCIe middleware: used to determine the legality of PCIe read / write requests according to PCIe read / write requests. If legal, convert the PCIe read / write requests into standard DMA write operations; and send the DMA write operations to the second endpoint EP; determine whether the PCIe read / write task is completed according to the EP-side interrupt notification feedback returned by the second endpoint EP, and return the first EP-side move request to the first EP-side business driver layer and determine whether the first EP-side move is completed. If completed, wait for the next batch of data transmission; The network communication system includes a first endpoint EP, a PCIe middleware, and a second endpoint EP; The first endpoint EP includes a first EP side user state area and a first EP side kernel state area, wherein the first EP side user state area includes a first EP side user layer, and the first EP side kernel state area includes a first EP side service driver layer, a PCIe domain resource management module of the PCIe middleware, and a first EP side hardware driver layer.

9. A network communication system based on peer-to-peer network communication middleware, characterized in that: include: The second endpoint EP directly communicates with the first endpoint EP through the PCIe middleware, including: Initialization process: The second EP side service driver layer: converts the second EP side hardware queue initialization request sent by the PCIe middleware into a second EP side hardware initialization operation, and forwards the second EP side hardware queue initialization request to the second EP side application layer; The second EP side application layer: performs the second EP side queue initialization operation according to the second EP side hardware queue initialization request forwarded by the second EP side service driver layer; Data exchange process: The second EP side service driver layer: according to the DMA write operation issued by the PCIe middleware, converts the DMA write operation into a DMA descriptor, and sends the DMA descriptor to the second EP side application layer; converts the second EP side service data address of the DMA write operation into the second EP side service data address, and sends the second EP side service data address to the second EP side application layer; The second EP side application layer: in response to the upper layer second EP side service data address, saves the second EP side service data based on the second EP side service data address locally.

10. The network communication system according to claim 9, characterized in that: include: Initialization process: The second EP side service driver layer: according to the second EP side hardware queue initialization request issued by the PCIe middleware, convert it into the second EP side hardware initialization operation, and forward the second EP side hardware queue initialization request to the second EP side application layer; according to the second EP side queue initialization status returned by the second EP side application layer, forward the second EP side queue initialization status to the PCIe middleware; according to the second EP side queue mapping issued by the PCIe middleware, save the application source data memory mapping, and return the current operation status to the PCIe middleware; The second EP side application layer: performs the second EP side queue initialization operation according to the second EP side hardware queue initialization request forwarded by the second EP side service driver layer, and returns the second EP side queue initialization state to the second EP side service driver layer; Data exchange process: The second EP side service driver layer: according to the DMA write operation sent by the PCIe middleware, convert the DMA write operation into a DMA descriptor, and send the DMA descriptor to the second EP side application layer; convert the second EP side service data address of the DMA write operation into the second EP side service data address, and send the second EP side service data address to the second EP side application layer, split the DMA write operation into each DMA sub-operation, and trigger each DMA sub-operation; in response to the lower layer interrupt notification feedback, forward the second EP side interrupt notification to the PCIe middleware and determine whether the second EP side data polling is completed; The second EP side application layer: in response to the upper layer DMA descriptor, forwards the DMA descriptor to the second EP side base layer; in response to the upper layer second EP side service data address, saves the second EP side service data based on the second EP side service data address locally; returns the second EP side interrupt notification to the second EP side service driver layer and determines whether the second EP side data polling is completed; The network communication system includes a first endpoint EP, a PCIe middleware, and a second endpoint EP; The second endpoint EP includes a second EP side inner kernel state area and a second EP side application state area, wherein the second EP side inner kernel state area includes a second EP side hardware driver layer and a second EP node resource management module of the second EP side base layer, and the second EP side application state area includes a second EP side application layer.

Images