A USB image data cooperative transmission method, system, device and storage medium

Abstract

The application provides a USB image data cooperative transmission method, system, device and storage medium, the method comprising: determining a processing system and a programmable logic shared access image receiving buffer, based on the address information thereof, configuring and submitting transmission description information to a USB host controller; the USB host controller receives USB image data output by a USB image device, and initiates a PCIe memory write transaction with the image receiving buffer as the write target according to the USB image data; the PCIe memory write transaction is converted into an on-chip bus write transaction through a PCIe root complex to write the USB image data into the image receiving buffer; and in response to a transmission completion event of the USB host controller, the programmable logic reads and processes the USB image data from the image receiving buffer through the processing system. The application reduces intermediate data migration and improves the cooperative efficiency of USB image reception and programmable logic processing.

Description

Technical Field

[0001] This application relates to the field of image data transmission technology, and in particular to a USB image data collaborative transmission method, system, device and storage medium. Background Technology

[0002] With the increasing demands for real-time image data processing capabilities in applications such as automotive vision, industrial inspection, and robotic perception, image data generated by external imaging devices typically features large data volumes, high transmission rates, and stringent real-time requirements. To meet the needs of image data acquisition, caching, processing, and output, Field Programmable Gate Arrays (FPGAs) or System-on-Chip (SoC) platforms that include processing systems and programmable logic are commonly used, leveraging their parallel processing capabilities and high-speed interface resources to complete image data processing.

[0003] In existing technologies, a processing system can receive camera images and write them to memory, then transfer the image data to programmable logic for processing via direct memory access and on-chip bus; alternatively, data from downstream devices can be written to the host-side storage space via a PCIe link. However, when a USB imaging device is connected to a SoC FPGA architecture platform via a USB host controller, the image data received by the USB host controller is typically first written to storage space managed by the processing system, and then copied, organized, or transferred by the processing system before being provided to the programmable logic for processing. This increases the data handling burden on the processing system and affects the transmission efficiency of USB image data into the programmable logic processing link.

[0004] Therefore, how to improve the efficiency of USB image data transmission from the USB host controller to the programmable logic for processing is a technical problem that needs to be solved in this field. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this application provides a USB image data collaborative transmission method, system, device, and storage medium, which at least solves the problems of heavy data handling burden and low link transmission efficiency in the process of transmitting USB image data from the USB host controller to the programmable logic.

[0006] To achieve the above objectives and other advantages, some embodiments of this application provide the following aspects: In a first aspect, some embodiments of this application provide a USB image data collaborative transmission method, including: The image receiving buffer shared by the processing system and the programmable logic is determined, the transmission description information is configured based on the address information of the image receiving buffer, and the transmission description information is submitted to the USB host controller. The USB host controller receives USB image data output from the USB image device and initiates a PCIe memory write transaction carrying the USB image data and targeting the image receive buffer, based on the transmission description information. The PCIe memory write transaction is converted into an on-chip bus write transaction through the PCIe root complex in order to write the USB image data into the image receive buffer. In response to the transfer completion event of the USB host controller, the processing system controls the programmable logic to read and process the USB image data from the image receive buffer.

[0007] Secondly, some embodiments of this application provide a USB image data cooperative transmission system, including a processing system, programmable logic, memory, PCIe root complex, and at least one USB host controller. The memory is provided with at least one image receiving buffer that is shared and accessed by the processing system and the programmable logic; Each of the USB host controllers is connected to the PCIe root complex via a corresponding PCIe link and has a USB interface for connecting USB imaging devices. The processing system is used to determine the image receiving buffer corresponding to the USB host controller, configure transmission description information based on the address information of the image receiving buffer, and submit the transmission description information to the USB host controller. Any of the USB host controllers is configured to receive USB image data output by the USB image device, and initiate a PCIe memory write transaction carrying the USB image data and targeting the image receive buffer, based on the transmission description information. The PCIe root complex is used to convert the PCIe memory write transaction into an on-chip bus write transaction, so as to write the USB image data output by different USB image devices into the image receiving buffer. The processing system is also configured to, in response to a transmission completion event of any of the USB host controllers, control the programmable logic to read from the image receive buffer and process the USB image data.

[0008] Thirdly, some embodiments of this application also provide an electronic device, the electronic device comprising: One or more processors; and a memory storing computer program instructions that, when executed, cause the processors to perform the USB image data cooperative transfer method as described above.

[0009] Fourthly, some embodiments of this application also provide a computer-readable storage medium having a computer program and / or instructions stored thereon, which, when executed by a processor, implement the USB image data collaborative transmission method as described above.

[0010] Compared with the prior art, the solution provided in this application determines the image receiving buffer shared by the processing system and the programmable logic, and configures the address information of the image receiving buffer in the transmission description information, so that the USB host controller can directly write the received USB image data into the image receiving buffer accessible by the programmable logic using PCIe memory write transactions; the PCIe root complex converts the PCIe memory write transactions into on-chip bus write transactions, so that external USB image data can enter the shared storage space inside the FPGA or SoC; after the USB image data transmission is completed, the processing system controls the programmable logic to read and process the USB image data from the same image receiving buffer. This improves the controllability of the USB image data write address, ensuring that the write target of the USB image data is consistent with the read source of the programmable logic. It reduces intermediate buffering, address translation, and duplicate data copying between the USB receiving side and the programmable logic processing side, thereby reducing the data transfer burden and storage access overhead borne by the processing system. At the same time, it shortens the data flow path of USB image data from reception to processing, reduces end-to-end transmission latency, and coordinates the read processing of the programmable logic with the completion status of USB image data transmission. This improves the collaborative efficiency, real-time performance, and orderly data handover between USB image data transmission and programmable logic processing. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other implementation methods can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is one of the flowcharts illustrating a USB image data collaborative transmission method provided in an embodiment of this application; Figure 2 This is a second schematic flowchart of a USB image data collaborative transmission method provided in an embodiment of this application; Figure 3 A schematic diagram illustrating the end-to-end transmission and processing of USB image data is provided for embodiments of this application; Figure 4 This application provides a schematic diagram of the structure of a multi-interface high-speed image data collaborative transmission system based on a SoC FPGA architecture. Figure 5 A schematic diagram provided for an embodiment of this application; Figure 6 A schematic diagram provided for an embodiment of this application; Figure 7 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0014] Some embodiments of this application relate to a USB image data collaborative transmission method, which can be applied to a multi-interface high-speed image data collaborative transmission system built on a SoC FPGA architecture platform. This method is used to transmit image data output from a USB image device to programmable logic for processing and is suitable for application scenarios that require large-scale image transmission and processing, such as vehicle vision, industrial inspection, robot perception, and high-speed camera acquisition.

[0015] In some embodiments, a multi-interface high-speed image data collaborative transmission system may include a processing system, programmable logic, a PCIe root complex, a USB host controller, and memory. The memory may include storage space shared by the processing system and the programmable logic. The programmable logic may include an image processing module, and may also include AXICDMA, a network communication module, and a peripheral management interface, depending on application requirements, to support USB image data reception, programmable logic processing, data transfer, and processing result output.

[0016] In some embodiments, the system can be configured with logically separate AXI control plane and AXI data plane. The AXI control plane can be used to carry out register configuration and status read transactions for the processing system to the PCIe root complex, image processing module, AXI CDMA, and other functional modules; the AXI data plane can be used to carry high-bandwidth data transactions such as USB image data writing, full frame reading, image processing result writing, and output data transfer. The AXI data plane may include one or more AXISmartConnects and connect to the corresponding AXI master device and a storage access interface for accessing shared storage space.

[0017] The PCIe root complex, image processing module, and AXI CDMA can all function as AXI master devices connecting to the AXI data plane and initiate corresponding AXI memory access transactions. Specifically, the PCIe root complex converts received PCIe memory write transactions into AXI write transactions to write USB image data to the image receive buffer; the image processing module initiates AXI read transactions to read complete frames from the image receive buffer and can also initiate AXI write transactions to write the output data generated from processing the complete frames to the output buffer; and the AXI CDMA initiates AXI read transactions for the source address to read the output data to be moved from the output buffer and initiates AXI write transactions for the destination address to write the output data to the destination storage area. AXI SmartConnect can perform address decoding, arbitration, and routing based on the addresses carried in each AXI memory access transaction, enabling different AXI master devices to access the corresponding storage areas.

[0018] Reference Figure 1 , Figure 2 As shown, this application may include the following steps: Step S1: Determine the image receiving buffer shared by the processing system and the programmable logic, configure the transmission description information based on the address information of the image receiving buffer, and submit the transmission description information to the USB host controller.

[0019] In some embodiments, the processing system can be a processor-side processing unit in a SoC FPGA architecture platform, such as the Processing System (PS) in a Xilinx Zynq series SoC, or other ARM processor systems integrated into the SoC FPGA architecture platform; the programmable logic can be the corresponding FPGA logic side (Programmable Logic, PL). The processing system and the programmable logic can access the same DDR memory space via the on-chip bus, and the image receiving buffer can be set in the DDR memory space. Before initiating USB image data transmission, the processing system can determine a storage area for receiving the current USB image data from the DDR address space shared by it and the programmable logic through a buffer management program running on it, as the image receiving buffer. For example, the processing system can allocate a corresponding storage area in the DDR address space, or select an available storage area from a pre-established buffer set as the image receiving buffer corresponding to the current USB image data transmission.

[0020] To enable the USB host controller to write USB image data to the image receive buffer and for programmable logic to access the written data, the image receive buffer can be located in contiguous physical memory, reserved shared memory, or DMA-coherent memory. The processing system can determine the physical address of the image receive buffer and establish a mapping between the virtual address and the physical address used by the processing system to access the image receive buffer. The virtual address allows software running on the processing system to access the image receive buffer; the physical address is used to configure the data write target of the USB host controller and can be provided to the programmable logic in subsequent processing so that the programmable logic can access the same image receive buffer.

[0021] After determining the image receiving buffer and its address information, the corresponding transmission description information can be configured and submitted according to the specific implementation of the USB host controller.

[0022] In a preferred embodiment, the USB host controller is an xHCI host controller; before submitting the transmission description information to the USB host controller, the xHCI host controller is initialized by the xHCI driver running on the processing system, the USB image devices are enumerated, the interface selection and image data transmission endpoint configuration are completed, and a transmission ring is established for the image data transmission endpoint.

[0023] In some embodiments, the xHCI host controller may be provided by a USB host controller chip connected to the PCIe root complex via a PCIe link. The USB host controller chip acts as a downstream PCIe device of the PCIe root complex on the PCIe side, providing xHCI host control functions and connecting to a USB graphics device via a USB port.

[0024] Before submitting the transfer description information to the xHCI host controller, the xHCI driver running on the processing system can initialize the xHCI host controller. Specifically, after the PCIe root complex completes device enumeration and resource configuration for the USB host controller chip, the xHCI driver can establish a command loop for issuing host control commands and an event loop for recording transfer completion events and error events, enabling the processing system to manage USB ports, USB devices, and the USB data transfer process through the xHCI driver.

[0025] After a USB imaging device is connected or powered on, the xHCI driver can detect the USB imaging device, assign the corresponding USB device address, read the device descriptor and configuration descriptor of the USB imaging device, and select the target interface for transmitting USB image data based on the interface and endpoint information recorded in the configuration descriptor, thus completing the configuration of the image data transmission endpoint in the target interface. The image data transmission endpoint can be the endpoint used by the USB imaging device to output depth data, infrared data, color data, or other image data.

[0026] After configuring the image data transmission endpoints, the xHCI driver can establish corresponding transport rings for each endpoint. The transport ring carries the transport descriptors submitted by the processing system, enabling the xHCI host controller to schedule the corresponding image data transmission endpoints according to the transport descriptors. When a USB imaging device provides multiple image data transmission endpoints, the xHCI driver can configure each endpoint separately and establish corresponding transport rings for each endpoint.

[0027] In some embodiments, after completing the device enumeration, interface selection, and image data transmission endpoint configuration of the USB image device, the processing system can also configure the data stream parameters of the USB image device and start the corresponding image data stream through the USB device control program running on it.

[0028] Specifically, the processing system can determine the type of data stream to be initiated and its corresponding data stream parameters based on the current image acquisition task, and generate a control request for the USB imaging device. The data stream type can include one or more of depth image data streams, color image data streams, infrared image data streams, or other image data streams; the data stream parameters can include one or more of image resolution, frame rate, data format, and endpoint transmission method. The control request can be submitted by the processing system to the xHCI host controller, and then sent by the xHCI host controller to the USB imaging device via USB control transmission, so that the USB imaging device can initiate the corresponding image data stream according to the data stream parameters. The image data stream configuration is used to determine the type of image data output by the USB imaging device and its corresponding transmission parameters, while the transmission ring established by the xHCI driver is used to carry the transmission descriptor for the corresponding image data transmission endpoint. Both provide the foundation for subsequent USB image data transmission from the perspectives of device data stream control and host-side data reception, respectively.

[0029] As an example, a USB imaging device can be a D435 imaging device. The processing system can send image data stream configuration requests to the D435 imaging device through libusb, librealsense, or a device control program with equivalent functionality to initiate depth image data streams, infrared image data streams, and / or color image data streams, and set the corresponding resolution, frame rate, data format, and endpoint transmission method. It should be noted that the D435 imaging device is only used to illustrate one example of a USB imaging device and does not constitute a limitation on the model or type of USB imaging device; a USB imaging device can also be a depth camera, color camera, infrared camera, multispectral camera, or other image acquisition device capable of outputting image data via a USB interface.

[0030] Reference Figure 3As shown, after the USB imaging device completes the data stream parameter configuration and starts the corresponding image data stream, the USB imaging device can output USB image data via the USB3 link and the corresponding image data transmission endpoint. The xHCI host controller can execute the USB image data transmission corresponding to the corresponding image data transmission endpoint according to the transmission descriptor in the transmission ring, and organize the received USB image data into a PCIe memory write transaction. After the PCIe root complex receives the PCIe memory write transaction, it converts the corresponding PCIe transaction layer data packet into an AXI write transaction, and writes the USB image data into the image receive buffer in the shared DDR via the AXI data plane. The data written in the image receive buffer can be the image payload received in a USB transmission or part of the data constituting a complete image frame. The processing system can perform frame synchronization, metadata parsing, or complete frame determination on the corresponding data according to the transmission completion event generated by the xHCI host controller. After determining that the corresponding data constitutes a complete frame, it controls the image processing module in the programmable logic to read and process the complete frame, or moves the corresponding data to other storage areas via AXI CDMA. The corresponding processing results can be further output via network path, local processing path, or other output path.

[0031] After completing the xHCI host controller initialization, USB image device enumeration, interface selection, image data transmission endpoint configuration, and transmission ring establishment, the processing system can configure the corresponding transmission descriptor based on the address information of the image receiving buffer and submit the transmission descriptor to the transmission ring corresponding to the image data transmission endpoint.

[0032] Step S1, which involves configuring transmission description information based on the address information of the image receiving buffer and submitting the transmission description information to the USB host controller, includes: Step S1101: Configure the address information of the image receiving buffer in the transmission descriptor for the image data transmission endpoint.

[0033] In some embodiments, the xHCI driver running on the processing system can determine the image data transmission endpoint corresponding to the USB image data to be received, and determine the transmission descriptor for carrying the current USB image data transmission request from the transmission ring established for the image data transmission endpoint.

[0034] The processing system can obtain the physical address of the image receiving buffer based on the pre-established mapping between virtual and physical addresses, and configure this physical address as the receiving address for USB image data in the transfer descriptor. This causes the transfer descriptor to designate the image receiving buffer as the write target for the current USB image data transmission. The address information configured in the transfer descriptor is the address information that the xHCI host controller can use when performing data writing, rather than a virtual address that is only accessible to the processing system software.

[0035] In one example, a USB imaging device can output depth data, infrared data, or color data. The processing system can determine the image data transmission endpoint corresponding to the currently active image data stream and configure the physical address of the image receive buffer used to receive the image data stream in the transmission descriptor for the corresponding image data transmission endpoint. If the USB imaging device has multiple image data transmission endpoints, corresponding transmission descriptors and image receive buffers can be configured separately for each endpoint.

[0036] The processing system can also record the correspondence between image data transmission endpoints, transmission descriptors, and image receive buffers. This allows it to determine the image receive buffer containing the corresponding USB image data based on the completed transmission descriptor when a transmission completion event is subsequently received. Thus, the transmission descriptor can instruct the xHCI host controller to write the USB image data received via the image data transmission endpoint into the image receive buffer.

[0037] Step S1102: Submit the configured transfer descriptor to the transfer ring so that the xHCI host controller can execute the USB image data transfer corresponding to the image data transfer endpoint according to the transfer descriptor.

[0038] In some embodiments, the xHCI driver can add the configured transfer descriptor to the transfer ring corresponding to the image data transfer endpoint, enabling the xHCI host controller to obtain the transfer descriptor from the transfer ring. Once submitted to the transfer ring, the transfer descriptor can serve as a pending transfer request for the image data transfer endpoint, awaiting scheduling by the xHCI host controller. When the USB imaging device outputs USB image data through the image data transfer endpoint, the xHCI host controller can execute the corresponding USB image data transfer according to the transfer descriptor in the transfer ring, using the image receive buffer configured in the transfer descriptor as the target for writing the received data.

[0039] In the solution provided in this embodiment, through steps S1101 and S1102, the address information of the image receiving buffer is configured in the transmission descriptor corresponding to the image data transmission endpoint, and the transmission descriptor is submitted to the corresponding transmission ring. This enables the xHCI host controller to perform USB image data transmission according to the pre-specified write address, writing the received USB image data into the image receiving buffer shared by the processing system and the programmable logic. Therefore, the correspondence between the image data transmission endpoint and the image receiving buffer is clearly defined, allowing USB image data to be directly written to a storage area that can be subsequently read by the programmable logic. This reduces intermediate buffering and redundant data movement of USB image data between the processing system and the programmable logic, lowering the data movement burden and storage access overhead of the processing system.

[0040] Step S2: Receive USB image data output by the USB image device through the USB host controller, and initiate a PCIe memory write transaction carrying the USB image data with the image receive buffer as the write target according to the transmission description information.

[0041] In some embodiments, after a transfer descriptor is submitted to the transfer ring corresponding to the image data transfer endpoint, the xHCI host controller can obtain and execute the transfer descriptor from the transfer ring, schedule the corresponding data transfer task according to the endpoint configuration information of the image data transfer endpoint, and receive the USB image data output by the USB image device through the image data transfer endpoint.

[0042] Specifically, the transfer descriptor may include a data buffer pointer indicating the image receive buffer, the data length corresponding to the current USB transfer, and the corresponding transfer control information. After receiving USB image data, the xHCI host controller can temporarily store the currently received data in its internal receive buffer, and its DMA write logic reads the transfer descriptor, determines the starting address of the image receive buffer based on the data buffer pointer, and determines the current write address based on the amount of data already written in the current transfer.

[0043] After determining the current write address, the xHCI host controller can construct one or more PCIe memory write request transaction layer packets through the PCIe transaction generation logic. The header of the PCIe memory write request transaction layer packet includes target address information determined based on the current write address, and the data payload of the PCIe memory write request transaction layer packet includes USB image data received via the image data transfer endpoint. Thus, the PCIe memory write transaction carries USB image data and targets the image receive buffer indicated by the transport descriptor.

[0044] When data received in a single USB transfer needs to be transmitted through multiple PCIe memory write transactions, the xHCI host controller can use the image receive buffer address indicated by the transfer descriptor as a reference, determine the target address corresponding to each PCIe memory write transaction based on the amount of data currently written, and organize the USB image data into the data payload portion of multiple PCIe memory write request transaction layer packets according to the data reception order. Thus, each PCIe memory write transaction can write its carried data into the corresponding address range in the image receive buffer, collectively completing the writing of the data received in the current USB transfer.

[0045] It should be noted that the data received in a single USB transfer may correspond to part or all of the data of a frame of image; when a frame of image is split into multiple USB transfers, the data written to the image receiving buffer through multiple USB transfers can together form a complete image frame.

[0046] Step S3: Convert the PCIe memory write transaction into an on-chip bus write transaction via the PCIe root complex to write the USB image data into the image receive buffer.

[0047] In some embodiments, the PCIe root complex can be a PCIe / XDMA Root Complex subsystem located within the SoC FPGA architecture platform. Its PCIe side connects to downstream PCIe devices providing xHCI host control functionality via PCIe links, and its on-chip bus side connects to the AXI data plane via an AXI data interface. The PCIe root complex can translate PCIe memory accesses initiated by downstream PCIe devices into corresponding AXI memory accesses based on a pre-established mapping between the PCIe address space and the AXI address space within the SoC FPGA architecture platform.

[0048] In a preferred embodiment, step S3 specifically includes: Step S301: Receive PCIe memory write transactions initiated by the xHCI host controller through the PCIe root complex.

[0049] In some embodiments, the xHCI host controller can send the PCIe memory write request generated in step S2 to the PCIe root complex via the PCIe interface of the PCIe device it resides in. The PCIe root complex receives the PCIe transaction layer data packet corresponding to the PCIe memory write transaction via the established PCIe link. The PCIe transaction layer data packet may include a transaction header and a data payload.

[0050] After receiving a PCIe transaction layer data packet, the PCIe root complex can parse its transaction type and destination address to determine that the transaction layer data packet is a PCIe memory write request for the system storage space, and determine the corresponding DDR address space shared by the processing system and programmable logic based on the destination address.

[0051] When a USB transfer is carried by multiple PCIe memory write transactions, the PCIe root complex can sequentially receive multiple corresponding PCIe transaction layer data packets. Each PCIe transaction layer data packet carries the target address information of the corresponding address range in the image receive buffer and the corresponding part of the USB image data.

[0052] Step S302: Convert the PCIe transaction layer data packet corresponding to the PCIe memory write transaction into an AXI write transaction.

[0053] In some embodiments, the PCIe root complex can generate a corresponding AXI write transaction based on the target address, write data length, and data payload carried in the PCIe transaction layer data packet. Specifically, the PCIe root complex can convert the target address in the PCIe transaction layer data packet into an AXI target address within the SoC FPGA architecture platform based on a pre-configured mapping relationship between the PCIe address space and the AXI address space; convert the data payload into AXI write data; and determine the transmission length of the AXI write transaction based on the corresponding data length, thereby maintaining the correspondence between the target address in the PCIe memory write transaction and the USB image data.

[0054] In one specific implementation, the PCIe root complex can output the AXI target address and corresponding write control information through the AXI write address channel, and output the USB image data obtained from the data payload of the PCIe transaction layer packet through the AXI write data channel. When the data payload of a PCIe transaction layer packet needs to be transmitted through multiple AXI data cycles, the PCIe root complex can divide the data payload into multiple AXI write data cycles according to the data bit width of the AXI interface, and output it in the form of a single AXI burst write transaction or multiple AXI write transactions. Thus, one PCIe memory write transaction can correspond to one or more AXI write data cycles.

[0055] Step S303: Route the AXI write transaction to the image receive buffer via the AXI data plane to write the USB image data to the image receive buffer.

[0056] In some embodiments, the PCIe root complex can output the converted AXI write transactions to the AXI data plane through its AXI data interface. The AXI data plane may include one or more AXI SmartConnects and connect the AXI data interface of the PCIe root complex to the storage access interface for accessing shared DDR.

[0057] AXI SmartConnect can decode the AXI destination address in an AXI write transaction. Based on the decoding result, it determines that the AXI destination address corresponds to the image receive buffer in the shared DDR and routes the AXI write transaction to the corresponding storage access interface. After the AXI write transaction arrives at the shared DDR storage access interface, the USB image data carried in the AXI write data can be written to the corresponding address range of the image receive buffer according to the AXI destination address. When the data payload of a PCIe transaction layer packet exceeds the data volume that a single AXI write data tick can carry, the PCIe root complex can organize the data payload into multiple AXI write data ticks according to the data bit width of the AXI interface and output them through one or more AXI write transactions.

[0058] The AXI data plane can be logically separated from the AXI control plane used for register configuration and status reading. This allows high-bandwidth memory access transactions for writing USB image data to be transmitted via the AXI data plane, while register configuration transactions for xHCI, PCIe root complex, and other modules are transmitted via the AXI control plane, thereby reducing the mutual interference between high-bandwidth data writes and low-bandwidth control accesses.

[0059] In the solution provided in this embodiment, through steps S301 to S303, the PCIe root complex receives the PCIe memory write transaction initiated by the xHCI host controller, converts the target address and data payload in the corresponding PCIe transaction layer data packet into an AXI write address and AXI write data, and routes the AXI write transaction to the image receive buffer in the shared DDR via the AXI data plane. Therefore, without changing the target of the USB image data write, the connection between the PCIe transaction and the on-chip bus transaction within the SoC FPGA architecture platform can be completed. This allows USB image data to be directly written to the storage area shared by the processing system and programmable logic via the PCIe link, reducing the need for the processing system to participate in protocol conversion and data transfer.

[0060] Step S4: In response to the transfer completion event of the USB host controller, read and process the USB image data from the image receive buffer by processing the system control programmable logic.

[0061] In some embodiments, after the xHCI host controller completes the USB image data transmission indicated by the transmission description information, it can generate a corresponding transmission completion event. The processing system determines the image receiving buffer where the transmitted data is located based on the transmission completion event, and after confirming that the corresponding data constitutes a complete image frame, provides the storage address of the complete image frame and related processing parameters to the image processing module in the programmable logic to trigger the image processing module to read and process the complete image frame.

[0062] In a preferred embodiment, step S4 specifically includes: Step S401: After the xHCI host controller completes one or more USB transfers corresponding to the transfer description information, a transfer completion event is generated, and an interrupt is reported to the PCIe root complex to notify the processing system to obtain the transfer completion event.

[0063] In some embodiments, the xHCI host controller can execute the transfer descriptor in the transfer ring and receive USB image data output by the USB imaging device through the corresponding image data transfer endpoint. When the transfer request indicated by the transfer descriptor is completed, or when an exception occurs during the transfer, the xHCI host controller can generate a corresponding transfer completion event and write the transfer completion event into the event ring.

[0064] A transfer completion event can include information such as identifying the completed transfer descriptor, the transfer completion status, and the actual length of data transferred, enabling the processing system to determine the image data transfer endpoint, transfer descriptor, and image receive buffer corresponding to this completion event. After the transfer completion event is written to the event ring, the xHCI host controller can send an interrupt request to the PCIe root complex via the PCIe link using MSI, MSI-X, or a traditional interrupt method. Upon receiving the interrupt request, the PCIe root complex can provide the corresponding interrupt signal to the processing system. In response to the interrupt signal, the processing system reads the event ring through the xHCI driver running on it, obtains the transfer completion event, and determines the completed USB image data transfer accordingly.

[0065] It should be noted that a single transmission completion event can correspond to a transmission request indicated by a transmission descriptor; when an image frame needs to be received through multiple transmission descriptors or multiple USB transmissions, the processing system can sequentially obtain multiple corresponding transmission completion events.

[0066] Step S402: The processing system determines, based on the transmission completion event, whether the USB image data written in the image receiving buffer is an image payload, a frame fragment, or a complete frame.

[0067] In some embodiments, the processing system can determine the image receiving buffer corresponding to this USB transmission based on the transmission descriptor associated with the transmission completion event, and obtain the actual completed data length, transmission status, and corresponding buffer write range.

[0068] The processing system can combine one or more transmission completion events to identify USB image data already written to the image receive buffer. Specifically, data received in a single USB transmission can be used as the corresponding image payload; when the image payload constitutes only a portion of the data in a frame, it can be identified as a frame fragment; when the data received in one or more USB transmissions meets preset complete frame conditions, the corresponding data can be identified as a complete frame. Complete frame conditions can be determined based on the data organization method of the USB image device. For example, the processing system can determine whether the currently written data forms a complete frame based on one or more of the following: frame boundary information in the received data, cumulative data length, continuous transmission relationship, and frame size corresponding to the image format.

[0069] If the currently written data consists only of image payloads or frame fragments, the processing system can retain the corresponding image receiving buffer and its data state, and wait for subsequent transmission completion events. After determining that the corresponding data constitutes a complete frame, the processing system can further determine the physical address, data length, frame number, and format parameters of the complete frame in the image receiving buffer. The frame number can be used to identify the order of the complete frame in a continuous image sequence, and the format parameters can include one or more of the following: image width, image height, pixel format, channel type, or data arrangement method.

[0070] Step S403: If it is determined that the written USB image data is a complete frame, the processing system writes the physical address of the image receiving buffer, as well as the data length, frame number, and format parameters of the complete frame, into the register of the image processing module set in the programmable logic, so as to trigger the image processing module to read the complete frame.

[0071] In some embodiments, after the processing system determines that the corresponding USB image data in the image receive buffer constitutes a complete frame, it can generate access parameters to indicate the complete frame. The access parameters may include the physical start address of the complete frame in the image receive buffer, the data length of the complete frame, the frame number, and format parameters. The processing system can access the registers of the image processing module set in the programmable logic through the AXI control plane and write the physical start address, data length, frame number, and format parameters into the corresponding parameter registers. The physical start address and data length are used to limit the range of storage addresses that the image processing module needs to read, the frame number is used to identify the current image frame to be processed, and the format parameters are used to indicate the data organization method and corresponding processing method of the complete frame.

[0072] In one specific implementation, after writing the aforementioned parameters, the processing system can also write a start field or update the data validity flag to the image processing module to notify the image processing module that a complete frame to be processed currently exists. Upon detecting the start field or data validity flag, the image processing module can latch the corresponding parameters and enter the complete frame reading and processing state.

[0073] In this way, the processing system is mainly responsible for identifying the complete frame and issuing the corresponding access parameters, without having to copy the complete frame to another storage area on the programmable logic side.

[0074] Step S404: The image processing module initiates an AXI read transaction based on the physical address and data length, reads the complete frame from the image receiving buffer, and processes the complete frame according to the format parameters.

[0075] In some embodiments, the image processing module can read the corresponding parameter registers to obtain the physical start address, data length, frame number, and format parameters of the complete frame, and generate one or more AXI read transactions based on the physical start address and data length. The image processing module can send the AXI read transactions to the AXI data plane through its AXI master interface. The AXI data plane can route the AXI read transactions to the memory access interface corresponding to the shared DDR based on the address carried by the AXI read transactions, and read the complete frame from the corresponding address range in the image receive buffer.

[0076] When the data volume of a complete frame needs to be read through multiple AXI read transactions, the image processing module can update the read address of subsequent AXI read transactions according to the length of the data already read, and read the complete frame in batches according to the address order until the data reading limited by the data length is completed.

[0077] After obtaining the complete frame, the image processing module can process it according to format parameters. For example, based on the image width, image height, pixel format, and data arrangement, it can perform one or more of the following: frame data processing, pixel format conversion, protocol encapsulation, image preprocessing, or image algorithm processing. The processing results can be further written to the output buffer, network transmission buffer, or other system-accessible storage areas in shared DDR.

[0078] After the image processing module completes the reading and processing of the entire frame, it can also generate a frame processing completion event to notify the processing system to reclaim the corresponding image receiving buffer or reconfigure the image receiving buffer in a subsequent transmission descriptor.

[0079] In the solution provided in this embodiment, through steps S401 to S404, after the xHCI host controller completes the USB image data transmission, the processing system identifies the data status in the corresponding image receiving buffer based on the transmission completion event. After forming a complete frame, the physical address and processing parameters of the complete frame are provided to the image processing module in the programmable logic, enabling the image processing module to directly read and process the complete frame from the same image receiving buffer. This allows the USB image data transmission completion status, frame integrity judgment, and programmable logic processing to be interconnected, reducing redundant data copying between the processing system and the programmable logic, and improving the collaborative efficiency and real-time performance between USB image reception and programmable logic processing.

[0080] In a preferred embodiment, the programmable logic includes an AXI CDMA, the AXI CDMA control interface is connected to the AXI control plane, and the AXI CDMA main interface is connected to the AXI data plane, which is logically separate from the AXI control plane, via AXI SmartConnect.

[0081] In some embodiments, the AXI CDMA direct memory access module for moving data between different AXI memory-mapped address spaces is located in programmable logic and includes an AXI-Lite slave interface for register configuration and status reading, and an AXI master interface for data reading and writing. The AXI-Lite slave interface is connected to the AXI control plane to allow the processing system to configure the source address, destination address, and transfer length and read the moving status. The AXI master interface is connected to the AXI data plane, which is logically separated from the AXI control plane, through AXISmartConnect to initiate AXI read transactions for the source address and AXI write transactions for the destination address.

[0082] In a preferred embodiment, the method further includes: Step A1: Write the output data generated by processing the complete frame into the output buffer through the image processing module.

[0083] In some embodiments, the image processing module can perform one or more of the following on the complete frame read from the image receiving buffer according to the format parameters configured in step S403: frame data processing, format conversion, protocol encapsulation, image preprocessing, or image algorithm processing, and generate corresponding output data. The output data can be a formatted image frame, data to be sent formed by protocol encapsulation, image preprocessing results, or image algorithm processing results. The image processing module can initiate an AXI write transaction through its AXI main interface to write the output data to a predetermined output buffer.

[0084] The output buffer can be located in the shared DDR, or in the image processing buffer, or in another AXI address space that can be accessed by both the image processing module and AXI CDMA. The image processing module can write the output data into the output buffer in the order in which the output data is generated, within a continuous or corresponding address range based on the corresponding starting address.

[0085] After completing the output data writing, the image processing module can also determine the starting address and actual data length of the output data in the output buffer, and notify the processing system through the status register, buffer status information or frame completion event, so that the processing system can configure subsequent AXI CDMA migration tasks.

[0086] Step A2: The processing system, via the AXI control plane, configures the starting address of the output data in the output buffer as the source address of AXI CDMA, configures the starting address of the destination storage area used to receive the output data as the destination address of AXI CDMA, and configures the data length of the output data as the transmission length of AXI CDMA.

[0087] In some embodiments, the processing system can determine the starting address and data length of the output data to be moved based on the output buffer information provided by the image processing module, and determine the corresponding destination storage area according to the subsequent use of the output data. For example, if the output data needs to be further read by the processing system software, the destination storage area can be another storage area in the shared DDR; if the output data needs to be sent via a network path, the destination storage area can be a network transmission buffer; if the output data needs to be further processed by other programmable logic modules or output through other data paths, the destination storage area can also be other AXI address spaces that the corresponding modules can access.

[0088] The processing system can access the AXI-Lite slave interface of the AXI CDMA through the AXI control plane, write the starting address of the output data in the output buffer to the AXI CDMA source address register, write the starting address of the destination memory area to the AXI CDMA destination address register, and write the data length of the output data to the AXI CDMA transmission length register. Both the source and destination addresses can be addresses that the AXI data plane can recognize and route. After configuring the above transmission parameters, the processing system can issue a migration start command to the AXI CDMA, or set the corresponding start state to enable the AXI CDMA to execute the current output data migration task according to the configured source address, destination address, and transmission length.

[0089] It should be noted that the processing system configures the AXI CDMA transfer parameters in this step, rather than transmitting the output data itself through the AXI control plane. The actual reading and writing of the output data are completed by AXI CDMA through the AXI data plane, thereby avoiding mixing high-bandwidth data transactions into the register configuration path.

[0090] Step A3: Using AXI CDMA, the output data is moved from the output buffer to the destination storage area via the AXI data plane based on the source address, destination address, and transmission length.

[0091] In some embodiments, after receiving the migration start instruction, AXI CDMA can generate one or more AXI read transactions based on the source address and send the AXI read transactions to the AXI data plane through its AXI master interface.

[0092] In the AXI data plane, AXI SmartConnect decodes the source address carried in an AXI read transaction and routes it to the storage access interface corresponding to the output buffer. AXI CDMA can then read output data of the specified transmission length, starting from the source address. After obtaining the corresponding output data, AXI CDMA can generate one or more AXI write transactions based on the destination address. AXI SmartConnect decodes the destination address carried in the AXI write transaction and routes it to the storage access interface corresponding to the destination storage area, thereby writing the read output data to the destination storage area.

[0093] When the length of the output data needs to be completed through multiple AXI reads and writes, AXI CDMA can update the subsequent read address and write address according to the amount of data that has been moved so that the source side read address and the destination side write address change synchronously according to the corresponding data offset until all the output data limited by the transmission length is moved.

[0094] In some embodiments, the destination storage area may be configured as a local processing buffer, data storage area, network transmission buffer, or storage area corresponding to other data output paths in a shared DDR, depending on how the image processing results are subsequently used.

[0095] When the destination address points to the local processing buffer, AXI CDMA can move the output data to the local processing buffer for subsequent access by the processing system software or other processing modules in the programmable logic, and perform display, analysis, format conversion or other processing on the output data.

[0096] When the destination address points to a data storage area, AXI CDMA can move the output data to the data storage area. After confirming that the data move is complete, the processing system can update the data storage area to a valid data state and associate the output data with one or more of the following: frame number, timestamp, data length, data format, or processing result type, for subsequent reading, playback, or offline processing.

[0097] When the destination address points to the network transmit buffer, AXI CDMA can move the output data to the network transmit buffer. The network communication module can access the network transmit buffer through the AXI data plane, read the output data, and send the output data to external network devices through the corresponding GEM network interface and SGMII link.

[0098] In some embodiments, the network communication module may include a first GEM network interface and a second GEM network interface, which are respectively connected to an external network via corresponding SGMII links.

[0099] In one configuration, the first GEM network interface can be used to send image data or image processing results, and the second GEM network interface can be used to transmit control commands, system status, or diagnostic information, so as to separate business data communication from maintenance and management communication.

[0100] In another configuration, the first GEM network interface and the second GEM network interface can serve as the primary network interface and the redundant network interface, respectively. When the network link corresponding to the primary network interface becomes unavailable, the processing system can activate the redundant network interface to continue forwarding image processing results, reporting status, or performing remote maintenance.

[0101] In some embodiments, the destination address may also point to a storage area corresponding to other DMA paths or PCIe backhaul paths, so that the output data can be transmitted to an external computing device through the corresponding data path.

[0102] Step A4: When the output data transfer is completed or an abnormality occurs during the transfer process, an interrupt is generated through AXI CDMA, and the status information of AXI CDMA is obtained by the system handling the interrupt to determine the transfer status of the output data.

[0103] In some embodiments, when AXI CDMA completes the reading and writing of output data corresponding to the transmission length, a transfer completion interrupt can be generated; when an AXI read / write error, address access exception, transmission interruption, or other transfer exception occurs during the transfer process, AXI CDMA can generate a corresponding error interrupt.

[0104] Interrupts generated by AXI CDMA can be sent to the interrupt input of the processing system via the interrupt aggregation module. After responding to the interrupt, the processing system can read the status register of AXI CDMA via the AXI control plane to obtain the status information corresponding to the current transfer task, and determine whether the output data has been transferred successfully or an exception occurred during the transfer process.

[0105] Once the status information indicates that the output data transfer is complete, the processing system can update the destination storage area to a valid data state and notify subsequent modules using that destination storage area to read the corresponding output data. At the same time, the output buffer can be released or reclaimed for continued use by subsequent image processing results.

[0106] If an anomaly is detected during the transfer process based on the status information, the processing system can record the corresponding anomaly status, clear the interrupt status, and reconfigure AXI CDMA, resubmit the transfer task, or restore the relevant data paths according to the anomaly type. The specific anomaly recovery method can be determined based on the system's preset anomaly handling strategy.

[0107] In the solution provided in this embodiment, through steps A1 to A4, the output data generated by the image processing module is written to the output buffer. The processing system configures the source address, destination address, and transmission length of the AXI CDMA via the AXI control plane. Then, the AXI CDMA completes the output data transfer via the AXI data plane, which is logically separate from the control plane. The transfer status is fed back to the processing system through completion interrupt or error interrupt. Therefore, the low-bandwidth transfer parameter configuration and the high-bandwidth output data transfer can be transmitted via the control plane and data plane respectively, reducing the mutual interference between control transactions and large data access, and improving the efficiency and controllability of transferring image processing results between shared DDR, network transmit buffer, and other AXI address spaces.

[0108] In a preferred embodiment, in order to uniformly manage different interrupt events generated during USB image data transmission, image processing, and output data transfer, the multiple interrupt inputs of the interrupt aggregation module are respectively connected to one or more of the PCIe root complex, AXI CDMA, image processing module, and network communication module, and the interrupt output of the interrupt aggregation module is connected to the interrupt input of the processing system.

[0109] Specifically, the PCIe root complex can provide the interrupt aggregation module with transmission completion interruptions, transmission error interruptions, or PCIe link error interruptions reported by the xHCI host controller; the image processing module can provide the interrupt aggregation module with frame processing completion interruptions or image processing error interruptions when a complete frame processing is completed or an error occurs during processing; and AXICDMA can provide the interrupt aggregation module with transfer completion interruptions or error interruptions when an error occurs during output data transfer. The interrupt aggregation module can aggregate the received different interrupt signals and output an interrupt request to the processing system.

[0110] The interrupt aggregation module can set corresponding interrupt status bits, interrupt flags, or interrupt input channels for different interrupt sources. Upon responding to an interrupt request, the processing system can read the interrupt status information from the interrupt aggregation module and determine the interrupt source based on this information. If the interrupt source corresponds to a PCIe root complex or xHCI host controller, the processing system can read the corresponding event ring through the xHCI driver to obtain a transmission completion event or a transmission exception event. If the interrupt source corresponds to an image processing module, the system can read the image processing module's status information to determine the processing status of the complete frame. If the interrupt source corresponds to AXI CDMA, the system can read the AXI CDMA's status information to determine the transfer status of the output data.

[0111] After the processing system completes the corresponding interrupt handling, it can clear the corresponding interrupt status information. If there are still other unhandled interrupt sources, the processing system can continue to identify and handle the next interrupt source until the processing of all current interrupt events is completed.

[0112] In the solution provided in this embodiment, the interrupt signal is uniformly provided to the processing system through the interrupt aggregation module, and the processing system identifies the interrupt source and performs corresponding processing based on the interrupt status information. This can reduce the occupation of the interrupt interface of the processing system, avoid confusion between different interrupt events, and enable the processes such as USB image data reception, complete frame processing and output data transfer to be interconnected through a unified interrupt entry point, thereby reducing the interrupt management complexity of the multi-interface image transmission system.

[0113] In a preferred embodiment, the image receiving buffer uses contiguous physical memory, reserved shared memory, or DMA-coherent memory, and the processing system records the correspondence between the virtual address and the physical address of the image receiving buffer.

[0114] In some embodiments, the processing system may pre-allocate one or more image receive buffers in memory. These image receive buffers may utilize contiguous physical memory, reserved shared memory, or DMA-coherent memory. The processing system may record the virtual address, physical address, buffer length, and usage status of each image receive buffer, and establish a mapping between virtual and physical addresses. The virtual address allows software running on the processing system to access and manage the image receive buffers, while the physical address is used to configure the xHCI host controller's transfer descriptor and is provided to the image processing module in the programmable logic, enabling both the xHCI host controller and the image processing module to access the same image receive buffer.

[0115] It should be noted that the image receive buffer lacks DMA consistency, meaning that the storage access path containing the image receive buffer does not provide hardware consistency guarantees between the processing system cache and DMA access. In this case, when the xHCI host controller writes USB image data to the image receive buffer via DMA, it does not automatically update or invalidate the cached data corresponding to the corresponding address range in the processing system cache. Therefore, the processing system needs to perform cache synchronization processing on the corresponding address range according to the data access direction to ensure that subsequent accesses can obtain the data corresponding to this DMA write.

[0116] In a preferred embodiment, the method further includes: Step B1: If the image receiving buffer does not have DMA consistency, before triggering the image processing module to read the complete frame, the processing system performs buffer synchronization processing on the address range in the image receiving buffer corresponding to the complete frame.

[0117] In some embodiments, after the processing system determines that the USB image data written to the image receive buffer constitutes a complete frame based on the transmission completion event, it can determine the range of addresses to be synchronized corresponding to the complete frame based on the physical start address and data length of the complete frame in the image receive buffer. If the image receive buffer lacks DMA consistency, the processing system can perform cache synchronization processing on the range of addresses to be synchronized, matching the current data access direction, before providing the physical start address, data length, frame number, and format parameters of the complete frame to the image processing module. Cache synchronization processing can include one or more of cache cleanup, cache invalidation, cache cleanup and invalidation, and memory access synchronization. The specific processing method can be determined based on the cache attributes of the image receive buffer, the cache structure of the processing system, and the memory consistency mechanism adopted by the SoC FPGA architecture platform.

[0118] For example, after the xHCI host controller writes a complete frame to the image receive buffer via the PCIe root complex and AXI data plane, the processing system can invalidate the cached data corresponding to the address range to be synchronized in its cache. If there is data in the corresponding cache that may be written back to memory, the system performs necessary cache cleanup to prevent old data or data to be written back from affecting the complete frame in the image receive buffer. The processing system can also perform corresponding memory access synchronization operations to ensure that the data write corresponding to the aforementioned PCIe memory write transaction has been completed and is visible to subsequent AXI read transactions initiated by the image processing module.

[0119] After completing the cache synchronization process, the processing system can write the physical start address, data length, frame number and format parameters of the complete frame into the corresponding register of the image processing module according to step S403, so as to trigger the image processing module to read the complete frame.

[0120] It should be noted that when the image receiving buffer uses DMA-coherent memory, or when the corresponding storage access path can guarantee data visibility between the xHCI host controller, processing system, and image processing module through hardware consistency mechanisms, additional cache synchronization processing is not required.

[0121] Step B2: After the image processing module completes the processing of the complete frame and generates a frame completion event, the processing system responds to the frame completion event by reclaiming the image receiving buffer.

[0122] In some embodiments, after the image processing module completes reading and processing of the complete frame, it can update the corresponding frame processing status and generate a frame completion event. The frame completion event can be transmitted to the processing system via an interrupt output of programmable logic to notify the processing system that the current complete frame has been processed. In response to the frame completion event, the processing system can read the status information of the image processing module through the AXI control plane and determine the image receiving buffer storing the complete frame based on one or more of the frame number associated with the frame completion event, the buffer identifier, the physical address of the complete frame, or the image processing task information.

[0123] After determining that the image processing module has completed accessing the image receive buffer, the processing system can detach the image receive buffer from the current complete frame, the current transport descriptor, and the current image processing task, and update the usage status of the image receive buffer from "processing" to "available," thus completing the reclamation of the image receive buffer. Reclaiming the image receive buffer can mean adding it back to the free buffer queue or the available buffer set, rather than releasing the physical memory corresponding to the image receive buffer back to the memory management system. Therefore, the reclaimed image receive buffer can continue to be used to receive subsequent USB image data.

[0124] After updating the state of the image receiving buffer, the processing system can also clear the interrupt status corresponding to the frame completion event and update the corresponding buffer management information. When the processing result generated by the image processing module still depends on the original complete frame, the processing system can update the corresponding image receiving buffer to an available state only after all relevant data accesses are completed, in order to avoid the buffer being reused prematurely.

[0125] Step B3: Based on the correspondence between virtual addresses and physical addresses, configure the physical address of the recycled image receive buffer in the next transfer descriptor, and submit the next transfer descriptor to the transfer ring so that the xHCI host controller can write subsequent USB image data into the image receive buffer.

[0126] In some embodiments, the processing system may select a recycled image receiving buffer from a free buffer queue or an available buffer set, and obtain the physical starting address and buffer length of the image receiving buffer according to a pre-recorded correspondence between virtual addresses and physical addresses.

[0127] The processing system can configure the physical start address in the data buffer address field of the next transfer descriptor, and configure the writable data length indicated by the next transfer descriptor according to the buffer length or the expected receive data length corresponding to the next USB transfer. The processing system can also record the correspondence between the next transfer descriptor, the corresponding image data transfer endpoint, and the recycled image receive buffer, so as to determine the write position of the corresponding USB image data when a transfer completion event is generated subsequently.

[0128] After configuring the next transfer descriptor, the processing system can submit the next transfer descriptor to the transfer ring corresponding to the image data transmission endpoint, and update the usage status of the image receive buffer from the available state to the submitted state or the awaited state. The xHCI host controller can obtain and execute the next transfer descriptor from the transfer ring. When it subsequently receives USB image data output from the USB imaging device, it writes the subsequent USB image data into the recycled image receive buffer according to the physical address configured in the next transfer descriptor.

[0129] When multiple image receive buffers are configured, the processing system can cyclically select available image receive buffers based on their usage status. For example, while one image receive buffer is used to receive USB image data, another can be used to store complete frames awaiting processing, and yet another can be read and processed by the image processing module. After processing is completed and the corresponding image receive buffer is recycled, the processing system can reconfigure it in subsequent transmission descriptors, thus forming a cyclical processing process that seamlessly integrates image reception, complete frame processing, and buffer reuse.

[0130] In the solution provided in this embodiment, through steps B1 to B3, when the image receiving buffer lacks DMA consistency, cache synchronization processing is performed on the address range corresponding to the complete frame, ensuring that the USB image data written by the xHCI host controller can be correctly read by the image processing module. After the image processing module completes the processing of the complete frame, the corresponding image receiving buffer is reclaimed, and its physical address is reconfigured in the subsequent transport descriptor. This avoids invalid or erroneous data read by the image processing module due to cached data asynchrony, prevents the image receiving buffer still in processing from being prematurely overwritten by subsequent USB image data, and reduces memory redistribution and data copying overhead through cyclic reuse of the image receiving buffer, thereby improving the stability and coordination efficiency between continuous USB image data reception and programmable logic processing.

[0131] In a preferred embodiment, such as Figure 4 As shown, the PCIe root complex is connected to at least two PCIe connection branches via at least two PCIe links. Each PCIe connection branch includes two cascaded PCIe-USB bridge chips. The upstream PCIe-USB bridge chip is connected to the PCIe root complex via a corresponding PCIe link, and the downstream PCIe-USB bridge chip is connected to the upstream PCIe-USB bridge chip via cascaded PCIe links. Each PCIe-USB bridge chip can provide an xHCI host controller and connect to a USB graphics device via a corresponding USB interface.

[0132] exist Figure 4In the specific implementation shown, the PCIe root complex connects two PCIe connection branches via PCIe link 0 and PCIe link 1 respectively. Each PCIe connection branch includes two PCIe-USB bridge chips, thus forming four PCIe-USB bridge chip access points. Each PCIe-USB bridge chip can provide an xHCI host controller and connect to a USB graphics device through a corresponding USB interface. Each xHCI host controller can maintain its corresponding command ring, event ring, and transport ring. In one specific implementation, the PCIe-USB bridge chip can be a PCI11101 chip, and the USB graphics device can be a D435 graphics device; the above chip models and device types are only examples. Figure 4 The number of PCIe links, PCIe connection branches, and USB graphics devices shown is only for illustrating the above connection structure and does not constitute a limitation on their quantity.

[0133] The image receive buffer includes multiple image receive buffers corresponding to each xHCI host controller. Each image receive buffer can be located in DDR memory space shared by the processing system and programmable logic. Each xHCI host controller can correspond to one or more image receive buffers. The processing system can establish the correspondence between the PCIe-USB bridge chip, xHCI host controller, USB imaging devices, transfer description information, and image receive buffers, so that the USB image data output by each USB imaging device is written to the corresponding image receive buffer, and the processing system can determine the image receive buffer to be read by the programmable logic based on the transfer completion event.

[0134] In a preferred embodiment, the method further includes: Step C1: Establish each PCIe link and the cascaded PCIe links within each PCIe connection branch through the PCIe root complex, and complete the device enumeration and resource configuration of each PCIe-USB bridge chip.

[0135] In some embodiments, after the system powers on, the processing system can release the reset signals of the PCIe root complex and each PCIe-USB bridge chip after the reference clock and on-chip bus clock have stabilized. The PCIe root complex performs PCIe link training with the upstream PCIe-USB bridge chip in each PCIe connection branch, and the upstream PCIe-USB bridge chip and downstream PCIe-USB bridge chip in the same PCIe connection branch perform cascaded PCIe link training.

[0136] After each PCIe link is established, the processing system can traverse the corresponding PCIe connection branches, identify each PCIe-USB bridge chip, and allocate PCIe device resources, BAR addresses, and interrupt resources to each PCIe-USB bridge chip. The xHCI driver running on the processing system can further initialize the xHCI host controllers provided by each PCIe-USB bridge chip, establish the corresponding command ring, event ring, and transport ring, and perform USB device enumeration, interface selection, and image data transmission endpoint configuration for the USB imaging devices connected to each xHCI host controller.

[0137] It should be noted that the PCIe device enumeration targets the PCIe-USB bridge chips in each PCIe connection branch, while the USB device enumeration targets the USB graphics devices connected to each xHCI host controller. These two belong to different levels of initialization processes.

[0138] Step C2: Configure the transmission description information for the corresponding xHCI host controller based on the address information of each image receiving buffer.

[0139] In some embodiments, the processing system may obtain the physical address and buffer length of each image receiving buffer, configure the physical address of each image receiving buffer in the transfer descriptor of the corresponding xHCI host controller, and submit the transfer descriptor to the transfer ring of the corresponding image data transmission endpoint.

[0140] The processing system can also record the correspondence between xHCI host controllers, image data transmission endpoints, transfer descriptors, and image receive buffers. When an xHCI host controller corresponds to multiple cyclically used image receive buffers, the processing system can configure transfer descriptors for each image receive buffer to form corresponding buffer queues. When the corresponding PCIe-USB bridge chip is not connected to a USB image device, the processing system can temporarily not submit transfer descriptors to its xHCI host controller, and configure the corresponding image receive buffers and transfer descriptor information after detecting the USB image device connection and completing USB device enumeration.

[0141] Step C3: Receive USB image data output by the corresponding USB image device through any xHCI host controller, and initiate a PCIe memory write transaction with the corresponding image receive buffer as the write target according to the corresponding transfer description information.

[0142] In some embodiments, each xHCI host controller can independently receive USB image data output by the corresponding USB imaging device. Any xHCI host controller can obtain and execute a transport descriptor from its transport ring, receive USB image data through the corresponding image data transmission endpoint, and initiate a PCIe memory write transaction with the corresponding image receive buffer as the write target, based on the physical address configured in the transport descriptor.

[0143] For an upstream PCIe-USB bridge chip, its initiated PCIe memory write transactions can be transmitted to the PCIe root complex through the corresponding PCIe link; for a downstream PCIe-USB bridge chip, its initiated PCIe memory write transactions can be transmitted to the PCIe root complex sequentially through the downstream PCIe-USB bridge chip, the corresponding cascaded PCIe link, and the upstream PCIe-USB bridge chip.

[0144] The PCIe root complex receives write transactions from various PCIe memory devices, converts the corresponding PCIe transaction layer data packets into AXI write transactions, and writes the USB image data to the image receive buffer corresponding to the destination address via the AXI data plane. When multiple PCIe memory write transactions overlap in time, the AXI SmartConnect in the AXI data plane can arbitrate the corresponding AXI write transactions and, based on the destination address carried by each AXI write transaction, write the USB image data output by different USB image devices to the corresponding image receive buffers respectively, thereby avoiding transmission conflicts between concurrent write transactions and cross-writing between different USB image data.

[0145] Step C4: In response to a transmission completion event from any xHCI host controller, the processing system determines the corresponding image receive buffer and controls the programmable logic to read USB image data from the image receive buffer.

[0146] In some embodiments, each xHCI host controller maintains its own event ring. When any xHCI host controller completes one or more USB transfers as indicated by the corresponding transfer description information, it can generate a transfer completion event, write the transfer completion event to the corresponding event ring, and report the interrupt to the PCIe root complex through its PCIe connection branch.

[0147] In response to the interrupt, the processing system can determine the xHCI host controller that generated the transmission completion event based on at least one of the PCIe device identifier, interrupt vector, xHCI event loop, and transport descriptor identifier, and determine the corresponding image receive buffer according to a pre-established correspondence.

[0148] The processing system can further obtain the physical address, actual data length, and transmission status of the data already written to the image receive buffer. When the written data constitutes a complete frame, the processing system can provide the programmable logic with the physical address, data length, frame number, and format parameters of the image receive buffer to trigger the programmable logic to read and process the complete frame. When the written data constitutes a frame fragment, the processing system can maintain the occupancy status of the image receive buffer and wait for the subsequent transmission completion event from the corresponding xHCI host controller. When multiple image receive buffers contain complete frames to be processed, the processing system can sequentially trigger the reading and processing of the corresponding complete frames according to the arrival order of the transmission completion events and the operating status of the programmable logic.

[0149] In the solution provided in this embodiment, steps C1 to C4 can be used to expand the number of USB image devices that can be connected by utilizing multiple PCIe connection branches and cascaded PCIe-USB bridge chips within the branches. Simultaneously, by establishing the correspondence between each xHCI host controller, transmission description information, image receiving buffer, and transmission completion event, multiple USB image data streams can be written to their respective image receiving buffers and accurately read by programmable logic. This avoids data mixing or buffer identification errors during concurrent transmission, improving the parallelism and reliability of multi-channel USB image data transmission. In a preferred embodiment, the method further includes detecting anomalies occurring during USB image data transmission and processing, and performing corresponding recovery processing based on the data path level where the anomaly occurs, specifically including: Step D1: Obtain abnormal information generated during USB image data transmission and processing, and determine the abnormal type and source based on the abnormal information.

[0150] In some embodiments, the abnormal information may include one or more of the following: transmission error events generated by the xHCI host controller, link status information generated by the PCIe root complex, error interruption generated by AXI CDMA, abnormal status generated by the image processing module, usage status of the image receiving buffer, and network abnormal status generated by the network communication module.

[0151] The processing system can respond to the interrupt request output by the interrupt aggregation module, read the status register, event loop, error register or buffer status information of the corresponding functional module, and determine the current exception corresponding to the USB imaging device, USB host controller, PCIe link, AXI data path or image processing module based on the reading result.

[0152] The exception type can include one or more of the following: USB endpoint transmission error, PCIe link exception, DMA transfer timeout or read / write error, image receive buffer overflow, incomplete image frame, and image processing exception.

[0153] Step D2: If the anomaly is determined to correspond to the data stream of the USB image device or the USB endpoint transmission, stop or reconfigure the corresponding USB image data stream and resubmit the corresponding transmission description information.

[0154] For example, when the processing system determines, based on the xHCI event loop, that a transmission error has occurred at the target USB endpoint, or if the corresponding transmission completion event is not obtained within a preset time range, it can stop the image data stream corresponding to the USB endpoint, clear the error state of the USB endpoint, and reconfigure the image data stream parameters or resubmit the transmission description information pointing to the image receive buffer. When the USB imaging device is a D435 imaging device, the processing system can stop the corresponding depth image stream, color image stream, or infrared image stream through libusb, librealsense, or equivalent control procedures, and restart the image data stream after the USB endpoint state is restored.

[0155] Step D3: If the anomaly is not eliminated by recovering the USB image data stream, perform recovery processing on the xHCI host controller that provides USB host functionality or the corresponding PCIe-USB host controller.

[0156] In some embodiments, the processing system can stop task submission for the corresponding transport ring, clear abnormal states in the event ring and transport ring, and reinitialize the corresponding USB port, transport endpoint, or xHCI host controller. After recovery, the USB imaging device enumeration, interface selection, and endpoint configuration can be re-executed, and the transport ring can be re-established.

[0157] Step D4: If the anomaly is determined to correspond to a PCIe link or AXI data path, perform recovery processing on the corresponding PCIe downstream port or AXI data path.

[0158] In some embodiments, when the PCIe root complex detects a link disconnection, link training failure, or transaction access anomaly in the corresponding PCIe-USB host controller, the processing system may stop submitting new transport description information to the corresponding device, reclaim the image receive buffer in the abnormal state, and perform link retraining, port reset, or device re-enumeration on the corresponding PCIe downstream port.

[0159] When the exception corresponds to an AXI CDMA timeout, AXI read / write error, or AXI data path access exception, the processing system can stop the current data transfer task, clear the error status of the AXI CDMA, and perform a reset on the AXI CDMA or the corresponding AXI data path according to the exception range. After the recovery is completed, the source address, destination address, and transmission length can be reconfigured, and the data transfer task can be restarted.

[0160] Step D5: If the exception is determined to correspond to the image receiving buffer or the image processing module, release the exception buffer or perform recovery processing on the image processing module.

[0161] In some embodiments, when the image receiving buffer overflows, the frame data is incomplete, or the buffer state is inconsistent with the transmission completion event, the processing system can mark the corresponding buffer as invalid, stop providing it to the image processing module, and add it back to the set of available buffers after clearing the invalid data.

[0162] When an exception occurs in the image processing module during the reading or processing of a complete frame, the processing system can stop sending new image frames to the image processing module, read the exception status of the image processing module, and perform status clearing, register reconfiguration, or module reset on the image processing module. After recovery, the system can re-provide the image processing module with the physical address of the image receive buffer, data length, frame number, and format parameters to resume subsequent image processing.

[0163] In some embodiments, the processing system can determine the initial recovery level based on the source of the anomaly and prioritize performing partial recovery on the functional modules directly corresponding to the source of the anomaly. If the same anomaly is still detected after partial recovery or data transmission is not restored, the recovery scope can be expanded to perform further recovery on the upstream control module or downstream data path of the functional module. The processing system can only expand the reset scope if normal data transmission cannot be restored after performing partial recovery on the level where the anomaly is located, or if multiple data paths are detected to be in an abnormal state simultaneously. This avoids a single module anomaly directly causing the entire SoC FPGA architecture platform system to power down and restart.

[0164] In the solution provided in this embodiment, through steps D1 to D5, abnormal information generated during USB image data transmission and processing is obtained, the abnormality type and the data path level where the abnormality occurs are determined, and local recovery is preferentially performed on the corresponding objects in the USB image data stream, xHCI host controller, PCIe downstream port, AXI data path, or image processing module. This limits the scope of abnormality handling to the functional level where the fault occurred. Therefore, the impact of local transmission errors on other normal data paths can be reduced, avoiding the need for a complete board power-down restart for single endpoint, link, DMA, or image processing abnormalities. This shortens the data path recovery time and improves the continuous operation capability and abnormality recovery reliability of the multi-interface image collaborative transmission system.

[0165] In summary, the USB image data collaborative transmission method provided in this application determines an image receiving buffer shared by the processing system and programmable logic, and configures the address information of the image receiving buffer in the transmission description information, enabling the USB host controller to directly write the received USB image data into the image receiving buffer accessible by the programmable logic using PCIe memory write transactions; the PCIe root complex converts the PCIe memory write transactions into on-chip bus write transactions, allowing external USB image data to enter the shared storage space inside the FPGA or SoC; after the USB image data transmission is completed, the processing system controls the programmable logic to read and process the USB image data from the same image receiving buffer. This improves the controllability of the USB image data write address, ensuring that the write target of the USB image data is consistent with the read source of the programmable logic. It reduces intermediate buffering, address translation, and duplicate data copying between the USB receiving side and the programmable logic processing side, thereby reducing the data transfer burden and storage access overhead borne by the processing system. At the same time, it shortens the data flow path of USB image data from reception to processing, reduces end-to-end transmission latency, and coordinates the read processing of the programmable logic with the completion status of USB image data transmission. This improves the collaborative efficiency, real-time performance, and orderly data handover between USB image data transmission and programmable logic processing.

[0166] The steps of the various methods described above are only for clarity. In practice, they can be combined into one step or some steps can be split into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this application. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, but without changing the core design of the algorithm and process, are also within the scope of protection of this application.

[0167] Reference Figure 4 As shown, some embodiments of this application relate to a USB image data collaborative transmission system. This system can be built on a SoC FPGA architecture platform, which serves as the system control and data exchange center, used to implement USB image device access, USB image data reception, PCIe transaction conversion, shared storage, programmable logic processing, and processing result output. The system includes a processing system, programmable logic, memory, a PCIe root complex, and at least one USB host controller. The memory contains at least one image receiving buffer that is shared and accessed by the processing system and the programmable logic; Each USB host controller is connected to the PCIe root complex via a corresponding PCIe link and has a USB interface for connecting USB graphics devices. The processing system is used to determine the image receiving buffer corresponding to the USB host controller, configure the transmission description information based on the address information of the image receiving buffer, and submit the transmission description information to the USB host controller. Any USB host controller is used to receive USB image data output by a USB image device and initiate a PCIe memory write transaction carrying the USB image data and targeting the image receive buffer, based on the transmission description information. The PCIe root complex is used to convert PCIe memory write transactions into on-chip bus write transactions, so as to write USB image data output by different USB image devices into the corresponding image receive buffer. The processing system is also used to control programmable logic to read and process USB image data from the image receive buffer in response to a transfer completion event of any USB host controller.

[0168] Specifically, the processing system can run a Linux operating system, device drivers, libusb, librealsense, or other equivalent software stacks to perform control operations such as PCIe device enumeration, USB host controller initialization, USB image device configuration, image receive buffer management, interrupt response, and image data scheduling. The programmable logic can include one or more of the following modules: image processing module, custom AXI module, AXI CDMA, AXI interconnect module, interrupt convergence module, and clock reset module.

[0169] The memory can be DDR memory accessible to both the processing system and programmable logic. An image receive buffer is provided within the memory, which can be located within a reserved shared DDR storage area. This buffer is used to receive and temporarily store USB image data output from USB imaging devices. When the system includes multiple USB host controllers, one or more image receive buffers can be configured for each USB host controller.

[0170] The processing system can obtain the physical address or bus address corresponding to each image receiving buffer and manage the occupancy status, data validity status, and recycling status of the image receiving buffer. The image processing module in the programmable logic can access the image receiving buffer through the on-chip bus to read the USB image data to be processed. Thus, the processing system and the programmable logic can collaboratively complete the reception and processing of image data based on the same image receiving buffer.

[0171] The USB host controller can be implemented using a PCIe-USB bridge chip. The PCIe-USB bridge chip acts as a downstream PCIe device in the PCIe root complex on the PCIe side, provides xHCI host control functions on the USB side, and has a USB interface for connecting USB graphics devices. As an example, the PCIe-USB bridge chip can be implemented using the PCI11101 chip. It should be noted that the PCI11101 is only one example of a USB host controller, and this application does not limit the specific chip model of the USB host controller.

[0172] In a preferred embodiment, the PCIe root complex can be connected to at least two PCIe connection branches via at least two PCIe links. Each PCIe connection branch includes two cascaded PCIe-USB bridge chips, wherein the upstream PCIe-USB bridge chip is connected to the PCIe root complex via a corresponding PCIe link, and the downstream PCIe-USB bridge chip is connected to the upstream PCIe-USB bridge chip via a cascaded PCIe link. Each PCIe-USB bridge chip can provide an xHCI host controller and connect to a USB graphics device via a corresponding USB interface. Figure 4 This illustrates a specific implementation of the aforementioned PCIe connection structure. Figure 4 In the example shown, the PCIe root complex is connected to two PCIe connection branches via PCIe link 0 and PCIe link 1, respectively. Each PCIe connection branch includes two PCIe-USB bridge chips that are cascaded in sequence. Figure 4 The number of PCIe links, PCIe connection branches, PCIe-USB bridge chips, and USB graphics devices shown are merely examples and do not constitute a limitation on the number of the corresponding components.

[0173] The USB imaging device can be a D435 depth camera, outputting one or more of depth image data, color image data, and infrared image data via a USB interface. The USB imaging device can also be other depth cameras, color cameras, infrared cameras, or high-speed image acquisition devices that output image data via a USB interface.

[0174] The PCIe root complex is located within the SoC FPGA architecture platform. The processing system can access the PCIe root complex's configuration registers via the AXI control plane to initialize each PCIe link, configure address windows, enumerate devices for PCIe-USB bridge chips in each PCIe connection branch, and allocate address and interrupt resources. The xHCI driver running on the processing system can initialize the xHCI host controllers provided by each PCIe-USB bridge chip, and perform USB device enumeration, interface selection, and transmission endpoint configuration for USB graphics devices connected to each xHCI host controller. The AXI control plane can use the AXI-Lite bus for transmitting register configurations, status reads, and control commands.

[0175] The PCIe root complex can also connect to the AXI data plane via the TLP / AXI interface to convert PCIe memory write transactions initiated by each USB host controller into AXI write transactions, and write the USB image data output by different USB imaging devices into the corresponding image receive buffer according to the target address carried by each AXI write transaction.

[0176] During system operation, the processing system determines the image receive buffer in memory and configures the transfer description information based on the address information of the image receive buffer. The transfer description information indicates the receiving location and transfer parameters of the USB image data, and may include one or more of the following: image receive buffer address, transfer length, and USB image data transfer endpoint information. The processing system can submit the transfer description information to the transfer ring established by the USB host controller through the xHCI driver running on it, so that the USB host controller performs the USB image data receiving task according to the transfer description information.

[0177] After the USB imaging device outputs USB image data, the USB host controller can receive the USB image data through the USB interface, and its provided xHCI host controller schedules the corresponding USB transmission according to the transmission description information. Based on the address information of the image receive buffer, the USB host controller initiates a PCIe memory write transaction on the PCIe side with the image receive buffer as the write target. The PCIe transaction layer data packet corresponding to the PCIe memory write transaction carries the USB image data.

[0178] After receiving a PCIe memory write transaction initiated by the USB host controller, the PCIe root complex can parse the target address and USB image data carried in the PCIe memory write transaction and convert it into an on-chip bus write transaction. In some embodiments, the on-chip bus can be an AXI bus, and the on-chip bus write transaction can be an AXI write transaction. The PCIe root complex can provide the generated AXI write transaction to the AXI data plane. The AXI SmartConnect or other AXI interconnect modules in the AXI data plane perform address decoding according to the target address carried in the AXI write transaction and route the AXI write transaction to the memory access interface corresponding to the shared DDR to write the USB image data into the image receive buffer.

[0179] After completing one or more USB image data transfers, the USB host controller can generate a transfer completion event and notify the processing system via PCIe message signal interrupt, event loop, or other event reporting methods. In response to the transfer completion event, the processing system can determine the completed transfer description information and its corresponding image receive buffer, and determine the data status in the image receive buffer based on the frame synchronization information, data length, or metadata of the corresponding USB image data.

[0180] When the processing system determines that a complete frame has been formed in the image receiving buffer or that the preset processing granularity is met, it can provide corresponding access parameters to the programmable logic. Access parameters may include the starting address and data length of the image receiving buffer, and may also include one or more of the following: frame number, image width, image height, pixel format, data arrangement, or buffer identifier. The processing system can write the access parameters into the control register corresponding to the image processing module in the programmable logic and trigger the image processing module to read USB image data from the image receiving buffer.

[0181] The image processing module can access the shared DDR via the AXI data plane and read the corresponding USB image data according to the access parameters provided by the processing system. The image processing module can perform one or more of the following image processing operations on the read USB image data: frame data processing, pixel format conversion, image preprocessing, image algorithm processing, protocol encapsulation, or other image processing operations. The processing results generated by the image processing module can be written to the output buffer in the shared DDR or provided to the subsequent data transfer module and network communication module.

[0182] In some embodiments, AXI CDMA can also be configured in the programmable logic. The AXI CDMA control interface can connect to the AXI control plane, and its AXI main interface can connect to the AXI data plane, allowing the processing system to configure the source address, destination address, and transmission length via the AXI control plane. The AXI CDMA then uses the AXI data plane to move output data between the output buffer, network transmit buffer, or other AXI address spaces. The system may also include dual SGMII / GEM network modules for sending image processing results or other service data to external network devices or a host computer.

[0183] In some embodiments, the system may further include an interrupt aggregation module. The interrupt aggregation module can receive PCIe-related interrupts, AXI CDMA interrupts, image processing module interrupts, and interrupt signals generated by other functional modules, and provide a unified interrupt request to the processing system. In response to the interrupt request, the processing system can identify the interrupt source based on the corresponding interrupt status information and perform transmission completion processing, data transfer status updates, frame processing status updates, buffer reclamation, or exception recovery. The clock reset module can provide corresponding clock and reset signals to the PCIe root complex, AXI interconnect module, AXI CDMA, image processing module, and other functional modules to support system initialization and exception recovery.

[0184] For details on the specific implementation of the processing system configuration transmission description information, the USB host controller executing USB transmission and initiating PCIe memory write transactions, the PCIe root complex performing transaction conversion, and the processing system controlling the programmable logic to read image data based on the transmission completion event, please refer to the relevant descriptions in the foregoing method embodiments, which will not be repeated here.

[0185] Furthermore, referring to Figure 5 As shown, in some embodiments, the USB image data cooperative transmission system can be configured with a control plane and a data plane in a layered manner. Solid arrows in the figure represent data transmission paths, while dashed arrows represent control access paths. The data plane carries image data output by the USB image device, data carried by PCIe memory access transactions, AXI data blocks, and image processing results. The control plane carries control information such as register configuration, status queries, address window settings, device initialization commands, and transmission parameter configurations.

[0186] In one specific implementation, the xHCI host controller in the USB host controller performs USB transmission scheduling according to the transmission description information pre-configured by the processing system and receives USB image data output by the USB imaging device. After receiving the USB image data, the USB host controller can initiate a corresponding PCIe memory access transaction on the PCIe side according to the image receive buffer address and transmission length recorded in the transmission description information, and transmit the USB image data to the PCIe root complex through PCIe transaction layer packets. After receiving the PCIe memory write transaction, the PCIe root complex can complete the conversion of the PCIe transaction layer packets into an AXI write transaction and provide the AXI write transaction to the AXI data plane. The AXISmartConnect or other high-bandwidth interconnect structure in the AXI data plane can perform address decoding according to the target address carried by the AXI write transaction and route the AXI write transaction to the memory access interface corresponding to the shared DDR to write the USB image data into the image receive buffer in the shared DDR. The image processing module in the programmable logic can read and process USB image data from the image receiving buffer. The image processing results can be written to the output buffer in the shared DDR, and then moved by AXI CDMA to the network sending buffer, other shared storage areas, or other AXI address spaces. The image processing results can also be sent to external network devices or host computers through the SGMII / GEM network path, or continued to be read and forwarded by the processing system software and other AXI master devices.

[0187] In contrast to the data transmission process described above, the processing system can initiate control access via the AXI HPM master interface. The AXI-Lite control plane interconnect routes this control access to functional modules such as the PCIe root complex, AXI CDMA, and image processing module, allowing access to their register spaces. Through this control access, the processing system can configure the PCIe root complex's address window and downstream port parameters, configure the AXI CDMA's source address, destination address, and transmission length, configure the image receiving buffer address, data length, and image format for the image processing module, and perform initialization, status queries, or operational parameter configuration for peripheral control modules such as the GPIO control module, I2C controller, and MDIO management interface module. Specifically, the GPIO control module can be used to enable, reset, or detect the status of external devices; the I2C controller can be used to configure or read data from cameras, clock chips, or other I2C peripherals; and the MDIO management interface module can be used to configure the Ethernet PHY and obtain network link status.

[0188] The processing system can also enumerate and configure resources for the PCIe-USB host controller and its xHCI functions connected downstream of the PCIe root complex through the PCIe configuration space. When the USB host controller is implemented using the PCI11101 chip, the processing system can complete the device identification, base address register configuration, and address window setting of the PCI11101 chip, and initialize the xHCI host control functions it provides.

[0189] After completing the PCIe side configuration, the processing system can use the xHCI driver, libusb, librealsense, or other equivalent software stacks to perform device enumeration, interface selection, image data transfer endpoint configuration, and image data stream configuration for the USB imaging device. For example, when the USB imaging device is a D435 imaging device, the processing system can enable depth image stream, color image stream, or infrared image stream, and configure the corresponding image format, resolution, frame rate, and transfer buffer address.

[0190] The aforementioned control plane is used to establish and manage image data transmission tasks, and does not directly carry the large volume of valid image data output by the USB image device. USB image data is written to the image receive buffer along a data transmission path consisting of the USB host controller, PCIe memory write transactions, the PCIe root complex, the AXI data plane, and shared DDR. Control information such as device initialization, address window configuration, register access, and status queries are transmitted through the AXI-Lite control access path. Therefore, image data transmission does not occupy the AXI-Lite control channel, and control access does not directly carry or block large-bandwidth image data streams, thereby reducing the mutual interference between control access and large-volume image transmission, and improving the collaborative efficiency of USB image data reception, programmable logic processing, and processing result output.

[0191] Furthermore, referring to Figure 6 As shown, in some embodiments, the USB image data cooperative transmission system can hierarchically organize clock, reset, and interrupt to support PCIe data transmission, AXI access, image processing, network communication, and device initialization and fault recovery.

[0192] Regarding clocking, the PCIe reference clock can be provided to the PCIe root complex and PCIe links after differential clock buffering and high-speed transceiver clock processing; the processing system clock can be provided to the AXI-Lite control plane, AXI data plane, AXICDMA, and image processing modules; and the SGMII reference clock can be provided to the GEM / SGMII network modules. Different functional modules can operate in their respective clock domains, and data or control signals transmitted across clock domains can be transferred through synchronizers, asynchronous FIFOs, or other cross-clock domain processing circuits.

[0193] Regarding reset, the system can be equipped with a synchronous reset module. The synchronous reset module receives and processes the system reset signal, the external reset signal, and the lock signal used to characterize clock stability. After the corresponding clocks stabilize, it synchronously releases the AXI interconnect reset, PCIe root complex reset, image processing module reset, and AXI CDMA reset, respectively.

[0194] For the USB host controller downstream of the PCIe root complex, the system can also provide a PCIe device reset signal, such as the PERST# reset signal. After the PCIe reference clock stabilizes and the PCIe root complex completes its reset, the PCIe device reset signal is released to initiate PCIe link training and USB host controller initialization. Subsequently, the processing system can perform USB bus reset and USB image device enumeration via the xHCI driver, and start the corresponding image data stream after completing the device parameter configuration.

[0195] In some embodiments, the system can complete initialization in the following order: first, after the clock stabilizes, release the AXI interconnect and related on-chip module resets; then, release the PCIe root complex reset and USB host controller reset; after completing PCIe device enumeration and xHCI initialization, execute the USB bus reset and USB graphics device enumeration; and finally, start the USB graphics data transmission and image processing tasks. This avoids downstream devices starting prematurely before the upstream clock or data path has stabilized.

[0196] Regarding interrupts, an interrupt aggregation module can be set up to aggregate PCIe root complex link status interrupts or MSI interrupts, USB host controller xHCI event interrupts, USB image data transmission completion or abnormal events, AXI CDMA completion or error interrupts, image processing module frame completion or abnormal interrupts, and SGMII / GEM network interrupts, and output a unified interrupt request to the processing system.

[0197] Upon responding to an interrupt request, the processing system can read the status information of the interrupt aggregation module and the status registers or event loops of the corresponding functional modules to determine the specific source of the event. When USB image data transmission is complete, the system can update the data status of the image receiving buffer and initiate subsequent image processing. When image processing or data transfer is complete, the system can perform buffer reclamation, result transfer, or output. In the event of PCIe link, USB transmission, AXI CDMA, image processing, or network anomalies, the system can perform status clearing, task retry, module reset, link recovery, or device re-enumeration.

[0198] This embodiment improves the stability of PCIe link training, USB host controller initialization, and USB image device startup by providing corresponding clocks for different functional modules and releasing resets at each level synchronously in a preset order. By unifying different interrupts and abnormal events into the processing system, image data reception, image processing, data transfer, network output, and abnormal recovery can be interconnected, improving the system's continuous operation capability and abnormal recovery reliability.

[0199] In some embodiments, the system may further include a first network path and a second network path. Each network path may include a GEM network controller and an SGMII interface module connected to the GEM network controller. The GEM network controller can connect to a corresponding PCS / PMA module through its GMII interface, and the PCS / PMA module converts the GMII-side signals into SGMII differential signals to connect to external PHYs or network devices. The processing system can configure external PHYs and read network link status through the MDIO management interface. The first network path and the second network path can be used for service data transmission and maintenance management communication, respectively, or as the primary network path and the redundant network path, respectively.

[0200] In some embodiments, the system may further include a peripheral interface control module, which may include one or more of an I2C controller, an MDIO management interface module, a UART controller, and a GPIO control module. The processing system can configure registers for cameras, image sensors, serializers, or other I2C peripherals through the I2C controller, configure external Ethernet PHYs and read network link status through the MDIO management interface module, perform debugging or maintenance communication through the UART controller, and output reset, enable, or mode selection signals through the GPIO control module. For bidirectional management buses such as I2C and MDIO, the system can also include a bidirectional bus encapsulation module. The bidirectional bus encapsulation module can connect the system's internal input signals, output signals, and tri-state control signals to an IOBuf or equivalent bidirectional buffer structure to select whether to drive external bidirectional pins or acquire input signals from external bidirectional pins based on the tri-state control signals.

[0201] In some embodiments, the system may further include a top-level pin adapter module for connecting the PCIe, SGMII, I2C, MDIO, UART, GPIO, reference clock, and reset signals within the SoC FPGA architecture platform to the corresponding board-level input / output pins. The top-level pin adapter module may also provide a reserved image input interface to support the access of other types of image acquisition devices or subsequent expansion.

[0202] In summary, this embodiment, through the coordinated operation of the PCIe-USB host controller, PCIe root complex, AXI control plane, AXI data plane, shared DDR, image processing module, AXI CDMA, and network communication module, enables image data output from the USB imaging device to be written to the image receiving buffer shared by the processing system and programmable logic via the PCIe link and AXI data plane, and then read and processed by the programmable logic. Simultaneously, through interrupt aggregation, hierarchical reset, clock domain management, and peripheral interface adaptation, unified control is achieved for the image data reception, processing, transfer, output, and anomaly recovery processes. This reduces the amount of large-volume image copying and transfer involving the processing system, improving the collaborative efficiency, real-time performance, and operational reliability of multi-interface image data transmission and processing.

[0203] The aforementioned USB image data collaborative transmission system can be implemented as an independent image acquisition and processing device, or it can be integrated as a functional subsystem into vehicle control equipment, industrial inspection equipment, robot control equipment, edge computing equipment, servers, or other electronic devices with image data reception and processing requirements.

[0204] Furthermore, some embodiments of this application also provide an electronic device. The electronic device can be various forms of digital computer, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, etc. The electronic device can also be various forms of mobile devices, such as cellular phones, smartphones, wearable devices, and other similar computing devices.

[0205] The electronic device includes: one or more processors; and a memory storing computer program instructions, which, when executed, cause the processor to perform a USB image data collaborative transmission method as provided in any one or more of the above embodiments. Figure 7An exemplary structural diagram of the electronic device is disclosed. The electronic device includes one or more processors 1101, a memory 1102, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components are interconnected via different buses 1105 and can be mounted on a common motherboard or otherwise as required. The processors can process instructions executed within the electronic device, including instructions stored in or on memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to the interface). In some other embodiments, multiple processors and / or multiple buses 1105 can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple electronic devices can be connected, each providing some of the necessary operations. The components, their connections and relationships, and their functions shown herein are merely examples and are not intended to limit the implementation of the present application described and / or claimed herein.

[0206] The electronic device may further include an input device 1103 and an output device 1104. The processor 1101, memory 1102, input device 1103, and output device 1104 may be connected via a bus 1105 or other means. Figure 7 Taking the connection between China and Israel via bus 1105 as an example.

[0207] Input device 1103 can receive input numerical or character information, and generate key signal inputs related to user settings and function control of the electronic device, such as a touch screen, keypad, mouse, trackpad, touchpad, joystick, one or more mouse buttons, trackball, joystick, etc. Output device 1104 may include a display device, auxiliary lighting device (e.g., LED), and haptic feedback device (e.g., vibration motor). The display device may include, but is not limited to, a liquid crystal display, a light-emitting diode display, and a plasma display. In some embodiments, the display device may be a touch screen.

[0208] To provide interaction with the user, the electronic device can be a computer. The computer has: a display device (e.g., a cathode ray tube or LCD monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback); and input from the user can be received in any form (e.g., voice input or tactile input).

[0209] In this embodiment, a computer-readable medium stores a computer program / instructions, which, when executed by a processor, implement a USB image data collaborative transmission method provided in any one or more of the above embodiments. The computer-readable medium may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more computer-readable instructions.

[0210] The memory 1102 can serve as a non-transitory computer-readable storage medium, used to store non-transitory software programs, non-transitory computer-executable programs, and modules. The processor 1101 executes various functional applications and data processing of the server by running the non-transitory software programs, instructions, and modules stored in the memory 1102, thereby implementing the program instructions / modules corresponding to the methods provided in any one or more of the embodiments described above in this application.

[0211] The memory 1102 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the electronic device. Furthermore, the memory 1102 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 1102 may optionally include memory remotely located relative to the processor 1101, and these remote memories can be connected to the electronic device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0212] It should be noted that the computer-readable medium described in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. Computer-readable media can be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, electrical connections having one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory, optical fibers, portable compact disk read-only memory, optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0213] Computer-readable media include both permanent and non-permanent, removable and non-removable media, which can store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory, static random access memory, dynamic random access memory, other types of random access memory, read-only memory, electrically erasable programmable read-only memory, flash memory or other memory technologies, read-only optical discs, digital versatile optical discs or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0214] The flowcharts or block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of devices, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-specific system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0215] The scope of this application is defined by the appended claims rather than the foregoing description, and is therefore intended to encompass all variations falling within the meaning and scope of equivalents of the claims. No reference numerals in the claims should be construed as limiting the scope of the claims. Furthermore, it is clear that the word "comprising" does not exclude other elements or steps, and the singular does not exclude the plural. Terms such as "first," "second," etc., are used only to distinguish descriptions and do not indicate any particular order, nor should they be construed as indicating or implying relative importance.

[0216] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily made by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims, and the above embodiments should be regarded as exemplary and non-limiting.

Claims

1. A USB image data collaborative transmission method, characterized in that, include: The image receiving buffer shared by the processing system and the programmable logic is determined, the transmission description information is configured based on the address information of the image receiving buffer, and the transmission description information is submitted to the USB host controller. The USB host controller receives USB image data output from the USB image device and initiates a PCIe memory write transaction carrying the USB image data and targeting the image receive buffer, based on the transmission description information. The PCIe memory write transaction is converted into an on-chip bus write transaction through the PCIe root complex in order to write the USB image data into the image receive buffer. In response to the transfer completion event of the USB host controller, the processing system controls the programmable logic to read and process the USB image data from the image receive buffer.

2. The USB image data collaborative transmission method according to claim 1, characterized in that, The USB host controller is an xHCI host controller; Before submitting the transmission description information to the USB host controller, the xHCI host controller is initialized by the xHCI driver running on the processing system, the USB image device is enumerated, the interface is selected and the image data transmission endpoint is configured, and a transmission loop is established for the image data transmission endpoint. The step of configuring transmission description information based on the address information of the image receiving buffer and submitting the transmission description information to the USB host controller includes: Configure the address information of the image receiving buffer in the transmission descriptor for the image data transmission endpoint; The configured transport descriptor is submitted to the transport ring so that the xHCI host controller can execute the USB image data transmission corresponding to the image data transmission endpoint according to the transport descriptor.

3. The USB image data collaborative transmission method according to claim 2, characterized in that, The step of converting the PCIe memory write transaction into an on-chip bus write transaction via the PCIe root complex to write the USB image data into the image receive buffer includes: The PCIe root complex receives PCIe memory write transactions initiated by the xHCI host controller. Convert the PCIe transaction layer data packet corresponding to the PCIe memory write transaction into an AXI write transaction; The AXI write transaction is routed to the image receive buffer via the AXI data plane to write the USB image data into the image receive buffer.

4. The USB image data collaborative transmission method according to claim 2, characterized in that, The step of controlling the programmable logic to read and process the USB image data from the image receive buffer in response to the transmission completion event of the USB host controller includes: After the xHCI host controller completes one or more USB transmissions corresponding to the transmission description information, it generates a transmission completion event and reports an interrupt to the PCIe root complex to notify the processing system to obtain the transmission completion event. The processing system determines, based on the transmission completion event, whether the USB image data written in the image receiving buffer is an image payload, a frame fragment, or a complete frame. If it is determined that the written USB image data is a complete frame, the processing system writes the physical address of the image receiving buffer, as well as the data length, frame number, and format parameters of the complete frame, into the register of the image processing module set in the programmable logic, so as to trigger the image processing module to read the complete frame. The image processing module initiates an AXI read transaction based on the physical address and the data length, reads the complete frame from the image receiving buffer, and processes the complete frame according to the format parameters.

5. The USB image data collaborative transmission method according to claim 4, characterized in that, The programmable logic includes an AXI CDMA, the AXI CDMA's control interface is connected to the AXI control plane, and the AXI CDMA's main interface is connected to the AXI data plane, which is logically separate from the AXI control plane, via AXI SmartConnect; the method further includes: The image processing module writes the output data generated from processing the complete frame into the output buffer. The processing system, via the AXI control plane, configures the starting address of the output data in the output buffer as the source address of the AXI CDMA, configures the starting address of the destination storage area used to receive the output data as the destination address of the AXI CDMA, and configures the data length of the output data as the transmission length of the AXI CDMA. The AXI CDMA uses the source address, the destination address, and the transmission length to move the output data from the output buffer to the destination storage area via the AXI data plane. When the output data transfer is completed or an abnormality occurs during the transfer process, an interrupt is generated by the AXI CDMA, and the processing system responds to the interrupt to obtain the status information of the AXI CDMA in order to determine the transfer status of the output data.

6. The USB image data collaborative transmission method according to claim 4, characterized in that, The image receiving buffer uses contiguous physical memory, reserved shared memory, or DMA-coherent memory; the processing system records the correspondence between the virtual address and physical address of the image receiving buffer; the method further includes: If the image receiving buffer does not have DMA consistency, before triggering the image processing module to read the complete frame, the processing system performs cache synchronization processing on the address range in the image receiving buffer corresponding to the complete frame. After the image processing module completes the processing of the complete frame and generates a frame completion event, the processing system responds to the frame completion event by reclaiming the image receiving buffer. Based on the correspondence between the virtual address and the physical address, the physical address of the recycled image receiving buffer is configured in the next transfer descriptor, and the next transfer descriptor is submitted to the transfer ring so that the xHCI host controller can write subsequent USB image data into the image receiving buffer.

7. The USB image data collaborative transmission method according to claim 2, characterized in that, The PCIe root complex is connected to at least two PCIe connection branches via at least two PCIe links. Each PCIe connection branch includes two cascaded PCIe-USB bridge chips. The upstream PCIe-USB bridge chip is connected to the PCIe root complex via a corresponding PCIe link. Each PCIe-USB bridge chip provides an xHCI host controller and is connected to a corresponding USB graphics device via a USB interface. The image receiving buffer includes multiple image receiving buffers corresponding to each of the xHCI host controllers; The method further includes: The PCIe root complex is used to establish each PCIe link and the cascaded PCIe links within each PCIe connection branch, and to complete the device enumeration and resource configuration of each PCIe-USB bridge chip. Based on the address information of each image receiving buffer, transmission description information is configured for the corresponding xHCI host controller; The xHCI host controller receives USB image data output from the corresponding USB image device and initiates a PCIe memory write transaction with the corresponding image receive buffer as the write target according to the transmission description information. In response to a transmission completion event of any of the xHCI host controllers, the processing system determines the corresponding image receive buffer and controls the programmable logic to read the USB image data from the image receive buffer.

8. A USB image data collaborative transmission system, characterized in that, It includes a processing system, programmable logic, memory, a PCIe root complex, and at least one USB host controller; The memory is provided with at least one image receiving buffer that is shared and accessed by the processing system and the programmable logic; Each of the USB host controllers is connected to the PCIe root complex via a corresponding PCIe link and has a USB interface for connecting USB imaging devices. The processing system is used to determine the image receiving buffer corresponding to the USB host controller, configure transmission description information based on the address information of the image receiving buffer, and submit the transmission description information to the USB host controller. Any of the USB host controllers is configured to receive USB image data output by the USB image device, and initiate a PCIe memory write transaction carrying the USB image data and targeting the image receive buffer, based on the transmission description information. The PCIe root complex is used to convert the PCIe memory write transaction into an on-chip bus write transaction, so as to write the USB image data output by different USB image devices into the corresponding image receiving buffer. The processing system is also configured to, in response to a transmission completion event of any of the USB host controllers, control the programmable logic to read from the image receive buffer and process the USB image data.

9. An electronic device, characterized in that, The electronic device includes: One or more processors; and a memory storing computer program instructions that, when executed, cause the processors to perform the USB image data cooperative transmission method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program and / or instructions stored thereon, characterized in that, When the computer program and / or instructions are executed by the processor, they implement the USB image data collaborative transmission method as described in any one of claims 1-7.

Images