A bridge method and system based on FPGA

Abstract

The application provides a bridge method and system based on FPGA, the bridge method is applied to a bridge system, the bridge system comprises a master SoC and an FPGA bridge device; the FPGA bridge device enumerates a camera through a USB host controller, acquires and caches a UVC capability mirror; a bus device controller of the FPGA bridge device is registered as a function device on a target bus; after the master SoC discovers and loads a bridge driver to read the UVC capability mirror, the master SoC registers a standard video device node to a V4L2 framework, so that the node capability is consistent with a real camera; the master SoC controls the camera to collect video based on the standard video device node, a control path converter is used to realize bidirectional protocol conversion of control signaling, and a data path converter is used to realize uplink format conversion of a video stream; so that a USB UVC camera can be connected through a non-USB interface, and a kernel data path section is short.

Description

Technical Field

[0001] This application relates to the field of information processing technology, and more specifically, to a bridging method and system based on FPGA. Background Technology

[0002] In the hardware design of portable smart terminals (such as industrial tablets, handheld detectors, vehicle terminals, medical imaging terminals, AI edge boxes, etc.), the peripheral interface resources of the main control SoC chip are prioritized. The USBHost controller port of the main control SoC is often occupied by other functional modules (such as 4G / 5G modules, debugging interfaces, OTG, external storage, etc.), while the system still needs to connect a USB UVC camera as an image acquisition device. At this time, the main control SoC only has non-USB interfaces such as the SDIO interface available, but the SDIO interface cannot be directly connected to a USB camera, and the application layer cannot access such cameras through the standard V4L2 interface. Summary of the Invention

[0003] In view of this, the purpose of this application is to provide a bridging method and system based on FPGA, which can connect a USB UVC camera through a non-USB interface, and has a short kernel data path, low CPU usage, and low end-to-end latency.

[0004] This application provides an FPGA-based bridging method applied to a bridging system. The bridging system includes a main control SoC and an FPGA bridging device. The FPGA bridging device includes a USB host controller, a bus device controller, a control path converter, and a data path converter. The FPGA bridging device is connected to a camera and the target bus interface of the main control SoC, respectively. The bridging method includes: The FPGA bridging device enumerates the cameras through the USB host controller, obtains and caches the UVC capability image of the cameras; the bus device controller of the FPGA bridging device registers as a target bus functional device on the target bus; The bus subsystem of the main control SoC enumerates the target bus, discovers the target bus functional devices, and loads the corresponding bridge driver; the bridge driver reads the UVC capability image from the FPGA bridge device through the target bus; The bridging driver registers standard video device nodes with the V4L2 framework based on the UVC capability mirror, so that the video capabilities presented by the registered standard video device nodes are consistent with the actual capabilities of the camera. The main control SoC controls the camera to perform video acquisition based on the standard video device node; wherein, the control path converter of the FPGA bridging device realizes bidirectional protocol conversion of control signaling during the acquisition process, and the data path converter realizes uplink format conversion of the video stream.

[0005] In some embodiments, in the FPGA-based bridging method, the bridging driver registers standard video device nodes with the V4L2 framework according to the UVC capability mirror, including: The bridging driver extracts the video capability parameters of the camera from the UVC capability image; the UVC capability image includes: the camera's video control descriptor, video stream descriptor, a list of supported pixel formats, a list of resolutions corresponding to each pixel format, a list of frame rates corresponding to each resolution, a list of controllable controls, and a value range for each controllable control. The bridging driver fills the extracted video capability parameters into the capability query callback of the standard video device node, so that the video capabilities presented by the standard video device node are consistent with the actual capabilities of the camera.

[0006] In some embodiments, the FPGA-based bridging method, wherein the main control SoC controls the camera to perform video acquisition based on the standard video device node, includes: The application on the main control SoC initiates video capture through the standard video device node; The bridge driver responds to the video acquisition by sending a stream start command to the FPGA bridge device via the target bus; The control path converter of the FPGA bridging device converts the stream start instruction into a UVC start control request executable by the USB host controller and sends it to the camera to start the video stream output of the camera; The data path converter of the FPGA bridging device responds to the video stream output of the camera by converting the UVC video data received by the USB host controller from the camera into uplink data frames that conform to the target bus transmission format, and then sending them to the main control SoC via the bus device controller.

[0007] In some embodiments, the FPGA-based bridging method, wherein converting the UVC video data received by the USB host controller from the camera into uplink data frames conforming to the target bus transmission format, includes: The data path converter retains the original UVC payload head of the camera; The data path converter adds a bridging frame header to the outer layer of the original UVC payload header and encapsulates it into the uplink data frame.

[0008] In some embodiments, the FPGA-based bridging method further includes: The bridging driver strips the bridging frame header of the uplink data frame and extracts the video frame including the original UVC payload header; The bridging driver places the extracted video frames into the vb2 queue of the main control SoC for the application to read.

[0009] In some embodiments, the FPGA-based bridging method further includes: The control path converter maps the UVC control request six-tuple in the downlink control instruction to the register group inside the control path converter; wherein, the register group includes at least a command word, a parameter FIFO, a status word, and a completion interrupt bit; the UVC control request six-tuple includes a request type, a request number, a parameter value, an index, a data length, and a data content; The control path converter sends the mapping results in the register group to the USB host controller, which generates the corresponding UVC control request and sends it to the camera.

[0010] In some embodiments, in the FPGA-based bridging method, the command word includes the sequence number assigned by the control path converter to the downlink control instruction; The completion interrupt bit is used so that after the downlink control command is sent to the camera, the control path converter sets the completion interrupt bit and sends a completion interrupt carrying the sequence number to the main control SoC via the bus device controller.

[0011] In some embodiments, in the FPGA-based bridging method, before the main control SoC controls the camera to perform video acquisition based on the standard video device node, the method further includes: The main control SoC sends a video format setting request through the standard video device node; The bridging driver maps the video format setting request to a pair of UVC standard VS_PROBE_CONTROL control requests and VS_COMMIT_CONTROL control requests, and sends them to the FPGA bridging device via the target bus; The control path converter of the FPGA bridging device converts the control request into a UVC control request that can be executed by the USB host controller and sends it to the camera, thus completing the format negotiation; The FPGA bridging device returns the negotiation result to the main control SoC via the target bus.

[0012] In some embodiments, the FPGA-based bridging method further includes the following steps during the process of the main control SoC controlling the camera to perform video acquisition based on the standard video device node: The application on the main control SoC sends control setting requests through the standard video device node; The bridge driver encapsulates the control setting request into a downlink control frame and sends it to the FPGA bridge device via the target bus; The control path converter of the FPGA bridging device maps the control setting parameters in the downlink control frame to a UVC control request that can be executed by the USB host controller, which is then sent to the camera for execution by the USB host controller.

[0013] In some embodiments, an FPGA-based bridging system is also provided, the bridging system including a host SoC and an FPGA bridging device, the FPGA bridging device including a USB host controller, a bus device controller, a control path converter, and a data path converter; the FPGA bridging device is respectively connected to the camera and the target bus interface of the host SoC; The FPGA bridging device is used to enumerate the camera through the USB host controller, obtain and cache the UVC capability image of the camera; the bus device controller of the FPGA bridging device is registered as a target bus functional device on the target bus; The main control SoC is used to enumerate the target bus through the bus subsystem, discover the target bus functional devices, and load the corresponding bridge driver; the bridge driver reads the UVC capability image from the FPGA bridge device through the target bus; the bridge driver registers standard video device nodes with the V4L2 framework according to the UVC capability image, so that the video capabilities presented by the registered standard video device nodes are consistent with the actual capabilities of the camera. The main control SoC is used to control the camera to perform video acquisition based on the standard video device node; The FPGA bridging device is used to realize bidirectional protocol conversion of control signaling during the acquisition process through the control path converter, and to realize uplink format conversion of video stream through the data path converter.

[0014] This application provides an FPGA-based bridging method and system. The bridging method is applied to a bridging system, which includes a main control SoC and an FPGA bridging device. The FPGA bridging device includes a USB host controller, a bus device controller, a control path converter, and a data path converter. The FPGA bridging device is connected to a camera and the target bus interface of the main control SoC. The FPGA bridging device enumerates the camera through the USB host controller, obtains and caches the UVC capability image of the camera. The bus device controller of the FPGA bridging device registers as a target bus functional device on the target bus. The bus subsystem of the main control SoC enumerates the target bus, discovers the target bus functional device, and loads the corresponding bridging driver. The bridging driver is connected through the... The target bus reads the UVC capability image from the FPGA bridging device; the bridging driver registers standard video device nodes with the V4L2 framework based on the UVC capability image, so that the video capabilities presented by the registered standard video device nodes are consistent with the actual capabilities of the camera; the main control SoC controls the camera to perform video acquisition based on the standard video device nodes; wherein, the control path converter of the FPGA bridging device realizes bidirectional protocol conversion of control signaling during the acquisition process, and the data path converter realizes uplink format conversion of the video stream; since the bridging driver directly registers the standard video device nodes with the V4L2 framework based on the UVC capability image, the video data reaches the user space on the host side only through SDIO interrupt → driver → vb2 queue, without going through the USB core stack and UVC. The video driver presents a standard camera node to user space with the shortest kernel data path, resulting in low CPU usage and low end-to-end latency. Applications and middleware only need to access the standard video device node to use the camera without any modifications. The UVC capability mirroring mechanism ensures that the node capability set is completely consistent with the real camera. At the same time, no virtual USB device is introduced, so there is no semantic confusion of multiple device nodes coexisting in user space. Attached Figure Description

[0015] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 A flowchart of the FPGA-based bridging method described in an embodiment of this application is shown; Figure 2 A schematic diagram of the overall structure of the bridging system described in an embodiment of this application is shown; Figure 3 A schematic diagram of the host-side software layered architecture described in an embodiment of this application is shown; Figure 4 A schematic diagram of the internal architecture of the FPGA bridge described in an embodiment of this application is shown; Figure 5 An end-to-end workflow diagram of the FPGA-based bridging method described in an embodiment of this application is shown. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0018] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0019] It should be noted that the term "comprising" will be used in the embodiments of this application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0020] In the hardware design of portable smart terminals (such as industrial tablets, handheld detectors, vehicle terminals, medical imaging terminals, AI edge boxes, etc.), the peripheral interface resources of the main control SoC chip are prioritized. The USB Host controller port of the main control SoC is often occupied by other functional modules (such as 4G / 5G modules, debugging interfaces, OTG, external storage, etc.), while the system still needs to connect a USB UVC camera as an image acquisition device. At this time, only the SDIO interface of the main control SoC is available.

[0021] The application layer accessing the camera wants to access the image source in a "standard camera" manner. In the Linux / Android ecosystem, the de facto interface for a standard camera is the V4L2 device node ` / dev / videoX`; most USB cameras are implemented using the USB Video Class (UVC) specification and are registered as V4L2 devices by the kernel's uvc video driver. Applications (including open-source middleware, video conferencing software, HAL camera services, etc.) only adapt to the V4L2 interface and are not concerned with the physical bus. However, the SDIO interface cannot directly connect to a USB camera, and the application layer cannot access this type of camera through the standard V4L2 interface.

[0022] Based on this, this application provides an FPGA-based bridging method and system. The bridging method is applied to a bridging system, which includes a main control SoC and an FPGA bridging device. The FPGA bridging device includes a USB host controller, a bus device controller, a control path converter, and a data path converter. The FPGA bridging device is connected to a camera and the target bus interface of the main control SoC. The FPGA bridging device enumerates the camera through the USB host controller, obtains and caches the UVC capability image of the camera. The bus device controller of the FPGA bridging device registers as a target bus functional device on the target bus. The bus subsystem of the main control SoC enumerates the target bus, discovers the target bus functional device, and loads the corresponding bridging driver. The bridging driver is implemented through... The target bus reads the UVC capability image from the FPGA bridging device; the bridging driver registers standard video device nodes with the V4L2 framework based on the UVC capability image, so that the video capabilities presented by the registered standard video device nodes are consistent with the actual capabilities of the camera; the main control SoC controls the camera to perform video acquisition based on the standard video device nodes; wherein, the control path converter of the FPGA bridging device realizes bidirectional protocol conversion of control signaling during the acquisition process, and the data path converter realizes uplink format conversion of the video stream; since the bridging driver directly registers the standard video device nodes with the V4L2 framework based on the UVC capability image, the video data reaches the user space on the host side only through SDIO interrupt → driver → vb2 queue, without going through the USB core stack and UVC. The video driver presents a standard camera node to user space with the shortest kernel data path, resulting in low CPU usage and low end-to-end latency. Applications and middleware only need to access the standard video device node to use the camera without any modifications. The UVC capability mirroring mechanism ensures that the node capability set is completely consistent with the real camera. At the same time, no virtual USB device is introduced, so there is no semantic confusion of multiple device nodes coexisting in user space.

[0023] Please refer to Figure 1 , Figure 1 A flowchart of the FPGA-based bridging method according to an embodiment of this application is shown. The bridging method is applied to a bridging system, which includes a main control SoC and an FPGA bridging device. The FPGA bridging device includes a USB host controller, a bus device controller, a control path converter, and a data path converter. The FPGA bridging device is connected to the camera and the target bus interface of the main control SoC, respectively. like Figure 1 As shown, the bridging method includes the following steps S101-S104: S101, The FPGA bridging device enumerates the camera through the USB host controller, obtains and caches the UVC capability image of the camera; the bus device controller of the FPGA bridging device registers as a target bus functional device on the target bus; S102. The bus subsystem of the main control SoC enumerates the target bus, discovers the target bus functional device, and loads the corresponding bridge driver; the bridge driver reads the UVC capability image from the FPGA bridge device through the target bus; S103. The bridging driver registers a standard video device node with the V4L2 framework according to the UVC capability mirror, so that the video capability presented by the registered standard video device node is consistent with the actual capability of the camera. S104. The main control SoC controls the camera to perform video acquisition based on the standard video device node; wherein, the control path converter of the FPGA bridging device realizes bidirectional protocol conversion of control signaling during the acquisition process, and the data path converter realizes uplink format conversion of the video stream.

[0024] In other words, the bridging system consists of two parts: a bridging driver on the main control SoC and an FPGA bridging device; the two are connected by a bus and communicate in a custom packaging format.

[0025] The buses include: SDIO bus: utilizing the idle SDIO interface of the main control SoC, which is a preferred implementation of this application embodiment, and the subsequent parts of the embodiment will be described using the SDIO bus as an example; eMMC bus: presenting the FPGA bridging device as a special eMMC device segment, with corresponding modifications made to the host side under the MMC subsystem; SPI bus: suitable for low-bandwidth scenarios (such as low-resolution, low-frame-rate applications), including QSPI or Octal SPI modes; UART serial bus: suitable for extremely low-bandwidth scenarios, serving as a protection boundary.

[0026] Please refer to Figure 2 , Figure 2A schematic diagram of the overall structure of the bridging system described in an embodiment of this application is shown; as follows: Figure 2 As shown, the bridging system provided in this application embodiment includes two parts: a bridging driver FPGA bridging device for the main control SoC. The main control SoC and the FPGA bridging device are connected via an SDIO bus, and the FPGA bridging device is externally connected to a USB UVC camera module.

[0027] The main control SoC side includes, from top to bottom, the application layer, V4L2 standard device nodes, bridge driver, MMC / SDIO Core, and SDIO Host Controller.

[0028] Please refer to Figure 3 , Figure 3 The diagram illustrates the layered architecture of the host-side software. The application layer includes Camera HAL (Camera Hardware Abstraction Layer), image processing libraries such as OpenCV, and video conferencing applications, used for initiating video capture, formatting, and control adjustments. The V4L2 standard device node is / dev / videoX, which is the standard camera access interface provided by the host SoC to the application layer. The bridge driver is directly registered under the MMC / SDIO Core and is responsible for SDIO bus communication with the FPGA bridge device. The MMC / SDIO Core is the core subsystem in the Linux kernel that manages the MMC / SD / SDIO bus, and the SDIO HostController is the SDIO host controller hardware on the host SoC.

[0029] like Figure 3 As shown, the main control SoC bridge driver consists of four software layers from bottom to top: A1. SDIO Function Driver: Registered as an SDIO Function driver in the MMC subsystem, it is attached to the SDIO Core. It is responsible for reading and writing the FPGA's function register space and data FIFO using CMD52 / CMD53; and for receiving SDIO interrupts (DAT1 line asynchronous interrupts) initiated by the FPGA, waking up the upper-layer receive task.

[0030] The SDIO function driver layer works directly under the MMC / SDIO subsystem framework without relying on the USB subsystem, providing a low-latency, high-throughput data channel for the upper layer, and is the underlying foundation for "shortest data path".

[0031] A2. Bridge Protocol Layer: Implements the parsing and encapsulation of "UVC-on-SDIO" frames. Uplink: Identifies the start / end of frames from the SDIO data stream, removes the bridge frame header, and retains the UVC Payload Header; Downlink: Packages V4L2 control requests (including PROBE / COMMIT, brightness / exposure / focus / white balance, and other Class-Specific Requests) into downlink control frames.

[0032] The bridging protocol layer carries the UVC protocol semantics losslessly on the SDIO bus. The uplink retains the timestamp information in the original UVC payload header without recalculation, ensuring the isochronism of the video stream. The downlink maintains the integrity of the UVC control semantics, so that all camera control operations of the application layer can be transmitted to the real camera.

[0033] A3. V4L2 Sub-Device Registration Layer (V4L2Provider): Registers the above SDIO functions with the V4L2 framework as a video_device, and implements callbacks such as v4l2_file_operations, vb2_queue_ops, and v4l2_ioctl_ops; in particular, VIDIOC_ENUM_FMT / G_FMT / S_FMT / TRY_FMT / REQBUFS / QBUF / DQBUF / STREAMON / STREAMOFF all use the vb2 queue and DMA-able buffer.

[0034] The V4L2 sub-device registration layer allows the application layer to access the camera through the standard V4L2 interface without any modification, achieving the beneficial effect of "completely transparent to the application layer and zero-intrusive access".

[0035] A4. UVC Capability Mirror Layer: During the initial device enumeration, the "UVC Capability Mirror" (i.e., the video stream endpoint descriptor summary of the real UVC camera: supported formats, resolution list, frame rate list, controllable control list and value range) is retrieved from the FPGA via the downlink channel and populated into the query callbacks such as enum_fmt / enum_frame_sizes / enum_frame_intervals / queryctrl of the V4L2 framework, so that the "capability set" seen by the upper layer application is completely consistent with the original UVC camera.

[0036] The UVC capability mirroring layer ensures that the video capabilities presented by standard video device nodes are completely consistent with those of a real USB UVC camera. All capability information queried by upper-layer applications through the V4L2 interface is equivalent to the actual capabilities of the camera, which is the key mechanism for achieving "zero differences at the application layer".

[0037] Please refer to Figure 4 , Figure 4 A schematic diagram of the internal architecture of the FPGA bridge is shown; the FPGA bridge device side includes an SDIODevice controller (i.e., a bus device controller), a control path converter, a data path converter, and a USB Host controller (i.e., a USB host controller).

[0038] In this embodiment of the application, the USB host controller is used to connect to an external camera, enumerate and configure the external camera, and transmit UVC format data with the camera; The bus device controller is used to connect to the host end, present the bridging device as a target bus functional device on the host end, and perform data transmission between the target bus and the host end; The control path converter is used to convert downlink control commands sent by the host via the target bus into UVC control requests executable by the USB host controller, and to convert uplink event information from the camera into a format recognizable by the host. The data path converter is used to convert the UVC video data received by the USB host controller from the camera into uplink data frames that conform to the target bus transmission format.

[0039] Specifically, the internal logic of the bridging FPGA consists of four parallel processing modules: B1. USB Host Controller + UVC Client: Completes enumeration, configuration, control transmission, and isochronous transmission of external USB UVC camera modules; extracts the Video Control / Video Streaming descriptors of the real camera and caches them as "capability images".

[0040] The USB host controller enables the FPGA to independently complete the full identification and control of a standard USB UVC camera without the host SoC participating in the USB enumeration process, thus allowing the host SoC to connect to a USB camera even without a USB Host port.

[0041] B2. SDIO Device Controller: Implements the CIA (Common I / O Area), Function 0 / Function N register space, CMD52 / CMD53 parsing, 4-bit data transfer on the DAT line, DAT1 interrupt, and Block-mode DMA as required by the SDIO specification. This design only requires one function (typically Function 1) to operate.

[0042] The SDIO device controller enables the FPGA to appear as a standard SDIO functional device on the host SoC side. It can be normally enumerated and recognized by the host SoC's MMC subsystem without modifying the MMC core layer code on the host SoC side, and has good platform compatibility.

[0043] B3. Control Path Translator: Maps downlink V4L2 control requests to Class-Specific Control Requests for UVC cameras using a set of registers (including command word, parameter FIFO, status word, and completion interrupt bit); reports uplink events (such as auto exposure convergence events) to the SoC.

[0044] The control path converter implements a lightweight mapping from V4L2 control semantics to UVC control semantics. With the help of rolling sequence numbers, it supports the main control SoC to issue multiple control requests in a pipelined manner. While preserving the integrity of UVC control semantics, it makes the control plane latency controllable and does not affect the real-time transmission of video data.

[0045] B4. Data Path Translator: Contains a set of Ping-Pong buffers at the video frame level; performs lightweight repackaging of UVC payload from USB isochronous endpoints—retaining the original UVC payload header (including BFH, PTS, SCR), and only appending a bridge frame header (including Function ID, frame sequence number, total number of bytes in this frame, and CRC) to the outermost layer, then writing it to the SDIO data FIFO and triggering an interrupt.

[0046] The data path converter uses a Ping-Pong double buffering mechanism to achieve pipelined parallel processing of "receiving USB data" and "sending SDIO data", eliminating inter-frame waiting time and avoiding frame loss. At the same time, it transmits the timestamp in the original UVC payload header without recalculating it on the host side, ensuring the isochronism and synchronization accuracy of the video stream, making the video frames obtained by the host side completely equivalent to those when directly connected to a USB camera.

[0047] In steps S101-S103, the bridging FPGA is powered on and enumerated to complete the registration of the standard video device node.

[0048] In step S101, the FPGA bridging device enumerates the camera through the USB host controller, obtains and caches the UVC capability image of the camera; the bus device controller of the FPGA bridging device registers as a target bus functional device on the target bus; After the FPGA bridging device is powered on, its internal USB host controller acts as a USB host and performs a standard enumeration process on the external USB UVC camera. Specifically, it reads the device descriptor, allocates a USB address, and obtains the video control descriptor and video stream descriptor to acquire all the capability information supported by the camera. The complete capability information includes pixel format, resolution, frame rate, and adjustable parameters such as brightness, exposure, and focus, as well as their value ranges. All capability information constitutes the "UVC capability image" of the camera, which is cached in the internal storage unit of the FPGA.

[0049] Meanwhile, the FPGA's bus device controller registers itself as a standard functional device on the target bus.

[0050] In step S102, the bus subsystem of the main control SoC enumerates the target bus, discovers the target bus functional device, and loads the corresponding bridge driver; the bridge driver reads the UVC capability image from the FPGA bridge device through the target bus.

[0051] When the main control SoC's bus subsystem (such as the MMC subsystem) enumerates the target bus, it discovers the functional devices registered by the FPGA and loads the bridge driver that matches the device. After the driver is loaded, it reads the previously cached UVC capability image from the FPGA's internal storage unit through the target bus to obtain the camera's complete capability information.

[0052] In step S103, the bridging driver registers a standard video device node with the V4L2 framework according to the UVC capability mirror, so that the video capabilities presented by the registered standard video device node are consistent with the actual capabilities of the camera.

[0053] In some embodiments, the bridging driver registers standard video device nodes with the V4L2 framework based on the UVC capability mirror, including: The bridging driver extracts the video capability parameters of the camera from the UVC capability image; the UVC capability image includes: the camera's video control descriptor, video stream descriptor, a list of supported pixel formats, a list of resolutions corresponding to each pixel format, a list of frame rates corresponding to each resolution, a list of controllable controls, and a value range for each controllable control. The bridging driver fills the extracted video capability parameters into the capability query callback of the standard video device node, so that the video capabilities presented by the standard video device node are consistent with the actual capabilities of the camera.

[0054] After the video capability parameters are filled in, the video capabilities presented by the standard video device node are completely consistent with those of a real USB UVC camera. Subsequently, when upper-layer applications or middleware (such as Camera HAL, OpenCV, etc.) query the camera's capabilities through the V4L2 interface, the answers obtained are completely equivalent to those of a real camera—what formats are supported, what resolution is available, what frame rate is available, and what parameters can be adjusted—the application layer cannot perceive any difference.

[0055] After completing the registration of the standard video device node, in some embodiments, before the main control SoC controls the camera to perform video acquisition based on the standard video device node, the method further includes: The main control SoC sends a video format setting request through the standard video device node; The bridging driver maps the video format setting request to a pair of UVC standard VS_PROBE_CONTROL control requests and VS_COMMIT_CONTROL control requests, and sends them to the FPGA bridging device via the target bus; The control path converter of the FPGA bridging device converts the control request into a UVC control request that can be executed by the USB host controller and sends it to the camera, thus completing the format negotiation; The FPGA bridging device returns the negotiation result to the main control SoC via the target bus.

[0056] In other words, after registering the standard video device node, the application can configure the camera's parameters before video capture through this node. The configuration process includes a pair of standard control requests—VS_PROBE_CONTROL (probing negotiation) and VS_COMMIT_CONTROL (confirming lock). After receiving VS_PROBE_CONTROL, the camera evaluates whether the format is feasible. If it is not feasible, it returns the closest feasible format (such as 720p). After the application confirms, it completes the lock through VS_COMMIT_CONTROL, and the negotiation result is returned through the original path. This avoids the application directly setting a format that the camera does not support, which would lead to failure. By using a probing mechanism to obtain the actual format that the camera can provide before confirming, it ensures that video capture can start normally.

[0057] Meanwhile, through the protocol conversion function of the control path converter, the application only needs to issue a standard V4L2 format setting request, without having to worry about the differences in the underlying bus.

[0058] In some embodiments, in step S104, the main control SoC controls the camera to perform video acquisition based on the standard video device node; wherein, the control path converter of the FPGA bridging device realizes bidirectional protocol conversion of control signaling during the acquisition process, and the data path converter realizes uplink format conversion of the video stream.

[0059] In some embodiments, the main control SoC controls the camera to perform video acquisition based on the standard video device node, including: The application on the main control SoC initiates video capture through the standard video device node; The bridge driver responds to the video acquisition by sending a stream start command to the FPGA bridge device via the target bus; The control path converter of the FPGA bridging device converts the stream start instruction into a UVC start control request executable by the USB host controller and sends it to the camera to start the video stream output of the camera; The data path converter of the FPGA bridging device responds to the video stream output of the camera by converting the UVC video data received by the USB host controller from the camera into uplink data frames that conform to the target bus transmission format, and then sending them to the main control SoC via the bus device controller.

[0060] In this process, the stream start command is issued through the control path, and the video data is uploaded through the data path—the two paths are independent of each other, realizing the separation of control signaling and video data; the control path converter is responsible for converting the start command, and the data path converter is responsible for the continuous transfer of video data. The two perform their respective functions and work in parallel, ensuring that the control operation does not affect the real-time transmission of video data during the video acquisition process.

[0061] In some embodiments, a single low-to-mid-range FPGA simultaneously implements the USBHost controller and the SDIODevice controller, and works in combination with a set of Ping-Pong frame buffers and control / data dual-channel converters; this hardware architecture is strictly symmetrical with the SoC driver structure in terms of protocol encapsulation.

[0062] In some embodiments, converting the UVC video data received by the USB host controller from the camera into uplink data frames conforming to the target bus transmission format includes: The data path converter retains the original UVC payload head of the camera; The data path converter adds a bridging frame header to the outer layer of the original UVC payload header and encapsulates it into the uplink data frame.

[0063] Correspondingly, in some embodiments, the FPGA-based bridging method further includes: The bridging driver strips the bridging frame header of the uplink data frame and extracts the video frame including the original UVC payload header; The bridging driver places the extracted video frames into the vb2 queue of the main control SoC for the application to read.

[0064] In other words, the UVCPayloadHeader (including FID / EOF / PTS / SCR) on the real USB isochronous endpoint is preserved in the bridging frame as is. After the bridging frame header is stripped at the host end, it can be enqueued as standard V4L2 buffer content. This avoids secondary encapsulation errors of already compressed payloads such as H.264 / MJPEG and preserves the original timestamp semantics.

[0065] In some embodiments, the FPGA-based bridging method further includes: The control path converter maps the UVC control request six-tuple in the downlink control instruction to the register group inside the control path converter; wherein, the register group includes at least a command word, a parameter FIFO, a status word, and a completion interrupt bit; the UVC control request six-tuple includes a request type, a request number, a parameter value, an index, a data length, and a data content; The control path converter sends the mapping results in the register group to the USB host controller, which generates the corresponding UVC control request and sends it to the camera.

[0066] In some embodiments, the command word includes a sequence number assigned by the control path converter to the downlink control instruction; The completion interrupt bit is used so that after the downlink control command is sent to the camera, the control path converter sets the completion interrupt bit and sends a completion interrupt carrying the sequence number to the main control SoC via the bus device controller.

[0067] Here, the six-tuple of the UVC control request (bmRequestType, bRequest, wValue, wIndex, wLength, Data, i.e., request type, request number, parameter value, index, data length, and data content) is mapped to a pair of "command register group + data FIFO + completion interrupt" on SDIO, and pipelined delivery is supported in conjunction with rolling sequence numbers. While preserving the completeness of UVC control semantics, the control plane latency is controllable.

[0068] In some embodiments, the FPGA-based bridging method further includes the following steps during the process of the main control SoC controlling the camera to perform video acquisition based on the standard video device node: The application on the main control SoC sends control setting requests through the standard video device node; The bridge driver encapsulates the control setting request into a downlink control frame and sends it to the FPGA bridge device via the target bus; The control path converter of the FPGA bridging device maps the control setting parameters in the downlink control frame to a UVC control request that can be executed by the USB host controller, which is then sent to the camera for execution by the USB host controller.

[0069] During video stream acquisition, users may need to adjust the camera's image parameters at any time—for example, increasing brightness in low-light environments or triggering focus at different shooting distances. During parameter adjustment, the control path converter internally uses a set of registers (including command words, parameter FIFOs, status words, and completion interrupt bits) to quickly map control setting requests from V4L2 control semantics to UVC control semantics. The main control SoC can continue subsequent operations without waiting for the camera to complete the adjustment, and the control plane latency is controllable. Since the control path and data path are independent of each other within the FPGA, control operations such as brightness adjustment and focus triggering do not interfere with the continuous transmission of video data. While the parameter adjustment command is issued, the video stream acquisition is not affected in any way, and there will be no screen stuttering or frame drops, ensuring a smooth user experience.

[0070] Please refer to Figure 5 , Figure 5 This paper illustrates an end-to-end workflow diagram of the FPGA-based bridging method described in an embodiment of this application, as follows: Figure 5 As shown, the end-to-end workflow includes: Phase 1, Power-on enumeration: C1. When the bridged FPGA is powered on, it enumerates the external UVC camera as a USB Host, obtains and caches its VC / VS descriptor and the value range of the controllable control (forming a "UVC capability mirror").

[0071] The C2.SoC-side MMC subsystem enumerates the SDIO bus, discovers Function1, and matches the SDIO Function driver of this device; the driver reads the "UVC capability image" through the downlink channel.

[0072] C3. Based on this, the driver registers the / dev / videoX node with the V4L2 framework and populates all capability query callbacks such as enum_fmt and queryctrl.

[0073] Phase 2, Parameter Negotiation: C4. The application opens / dev / videoX and calls VIDIOC_S_FMT to set the resolution / pixel format; the driver maps it to a pair of UVCVS_PROBE_CONTROL / VS_COMMIT_CONTROLClass-SpecificRequest, which are then sent to the FPGA via the downlink channel.

[0074] The C5.FPGA control path converter (B3) performs the PROBE-COMMIT negotiation on a real UVC camera and reports the negotiation result.

[0075] Phase 3, Video Stream Acquisition: C6. The application calls VIDIOC_STREAMON, and the driver sends a stream start control frame to the FPGA.

[0076] The C7.FPGA continuously receives the UVC payload at the USB and other time endpoints. After the B4 module aligns it into a complete video frame, it writes it into the SDIO data FIFO in the uplink frame format and notifies the SoC via the DAT1 interrupt.

[0077] The C8 SoC's SDIO Function driver responds to interrupts, using CMD53 block mode DMA to read the entire frame data into the vb2 buffer; the bridging protocol layer (A2) strips the bridging frame header and puts the clean UVC frame payload containing the UVC Payload Header into the vb2 queue; the application can then call VIDIOC_DQBUF to obtain the video frame that is completely equivalent to the "real USB UVC camera" path.

[0078] Phase 4, controlling the downside: C9. During operation, the application's settings for controls such as brightness and focus (VIDIOC_S_CTRL) are all transmitted directly to the FPGA via the downlink control frame channel and then passed through to the actual camera.

[0079] Based on the same inventive concept, this application also provides an FPGA-based bridging device corresponding to the FPGA-based bridging method. Since the principle of the device in this application is similar to the FPGA-based bridging method described above in this application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0080] Please refer to Figure 2 , Figure 2 The diagram shows the overall structure of the FPGA-based bridging system according to an embodiment of this application. The FPGA-based bridging system includes a main control SoC and an FPGA bridging device. The FPGA bridging device includes a USB host controller, a bus device controller, a control path converter, and a data path converter. The FPGA bridging device is connected to the camera and the target bus interface of the main control SoC, respectively. The FPGA bridging device is used to enumerate the camera through the USB host controller, obtain and cache the UVC capability image of the camera; the bus device controller of the FPGA bridging device is registered as a target bus functional device on the target bus; The main control SoC is used to enumerate the target bus through the bus subsystem, discover the target bus functional devices, and load the corresponding bridge driver; the bridge driver reads the UVC capability image from the FPGA bridge device through the target bus; the bridge driver registers standard video device nodes with the V4L2 framework according to the UVC capability image, so that the video capabilities presented by the registered standard video device nodes are consistent with the actual capabilities of the camera. The main control SoC is used to control the camera to perform video acquisition based on the standard video device node; The FPGA bridging device is used to realize bidirectional protocol conversion of control signaling during the acquisition process through the control path converter, and to realize uplink format conversion of video stream through the data path converter.

[0081] In some embodiments, in the FPGA-based bridging device, The main control SOC, when the bridging driver registers standard video device nodes with the V4L2 framework according to the UVC capability mirror, is specifically used for: The bridging driver extracts the video capability parameters of the camera from the UVC capability image; the UVC capability image includes: the camera's video control descriptor, video stream descriptor, a list of supported pixel formats, a list of resolutions corresponding to each pixel format, a list of frame rates corresponding to each resolution, a list of controllable controls, and a value range for each controllable control. The bridging driver fills the extracted video capability parameters into the capability query callback of the standard video device node, so that the video capabilities presented by the standard video device node are consistent with the actual capabilities of the camera.

[0082] In some embodiments, in the FPGA-based bridging device, When the main control SoC controls the camera to perform video acquisition based on the standard video device node, it is specifically used for: The application on the main control SoC initiates video capture through the standard video device node; The bridge driver responds to the video acquisition by sending a stream start command to the FPGA bridge device via the target bus; The control path converter of the FPGA bridging device converts the stream start instruction into a UVC start control request executable by the USB host controller and sends it to the camera to start the video stream output of the camera; The data path converter of the FPGA bridging device responds to the video stream output of the camera by converting the UVC video data received by the USB host controller from the camera into uplink data frames that conform to the target bus transmission format, and then sending them to the main control SoC via the bus device controller.

[0083] In some embodiments, in the FPGA-based bridging device, The data path converter of the FPGA bridging device, when converting the UVC video data received by the USB host controller from the camera into uplink data frames conforming to the target bus transmission format, is specifically used for: The data path converter retains the original UVC payload head of the camera; The data path converter adds a bridging frame header to the outer layer of the original UVC payload header and encapsulates it into the uplink data frame.

[0084] In some embodiments, in the FPGA-based bridging device, When the main control SOC strips the bridging frame header of the uplink data frame through the bridging driver and extracts the video frame including the original UVC payload header, it is specifically used for: The bridging driver places the extracted video frames into the vb2 queue of the main control SoC for the application to read.

[0085] In some embodiments, in the FPGA-based bridging device, The control path converter is used to map the UVC control request six-tuple in the downlink control instruction to the register group inside the control path converter; wherein, the register group includes at least a command word, a parameter FIFO, a status word, and a completion interrupt bit; the UVC control request six-tuple includes a request type, a request number, a parameter value, an index, a data length, and a data content; The mapping result in the register group is sent to the USB host controller, which generates the corresponding UVC control request and sends it to the camera.

[0086] In some embodiments, in the FPGA-based bridging device, the command word includes the sequence number assigned by the control path converter to the downlink control instruction; The completion interrupt bit is used so that after the downlink control command is sent to the camera, the control path converter sets the completion interrupt bit and sends a completion interrupt carrying the sequence number to the main control SoC via the bus device controller.

[0087] In some embodiments, in the FPGA-based bridging device, the main control SoC is further configured to send a video format setting request through the standard video device node before controlling the camera to perform video acquisition based on the standard video device node; The bridging driver maps the video format setting request to a pair of UVC standard VS_PROBE_CONTROL control requests and VS_COMMIT_CONTROL control requests, and sends them to the FPGA bridging device via the target bus; The control path converter of the FPGA bridging device converts the control request into a UVC control request that can be executed by the USB host controller and sends it to the camera, thus completing the format negotiation; The FPGA bridging device returns the negotiation result to the main control SoC via the target bus.

[0088] In some embodiments, in the FPGA-based bridging device, the main control SoC, during the process of controlling the camera to perform video acquisition based on the standard video device node, is specifically used for: The application on the main control SoC sends control setting requests through the standard video device node; The bridge driver encapsulates the control setting request into a downlink control frame and sends it to the FPGA bridge device via the target bus; The control path converter of the FPGA bridging device is also used to map the control setting parameters in the downlink control frame into a UVC control request that can be executed by the USB host controller, which is then sent to the camera for execution by the USB host controller.

[0089] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0090] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0091] In addition, the functional units in the embodiments provided in this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0092] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0093] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0094] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The protection scope of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application. All should be covered within the protection scope of this application. Therefore, the protection scope of this application should be determined by the protection scope of the claims.

Claims

1. A bridging method based on FPGA, characterized in that, The bridging method is applied to a bridging system, which includes a main control SoC and an FPGA bridging device. The FPGA bridging device includes a USB host controller, a bus device controller, a control path converter, and a data path converter. The FPGA bridging device is connected to the camera and the target bus interface of the main control SoC respectively; The bridging method includes: The FPGA bridging device enumerates the cameras through the USB host controller, obtains and caches the UVC capability image of the cameras; the bus device controller of the FPGA bridging device registers as a target bus functional device on the target bus; The bus subsystem of the main control SoC enumerates the target bus, discovers the target bus functional devices, and loads the corresponding bridge driver; the bridge driver reads the UVC capability image from the FPGA bridge device through the target bus; The bridging driver registers standard video device nodes with the V4L2 framework based on the UVC capability mirror, so that the video capabilities presented by the registered standard video device nodes are consistent with the actual capabilities of the camera. The main control SoC controls the camera to perform video acquisition based on the standard video device node; wherein, the control path converter of the FPGA bridging device realizes bidirectional protocol conversion of control signaling during the acquisition process, and the data path converter realizes uplink format conversion of the video stream.

2. The FPGA-based bridging method according to claim 1, characterized in that, The bridging driver registers standard video device nodes with the V4L2 framework based on the UVC capability mirror, including: The bridging driver extracts the video capability parameters of the camera from the UVC capability image; the UVC capability image includes: the camera's video control descriptor, video stream descriptor, a list of supported pixel formats, a list of resolutions corresponding to each pixel format, a list of frame rates corresponding to each resolution, a list of controllable controls, and a value range for each controllable control. The bridging driver fills the extracted video capability parameters into the capability query callback of the standard video device node, so that the video capabilities presented by the standard video device node are consistent with the actual capabilities of the camera.

3. The FPGA-based bridging method according to claim 1, characterized in that, The main control SoC controls the camera to perform video acquisition based on the standard video device node, including: The application on the main control SoC initiates video capture through the standard video device node; The bridge driver responds to the video acquisition by sending a stream start command to the FPGA bridge device via the target bus; The control path converter of the FPGA bridging device converts the stream start instruction into a UVC start control request executable by the USB host controller and sends it to the camera to start the video stream output of the camera; The data path converter of the FPGA bridging device responds to the video stream output of the camera by converting the UVC video data received by the USB host controller from the camera into uplink data frames that conform to the target bus transmission format, and then sending them to the main control SoC via the bus device controller.

4. The FPGA-based bridging method according to claim 3, characterized in that, The step of converting the UVC video data received by the USB host controller from the camera into uplink data frames conforming to the target bus transmission format includes: The data path converter retains the original UVC payload head of the camera; The data path converter adds a bridging frame header to the outer layer of the original UVC payload header and encapsulates it into the uplink data frame.

5. The FPGA-based bridging method according to claim 4, characterized in that, The bridging method further includes: The bridging driver strips the bridging frame header of the uplink data frame and extracts the video frame including the original UVC payload header; The bridging driver places the extracted video frames into the vb2 queue of the main control SoC for the application to read.

6. The FPGA-based bridging method according to claim 1, characterized in that, The bridging method further includes: The control path converter maps the UVC control request six-tuple in the downlink control instruction to the register group inside the control path converter; wherein, the register group includes at least a command word, a parameter FIFO, a status word, and a completion interrupt bit; the UVC control request six-tuple includes a request type, a request number, a parameter value, an index, a data length, and a data content; The control path converter sends the mapping results in the register group to the USB host controller, which generates the corresponding UVC control request and sends it to the camera.

7. The FPGA-based bridging method according to claim 6, characterized in that, The command word includes the sequence number assigned by the control path converter to the downlink control command; The completion interrupt bit is used so that after the downlink control command is sent to the camera, the control path converter sets the completion interrupt bit and sends a completion interrupt carrying the sequence number to the main control SoC via the bus device controller.

8. The FPGA-based bridging method according to claim 1, characterized in that, Before the main control SoC controls the camera to perform video acquisition based on the standard video device node, the method further includes: The main control SoC sends a video format setting request through the standard video device node; The bridging driver maps the video format setting request to a pair of UVC standard VS_PROBE_CONTROL control requests and VS_COMMIT_CONTROL control requests, and sends them to the FPGA bridging device via the target bus; The control path converter of the FPGA bridging device converts the control request into a UVC control request that can be executed by the USB host controller and sends it to the camera, thus completing the format negotiation; The FPGA bridging device returns the negotiation result to the main control SoC via the target bus.

9. The FPGA-based bridging method according to claim 1, characterized in that, During the process of the main control SoC controlling the camera to perform video acquisition based on the standard video device node, the following are also included: The application on the main control SoC sends control setting requests through the standard video device node; The bridge driver encapsulates the control setting request into a downlink control frame and sends it to the FPGA bridge device via the target bus; The control path converter of the FPGA bridging device maps the control setting parameters in the downlink control frame to a UVC control request that can be executed by the USB host controller, which is then sent to the camera for execution by the USB host controller.

10. An FPGA-based bridging system, characterized in that, The bridging system includes a main control SoC and an FPGA bridging device. The FPGA bridging device includes a USB host controller, a bus device controller, a control path converter, and a data path converter. The FPGA bridging device is connected to the camera and the target bus interface of the main control SoC, respectively. The FPGA bridging device is used to enumerate the camera through the USB host controller, obtain and cache the UVC capability image of the camera; The bus device controller of the FPGA bridging device is registered as a target bus functional device on the target bus; The main control SoC is used to enumerate the target bus through the bus subsystem, discover the target bus functional devices, and load the corresponding bridge driver; the bridge driver reads the UVC capability image from the FPGA bridge device through the target bus; The bridging driver registers standard video device nodes with the V4L2 framework based on the UVC capability mirror, so that the video capabilities presented by the registered standard video device nodes are consistent with the actual capabilities of the camera. The main control SoC is used to control the camera to perform video acquisition based on the standard video device node; The FPGA bridging device is used to realize bidirectional protocol conversion of control signaling during the acquisition process through the control path converter, and to realize uplink format conversion of video stream through the data path converter.

Images