Video transmission method, system, electronic device and readable storage medium

Abstract

The application discloses a video transmission method and system, electronic equipment and readable storage medium, and relates to the technical field of video processing. The video transmission method is applied to a UVC device. The UVC device at least comprises a UVC driver, a USB controller, a memory pool and a buffer. The video transmission method comprises the following steps: in response to receiving original image data, preprocessing and encoding the original image data to obtain memory information, wherein the memory information at least comprises a first physical memory, a memory block handle and metadata; converting the memory block handle into a file descriptor; in response to receiving an enqueue request of the buffer, obtaining a memory address according to the file descriptor and the metadata, and packeting the first physical memory according to state information of the buffer in the enqueue request to obtain a second physical memory; and transmitting stream data in the second physical memory to a video terminal according to the memory address. The application solves the problem of high delay caused by CPU memory copying.

Description

Technical Field

[0001] This application relates to the field of video processing technology, and in particular to a video transmission method, system, electronic device, and computer-readable storage medium. Background Technology

[0002] With the rapid development of high-definition video conferencing, telemedicine, and industrial visual inspection, USB (Universal Serial Bus) video transmission has become the core channel for image data interaction between terminal devices and external devices. Various UVC (USB Video Class) cameras transmit video streams to the host in real time via the USB interface. The efficiency and latency of video transmission directly affect the smoothness of business processing and user experience.

[0003] Currently, most UVC video transmission solutions on the market adopt the traditional path of CPU (Central Processing Unit) memory copying. After the encoded frame data is output from the hardware encoder, it needs to be copied from the encoder output buffer to the UVC driver by the CPU, and finally transmitted by the USB controller. In this process, the CPU needs to participate in a complete frame data copy operation. However, this CPU memory copying solution has significant drawbacks when dealing with high-resolution, high-frame-rate video streams: for each frame data transmission, the CPU needs to move the entire frame data from the encoder output buffer to the UVC driver buffer. In high-bandwidth scenarios, this frequent CPU memory copying leads to a significant increase in CPU load and a substantial increase in end-to-end transmission latency, seriously affecting the real-time performance of video transmission. Especially in scenarios such as large-scale video surveillance and multi-channel concurrent live streaming, CPU resources are consumed in large quantities for data movement, the overall system performance drops sharply, and the user experience is severely affected.

[0004] Therefore, overcoming the high latency caused by CPU memory copying in existing technologies is a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0005] The main purpose of this application is to provide a video transmission method that aims to solve the technical problem of high latency caused by CPU memory copying.

[0006] To achieve the above objectives, this application proposes a video transmission method applied to a UVC device, wherein the UVC device includes at least a UVC driver, a USB controller, a memory pool, and a buffer, and the video transmission method includes: In response to receiving raw image data, the raw image data is preprocessed to obtain raw image frames; The original image frame is encoded to obtain memory information, and the memory information is stored in the memory pool. The memory information includes at least the first physical memory, memory block handles, and metadata. The memory block handle is converted into a file descriptor, and the file descriptor and the metadata are pushed to the UVC driver; In response to receiving the enqueue request of the buffer, the UVC driver obtains the memory address based on the file descriptor and the metadata, and packages the first physical memory according to the status information of the buffer in the enqueue request to obtain the second physical memory; A transport descriptor is generated based on the memory address, and the bitstream data in the second physical memory is transmitted to the video terminal according to the transport descriptor.

[0007] In one embodiment, the buffer includes at least a first buffer and a second buffer; before the step of preprocessing the original image data to obtain the original image frame, the method further includes: In response to receiving a format negotiation request, a pixel format is determined according to the format negotiation request, the pixel format including an encoded format and a raw format; If the pixel format is an encoded format, return to the step of preprocessing the original image data; When the pixel format is the raw format, the raw image data is acquired and preprocessed to obtain encoded frame data, which is located in the first buffer. The encoded frame data in the first buffer is copied to the second buffer, and the encoded frame data in the second buffer is pushed to the video terminal.

[0008] In one embodiment, after the step of determining the pixel format according to the format negotiation request in response to receiving the format negotiation request, the method further includes: In response to receiving a rate negotiation request, rate negotiation information and single-frame load value are determined based on the rate negotiation request, wherein the rate negotiation information includes at least link rate and endpoint parameters; The rate negotiation information is calculated based on preset calculation rules to obtain the basic load value, and the basic load value is compared with the single frame load value to obtain the single transmission capacity. Response data is constructed based on the single transmission capacity, and the response data is pushed to the video terminal.

[0009] In one embodiment, when the pixel format is an encoded format, after the step of preprocessing the original image data, the method further includes: The buffer capacity is determined based on the resolution of the original image frame; Allocate the capacity of the first physical memory according to the buffer capacity.

[0010] In one embodiment, the step of assembling the first physical memory according to the status information to obtain the second physical memory includes: A protocol header is generated based on the status information, which includes at least a frame identifier, buffer metadata, and queue error status. The protocol header is filled into the starting position of the first physical memory to obtain the second physical memory.

[0011] In one embodiment, after the step of transmitting the bitstream data in the second physical memory to the video terminal according to the transport descriptor, the method further includes: Obtain the index of the bitstream data and find the memory block handle corresponding to the index; The frame corresponding to the memory block handle is pushed to the memory pool.

[0012] In one embodiment, the video transmission method further includes: Allocate at least one buffer index for the file descriptor; The memory block handle is recorded in a preset tracking table, which contains at least the memory block handle and the correspondence between the buffer index and the memory block handle; If the UVC driver refuses to accept the enqueue request of the buffer, it looks up the memory block handle in the tracking table according to the buffer index, obtains the memory information, and pushes the memory information to the memory pool.

[0013] Furthermore, to achieve the above objectives, this application also proposes a video transmission system applied to a UVC device, wherein the UVC device includes at least a UVC driver, a USB controller, a memory pool, and a buffer, and the video transmission system includes: The video acquisition and processing module is used to receive raw image data and preprocess the raw image data to obtain raw image frames; The video encoding module is used to encode the original image frames to obtain memory information and store the memory information in the memory pool. The memory information includes at least a first physical memory, a memory block handle, and metadata. The zero-copy bridging module is used to convert the memory block handle into a file descriptor and push the file descriptor and the metadata to the UVC driver; The packet assembly module is used to receive the enqueue request of the buffer, obtain the memory address through the UVC driver according to the file descriptor and the metadata, and assemble the first physical memory according to the status information of the buffer in the enqueue request to obtain the second physical memory; The transmission module is used to generate a transmission descriptor based on the memory address, and to transmit the bitstream data in the second physical memory to the video terminal according to the transmission descriptor.

[0014] In addition, to achieve the above objectives, this application also proposes an electronic device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the video transmission method as described above.

[0015] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and which, when executed by a processor, implements the steps of the video transmission method described above.

[0016] This application proposes a video transmission method applied to a UVC device. The UVC device includes at least a UVC driver, a USB controller, a memory pool, and a buffer. The video transmission method includes: in response to receiving raw image data, preprocessing the raw image data to obtain raw image frames; encoding the raw image frames to obtain memory information, and storing the memory information in the memory pool, the memory information including at least a first physical memory, a memory block handle, and metadata; converting the memory block handle into a file descriptor, and pushing the file descriptor and the metadata to the UVC driver; in response to receiving an enqueue request from the buffer, obtaining a memory address through the UVC driver based on the file descriptor and the metadata, and assembling the first physical memory into packets based on the status information of the buffer in the enqueue request to obtain a second physical memory; generating a transmission descriptor based on the memory address, and transmitting the bitstream data in the second physical memory to a video terminal based on the transmission descriptor.

[0017] This application preprocesses and encodes the raw image data, storing the encoded bitstream directly in the physical memory of the memory pool, thus providing a data foundation and unique identifier for subsequent cross-driver sharing. By converting memory block handles into file descriptors and pushing them to the UVC driver, the UVC driver gains direct access to the same physical memory without CPU involvement in data transfer, achieving memory sharing between the encoder driver and the UVC driver. Upon receiving a buffer enqueue request, the physical memory address is obtained based on the file descriptor, and a packet assembly operation is performed on the first physical memory to obtain the second physical memory. This process does not require additional memory allocation or data copying; packet assembly is completed simply by modifying the memory content in place. A transfer descriptor is generated from the memory address, and the bitstream data with the protocol header is transferred to the video terminal in one go via DMA (Direct Memory Access), completely eliminating CPU involvement in data transfer and overcoming the high latency problem caused by CPU copying during video transmission. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating an embodiment of the video transmission method of this application. Figure 2 This is a schematic diagram of the system architecture provided in Embodiment 1 of the video transmission method of this application; Figure 3 This is a schematic diagram of the data transmission process of the video transmission method provided in Embodiment 1 of this application; Figure 4 This diagram illustrates the I / O (Input / Output) mode switching of the video transmission method provided in Embodiment 2 of this application. Figure 5 This is a schematic diagram of the link rate adaptive process of the video transmission method provided in Embodiment 2 of this application; Figure 6 This is a schematic diagram of the module structure of the video transmission system according to an embodiment of this application; Figure 7 This is a schematic diagram of the device structure of the hardware operating environment involved in the video transmission method in the embodiments of this application. Detailed Implementation

[0021] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0022] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0023] The main solution of this application embodiment is as follows: In response to receiving raw image data, the raw image data is preprocessed to obtain raw image frames; the raw image frames are encoded to obtain memory information, and the memory information is stored in a memory pool, the memory information including at least a first physical memory, a memory block handle, and metadata; the memory block handle is converted into a file descriptor, and the file descriptor and metadata are pushed to the UVC driver; in response to receiving an enqueue request from the buffer, the UVC driver obtains the memory address based on the file descriptor and metadata, and assembles the first physical memory into packets based on the buffer status information in the enqueue request to obtain a second physical memory; a transmission descriptor is generated based on the memory address, and the bitstream data in the second physical memory is transmitted to the video terminal based on the transmission descriptor.

[0024] In this embodiment, for ease of description, the following description uses a UVC device as the execution subject.

[0025] Because current technologies require the CPU to participate in a complete frame data copy operation during video transmission, this CPU memory copying scheme has significant drawbacks when dealing with high-resolution, high-frame-rate video streams: for each frame data transmission, the CPU needs to move the entire frame data from the encoder output buffer to the UVC driver buffer. In high-bandwidth scenarios, this frequent CPU memory copying leads to a significant increase in CPU load and a substantial increase in end-to-end transmission latency, severely impacting the real-time performance of video transmission.

[0026] This application provides a solution that preprocesses and encodes raw image data, storing the encoded bitstream directly in the physical memory of a memory pool, thus providing a data foundation and unique identifier for subsequent cross-driver sharing. By converting memory block handles into file descriptors and pushing them to the UVC driver, the UVC driver gains direct access to the same physical memory without CPU involvement in data transfer, achieving memory sharing between the encoder driver and the UVC driver. Upon receiving a buffer enqueue request, the physical memory address is obtained based on the file descriptor, and a packet assembly operation is performed on the first physical memory to obtain the second physical memory. This process does not require additional memory allocation or data copying; packet assembly is completed simply by modifying the memory content in-situ. A transfer descriptor is generated from the memory address, and the bitstream data with the protocol header is transferred to the video terminal in one go by DMA, completely eliminating CPU involvement in data transfer and overcoming the high latency problem caused by CPU copying during video transmission.

[0027] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device capable of performing the above functions. The following description uses a UVC device as an example to illustrate this embodiment and the subsequent embodiments.

[0028] Based on this, embodiments of this application provide a video transmission method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the video transmission method of this application.

[0029] In this embodiment, the video transmission method includes the following steps: Step S110: In response to receiving the raw image data, preprocess the raw image data to obtain the raw image frame; It should be noted that preprocessing refers to image signal processing and video subsystem processing of the raw image data acquired by the image sensor. This includes, but is not limited to, 3A (Auto Focus, Auto Exposure, Auto White Balance) processing, scaling, cropping, rotation, and color space conversion, to ensure that the data meets the input format requirements of the subsequent encoder. On embedded platforms, these processes are typically performed by dedicated hardware modules, and the CPU does not participate in pixel data processing.

[0030] For example, the image sensor outputs raw image data, which is then processed by the ISP (Image Signal Processor) using 3A (autofocus, auto exposure, auto white balance) technology. The VPSS (Video Process Sub-System) then scales the data to 1920×1080 resolution and converts it to H.264 (a video compression coding standard) format to obtain a single raw image frame. Furthermore, to address transmission delays during the initial streaming phase, the UVC transport layer pre-fills several frames (e.g., 2 frames) of DMA-buf (Direct Memory Access Buffer) into the transmission queue at streaming start. This ensures that frames are always available for the host (video terminal) to consume during the turnaround period, preventing queue starvation in a single cycle and maintaining a stable full frame rate output.

[0031] Additionally, it should be noted that raw image frames refer to image data frames that have undergone preprocessing but have not yet been encoded or compressed. In traditional schemes, if such raw image frames are directly output in an encoded format, such as MJPEG (a video compression coding standard) or H.264, their bitstream is a continuous byte stream without line alignment padding, which meets the conditions for direct transfer by the DMA engine. Therefore, the DMA-BUF zero-copy path can be used.

[0032] Understandably, the preprocessing step converts the raw data output by the image sensor into raw image frames in a standard format for subsequent encoding.

[0033] Step S120: Encode the original image frame to obtain memory information, and store the memory information in the memory pool. The memory information includes at least the first physical memory, memory block handles, and metadata. It should be noted that encoding refers to the compression and encoding of the original image frames by a hardware encoder, outputting an encoded bitstream. After encoding, the bitstream data is stored in a hardware shared memory block, and the encoder driver returns a handle to the memory block to identify it.

[0034] For example, the hardware encoder encodes a 1920×1080 MJPEG raw image frame, outputting approximately 80KB (kilobyte) of bitstream data, stored in a contiguous physical memory block of 6MB (megabyte) starting at address 0x8000_0000. This exported hardware memory block should be contiguous physical memory, such as CMA (Contiguous Memory Allocator), to ensure the driver can map it entirely to a single DMA address. The encoder driver returns a memory block handle H1, along with metadata: timestamp 0xABCDEF, data length 81920 bytes, pixel format MJPEG, and frame number 1024.

[0035] Additionally, it should be noted that metadata refers to auxiliary information describing the attributes of the encoded frame, including at least timestamps, data length, pixel format, and frame sequence number. Metadata is used for subsequent UVC protocol header generation and transmission control. The memory pool refers to the physical memory resource pool managed by the encoder driver, used to store the encoded output frame data, enabling the circular reuse of physical memory.

[0036] For example, metadata is returned to the upper-layer application along with the memory block handle and stored in a memory pool for management. The memory pool maintains a list of free handles, allocates memory blocks from the free list during encoding, and reclaims handles to the free list upon return.

[0037] Understandably, this step provides the core resources (contiguous physical memory + handles) for subsequent DMA-BUF export and zero-copy transfer, and is the data foundation for the entire zero-copy scheme.

[0038] Step S130: Convert the memory block handle into a file descriptor, and push the file descriptor and metadata to the UVC driver; It should be noted that a file descriptor refers to a standard Linux (UNIX-like operating system) file descriptor (fd) that exports a memory block handle via the DMA-BUF mechanism. DMA-BUF is a cross-driver shared memory mechanism provided by the Linux kernel. An fd can be transferred between different drivers and different processes, while the underlying mechanism points to the same physical memory block. The entire export process does not involve copying the frame data itself.

[0039] For example, the platform memory management interface is invoked to export the memory block handle H1 as a file descriptor fd=5, which corresponds to 6MB of contiguous physical memory starting at the underlying 0x8000_0000. Then, fd=5 and the metadata are pushed to the UVC driver through a custom callback interface. For encoding formats with variable frame lengths, the total buffer size must be explicitly specified when setting the output format; otherwise, the kernel driver will refuse DMA-BUF enqueueing because it cannot determine the buffer length.

[0040] Understandably, the handle-to-file-descriptor conversion is a crucial bridging step for achieving zero-copy transfer. Through the DMA-BUF mechanism, the memory block handle of the encoder driver domain is converted into a file descriptor recognizable by the UVC driver domain, allowing the UVC driver to directly access the physical memory output by the encoder without CPU copying.

[0041] Step S140: In response to receiving the enqueue request of the buffer, the UVC driver obtains the memory address based on the file descriptor and metadata, and assembles the first physical memory into a packet based on the status information of the buffer in the enqueue request to obtain the second physical memory; It should be noted that packet assembly refers to generating a UVC payload protocol header based on the buffer's state information at the beginning of the first physical memory, and writing the protocol header in place at the beginning of the first physical memory, so that the first physical memory, which originally only contained the encoded bitstream, becomes the second physical memory containing both the UVC protocol header and the encoded bitstream.

[0042] For example, the UVC driver uses the file descriptor fd=5 to call the DMA-BUF subsystem interface to attach the file descriptor to the UVC device, obtaining the address 0x8000_0000 and length 6MB of the first physical memory. Based on the frame identifier FID=1 (toggled at the start of each frame transmission), metadata (timestamp 0xABCDEF, data length 81920 bytes), and queue error status=0, the UVC driver generates a 2-byte UVC protocol header, containing bHeaderLen (protocol header length) and bmHeaderInfo (protocol header information flags), including EOH / FID / EOF flags, and writes this protocol header to the starting address 0x8000_0000 of the first physical memory. After writing, the 2-byte protocol header is at address 0x8000_0000, and the 80KB encoded bitstream starts at address 0x8000_0002, together forming the second physical memory. The entire process involves only the 2-byte protocol header being written by the kernel driver; the frame data itself is copied zero-time and involves zero CPU intervention.

[0043] Understandably, the packet assembly step uses the in-situ writing method of the protocol header, and the DMA directly transports the entire frame bitstream containing the protocol header, achieving zero copying of the packet assembly and zero overhead of the protocol header.

[0044] Step S150: Generate a transfer descriptor based on the memory address, and transfer the bitstream data in the second physical memory to the video terminal according to the transfer descriptor.

[0045] It should be noted that a transfer descriptor is a descriptor used by the USB controller driver to describe a DMA transfer task, and it contains at least information such as the source memory address (i.e., the DMA address of the second physical memory), the transfer length, and the endpoint number. The USB controller hardware starts the DMA engine based on the transfer descriptor to move the data from the second physical memory to the USB bus.

[0046] For example, the UVC driver calls the endpoint queue interface of the USB controller driver, passing in the DMA address of the second physical memory and a transfer length of 81922 bytes (2-byte protocol header + 81920-byte bitstream). The USB controller driver fills in the DMA transfer descriptor based on this information: the source address is the DMA mapping address of the second physical memory, the length is 81922, and the endpoint number is ep1. The USB controller hardware starts the DMA based on this transfer descriptor, transferring the entire frame bitstream, including the protocol header, to the USB bus and sending it to the video terminal.

[0047] Understandably, the transmission step is the final link in the entire zero-copy path. The entire process of the encoded bitstream from the hardware encoder output to the USB transmission is directly completed by the DMA engine. The CPU does not participate in any data transfer, eliminating the CPU memory copying step and significantly reducing end-to-end latency and CPU utilization.

[0048] In one possible implementation, step S150 is followed by: Step S151: Obtain the index of the bitstream data and find the memory block handle corresponding to the index; It should be noted that the index refers to the buffer index allocated by the UVC driver for each buffer, which corresponds one-to-one with the entry index in the tracking table (inflight table). In the zero-copy path, since the hardware can only provide the buffer index when the USB transfer is completed and cannot directly provide the memory block handle of the encoder domain, the tracking table is needed to translate the index to the handle.

[0049] For example, after the USB transfer is complete, the UVC driver retrieves buffer index 0 from the DQBUF (VIDIOC_DQBUF: buffer dequeue) callback. Based on the one-to-one correspondence between entry index 0 and buffer index 0 in the trace table, the memory block handle H1 recorded in entry 0 is found.

[0050] Understandably, step S151 is the core action of the "return phase" in frame lifecycle management. It translates the buffer index of the USB domain into the memory block handle of the encoder domain through the tracking table, thus solving the problem of frame identity translation across driver domains.

[0051] Step S152: Push the frame corresponding to the memory block handle to the memory pool.

[0052] For example, the platform memory management interface is invoked to release the memory block handle H1 to the encoder memory pool. The memory pool marks the physical memory block corresponding to H1 (6MB starting from 0x8000_0000) as reallocatable and clears the occupied mark of entry 0 in the tracking table, so that the buffer index can be reused in subsequent frames.

[0053] Understandably, step S152 achieves the circular reuse of physical memory and successfully executes a complete frame return operation.

[0054] In this embodiment, by establishing a tracking table to record the correspondence between buffer indexes and memory block handles, and after the transmission is completed, the tracking table is searched according to the buffer index to obtain the memory block handle, and the platform memory management interface is called to release the memory block handle to the memory pool, the technical problem of the frame not being accurately returned after asynchronous transmission due to the different identification systems across driver domains (UVC driver domain uses buffer index, encoder driver domain uses memory block handle) under zero-copy path is solved, which leads to memory leaks. This achieves safe reclamation and cyclic reuse of physical memory.

[0055] Furthermore, in some embodiments, reference is made to Figure 2 , Figure 2 This is a schematic diagram of the system architecture of the video transmission method of this application.

[0056] The process begins with the sensor acquiring raw image data and transmitting it to the ISP module for initial processing. Subsequently, the data flows to the VPSS module for further image scaling, cropping, or rotation. After VPSS, the data flow splits into two paths: one continues to the VENC (Video Encoder) hardware encoder for MJPEG / H264 encoding, and the other flows directly to the UVC MMAP (Memory Map) copy path in either YUYV (a raw image pixel format) or NV12 (a raw image pixel format) raw format. After the image data processed by VPSS enters the VENC hardware encoder for MJPEG / H264 encoding, the encoded data is stored in a hardware memory block. Next, the memory block handle is converted to a DMA buffer file descriptor through a zero-copy bridging layer, following the path MB_BLK (memory block handle) → dma-buf fd, and then enters the UVC V4L2 DMABUF (UVC device zero-copy path) zero-copy path. In this process, the UVC protocol packet layer generates the UVC payload protocol header (including bHeaderLen and bmHeaderInfo, such as EOH / EOF / FID ​​flags) in situ on the memory where the encoded frame is located within the kernel UVC gadget (UVC device driver). Subsequently, the UVC transport layer directly reads the physical memory associated with dma-buf through the USB controller DMA engine, moves the entire frame bitstream containing the protocol header to the USB bus, and finally transmits it to the USB Host (PC (Personal Computer) / conference terminal). Simultaneously, the YUV / NV12 raw format data processed by VPSS directly enters the UVC MMAP copy path and is finally transmitted to the video terminal. This application achieves zero-copy transmission of video data, avoiding multiple copies of data between memory locations in traditional solutions, significantly reducing CPU load and memory bandwidth usage, and improving the efficiency and real-time performance of video transmission.

[0057] Furthermore, in some other embodiments, reference is made to Figure 3 , Figure 3 This is a schematic diagram of the data transmission process of the video transmission method of this application.

[0058] Efficient video transmission is achieved through the close collaboration of four modules: the VENC decoder, the zero-copy bridging layer, the UVC transport layer, and the USB Host. The prerequisite for this entire process is that the frame memory is a contiguous block of CMA physical memory. The process begins with the VENC decoder module completing the encoding of a frame of MJPEG / H264 image, writing the bitstream data into a contiguous block of physical memory, and outputting an MB_BLK memory block handle.

[0059] Subsequently, after receiving the MB_BLK handle from VENC, the zero-copy bridging layer converts the handle of the memory block into a DMA-buf file descriptor. This process only transmits the handle and does not involve actual data copying. Next, the zero-copy bridging layer uses this DMA-buf file descriptor to obtain the total length and effective data length of the buffer, preparing for subsequent enqueue operations. It also pre-fills several frames of DMA-buf into the transmission queue at stream startup to prevent first-queue starvation. After obtaining the information, the zero-copy bridging layer notifies the UVC transport layer via a callback, passing the DMA-buf file descriptor and its related information to the UVC transport layer for processing. After receiving the callback notification, the UVC transport layer fills the dma-buf file descriptor into the V4L2 buffer structure, sets the memory type to V4L2_MEMORY_DMABUF, and explicitly specifies sizeimage (total buffer size) when recording inflight[idx]QBUF. Otherwise, the kernel will reject the DMABUF enqueue and submit it to the UVC gadget driver via the VIDIOC_QBUF system call. This process only transmits the file descriptor and metadata, without copying the frame data itself. If QBUF enqueueing fails, i.e., the driver refuses to receive, an exception handling mechanism is triggered. The system will immediately return the just-acquired frame and synchronously roll back the inflight record to ensure the consistency of resource management. If QBUF enqueueing is successful, the UVC transport layer writes a 2-byte UVC payload protocol header in place at the beginning of the dma-buf memory block, including flag bits such as bHeaderLen and bmHeaderInfo. This process only requires the CPU to write the 2-byte protocol header; the frame data itself does not need to be copied or moved. After the protocol header is filled, the DMA address is obtained. The UVC transport layer attaches the file descriptor through the dma-buf subsystem and maps it to the device, obtaining its physical memory DMA address. Subsequently, the USB controller hardware directly reads the entire frame bitstream containing the protocol header from the physical memory associated with dma-buf, transports it to the USB bus through the DMA engine, and finally delivers it to the USB Host. After the transmission is completed, the UVC transport layer looks up the corresponding memory block handle in the inflight trace table according to the buffer index returned by DQBUF, calls the platform memory management interface to return the frame to the encoder memory pool and releases it by releaseStream (release bitstream), and clears the corresponding occupied mark in the trace table. This completes the closed-loop management of a frame's lifecycle from encoder output to USB transmission completion.

[0060] Based on the embodiments of steps S110-S150 above, by preprocessing and encoding the original image data, the encoded bitstream is directly stored in the physical memory of the memory pool, thereby providing a data foundation and unique identifier for subsequent cross-driver sharing; by converting the memory block handle into a file descriptor and pushing it to the UVC driver, the UVC driver obtains direct access to the same physical memory without CPU involvement in data transfer, realizing memory sharing between the encoder driver and the UVC driver; after receiving the buffer enqueue request, the physical memory address is obtained according to the file descriptor, and the first physical memory is used for packet assembly to obtain the second physical memory. This process does not require additional memory allocation or data copying; packet assembly can be completed simply by modifying the memory content in place; a transmission descriptor is generated through the memory address, and the bitstream data with the protocol header is transferred to the video terminal in one go by DMA, completely eliminating the CPU's involvement in data transfer and overcoming the high latency problem caused by CPU copying during video transmission.

[0061] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description and will not be repeated hereafter. Based on this, before step S110, the video transmission method further includes the following step: Step S210: In response to receiving a format negotiation request, determine the pixel format according to the format negotiation request. The pixel format includes the encoded format and the raw format. It should be noted that the format negotiation request refers to the standard UVC protocol request initiated by the video terminal via USB control transmission to determine the video transmission format. The UVC device must respond to this request during the PROBE / COMMIT control negotiation phase, informing the host of the supported pixel formats and related parameters.

[0062] For example, the host initiates a PROBE control negotiation request to inquire about the formats supported by the device. The device responds by supporting both MJPEG encoded format and YUYV raw format, and the host ultimately selects the MJPEG format and sends an acknowledgment.

[0063] Additionally, it's important to note that the encoding format refers to the video format compressed by a hardware encoder, including MJPEG and H264. The output stream of an encoded format is a continuous byte stream without line alignment padding, and its frame length is variable, making it suitable for DMA-BUF zero-copy transfer. The raw format refers to the uncompressed, unencoded image format, including YUYV and NV12. Raw formats contain stride (line alignment padding), causing the DMA engine to fail to transfer the data as expected, requiring the use of the MMAP copy path.

[0064] For example, when the encoding format is MJPEG, the bitstream of a 1920×1080 frame is about 80KB, which is compactly arranged in memory; when the original format is YUYV, the data size of a 1920×1080 frame is 4MB, but due to line alignment padding, it actually occupies more memory space.

[0065] Understandably, format negotiation is the starting point for adaptive I / O mode switching. By determining the pixel format through format negotiation, the system can decide whether to follow the DMA-BUF zero-copy path or the MMAP copy path, thus implementing an adaptive transmission strategy transparent to upper-layer applications.

[0066] In one possible implementation, step S210 is followed by: Step S211: In response to receiving a rate negotiation request, determine the rate negotiation information and single-frame load value according to the rate negotiation request. The rate negotiation information includes at least the link rate and endpoint parameters. It should be noted that the rate negotiation request refers to the UVC protocol request initiated by the host after completing device configuration (SET_CONFIGURATION) to negotiate the single-transmission capacity. The device needs to respond to this request in the PROBE / COMMIT control negotiation, informing the host of the single-transmission capacity committed by the protocol layer.

[0067] Additionally, it should be noted that the rate negotiation information refers to the link rate and endpoint parameters obtained from the USB controller driver. The link rate refers to the actual negotiated transmission rate of the USB bus, including High-Speed ​​and SuperSpeed. Endpoint parameters refer to the maximum packet length per transaction and burst multiplier preset in the USB endpoint configuration. The single-frame payload value refers to the total number of bytes in a single frame of data under the current pixel format.

[0068] For example, under a USB 2.0 High-Speed ​​link, with endpoint parameters of 1024 bytes maximum packet length per transaction and burst multiplier of 3, the basic link payload is 1024 × 3 = 3072 bytes. The single-frame payload in MJPEG format is 81920 bytes.

[0069] Understandably, rate negotiation provides input parameters for link rate adaptation, enabling the single transmission size promised by the protocol layer to be dynamically aligned with the actual link capacity and endpoint configuration. This is a prerequisite for preventing stalls / underruns and maximizing throughput.

[0070] Step S212: Calculate the rate negotiation information based on the preset calculation rules to obtain the basic load value, and compare the basic load value with the single frame load value to obtain the single transmission capacity. It should be noted that the preset calculation rule is: base load value = burst multiplier × maximum packet length per transaction, which is clamped to a high bandwidth limit of 3072 bytes per microframe under the USB 2.0 High-Speed ​​link; the single transmission capacity is the larger of the base load value and the single frame load value, but does not exceed the actual capacity limit of the endpoint.

[0071] For example, in a USB 2.0 High-Speed ​​link, with a burst multiplier of 3 and a maximum packet length of 1024 bytes per transaction, the base payload value is 3 × 1024 = 3072 bytes. The MJPEG single frame payload is 81920 bytes, which is greater than 3072 bytes, so the single transmission capacity is taken as 81920 bytes.

[0072] Understandably, through preset calculation rules, the system can dynamically calculate the single transmission capacity that matches the current link rate and endpoint capacity, avoiding the host's expected value from exceeding the endpoint capacity and causing stall / underrun, while making full use of transmission capacity on high-bandwidth links and reducing interruption and scheduling overhead.

[0073] Step S213: Construct response data based on the single transmission capacity and push the response data to the video terminal.

[0074] For example, the calculated single transfer capacity of 81920 bytes is filled into the dwMaxPayloadTransferSize field of the UVC PROBE / COMMIT control negotiation response, the response data is constructed and returned to the Host via USB control transfer.

[0075] Understandably, by informing the Host of the calculated single transmission capacity through the standard UVC protocol mechanism, the transmission size promised by the protocol layer is strictly aligned with the actual capacity of the endpoint, thus achieving adaptive link throughput.

[0076] Step S220: If the pixel format is an encoded format, return to the step of preprocessing the original image data; It should be noted that when the pixel format determined by format negotiation is an encoding format (MJPEG / H264), the system selects the DMA-BUF zero-copy path, that is, returns to step S110 of the first embodiment to continue execution. At this time, the bitstream of the encoding format is a continuous byte stream with no line alignment padding, which meets the conditions for direct transfer by the DMA engine.

[0077] In one possible implementation, step S220 is followed by: Step S221: Determine the buffer capacity based on the resolution of the original image frame; For example, the original image frame resolution is 1920×1080, and the encoding format is MJPEG. Considering that the MJPEG frame length is not fixed (depending on the image complexity), the buffer capacity is set to resolution × 3, i.e., 1920×1080×3≈6MB, to ensure that the encoder output buffer is large enough to accommodate the maximum possible bitstream.

[0078] Understandably, for encoding formats with variable frame lengths (MJPEG / H264), the sizeimage must be explicitly specified when setting the output format; otherwise, the kernel driver will refuse DMABUF enqueueing because it cannot determine the buffer length. Determining the buffer capacity based on the resolution is a necessary step to satisfy the DMABUF enqueueing constraints.

[0079] Step S222: Allocate the capacity of the first physical memory according to the buffer capacity.

[0080] For example, if a 6MB buffer capacity is determined, the encoder driver interface is called to allocate a contiguous block of physical memory (CMA memory) of size 6MB as the first physical memory. The starting address of this memory block is 0x8000_0000, and subsequent encoded bitstreams will be written to this memory block.

[0081] Understandably, the calculated buffer capacity is implemented in the actual physical memory allocation, providing a physical memory foundation for subsequent encoding output and DMA-BUF export.

[0082] Step S230: When the pixel format is the original format, acquire the original image data and preprocess the original image data to obtain the encoded frame data, which is located in the first buffer. It should be noted that the first buffer refers to the encoder output buffer, used to store the encoded bitstream data. The second buffer refers to the mmap buffer maintained by the UVC driver, used to store the frame data to be transmitted. In the original format, zero-copy is not possible due to stride issues; data must be moved from the first buffer to the second buffer via CPU copying.

[0083] For example, the original format is YUYV with a resolution of 1920×1080. The original image data output by VPSS has line alignment padding, with each line containing 1920×2=3840 bytes of effective pixels, but due to hardware alignment requirements, each line actually occupies 4096 bytes. The encoded frame data is located in the first buffer (encoder output buffer) and contains complete line alignment padding data.

[0084] Understandably, the original format caused the DMA engine to fail to transfer data as expected due to stride, necessitating a fallback to the MMAP copy path. By using adaptive I / O mode switching, the copy path is automatically selected under the original format, ensuring compatibility.

[0085] Step S240: Copy the encoded frame data from the first buffer to the second buffer, and push the encoded frame data from the second buffer to the video terminal.

[0086] For example, the encoded frame data in the first buffer is copied line by line to the second buffer (UVC mmap buffer) via CPU memcpy (CPU memory copy). During the copying process, line alignment padding is removed to obtain compactly arranged valid pixel data. The data in the second buffer is then sent to the video terminal via the USB bus.

[0087] Understandably, using the MMAP copy path in the original format is a compatibility guarantee. Although it sacrifices the performance advantage of zero-copy, it ensures that the system can still function normally in the original format, is completely transparent to upper-layer applications, and requires no application code intervention in the decision-making process.

[0088] Furthermore, in some embodiments, reference is made to Figure 4 , Figure 4 This is a schematic diagram of the I / O mode switching of the video transmission method of this application.

[0089] Figure 4 This demonstrates the complete logic of a USB UVC device driver based on the V4L2 framework, dynamically selecting the underlying data transmission path according to a preset video format when responding to a video stream initiation request. The process begins with the USB Host negotiating the video format. After the USB host sets the desired video format via standard control transmission, the system triggers the `streamon` event, marking the official opening of the video acquisition and transmission channel. At this point, the system reaches the core decision point, determining the currently effective video format `cur_format` and generating two distinct branches based on the format type.

[0090] When a video format such as MJPEG / H264 is detected, the system uses the DMABUF zero-copy path on the left. The core objective of this path is to avoid secondary data transfer in memory. First, it establishes that the transmission of this frame uses a DMA buffer sharing mechanism. The driver allocates a video buffer through VIDIOC_REQBUFS and explicitly specifies the memory type as V4L2_MEMORY_DMABUF. Then, it obtains the DMA-buf file descriptor representing the physical memory of this frame through a custom callback function acquire_frame_dmabuf and binds it to the V4L2 buffer structure. At this point, the data is usually written directly to this contiguous memory block by the hardware encoder.

[0091] When the video format is detected to be raw, such as YUYV / NV12, the system uses the MMAP copy path on the right. Since raw images often have line alignment padding issues, making direct DMA zero-copy operations unsuitable, a memory mapping mechanism is used for this frame transfer. The driver also allocates a buffer, but specifies the memory type as V4L2_MEMORY_MMAP. Then, the mmap system call maps the kernel-side physical memory to the user-space virtual address space, allowing the CPU to read and write directly. Next, the get_frame_mmap function retrieves the image data, and memcpy is used to copy the uncompressed raw image data from the source memory region to the DMA-compatible buffer prepared by the V4L2 driver.

[0092] Regardless of the path taken, the filled data buffer will eventually be submitted to the USB controller. Both paths ultimately converge at the VIDIOC_QBUF enqueue operation, where the system adds the prepared buffer to the driver's transfer queue. Once enqueueing is successful, the data in the buffer is considered ready.

[0093] Furthermore, in some other embodiments, reference is made to Figure 5 , Figure 5 This is a schematic diagram of the link rate adaptive process of the video transmission method of this application.

[0094] Figure 5This document describes the complete internal processing logic of a USB video device after receiving configuration commands from the host, including link parameter parsing and bandwidth negotiation. The process begins after the USB bus layer completes the SET_CONFIGURATION configuration of the device, signifying the establishment of a physical connection between the device and the host, and the two parties need to determine the actual data transmission strategy. Next, the system performs a crucial hardware abstraction layer read operation: obtaining the actual physical speed of the current USB link (distinguishing between High-Speed ​​and SuperSpeed), and parsing the configuration parameters of the current endpoint, focusing on extracting the maximum packet length per transaction and the burst factor—two core metrics determining throughput. After obtaining the aforementioned underlying hardware capability data, the system enters the core dynamic calculation phase, using a specific formula (burst factor multiplied by maximum packet length per transaction) to derive the maximum effective payload for a single transmission. During this process, a specific engineering trade-off is included for High-Speed ​​(HS) links: to prevent unnecessary bus overhead from frequent packet assembly, the calculation result is constrained within a high bandwidth limit of 3072 bytes per microframe, thus achieving a balance between theoretical performance and practical stability. Subsequently, the calculated optimal transmission parameters are encapsulated in a standard PROBE_COMMIT control response message, formally written back, and notified to the USB host. This process ensures that the single transmission capacity promised by the protocol layer is strictly aligned with the actual transmission capacity of the underlying endpoints, thereby effectively preventing stalls or underruns under high loads. At the same time, this mechanism also allows the device to adaptively increase the single transmission amount as the link rate changes, thus fully exploring and squeezing the available bandwidth potential of the USB bus.

[0095] In this embodiment, the pixel format is determined through format negotiation, providing a decision basis for subsequent path selection; the single transmission capacity is dynamically calculated through rate negotiation to achieve adaptive link throughput; a DMA-BUF zero-copy path is used under the encoding format, and buffer capacity is allocated according to the resolution to meet the DMABUF enqueue constraints; under the original format, it falls back to the MMAP copy path to ensure compatibility. This application achieves the advantages of low latency and low CPU usage with zero copy in the encoding format, maintains functional compatibility in the original format, and the mode switching is automatically completed at the lower layer, completely transparent to upper layer applications.

[0096] Based on any of the above embodiments, in the third embodiment of this application, the content that is the same as or similar to the above embodiments can be referred to the above description, and will not be repeated hereafter. Based on this, step S140, the video transmission method further includes the following step: Step S310: Generate a protocol header based on the status information, which includes at least the frame identifier, buffer metadata, and queue error status; It should be noted that the protocol header refers to the UVC payload protocol header, which consists of bHeaderLen and bmHeaderInfo, totaling 2 bytes. The protocol header contains essential control information for the UVC protocol, such as the start and end of frame flags, frame identifier, and timestamp presence flag, used to inform the video terminal of the frame boundaries and attributes of the current data packet.

[0097] Additionally, it should be noted that the frame identifier is a flag bit in the UVC protocol used to distinguish adjacent frames. It is toggled at the beginning of each frame, and the video terminal determines frame boundaries by detecting changes in the frame identifier. Buffer metadata refers to the frame attribute information pushed from the encoder to the UVC driver along with the file descriptor, including at least timestamp, data length, pixel format, and frame sequence number. Queue error status refers to the error flag of the UVC driver's buffer queue; it is 0 during normal transmission and set to 1 when a DMA error or timeout occurs.

[0098] For example, the identifier FID of the current frame is 1 (0 in the previous frame), and the buffer metadata includes a timestamp of 0xABCDEF, a data length of 81920 bytes, a pixel format of MJPEG, and a queue error status of 0. The UVC driver generates a 2-byte protocol header based on this information: bHeaderLen = 0x02 (the protocol header is 2 bytes long), bmHeaderInfo = 0x0C (setting the FID bit and PTS bit, indicating that FID = 1 and includes a timestamp).

[0099] Understandably, generating the protocol header is the first step in packet assembly. It constructs the protocol header content that conforms to the UVC protocol standard based on the frame identifier, metadata, and error status in the status information.

[0100] Step S320: Fill the protocol header into the starting position of the first physical memory to obtain the second physical memory.

[0101] For example, the UVC driver writes the generated 2-byte protocol header (0x02 0x0C) to the starting address 0x8000_0000 of the first physical memory. After writing, 0x8000_0000 to 0x8000_0001 is the protocol header, and 0x8000_0002 onwards is the original 80KB encoded bitstream. The first physical memory (containing only the bitstream) becomes the second physical memory (protocol header + bitstream), with a total length of 81922 bytes.

[0102] Alternatively, as an alternative, the kernel also supports assembling the protocol header and each physical page segment of the DMA-BUF into a scatter-gather table, which the DMA controller then uses to move data segment by segment. This method offers some flexibility when physical memory is not contiguous, but at the cost of increased transmission path complexity. The DMA engine needs to switch back and forth between multiple non-contiguous physical segments, introducing additional scheduling overhead and uncertainty. To reduce the complexity of the scatter-gather path and ensure the determinism of single-segment continuous transmission, this solution actively disables the scatter-gather path by configuration, forcing the use of the aforementioned "contiguous memory linear mapping + DMA direct read" method. In this way, the DMA engine only needs to read the entire frame of data from contiguous physical memory at once, without switching between multiple physical segments. This simplifies the transmission path and ensures the determinism of packet assembly and DMA, fundamentally avoiding transmission jitter problems that may be caused by discrete physical memory.

[0103] In this embodiment, through the coordinated operation of steps S310 to S320, this application achieves in-situ generation and writing of the UVC protocol header. The entire process involves only the CPU writing a 2-byte protocol header, with zero copying of the frame data itself and zero CPU involvement. Compared to the traditional approach where the protocol header needs to be assembled with the payload via memcpy or scatter-gather after generation, this significantly reduces packet assembly overhead and represents a key innovation in the zero-copy transmission path involving the coordination of the protocol header and DMA.

[0104] Based on any of the above embodiments, in the fourth embodiment of this application, the content that is the same as or similar to the above embodiments can be referred to the above description, and will not be repeated hereafter. In addition, the video transmission method further includes the following steps: Step S410: Allocate at least one buffer index for the file descriptor; For example, the UVC driver allocates buffer index 0 for the pushed file descriptor fd=5. This buffer index corresponds one-to-one with the buffer index in the V4L2 buffer queue, and will be used to identify the position of the current frame in the UVC driver buffer queue when QBUF is enqueued subsequently.

[0105] Understandably, allocating a buffer index is a marker indicating that a frame has entered the scope of UVC driver management. By assigning a unique buffer index to each file descriptor, the UVC driver can quickly locate the corresponding frame resource through the index in subsequent stages such as enqueuing, transmission, and return, laying the foundation for the establishment of the tracking table and frame lifecycle management.

[0106] Step S420: Record the memory block handle to a preset tracking table. The tracking table contains at least the memory block handle and the correspondence between the buffer index and the memory block handle. It's important to note that the tracking table refers to the InFlight tracking table, which records the complete lifecycle state of each frame from acquisition to return. The entry indices in the tracking table correspond one-to-one with the buffer indices. Each entry stores at least a memory block handle and an occupancy flag. The tracking table also contains the correspondence between memory block handles and buffer indices, ensuring accurate location of memory block handles when subsequently searching the tracking table using buffer indices. The tracking table is the core data structure for frame lifecycle management under zero-copy paths, solving the frame identity translation problem across driver domains (the encoder driver domain uses memory block handles, while the UVC driver domain uses buffer indices).

[0107] For example, memory block handle H1 is recorded in trace table entry 0, and the occupancy flag of this entry is set to "occupied". At this time, the trace table status is: entry 0 = {memory block handle = H1, status = occupied}, and entries 1 to N are all in the free state. Entry index 0 corresponds one-to-one with buffer index 0 allocated in step S410.

[0108] Understandably, recording memory block handles in the tracking table is the starting point for frame lifecycle management. By establishing a correspondence between buffer indices and memory block handles, the tracking table allows the memory block handle of the encoder domain to be retrieved by looking up the buffer index reported by the hardware during the subsequent USB transfer completion callback, thus ensuring accurate frame return. Simultaneously, the "occupied" flag ensures that the frame is not duplicated or mistakenly released before the transfer is complete.

[0109] Step S430: If the UVC driver refuses to accept the enqueue request of the buffer, it looks up the memory block handle in the tracking table according to the buffer index, obtains the memory information, and pushes the memory information to the memory pool.

[0110] For example, the UVC driver calls VIDIOC_QBUF to submit the frame corresponding to buffer index 0 to the UVC gadget driver, but the driver refuses to accept the enqueue request (e.g., due to buffer size mismatch, insufficient driver resources, etc.). At this time, the memory block handle H1 is still in the current context. Based on buffer index 0, the tracking table entry 0 is looked up to confirm the corresponding memory block information. The platform memory management interface is immediately called to release H1 to the encoder memory pool, and the occupied mark of tracking table entry 0 is cleared, restoring entry 0 to an idle state.

[0111] Understandably, rollback upon enqueue failure is a crucial aspect of anomaly protection in frame lifecycle management. When QBUF fails, although the frame has been acquired from the encoder and recorded in the tracking table, it has not successfully entered the UVC transmission queue. If not returned promptly, the physical memory corresponding to the frame will remain permanently in an "occupied" state, leading to memory leaks. By immediately returning the frame and clearing the tracking table record, this application ensures that frame acquisition and return remain strictly paired even in the abnormal path of enqueue failure, preventing encoder memory pool leaks due to anomalies. Furthermore, since the memory block handle is still available (recently acquired) when enqueue fails, it can be returned directly without needing to look up the handle in the tracking table, improving rollback efficiency.

[0112] In this embodiment, through the coordinated operation of steps S410 to S430, this application implements the acquisition phase and rollback protection for enqueue failure in frame lifecycle management. By allocating a buffer index to the file descriptor, the frame enters the UVC driver management scope; by recording the memory block handle in the tracking table and marking it as occupied, a "buffer index" is established. The mapping between "memory block handles" provides a data foundation for subsequent frame tracking and return. When an enqueue request fails, the frame information is confirmed by looking up the tracking table using the buffer index, the frame is immediately returned, and the tracking table record is cleared, preventing frame leakage. Through these steps, this application achieves closed-loop management of frame acquisition, tracking, and exception rollback under a zero-copy path, ensuring strict pairing of acquisition and return across all paths and eliminating DMA-BUF frame leakage.

[0113] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the video transmission method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0114] This application also provides a video transmission system applied to a UVC device. The UVC device includes at least a UVC driver, a USB controller, a memory pool, and a buffer. Please refer to [reference needed]. Figure 6 The video transmission system includes: The video acquisition and processing module 10 is used to receive raw image data and preprocess the raw image data to obtain raw image frames; The video encoding module 20 is used to encode the original image frames, obtain memory information, and store the memory information in the memory pool. The memory information includes at least the first physical memory, memory block handles, and metadata. The zero-copy bridging module 30 is used to convert memory block handles into file descriptors and push the file descriptors and metadata to the UVC driver; The packet assembly module 40 is used to receive the enqueue request of the buffer, obtain the memory address through the UVC driver based on the file descriptor and metadata, and assemble the first physical memory according to the status information of the buffer in the enqueue request to obtain the second physical memory; The transmission module 50 is used to generate a transmission descriptor based on the memory address and transmit the bitstream data in the second physical memory to the video terminal according to the transmission descriptor.

[0115] The video transmission system provided in this application, employing the video transmission method described in the above embodiments, can solve the technical problem of high latency caused by CPU memory copying. Compared with the prior art, the beneficial effects of the video transmission system provided in this application are the same as those of the video transmission method provided in the above embodiments, and other technical features of the video transmission system are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0116] This application provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the video transmission method in Embodiment 1 above.

[0117] The following is for reference. Figure 7 The diagram illustrates a structural schematic of an electronic device suitable for implementing embodiments of this application. The electronic devices in these embodiments may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 7 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0118] like Figure 7As shown, the electronic device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 or a program loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the electronic device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. The communication device 1009 allows the electronic device to communicate wirelessly or wiredly with other devices to exchange data. Although the diagrams show electronic devices with various systems, it should be understood that it is not required to implement or have all of the systems shown. More or fewer systems may be implemented alternatively.

[0119] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0120] The electronic device provided in this application, employing the video transmission method described in the above embodiments, can solve the technical problem of high latency caused by CPU memory copying. Compared with the prior art, the beneficial effects of the electronic device provided in this application are the same as those of the video transmission method provided in the above embodiments, and other technical features of this electronic device are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.

[0121] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0122] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0123] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the video transmission method described in the above embodiments.

[0124] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0125] The aforementioned computer-readable storage medium may be included in an electronic device or may exist independently without being assembled into an electronic device.

[0126] The aforementioned computer-readable storage medium carries one or more programs. When the aforementioned one or more programs are executed by an electronic device, the electronic device causes the following actions: In response to receiving raw image data, it preprocesses the raw image data to obtain raw image frames; it encodes the raw image frames to obtain memory information and stores the memory information in a memory pool, the memory information including at least a first physical memory, a memory block handle, and metadata; it converts the memory block handle into a file descriptor and pushes the file descriptor and metadata to the UVC driver; In response to receiving an enqueue request from the buffer, it obtains the memory address through the UVC driver based on the file descriptor and metadata, and assembles the first physical memory into a second physical memory according to the buffer status information in the enqueue request; it generates a transmission descriptor based on the memory address and transmits the bitstream data in the second physical memory to the video terminal according to the transmission descriptor.

[0127] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0128] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0129] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0130] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described video transmission method, thereby solving the technical problem of high latency caused by CPU memory copying. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the video transmission method provided in the above embodiments, and will not be repeated here.

[0131] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A video transmission method, characterized in that, Applied to UVC devices, the UVC device includes at least a UVC driver, a USB controller, a memory pool, and a buffer; the video transmission method includes: In response to receiving raw image data, the raw image data is preprocessed to obtain raw image frames; The original image frame is encoded to obtain memory information, and the memory information is stored in the memory pool. The memory information includes at least the first physical memory, memory block handles, and metadata. The memory block handle is converted into a file descriptor, and the file descriptor and the metadata are pushed to the UVC driver; In response to receiving the enqueue request of the buffer, the UVC driver obtains the memory address based on the file descriptor and the metadata, and packages the first physical memory according to the status information of the buffer in the enqueue request to obtain the second physical memory; A transport descriptor is generated based on the memory address, and the bitstream data in the second physical memory is transmitted to the video terminal according to the transport descriptor.

2. The video transmission method as described in claim 1, characterized in that, The buffer includes at least a first buffer and a second buffer; before the step of preprocessing the original image data to obtain the original image frame, the method further includes: In response to receiving a format negotiation request, a pixel format is determined according to the format negotiation request, the pixel format including an encoded format and a raw format; If the pixel format is an encoded format, return to the step of preprocessing the original image data; When the pixel format is the raw format, the raw image data is acquired and preprocessed to obtain encoded frame data, which is located in the first buffer. The encoded frame data in the first buffer is copied to the second buffer, and the encoded frame data in the second buffer is pushed to the video terminal.

3. The video transmission method as described in claim 2, characterized in that, After the step of responding to receiving a format negotiation request and determining the pixel format according to the format negotiation request, the method further includes: In response to receiving a rate negotiation request, rate negotiation information and single-frame load value are determined based on the rate negotiation request, wherein the rate negotiation information includes at least link rate and endpoint parameters; The rate negotiation information is calculated based on preset calculation rules to obtain the basic load value, and the basic load value is compared with the single frame load value to obtain the single transmission capacity. Response data is constructed based on the single transmission capacity, and the response data is pushed to the video terminal.

4. The video transmission method as described in claim 2, characterized in that, In the case where the pixel format is an encoded format, after the step of preprocessing the original image data, the method further includes: The buffer capacity is determined based on the resolution of the original image frame; Allocate the capacity of the first physical memory according to the buffer capacity.

5. The video transmission method as described in claim 1, characterized in that, The step of assembling the first physical memory according to the status information to obtain the second physical memory includes: A protocol header is generated based on the status information, which includes at least a frame identifier, buffer metadata, and queue error status. The protocol header is filled into the starting position of the first physical memory to obtain the second physical memory.

6. The video transmission method as described in claim 1, characterized in that, After the step of transmitting the bitstream data in the second physical memory to the video terminal according to the transmission descriptor, the method further includes: Obtain the index of the bitstream data and find the memory block handle corresponding to the index; The frame corresponding to the memory block handle is pushed to the memory pool.

7. The video transmission method as described in claim 1, characterized in that, The video transmission method further includes: Allocate at least one buffer index for the file descriptor; The memory block handle is recorded in a preset tracking table, which contains at least the memory block handle and the correspondence between the buffer index and the memory block handle; If the UVC driver refuses to accept the enqueue request of the buffer, it looks up the memory block handle in the tracking table according to the buffer index, obtains the memory information, and pushes the memory information to the memory pool.

8. A video transmission system, characterized in that, Applied to UVC devices, the UVC device includes at least a UVC driver, a USB controller, a memory pool, and a buffer; the video transmission system includes: The video acquisition and processing module is used to receive raw image data and preprocess the raw image data to obtain raw image frames; The video encoding module is used to encode the original image frames to obtain memory information and store the memory information in the memory pool. The memory information includes at least a first physical memory, a memory block handle, and metadata. The zero-copy bridging module is used to convert the memory block handle into a file descriptor and push the file descriptor and the metadata to the UVC driver; The packet assembly module is used to receive the enqueue request of the buffer, obtain the memory address through the UVC driver according to the file descriptor and the metadata, and assemble the first physical memory according to the status information of the buffer in the enqueue request to obtain the second physical memory; The transmission module is used to generate a transmission descriptor based on the memory address, and to transmit the bitstream data in the second physical memory to the video terminal according to the transmission descriptor.

9. An electronic device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the video transmission method as described in any one of claims 1 to 7.

10. A readable storage medium, characterized in that, The readable storage medium is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the video transmission method as described in any one of claims 1 to 7.

Images