A method for processing multi-path video stream data, a program product and an electronic device

Abstract

This application provides a method, program product, and electronic device for processing multi-channel video stream data. The method includes: determining a target frame buffer corresponding to the UDP data packet based on the video stream identifier; if the UDP data packet contains an intra-frame offset corresponding to its data frame, directly copying the payload data in the UDP data packet to the corresponding position in the target frame buffer according to the intra-frame offset, and simultaneously updating the fill status record of the target frame buffer to mark the position in the target frame buffer that has been filled with valid data; calculating the total amount of valid data filled in the target frame buffer after copying based on the fill status record; and determining that the data frame is a complete frame when the total amount of valid data reaches a preset total frame size. Based on this, each UDP data packet can be directly located to the correct position within the frame without waiting for preceding packets, naturally eliminating out-of-order problems caused by network jitter and significantly reducing reassembly latency and memory overhead.

Description

Technical Field

[0001] This application relates to the field of video transmission and processing technology, and in particular to a method, program product, and electronic device for processing multi-channel video stream data. Background Technology

[0002] In visual scenarios such as vehicle driving, robot environmental perception, and panoramic imaging, multiple cameras simultaneously acquire video streams and send multiple video stream data to the video receiving unit.

[0003] In embedded and high-bandwidth video scenarios, UDP protocol is commonly used to balance transmission overhead and real-time performance. A single frame of video data is split into multiple UDP packets and sent sequentially.

[0004] However, when multiple cameras transmit data frames to the video receiving device simultaneously, network latency and jitter can cause UDP packets belonging to the same data frame to be transmitted out of order. For example, packets sent earlier may arrive later. If the video receiving unit directly reassembles the frame data according to the order in which the packets are received, it will cause frame content to be disordered. In severe cases, it may also trigger parsing exceptions, memory errors, and ultimately lead to the receiving unit program crashing or even the system crashing. Summary of the Invention

[0005] This application provides a multi-channel video stream data processing method, program product, and electronic device to solve the problem of disordered received data frame content caused by out-of-order UDP packets due to network jitter or latency.

[0006] Firstly, a method for processing multi-channel video stream data is provided, applied to a video transmission system. The video transmission system is configured with a frame buffer corresponding to each video stream. The method includes: receiving UDP data packets generated from fragments of each video stream; determining the target frame buffer corresponding to the UDP data packet based on the video stream identifier parsed and extracted from the UDP data packet; if the UDP data packet contains an intra-frame offset corresponding to its data frame, directly copying the payload data in the UDP data packet to the corresponding position in the target frame buffer based on the intra-frame offset; updating the fill status record of the target frame buffer immediately after copying, the fill status record being used to mark the positions in the target frame buffer that have been filled with valid data; calculating the total amount of valid data filled in the target frame buffer after copying based on the fill status record; and determining that the data frame is a complete frame when the total amount of valid data reaches a preset total frame size.

[0007] In one possible implementation of the first aspect above, the method further includes: if there is no intra-frame offset in the UDP packet, appending the payload data in the UDP packet to the end of the current target frame buffer in the order of arrival.

[0008] In one possible implementation of the first aspect above, the fill status record is a bitmap, where each unit in the bitmap corresponds to a minimum storage unit in the frame buffer. The method for updating the fill status record of the target frame buffer includes: marking the unit at the corresponding position in the bitmap as filled after each copy or append of payload data; if the unit corresponding to the position to be operated on has been marked as filled, then the UDP packet to be stored is determined to be a duplicate packet, its payload data is discarded and the total amount of effective data is not increased.

[0009] In one possible implementation of the first aspect described above, the video transmission system further configures a frame buffer queue for each video stream. The frame buffer queue is used to cache complete frames obtained from the frame buffer, and the depth of the frame buffer queue is a preset first number. The system also includes: when it is determined that a data frame in the frame buffer is a complete frame, determining whether the frame buffer queue corresponding to the complete frame is full; if the frame buffer queue corresponding to the complete frame is not full, then the complete frame is enqueued into the frame buffer queue and waits for the display thread to consume it; if the frame buffer queue corresponding to the complete frame is full, then all cached complete frames in the frame buffer queue corresponding to the complete frame are discarded, and the current complete frame is enqueued.

[0010] In one possible implementation of the first aspect described above, the intra-frame offset is carried by an extension field in the UDP packet, and the total frame size is obtained from the protocol header or a preset configuration.

[0011] In one possible implementation of the first aspect above, receiving UDP packets generated from each video stream segment includes: receiving the UDP packets from each video stream, encapsulating each UDP packet into a structure containing the video stream identifier, a tag indicating the special attributes of the UDP packet, and payload data, and writing it into a lock-free circular queue.

[0012] In one possible implementation of the first aspect above, the payload data in the UDP packet is in raw pixel format, and the method further includes: identifying the pixel format type of the payload data; and selecting a corresponding display adaptation method to output the display based on the identified pixel format type.

[0013] In one possible implementation of the first aspect above, the method employs a multi-threaded parallel architecture, which includes: at least one UDP receiving thread for receiving the UDP data packets and writing them into a lock-free circular queue; at least one parsing and distributing thread for reading the UDP data packets from the lock-free circular queue and performing parsing and distributing operations; at least one frame combining thread for performing copy or append operations, as well as filling state updates and frame integrity determination; and multiple independent display threads, one display thread for each video stream, for rendering and displaying complete frames.

[0014] In one possible implementation of the first aspect above, at least one parsing and distribution thread is used to read the UDP data packets from the lock-free circular queue and perform parsing and distribution operations, specifically including: the at least one parsing and distribution thread is used to read the encapsulated data structure from the lock-free circular queue, parse the transmission header of the UDP data packets, and extract the sequence number, relative timestamp, and number of extended fields; parse the extended fields, and extract the number of sampling points and step size to inform the receiving end of the pixel arrangement of the frame; determine whether the extended fields carry the intra-frame offset; determine the corresponding frame buffer according to the video stream identifier, and distribute the detected intra-frame offset and the payload data to the frame combination thread.

[0015] In a second aspect, an electronic device is provided, including a processor and a memory, the memory storing a computer program / instruction, which, when executed by the processor, implements the multi-channel video stream data processing method as described in the first aspect and any of the various possible implementations of the first aspect.

[0016] Thirdly, a computer program product is provided, including a computer program / instruction, which, when executed by a processor, implements a multi-channel video stream data processing method as described in the first aspect and any of the various possible implementations of the first aspect.

[0017] Fourthly, a computer-readable storage medium is provided, on which instructions are stored, which, when executed on a computer, cause the computer to perform the multi-channel video stream data processing method as described in the second aspect above. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 According to some embodiments of this application, a schematic diagram of a scenario for multi-channel video stream transmission is shown.

[0020] Figure 2 According to some embodiments of this application, a schematic diagram is shown of multiple cameras transmitting video streams to a video receiving unit in a video transmission system.

[0021] Figure 3 According to some embodiments of this application, a schematic diagram of multiple data caching stages is shown.

[0022] Figure 4 According to some embodiments of this application, a schematic diagram of out-of-order UDP packet framing based on directly writing intra-frame offsets into the frame buffer is shown.

[0023] Figure 5 According to some embodiments of this application, a flowchart of a multi-channel video stream data processing method is shown.

[0024] Figure 6 According to some embodiments of this application, a schematic diagram of a frame dropping process in a screen display scenario is shown.

[0025] Figure 7 According to some embodiments of this application, a flowchart of another video stream data processing method is shown.

[0026] Figure 8 According to some embodiments of this application, a structural block diagram of an electronic device is shown. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0028] Figure 1 According to some embodiments of this application, a schematic diagram of a scenario for multi-channel video stream transmission is shown. (Reference) Figure 1 As shown, multiple cameras, namely Camera 1, Camera 2, and Camera 3, are deployed on different parts of the robot. Cameras 1, 2, and 3 respectively collect video data, resulting in corresponding video streams L1, L2, and L3. Cameras 1, 2, and 3 can function as part of a video acquisition unit, transmitting the video streams to an internal video receiving unit via the UDP protocol.

[0029] Understandable. Figure 1 This example illustrates the deployment of the video acquisition unit and the video receiving unit on the same electronic device. In another embodiment, the video acquisition unit and the video receiving unit can be deployed on different devices, and this is not a limitation.

[0030] Figure 2According to some embodiments of this application, a schematic diagram is shown of multiple cameras transmitting video streams to a video receiving unit in a video transmission system. It is understood that the video transmission system includes a video acquisition and transmission unit and a video receiving unit. This scenario illustrates an example where the video acquisition unit includes multiple cameras, and multiple cameras acquire and transmit multiple video streams via the UDP protocol. Specifically, each camera fragments each data frame in the video stream into multiple UDP data packets using the UDP protocol, and then numbers each UDP data packet before transmitting it to the video receiving unit. (Reference) Figure 2 In the video stream L1 of camera 1, there is a data frame S1 to be transmitted, which is divided into UDP packets B1 to Bn, where n is a positive integer. The video stream L2 of camera 2 includes a data frame H1 to be transmitted, which is divided into UDP packets C1 to Cm, where m is a positive integer. The video stream L3 of camera 3 includes a data frame G1 to be transmitted, which is divided into UDP packets D1 to Dk, where k is a positive integer. After receiving the UDP packets from each camera, the video receiving unit first temporarily stores them; then, according to the video stream identifier, it distributes the data from different streams to their respective frame queues in the frame buffer. For example, UDP packet B1 is stored in frame queue 1, UDP packet C1 in frame queue 2, and UDP packet D1 in frame queue 3. When the end-of-frame identifier of a video stream is detected, it is determined that all fragmented data packets of the current frame have been received, and a complete data frame can be obtained. The data content of the complete data frame is obtained by splicing all data packets according to the receiving order. The complete data frame is then pushed to the display unit for rendering output. For example, when the EOF flag is received from the L1 video stream, it is determined that the UDP packets belonging to the same data frame have been sent, and the complete data frame obtained by splicing the UDP packets in the order of receipt is displayed.

[0031] However, as shown in the background section, when multiple cameras simultaneously transmit data frames to a video receiving device, UDP data packets belonging to the same data frame may be transmitted out of order due to network latency, jitter, and other reasons. This can lead to frame data corruption when the video receiving unit reassembles the received UDP data packets, and may even cause the video receiving unit's system to crash. For example, due to network latency or jitter, during data transmission, multiple video streams may need to be transmitted simultaneously within a very short time, causing UDP data packets to be transmitted out of order. For instance, in data frame S1, data packet B8 may arrive after data packet B11. When the video receiving unit reassembles data frame S1 according to the receiving order, data frame S1 will be corrupted.

[0032] Therefore, to solve the above problems, this application provides a multi-channel video stream data processing method. In this method, the video receiving unit in the video transmission system pre-configures an independent frame buffer for each video stream. After receiving UDP data packets from each video stream, the UDP data packets of each video stream are stored in the corresponding target frame buffer based on the video stream identifier. Each UDP data packet selectively carries the absolute offset of its payload within its data frame (i.e., intra-frame offset), which can be extracted by parsing the extended field in the UDP data packet. If the UDP data packet carries the absolute offset, the payload data is directly copied into the target frame buffer at the position specified by the offset via memory copying; if the data packet does not carry an offset, the payload data is appended to the end of the frame buffer in the current state according to the arrival order of the data packets. At the same time, the system maintains a fill status record (e.g., a bitmap) for each frame buffer to accurately mark whether each byte position in the frame buffer has been covered by valid data. After each write operation, the record is updated: if the position to be written has been marked, the current data packet is determined to be a duplicate packet, its payload is discarded, and the valid padding count is not increased; if it has not been marked, the position is marked and the number of valid padding bytes is increased. When the number of valid padding bytes accumulated in the frame buffer reaches the preset total size of the frame, the data frame is determined to have been completely received.

[0033] Understandably, this method is completely independent of the arrival order of data packets. Even if all fragmented packets arrive in arbitrary out-of-order, they can be directly located to the correct position within the frame based on the offset, naturally eliminating out-of-order problems caused by network jitter. It eliminates the need to allocate an independent out-of-order reassembly buffer for each stream, and also eliminates the need to maintain expected sequence numbers or perform packet sorting and waiting. Since each packet can be located to its final position immediately upon arrival, it avoids the blocking latency introduced by waiting for missing packets in traditional solutions. The reassembly latency is only limited by the memory copy operation itself, significantly improving real-time performance and memory utilization efficiency in high-throughput scenarios, making it particularly suitable for latency-sensitive embedded multi-channel video processing systems.

[0034] Figure 3 According to some embodiments of this application, a schematic diagram of an architecture with multiple data caching stages is shown.

[0035] Combination Figure 3 The multiple data caching stages shown are illustrated with examples. (Reference) Figure 3As shown, the video receiving unit includes a lock-free circular queue and a frame buffer pool. The lock-free circular queue is used to buffer UDP packets from multiple video streams, and the received UDP packets are buffered sequentially according to the receiving order. The frame buffer pool includes multiple frame buffers corresponding to each video stream, such as frame buffer E1 for video stream L1, frame buffer E2 for video stream L2, and frame buffer E3 for video stream L3. Unlike traditional solutions that require an additional out-of-order reassembly buffer before the frame buffer to sort out-of-order packets, in this embodiment, after the parsing and distribution thread retrieves the UDP packet from the lock-free circular queue, it directly writes the payload data to the specified position in the corresponding frame buffer through a memory copy operation based on the intra-frame offset carried in the extended field of the UDP packet, without needing an additional sorting and caching step.

[0036] Figure 4 According to some embodiments of this application, a schematic diagram of out-of-order UDP packet framing based on direct writing of intra-frame offsets to the buffer is shown. The following description uses the example of out-of-order arrival of UDP packets in video stream L1 due to network jitter.

[0037] The following is combined Figure 3 as well as Figure 4This example illustrates how out-of-order arrival of UDP packets from video stream L1 in frame buffer E1 due to network jitter can be used as an example. At the sending end, a frame of video stream L1 is divided into multiple fragments based on sequence number (seq), such as UDP packet seq=1, UDP packet seq=9, UDP packet seq=10, UDP packet seq=15, etc. Each UDP packet carries an intra-frame offset (CHUNK_OFFSET) in its extended field. For example, the packet with seq=1 carries an offset of offset=0, indicating that its payload should be written to the beginning of the frame buffer; the packet with seq=15 carries an offset of offset=30720, indicating that its payload should be written to the address at offset 30720 in the frame buffer. Assume that due to network jitter, the packets arrive at the video receiving unit in the order of seq=15, seq=9, seq=10, seq=1, i.e., out-of-order arrival. The receiving thread encapsulates these UDP packets sequentially according to their arrival order and writes them into a lock-free circular queue. After retrieving each data packet from the lock-free circular queue, the parsing and distribution thread extracts the video stream identifier to determine the corresponding target frame buffer and parses the extended fields to extract the intra-frame offset. For the first arriving packet (seq=15), the frame assembly thread directly writes the payload data into the frame buffer at offset 30720 based on its offset 30720; the subsequent packet (seq=9) is written into the frame buffer at offset 24064 based on its offset 24064; the packet (seq=10) is written into the frame buffer at offset 27392 based on its offset 27392; and the packet (seq=1) is written into the starting position at offset 0. The entire process is independent of the arrival order of the data packets; each packet can be immediately located to its final intra-frame position upon arrival without waiting for other packets, achieving zero-wait out-of-order reassembly.

[0038] Understandably, during the copying process, the sequence number (seq) does not participate in the sorting of data packets in the frame buffer, but it can be used to detect packet loss rate or record logs. Specifically, when the sequence number of a data packet does not appear in the statistics window, it is determined to be a packet loss and an alarm message is recorded to promptly investigate network link quality anomalies.

[0039] After each write operation, the frame assembly thread updates the fill status record (e.g., the bitmap filled_map), marking the bits corresponding to the written positions as filled and incrementing the total effective data size by filled_size. When filled_size reaches the preset total size of the frame, the data frame is considered a complete frame. In some implementations, the frame buffer size is consistent with the preset total frame size. Specifically, a 1920×1536 resolution YUV422 format image frame has a total size of approximately 5.9MB. The corresponding fill status record is a bitmap (filled_map), where each bit in the bitmap corresponds to a minimum storage unit in the frame buffer. For example, if each byte corresponds to 1 bit, then the bitmap size is approximately 737KB.

[0040] Understandably, after each copy or append of payload data, based on the starting offset of the payload data in the frame buffer and the data length, all bits in the corresponding range of the bitmap are set to 1, indicating that these positions have been filled with valid data. If the bit corresponding to the position to be written is already marked as 1, it indicates that the UDP packet is a duplicate packet (e.g., a packet retransmitted by the sender), and in this case, the payload data is discarded without increasing the total amount of valid data. In this way, the filling progress of the frame buffer can be accurately tracked with minimal memory overhead.

[0041] In other embodiments, for certain UDP packets that do not carry intra-frame offsets (e.g., senders in some simplified protocol implementations), when the parsing and distribution thread detects the absence of an intra-frame offset in the extended field, the frame assembly thread appends the payload data from that UDP packet to the tail of the current target frame buffer in arrival order. Specifically, the frame assembly thread maintains a tail pointer (tail_ptr) for the current frame buffer, which initially points to the start address of the frame buffer. After each append of payload data, the tail pointer moves backward by an offset equal to the length of the payload data. Simultaneously, the bits corresponding to the appended range in the bitmap are marked as filled, and the total amount of valid data is accumulated. This backward-compatible processing method allows the system to function correctly without requiring all senders to support the extended field, improving the compatibility of the solution.

[0042] Understandably, at this point, user-defined UDP packets can be directly processed to transmit video streams with large amounts of data, without relying on standard protocols such as RTSP / RTP. Furthermore, the transmitted video streams natively support YUV422 and the RAW8 BGGR Bayer format commonly used in industrial applications, thus expanding the application scenarios.

[0043] Understandably, in addition to displaying images, data frames can also be combined with algorithms for processing, and the specific application scenarios of data frames are not limited here.

[0044] Figure 5 According to some embodiments of this application, a flowchart of a multi-channel video stream data processing method is shown. The execution entity of this method can be a video transmission system. This system includes at least a video receiving unit and may also include a video sending unit; no limitation is made here. When multiple video streams from multiple cameras in the video sending unit are transmitted to the video receiving unit, the video receiving unit can receive and process the data. The specific process is as follows: S501 receives UDP data packets from each video stream.

[0045] Understandably, each camera outputs one video stream, and each video stream consists of continuous data frames. Each data frame is sliced ​​into multiple UDP packets for transmission. That is, each UDP packet corresponds to a unique video stream. Each UDP packet contains the stream identifier (stream_id) of its respective video stream and the corresponding sequence number (seq) from the slicing process. Additionally, each UDP packet's extended fields carry parameters such as the intra-frame offset (chunk_offset) of its payload data within its respective data frame, the number of samples (sample_count), and the stride, which inform the receiving end of the pixel arrangement of the frame.

[0046] In some embodiments, after receiving UDP packets from each video stream from the video acquisition and transmission unit, the video receiving unit encapsulates each UDP packet into a structure containing a video stream identifier (stream_id), a packet sequence number (seq), a tag indicating special attributes of the UDP packet, and payload data. This structure is then written into a lock-free ring queue in the order of reception, thereby achieving high throughput and low lock contention in the data buffer. The tag can be used to identify whether the UDP packet is a keyframe fragment, whether it carries extended fields, or whether it is a start-of-frame fragment or an end-of-frame fragment, among other special attributes.

[0047] S502 determines the target frame buffer corresponding to the UDP packet based on the video stream identifier.

[0048] Understandably, the video receiving unit contains a frame buffer pool. The number of frame buffers configured in the frame buffer pool corresponds to the number of current video streams, and each frame buffer is large enough to hold at least one complete frame of data. There is a one-to-one correspondence between frame buffers and video streams. Each frame buffer has a buffer identifier, which represents the corresponding video stream. The parsing and distribution thread reads the encapsulated data structure from the lock-free circular queue, parses the UDP packet header, extracts the sequence number (seq), relative timestamp (timestamp), and the number of extended fields, and matches the video stream identifier (stream_id) with the corresponding buffer identifier of each frame buffer to determine the target frame buffer corresponding to the UDP packet.

[0049] S503: Determine if an intra-frame offset exists in the UDP packet. If yes, proceed to S504; otherwise, proceed to S505.

[0050] The parsing and distribution thread further parses the extended fields of the UDP packet to determine whether the extended fields carry an intra-frame offset (CHUNK_OFFSET). In one embodiment, the extended fields are organized in TLV (Type-Length-Value) format, where the type field (Type) identifies the type of the extended field. For example, Type=0x01 indicates that the extended field carries an intra-frame offset, Type=0x02 indicates that it carries the number of sampling points, and Type=0x03 indicates that it carries step size information. The parsing and distribution thread traverses all extended fields, checking for the existence of a TLV item of type CHUNK_OFFSET.

[0051] S504 directly copies the payload data to the corresponding position in the target frame buffer according to the intra-frame offset.

[0052] When an intra-frame offset exists in a UDP packet, the frame assembly thread determines the starting address of the payload data in the target frame buffer based on the intra-frame offset. Specifically, the target address is calculated by adding the intra-frame offset (chunk_offset) to the base address (base_addr) of the target frame buffer. The payload data is then directly copied to this target address using a memory copy operation (e.g., memcpy). Since each UDP packet carries its absolute offset within its frame, regardless of the order in which fragments arrive, they can be directly written to the correct position within the frame, without needing to wait for other fragments to arrive before reordering and reassembling.

[0053] S505 appends the payload data to the end of the current target frame buffer in the order of arrival.

[0054] When there is no intra-frame offset in the UDP packet, the frame assembly thread appends the payload data to the end of the current target frame buffer in arrival order. Specifically, the frame assembly thread maintains a current write position pointer (write_ptr) for each frame buffer. When a packet without an offset is received, the payload data is written starting from the address pointed to by write_ptr. After writing is complete, write_ptr is moved forward by the offset of the payload data length. This method is compatible with traditional senders that do not support extended fields.

[0055] S506, Update the fill status record of the target frame buffer.

[0056] The fill status record is used to mark the positions in the target frame buffer that have been filled with valid data. After each copy (S504) or append (S505) of payload data, the frame assembly thread updates the fill status record. In one embodiment, the fill status record is a bitmap (filled_map), where each unit in the bitmap corresponds to a minimum storage unit in the frame buffer. Updating the fill status record specifically includes: after each copy or append of payload data, based on the starting offset of the payload data in the frame buffer and the data length, marking all bits in the corresponding range of the bitmap as filled (set to 1); if the bit corresponding to the position to be operated on has already been marked as filled (i.e., 1), then the current UDP packet is determined to be a duplicate packet, the payload data is discarded, and the total amount of valid data is not increased.

[0057] Understandably, the sender may retransmit previously sent UDP packets under certain circumstances, such as when the receiver fails to respond with an acknowledgment in a timely manner due to network congestion. By using a bitmap-based duplicate packet detection mechanism, the system can safely discard duplicate packets, preventing data from being written to the same location multiple times, which could lead to content overwriting or statistical errors, thus enhancing the system's robustness.

[0058] S507, count the total amount of valid data that has been filled in the target frame buffer after copying.

[0059] The frame assembly thread counts the total number of bytes filled with valid data in the target frame buffer based on the fill status record. In some embodiments, the total amount of valid data (filled_size) is maintained by a separate counter. After each successful write of payload data, filled_size is incremented by the length of the data written. In other embodiments, the total amount of valid data can also be obtained by traversing the bitmap and counting the number of bits marked as filled. However, the counter maintenance method is more efficient and suitable for high-throughput scenarios.

[0060] S508: Determine whether the total amount of valid data has reached the preset total frame size. If yes, proceed to S509; otherwise, return to S501 to continue receiving subsequent data packets.

[0061] The frame assembly thread compares the total amount of valid data filled in the current frame buffer with the preset total frame size (total_size). The total frame size can be obtained from the frame length field carried in the protocol header, or it can be preset in the configuration file during system initialization. When the total amount of valid data reaches the preset total frame size, i.e., filled_size ≥ total_size, it indicates that all fragments of the data frame have been correctly received and written to the frame buffer, and the data frame is determined to be a complete frame.

[0062] S509, confirm that the data frame is a complete frame.

[0063] Once a data frame is determined to be a complete frame, subsequent processing can be performed based on it. For example, it can be pushed into the frame buffer queue to wait for the display thread to consume it, or it can be directly handed over to the algorithm processing module for image analysis. Specific applications will not be elaborated here.

[0064] Understandably, the specific order of steps S501 to S509 is not restricted here, and the steps can be recombinated and split, which is also not restricted here.

[0065] Understandably, to prevent data packet and frame loss caused by network jitter from affecting system robustness, the video receiving unit continuously parses and copies the data packets to be copied from the lockless circular queue to the frame buffer queue. Regardless of whether the current frame is complete, the frame buffer corresponding to each video stream will continuously acquire data packets and update its filling status until the frame is determined to be complete.

[0066] In some embodiments, to prevent the frame buffer from being continuously occupied by incomplete data frames due to packet loss, which could lead to system crashes, a timeout cleanup mechanism can be introduced at the receiving end to periodically release frame buffers that have not been updated for a long time, while retransmitting data packets and confirming the reception status.

[0067] In some embodiments, in a screen display scenario, in order to control the latency of video data transmission and display, all frame data in the buffer queue can be discarded when the current frame buffer queue is full, so that new frames can be received and displayed in real time, achieving deterministic low latency at the cost of losing frames.

[0068] Figure 6 This diagram illustrates the process of frame dropping in a video display scenario. The specific process is as follows: S601, determine if the frame buffer queue is not full. If yes, proceed to S602; otherwise, proceed to S603.

[0069] In some embodiments, the dequeue rate of complete frames from the frame buffer queue may be less than the enqueue rate, leading to backlog in the finite-length frame queue. If the backlog reaches a preset queue depth threshold, blocking may occur, resulting in slower rendering of new frames and higher system latency. In other embodiments, incomplete frames may exist in the frame buffer queue, interrupting the frame data dequeue process and preventing the frame buffer queue from releasing stored frame data normally, thus causing frame buffer queue blocking and latency. To control latency, the depth of the frame buffer queue for each video stream can be limited (e.g., 3 frames) to a preset depth range. This preset depth range can be set based on the number of frames with visual distortion perceived by the human eye. When the frame buffer queue is determined to be full, completed frames in the frame buffer queue are actively discarded, allowing new frames to continue to be received. Since the depth of the frame buffer queue is small, the latency of the video transmission system can be reduced while avoiding visual distortion, achieving deterministic low latency at the cost of dropped frames.

[0070] Understandably, if the frame is confirmed to be complete (e.g., in S507 above), and the frame buffer queue is not full, there is still some redundant buffer space in the frame buffer queue compared to the currently full data frame. For example, with a pre-configured depth of 3 frames, there is at least 2 frames of redundant space. In this case, S602 can be entered to display based on the complete frame. If the frame is confirmed to be complete, but the frame buffer queue is full, it means there is not enough buffer space in the frame buffer queue to buffer subsequent new frame data. In this case, S603 needs to be entered to discard all data frames in the frame buffer queue.

[0071] S602 displays based on complete frames.

[0072] In some embodiments, the display module of the video receiving unit is provided with a display buffer queue; complete frame data is first pushed to the display buffer queue for buffering, and then the display module reads the frame data in the buffer queue to complete the screen output display.

[0073] In another embodiment, the display module does not integrate a local display cache queue; instead, the display module directly reads the complete frame image data cached in the frame buffer queue and outputs it for display.

[0074] S603 actively discards all data frames in the frame buffer queue.

[0075] Understandably, pre-setting an appropriate frame buffer queue depth and discarding all frames in the queue when it is full can reduce frame latency while ensuring data integrity—this is an intelligent latency and frame drop balancing mechanism. This avoids the problem of latency accumulation caused by blocking and waiting to ensure complete data, as seen in existing technologies.

[0076] The embodiments of this application can perform low-latency reception, out-of-order reassembly, frame-level splicing, and parallel real-time display of multiple, high-resolution, custom-format raw video streams in an unreliable UDP network transmission environment.

[0077] Furthermore, it is understandable that traditional solutions often employ global locking or serial processing when handling multiple independent video streams, leading to high CPU load, severe memory contention, and inability to fully utilize multi-core processors. This application also proposes another video stream data processing method for handling multiple video streams to improve processing efficiency. The following is a combination of... Figure 7 To elaborate further.

[0078] Figure 7 According to some embodiments of this application, a flowchart of another video stream data processing method is shown, which employs a multi-threaded parallel architecture to improve processing efficiency. The video transmission system allocates independent threads for UDP packet reception, UDP packet parsing and distribution, frame assembly, and frame data display, respectively. The specific process is as follows: S701, the UDP receiving thread receives UDP data packets from various video streams and writes each UDP data packet into a lock-free circular queue according to a preset structure.

[0079] In some embodiments, the video transmission system is configured with a dedicated UDP receiving thread. This thread receives UDP packets from various video streams and encapsulates each packet into a structure containing a video stream identifier (stream_id), packet sequence number (seq), tag, and payload data. The data is then written to a lock-free circular queue in the order of reception to achieve high throughput and low lock contention in the data buffer. The lock-free circular queue uses atomic operations to synchronize producers and consumers, avoiding the context switching overhead of traditional mutex locks. The specific process of the UDP receiving thread receiving UDP packets can be found in S501 above and will not be elaborated upon here.

[0080] S702, the parsing and distribution thread distributes the parsed UDP packets to the frame buffer corresponding to the video stream.

[0081] The video transmission system is configured with at least one parsing and distribution thread dedicated to parsing and distributing data packets. The parsing and distribution thread reads the encapsulated data structure from a lock-free circular queue, parses the UDP packet header, and extracts the sequence number (seq), relative timestamp, and the number of extended fields. It then parses the extended fields, extracting the sample count (sample_count) and stride to inform the receiver of the frame's pixel arrangement, and determines whether the extended fields carry an intra-frame offset (CHUNK_OFFSET). Finally, it determines the corresponding frame buffer based on the video stream identifier and distributes the detected intra-frame offset and payload data to the frame assembly thread.

[0082] In some embodiments, when the parsing and distribution thread detects that the current UDP packet is the first fragment of a frame (e.g., identified by the Start of Frame (SOF) flag), it also creates or resets the corresponding frame buffer to prepare for receiving the data of that frame. When it detects that the current UDP packet is the last fragment of a frame (e.g., identified by the End of Frame (EOF) flag), the frame may meet the integrity condition after the fragment is written.

[0083] S703, the frame assembly thread writes the payload data of UDP packets into the frame buffer based on the intra-frame offset.

[0084] In some embodiments, the video transmission system configures at least one independent frame assembly thread specifically for assembling frame data. The frame assembly thread receives the intra-frame offset and payload data from the parsing and distribution thread, and directly copies the payload data to the corresponding position in the target frame buffer according to the intra-frame offset. After copying, the frame assembly thread updates the fill status record of the target frame buffer, counts the total amount of valid data filled, and determines whether the total amount of valid data reaches a preset total frame size to determine whether the data frame is a complete frame.

[0085] Understandably, the frame combining thread can be configured with a fill status record and a valid data total counter for each video stream to independently track the frame combining progress of each video stream. The frame combining operations between the video streams do not interfere with each other, naturally achieving parallel processing of multi-channel data.

[0086] Understandably, the specific process can be referred to in S503 to S505 above, and will not be repeated here.

[0087] S704, the rendering thread and the display thread render and display the images of various video streams in parallel.

[0088] In some embodiments, the video transmission system creates a separate set of rendering threads and display threads for each video stream, with each thread bound to a dedicated finite-depth frame buffer queue. Understandably, the rendering and display threads execute in parallel; for example, while the display thread is displaying frame 1, the rendering thread can render frame 2. The rendering thread reads the data of a complete frame from its associated frame buffer queue and sends the data frame to the display module, where the final image output is completed. Exemplary implementations include pushing frame data to the ffplay player for rendering via a data pipeline, and implementing hardware-accelerated rendering based on the SDL graphics library and OpenGL graphics interface.

[0089] In some embodiments, the system supports adaptive adaptation to multiple raw pixel formats, and can recognize and process image data such as YUV422 (16-bit) and RAW8 BGGR Bayer format. For RAW8 BGGR Bayer format, two output modes are supported: one is to display it directly as a grayscale image, and the other is to convert the Bayer raw data into an RGB image through a fast demosaic algorithm before outputting it for display.

[0090] Understandably, existing general-purpose players and display links (such as ffplay) cannot directly support non-standard formats (such as RAW8 BGGR Bayer format and other original video formats) output by devices such as industrial cameras.

[0091] S705, real-time screen display.

[0092] Understandably, by configuring different independent threads for different stages of the video data processing workflow—that is, adopting a model of "independent UDP receiving thread + independent parsing and distribution thread + independent frame combination thread + multi-channel parallel rendering and display thread"—it naturally adapts to multi-core CPUs, avoids global lock contention, and significantly reduces CPU load during multi-channel processing. Furthermore, these threads can be combined for execution; for example, the independent parsing and distribution thread and the frame combination thread can be the same thread, and the rendering and display threads can also be executed by the same thread. The specific configuration of different independent threads for different stages of the video data processing workflow can be arbitrarily set as needed, and this application does not impose any restrictions.

[0093] Furthermore, in some embodiments, all modules of this application can adopt standard C++ libraries and lightweight synchronization mechanisms, without external heavy framework dependencies, and are easy to deploy and port on different platforms such as embedded Linux and x86.

[0094] Understandably, in the embodiments of this application, the low-latency control mechanism of "limited frame buffer queue tolerance + active frame dropping" limits the frame buffer queue depth of each stream and actively discards all frames when the queue is full, instead of blocking and waiting, thereby achieving deterministic and predictable low-latency real-time display in unreliable network transmission.

[0095] Furthermore, the system natively integrates the recognition, caching, optional de-mosaicing, and direct grayscale display capabilities for RAW8 BGGR Bayer format data within the UDP packet reception, reassembly, and display chain, enabling end-to-end real-time display processing. Moreover, it can separate data reception, protocol parsing, and complex frame reassembly into a pipelined processing structure, maximizing data throughput.

[0096] The multi-channel video stream data processing method proposed in this application embodiment can avoid the problems of out-of-order delivery, high latency, high CPU load, poor multi-channel scalability, and weak format compatibility in the reception, reconstruction, and display of multi-channel, high-resolution, custom-format video streams transmitted based on UDP.

[0097] Figure 8 This is a structural block diagram of an electronic device provided in an embodiment of this application. The electronic device may include: a memory 110, a processor 120, and a communication interface 130. The memory 110, processor 120, and communication interface 130 are connected via internal connection paths. The memory 110 is used to store computer programs; in some implementations, the computer program may include code for implementing the methods of the embodiments of this application.

[0098] The processor 120 executes the computer program stored in the memory 110 to control the communication interface 130 to receive input data and information, and output operation results and other data. In some implementations, when the solutions of the embodiments of this application are implemented by software or firmware, the computer program used to implement the solutions of the embodiments of this application can be executed by the processor 120.

[0099] The memory 110 may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory may be random access memory (RAM). It should be noted that the memory 110 described herein is intended to include, but is not limited to, any memory of these and other suitable types. As an example, the memory 110 includes random access memory (RAM), cache memory, and read-only memory (ROM). The memory 110 stores a computer program that can be executed by processor 120, causing processor 120 to implement the steps of any of the methods described above.

[0100] This application also provides a computer-readable medium storing instructions that, when executed on a server, cause the server to perform the multi-channel video stream data processing method mentioned in any of the above embodiments of this application.

[0101] This application also provides a computer program product, including: a computer program / instructions, which, when executed by a processor, implement the multi-channel video stream data processing method mentioned in any of the above embodiments of this application.

[0102] It should be noted that all units / modules mentioned in the device embodiments of this application are logical units / modules. Physically, a logical unit / module can be a physical unit / module, a part of a physical unit / module, or a combination of multiple physical units / modules. The physical implementation of these logical units / modules themselves is not the most important factor; the combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed in this application. Furthermore, to highlight the innovative aspects of this application, the above-described device embodiments of this application have not introduced units / modules that are not closely related to solving the technical problems proposed in this application. This does not mean that the above-described device embodiments do not contain other units / modules.

[0103] It should be noted that, in the examples and description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0104] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for processing multi-channel video stream data, characterized in that, Applied to a video transmission system configured with a frame buffer independently maintained for each video stream, the method includes: Receive UDP data packets generated from each video stream segment; The target frame buffer corresponding to the UDP packet is determined based on the video stream identifier in the UDP packet; If the UDP packet contains an intra-frame offset corresponding to its data frame, the payload data in the UDP packet is directly copied to the corresponding position in the target frame buffer according to the intra-frame offset. Update the fill status record of the target frame buffer, the fill status record being used to mark the positions in the target frame buffer that have been filled with valid data; The total amount of valid data that has been filled into the target frame buffer after copying is calculated based on the filling status record. When the total amount of valid data reaches the preset total frame size, the data frame is determined to be a complete frame.

2. The method according to claim 1, characterized in that, The method further includes: If the intra-frame offset is not present in the UDP packet, the payload data in the UDP packet is appended to the end of the current target frame buffer in the order of arrival.

3. The method according to claim 1, characterized in that, The fill state record is a bitmap, and each unit in the bitmap corresponds to a minimum storage unit in the frame buffer. Updating the fill state record of the target frame buffer includes: After each copy or append of the payload data, the unit at the corresponding position in the bitmap is marked as filled. If the unit corresponding to the location to be operated on has been marked as filled, then the UDP packet is determined to be a duplicate packet, the payload data is discarded and the total amount of valid data is not increased.

4. The method according to claim 1, characterized in that, The video transmission system further configures a frame buffer queue for each video stream, the frame buffer queue being used to buffer complete frames obtained from the frame buffer, the depth of the frame buffer queue being a preset first number; and the method further includes: Once it is determined that the data frame is a complete frame, it is determined whether the frame buffer queue corresponding to the complete frame is full. If the frame buffer queue corresponding to the complete frame is not full, the complete frame is enqueued into the frame buffer queue and waits for the display thread to consume it; If the frame buffer queue corresponding to the complete frame is full, then all the complete frames already cached in the frame buffer queue corresponding to the complete frame are discarded, and the current complete frame is enqueued.

5. The method according to claim 1, characterized in that, The intra-frame offset is carried by an extended field in the UDP packet, and the total frame size is obtained from the protocol header or a preset configuration.

6. The method according to claim 1, characterized in that, The process of receiving UDP data packets generated from each video stream segment includes: The system receives UDP packets from various video streams and encapsulates each UDP packet into a structure containing the video stream identifier, a tag indicating the special attributes of the UDP packet, and the payload data, and writes it into a lock-free circular queue.

7. The method according to claim 1, characterized in that, The payload data in the UDP packet is in raw pixel format, and the method further includes: Identify the pixel format type of the payload data; Based on the identified pixel format type, select the corresponding display adaptation method for output display.

8. The method according to claim 1, characterized in that, The method employs a multi-threaded parallel architecture, which includes: At least one UDP receive thread is used to receive the UDP packets and write them to a lock-free circular queue; At least one parsing and distribution thread is used to read the UDP packets from the lock-free circular queue and perform parsing and distribution operations; At least one frame combination thread is used to perform copy or append operations, as well as fill state updates and frame integrity determination; Multiple independent rendering and display threads are used, with one rendering thread and one display thread corresponding to each video stream, for rendering and displaying complete frames.

9. The method according to claim 8, characterized in that, The at least one parsing and distribution thread is used to read the UDP packets from the lock-free circular queue and perform parsing and distribution operations, including: The at least one parsing and distribution thread is used to read the encapsulated data structure from the lock-free circular queue, parse the transmission header of the UDP packet, and extract the sequence number, relative timestamp, and number of extended fields. The extended field is parsed to extract the number of sampling points and the step size, which are used to inform the receiving end of the pixel arrangement of the frame; Determine whether the extended field carries the intra-frame offset; The corresponding frame buffer is determined based on the video stream identifier, and the detected intra-frame offset and the payload data are distributed to the frame combination thread.

10. An electronic device comprising a processor and a memory, the memory storing computer programs / instructions, characterized in that, When the computer program / instructions in the memory are executed by the processor, the multi-channel video stream data processing method as described in any one of claims 1-9 is implemented.

11. A computer program product, comprising a computer program / instructions, characterized in that, When a computer program / instruction is executed by a processor, it implements the multi-channel video stream data processing method as described in any one of claims 1-9 above.

Images