Method and system for processing 4K240 frame video stream

Abstract

The invention discloses a 4K240 frame video stream processing method and a 4K240 frame video stream processing system. The method comprises the following steps: receiving 4K, 240fps original video frames output by an image sensor, and generating frame level identification information containing a frame sequence number and odd-even labels for each frame; caching the original video frame to an external memory, and allocating the original video frame to a first processing channel and a second processing channel in a time sequence in a staggered manner according to odd-even tags; writing the frame level identification information into a user-defined data area of video frame data output by a channel to form two paths of frame data streams, and respectively carrying out image processing and video coding; and after coding, sending to a server, rearranging and interlacing and merging by the server according to the frame sequence number, and outputting a 4K240fps video stream. The system comprises an image sensor, a programmable logic device, a double-channel video processing unit and a server. Through parallel processing of frame-level identification and staggered shunting, high-frame-rate load sharing and sequence reliable recovery are realized, and the real-time performance and the transmission stability are improved.

Description

Technical Field

[0001] This application relates to the field of ultra-high-definition video acquisition and transmission technology, and in particular to a 4K 240 frame video stream processing method and system. Background Technology

[0002] With the increasing demand for high-speed motion detail capture capabilities in film and television production, sports events, scientific imaging, security monitoring, medical education, and VR / AR scenarios, 4K resolution combined with high frame rate video acquisition and network transmission is gradually becoming an important development direction for high-end network cameras. In a typical video link of a network camera, the raw video frame data usually needs to undergo image processing and encoding before it can be output as a transmittable video stream. This includes processing steps such as de-pixelation, white balance, noise reduction, cropping, and encoding.

[0003] However, limited by the computing power, storage bandwidth, and encoding / decoding throughput of existing dedicated image processing chips—for example, a single SoC integrating an ISP and a video encoder—often struggles to simultaneously meet the combined requirements of 4K resolution, 240 frames per second, and low-latency encoding output. For instance, some common market solutions offer encoding capabilities roughly equivalent to 8K 30 frames per second or 4K 120 frames per second, but further increasing to 4K 240 frames per second easily encounters bottlenecks in encoding throughput, memory access, interface bandwidth, system power consumption, and cost, making it difficult for the entire system to achieve the target specifications.

[0004] To improve ultra-high-definition and high-frame-rate processing capabilities, existing technologies have explored ways to increase system throughput through multi-channel parallel processing and parallel transmission. For example, in 8K 120P ultra-high-definition video scenarios, some solutions propose using multiple high-speed serial interfaces to transmit video frame data in parallel and then merging and outputting them on the server side to achieve smoother processing at higher frame rates or higher resolutions. However, such solutions are often geared towards specific playback and transmission links and generally have dedicated hardware interfaces. In practical applications, they may still face challenges such as timing consistency of multi-channel data, frame order restoration, alignment and reassembly under network jitter, and continuous stable output.

[0005] Therefore, there is an urgent need for a new processing solution to achieve stable processing and output of 4K 240fps video streams in the actual network camera link in an engineering-featured, cost-controllable, and scalable manner. Summary of the Invention

[0006] The embodiments of this application provide a 4K240 frame video stream processing method and system to solve the problem that a single image processing chip cannot handle the stable output of a 4K240 frame video stream.

[0007] To address the aforementioned technical problems, embodiments of this application disclose the following technical solutions: On one hand, a method for processing a 4K 240fps video stream is provided, comprising: acquiring raw video frame data at 4K resolution and 240 frames per second output by an image sensor; generating frame-level identification information for each of the raw video frames in a programmable logic device, wherein the frame-level identification information includes at least a frame sequence number and a parity tag corresponding to the frame sequence number; caching the raw video frame data to an external memory, and interleaving the raw video frames in chronological order to a first processing channel and a second processing channel according to the parity tag, such that any raw video frame is assigned to only the first processing channel or the second processing channel, and the first processing channel and the second processing channel are interleaved. The receiving frame rate of each channel is 120 frames per second; the frame-level identification information is written into the preset user-defined data area of ​​the video frame data output to the first processing channel and the second processing channel to form the first frame data stream and the second frame data stream, respectively; image processing and video encoding are performed on the first frame data stream and the second frame data stream, respectively, to obtain the first encoded data stream and the second encoded data stream; the first encoded data stream and the second encoded data stream are packaged into network data frames carrying the frame sequence number and sent to the server; based on the frame sequence number, the first encoded data stream and the second encoded data stream are rearranged and interleaved to output a 4K 240 frame video stream.

[0008] Furthermore, the parity tag is used to characterize the parity attribute of the frame sequence number, and the step of interleaving the original video frames to the first processing channel and the second processing channel according to the parity tag includes: allocating the original video frames with the parity tag of the first value to the first processing channel and allocating the original video frames with the parity tag of the second value to the second processing channel; or, allocating the original video frames with the parity tag of the first value to the second processing channel and allocating the original video frames with the parity tag of the second value to the first processing channel.

[0009] Furthermore, writing the frame-level identifier information into the preset user-defined data area includes: adding a preset number of user-defined data rows to the video frame data output to the first processing channel and the second processing channel according to a preset encapsulation format, and writing the frame sequence number, the parity tag, and the timestamp into the user-defined data rows.

[0010] Furthermore, the step of caching the original video frame data to external memory and the interleaving allocation includes: separating the payload data in the original video frame data from the protocol control information used for interface protocol encapsulation on the input interface side of the programmable logic device, writing the payload data into the external memory, and recording the protocol control information into a cache queue; reconstructing the interface protocol encapsulation of the payload data read from the external memory according to the protocol control information recorded in the cache queue on the output interface side of the programmable logic device, and continuously outputting the payload data corresponding to the first processing channel to the first processing channel and continuously outputting the payload data corresponding to the second processing channel to the second processing channel; wherein, the protocol control information includes at least control information for frame delimitation and / or channel alignment.

[0011] Furthermore, the cache queue is a first-in-first-out (FIFO) queue, and the protocol control information is recorded line by line in the FIFO queue. The reconstructing interface protocol encapsulation includes: reading the FIFO queue line by line, determining whether the current data line to be output is a user-defined data line based on the line type identifier in the protocol control information; when it is determined that the current data line to be output is a user-defined data line, the output interface side writes the frame sequence number, the parity tag, and / or timestamp into the current user-defined data line to be output; when it is determined that the current data line to be output is not a user-defined data line, the output interface side reads the corresponding payload data line from the external memory line by line through the bus interface and encapsulates it in conjunction with the protocol control information, while inserting control information for frame delimitation and / or channel alignment to achieve continuous output.

[0012] Furthermore, the timestamp is generated based on the synchronization time provided by the precision time protocol and is used for frame alignment and order restoration in the rearrangement and interleaving merging.

[0013] Furthermore, the first frame data stream is output to the first processing channel through a Scalable Low Voltage Signal Embedded Clock Interface, and the second frame data stream is output to the second processing channel through a Low Voltage Differential Signal Interface; the second frame data stream output by the Low Voltage Differential Signal Interface is in RAW10 format, which is formed by reassembling data read from the external memory.

[0014] On the other hand, a 4K 240fps video stream processing system is provided, comprising: an image sensor for outputting raw video frame data at 4K resolution and 240 frames per second; a programmable logic device having an input interface and an output interface, the input interface being coupled to the image sensor to receive the raw video frame data, and the output interface being coupled to a first processing channel and a second processing channel to transmit a first frame data stream and a second frame data stream respectively; a first video processing unit and a second video processing unit, respectively coupled to the first processing channel and the second processing channel, for performing image processing and video encoding on the first frame data stream and the second frame data stream respectively, to obtain a first encoded data stream and a second encoded data stream; a network transmission module for packaging the first encoded data stream and the second encoded data stream into network data frames carrying frame numbers and sending them to a server; and a server for processing the first encoded data stream and the second encoded data stream based on the frame numbers. The second encoded data stream is rearranged and interleaved to output a 4K 240 frame video stream. The programmable logic device (PLD) is configured to generate frame-level identification information for each original video frame and cache the original video frame data in external memory. The frame-level identification information includes at least a frame sequence number and a parity tag corresponding to the frame sequence number. The PLD allocates the original video frames to the first processing channel and the second processing channel in chronological order based on the parity tag, such that any original video frame is allocated to only the first processing channel or the second processing channel, and the frame rate received by both the first and second processing channels is 120 frames per second. The PLD is also configured to write the frame-level identification information into a preset user-defined data area of ​​the video frame data output to the first and second processing channels to form the first frame data stream and the second frame data stream, respectively.

[0015] Furthermore, the preset user-defined data area includes a preset number of user-defined data rows added according to a preset encapsulation format; the programmable logic device is configured to write the frame number, the parity tag, and the timestamp into the user-defined data rows, the timestamp being generated based on the synchronization time provided by the precision time protocol; the programmable logic device receives the original video frame data and outputs the first frame data stream through a scalable low-voltage embedded clock interface, and outputs the second frame data stream through a low-voltage differential signal interface.

[0016] Furthermore, the programmable logic device is configured to: separate the payload data in the original video frame data from the protocol control information used for interface protocol encapsulation on the input interface side, write the payload data into the external memory, and record the protocol control information into a buffer queue; on the output interface side, reconstruct the interface protocol encapsulation of the payload data read from the external memory according to the protocol control information recorded in the buffer queue, and continuously output the payload data corresponding to the first processing channel to the first processing channel and continuously output the payload data corresponding to the second processing channel to the second processing channel; wherein, the protocol control information includes at least control information for frame delimitation and / or channel alignment.

[0017] The above-mentioned technical solutions have at least the following beneficial effects: After acquiring raw video frame data at 4K resolution and 240 frames per second, frame-level identification information containing frame sequence number and its parity tag is generated for each raw video frame. Based on the parity tag, the raw video frames are interleaved and allocated to the first processing channel and the second processing channel in chronological order, so that the two channels process in parallel at a frame rate of 120 frames per second. Thus, without relying on a single chip to directly complete the 4K240 frame processing and encoding capability, the overall processing capability is improved by using multi-channel parallel processing. At the same time, frame-level identification information is written into the preset user-defined data area of ​​the video frame data output to the two processing channels, so that subsequent encoding, network transmission, and server-side rearrangement and interleaving have a clear frame-level order basis. Frame order recovery and stable merged output can be achieved under multi-processing and transmission links, thereby achieving the 4K240 frame video stream output capability with lower hardware cost and system modification complexity.

[0018] The above-mentioned technical solutions also have the following advantages: By separating the payload data in the original video frame data from the protocol control information used for interface encapsulation, and writing them into external memory and a first-in-first-out queue recorded by line respectively, and performing line-level encapsulation reconstruction and continuous output on the output side according to the protocol control information, it can take into account caching, scheduling and real-time encapsulation under high-bandwidth video data links, reduce the pressure on internal storage and processing and improve output continuity and protocol consistency; when determining user-defined data lines, frame sequence numbers, parity tags and / or timestamps can be inserted line by line, so that frame-level identification and synchronization information can be transmitted with the data frame, which is convenient for alignment and order recovery between multiple channels; in addition, by using scalable low-voltage signal embedded clock interface and low-voltage differential signal interface to output two data channels respectively, and configuring high-speed serial transceiver resources under resource-constrained conditions to achieve a transceiver mode with inconsistent receiving and sending rates, it can reduce the occupation of high-speed serial transceiver resources, improve the flexibility of device selection and cost control while meeting the dual-channel bandwidth requirements; and by reformatting some data to adapt to different interface output requirements, it enhances the system's adaptability to the interface side of sensors and subsequent processing units. Attached Figure Description

[0019] Figure 1 A flowchart illustrating a method for processing a 4K 240fps video stream provided in this application; Figure 2 A block diagram of a 4K 240fps video stream processing system provided in this application; Figure 3 This is a schematic diagram of the internal functional structure of a programmable logic device provided in this application. Detailed Implementation

[0020] To facilitate a better understanding of the technical solutions of this invention by those skilled in the art, some terms used in this specification are explained below. It should be understood that the explanations of the following terms are for description purposes only and do not constitute a substantial limitation on the scope of protection of this application.

[0021] In the description of this application, the terms "comprising," "including," and "having" are open-ended expressions, indicating that the object possesses at least the listed elements, but not excluding the possibility of possessing other elements. The terms "at least one" and "multiple" are used to describe a range of quantities; "at least one" means one or more, and "multiple" means two or more. The terms "first," "second," and "third," etc., are used only to distinguish identical or similar technical elements and do not indicate a specific quantity, order, or degree of importance. The terms "coupled," "connected," and "linked" should be interpreted broadly, encompassing electrical, data, or communication connections, as well as direct connections or indirect connections via an intermediate medium; unless otherwise explicitly defined, they should not be construed as limited to a specific connection method. The term "based on" indicates that it is based on at least one factor or condition, and does not exclude the combination of other factors or conditions. The term "preset" indicates parameters, rules, or structures that can be configured before factory delivery, deployment, or operation, or dynamically configured during operation, and does not limit the timing or method of configuration. The terms "may / can / can" indicate the possibility of implementation and do not constitute a limitation on the necessity of adopting a particular method or achieving a particular effect. Terms such as “exemplary” and “for example” are used for illustration and do not constitute a limitation on the scope of protection.

[0022] The following provides further clarification on the technical terms used in this application. It should be understood that the following explanations are intended to standardize the meaning and interpretation of the terms and are not intended to limit the scope of protection.

[0023] In this application, "4K" refers to video / image data that reaches 4K pixel resolution specifications, which generally corresponds to a pixel matrix of 3840×2160 or 4096×2160; "frame" refers to a single image unit in a video sequence; "frame rate" refers to the number of frames output / transmitted / processed per unit time, usually expressed in frames per second (fps); "video stream" refers to a video data sequence organized continuously over time, which can be presented in different data organization and encapsulation forms during the acquisition, transmission, processing, or encoding stages.

[0024] In this application, "raw video frame data" refers to video frame data output by an image sensor that has not yet been compressed and encoded, which may be raw pixel data or frame data with interface encapsulation; "video encoded data / bitstream" refers to the data sequence compressed by a video encoder, and its encoding standard may be H.264, H.265, or other equivalent standards; "image processing" refers to preprocessing or enhancement processing performed on the raw pixel data, typically including but not limited to noise reduction, color correction, sharpening, distortion correction, and other operations.

[0025] In this application, "Field Programmable Gate Array (FPGA)" refers to a device that can be configured to implement specific hardware logic functions for high-speed data access, cache scheduling, protocol parsing and encapsulation output, etc.; "External Memory" refers to memory resources connected to the programmable logic device for data caching; "Double Data Rate 4 Synchronous Dynamic Random Access Memory (DDR4)" refers to a typical implementation of DDR4 type external memory.

[0026] In this application, "frame-level identification information" refers to a set of identification fields corresponding to video frames, used for frame identification, association, and sequential management; "frame sequence number" refers to a sequence number field that represents the order of frames, usually increasing or cyclically increasing according to a preset rule; "odd / even tag" refers to a tag field associated with the odd / even attribute or mapping attribute of the frame sequence number, used to realize the division and management of frames among different channels / resources; "timestamp" refers to a time information field that represents a certain moment, which can be used for alignment, timing, or sequence assistance.

[0027] In this application, "Precision Time Protocol (PTP)" refers to a protocol system used to achieve time synchronization of multiple nodes, and common implementations can conform to standards such as IEEE 1588; "One Pulse Per Second (1PPS)" refers to a time reference pulse signal output once per second, which can be used as one of the sources of time synchronization or timing reference.

[0028] In this application, "Scalable Low Voltage Signaling with Embedded Clock (SLVS-EC)" refers to a high-speed serial interface form with an embedded clock, commonly used for high-speed data interconnection between image sensors and receivers; "Low Voltage Differential Signaling (LVDS)" refers to a low-voltage high-speed signaling technology using differential pair transmission; "RAW10" refers to a raw image data representation with an effective pixel bit width of 10 bits; and "processing channel" refers to a logical or physical path that carries data transmission and processing, which can correspond to different interface links, different processing units, or different input / output paths.

[0029] In this application, "payload" refers to the main business data portion carried in the encapsulated data unit, distinguished from control / synchronization fields; "protocol control information" refers to the set of control fields used to describe data encapsulation boundaries, alignment, synchronization, or control behavior, the specific content of which may vary depending on the interface encapsulation rules; "First-In-First-Out (FIFO) queue" refers to a data buffer structure that reads data in the order it is written, which can be used for rate matching, temporary storage, and scheduling; "row type identifier" refers to an identifier field used to distinguish different types of data rows (e.g., user-defined data rows and ordinary data rows), the field format and value rules of which can be set according to the encapsulation convention or engineering implementation; "user-defined data area / user-defined data row" refers to a writable area / data row reserved or extended within the scope allowed by the established encapsulation format, used to carry user-defined fields.

[0030] In this application, "Advanced eXtensible Interface 4 (AXI4)" refers to the on-chip interconnect bus protocol in the AMBA architecture, used for data transmission and memory access between modules; "Interconnect" refers to the bus interconnect structure used for multi-master multi-slave connections, arbitration, and bandwidth allocation; "Network data frame" refers to the encapsulated data unit used for network transmission, which can be an Ethernet frame or the upper-layer protocol data unit carried by it; "Ethernet" refers to a commonly used network communication technology system; "RJ45 (Registered Jack 45)" refers to a common Ethernet physical interface form; and "Server" refers to the software and hardware platform used to carry out data reception, caching, processing, or distribution, which can be a general-purpose computing platform or a dedicated computing platform.

[0031] Figure 1 A flowchart illustrating a method for processing a 4K 240fps video stream provided in this application. Figure 1As shown, the exemplary 4K 240fps video stream processing method provided in this application includes the following steps: S1, acquiring raw video frame data at 4K resolution and 240 frames per second output by an image sensor; S2, generating frame-level identification information for each raw video frame in a programmable logic device, the frame-level identification information including at least a frame sequence number and a parity tag corresponding to the frame sequence number; S3, buffering the raw video frame data into an external memory, and allocating the raw video frames to a first processing channel and a second processing channel in chronological order according to the parity tag, such that any raw video frame is allocated only to the first processing channel or the second processing channel, and the data received by the first processing channel and the second processing channel are processed in a manner that is consistent with each other. The frame rate is 120 frames per second. S4: Write the frame-level identification information into the preset user-defined data area of ​​the video frame data output to the first and second processing channels to form the first frame data stream and the second frame data stream, respectively. S5: Perform image processing and video encoding on the first and second frame data streams respectively to obtain the first encoded data stream and the second encoded data stream. S6: Pack the first and second encoded data streams into network data frames carrying frame sequence numbers and send them to the server. S7: Rearrange and interleave the first and second encoded data streams based on the frame sequence numbers to output a 4K240 frame video stream. In some embodiments, the network data frames can be sent to the server via an Ethernet link. For example, the first and second video processing units are connected to an Ethernet module via a Reduced Gigabit Media Independent Interface (RGMII), and the Ethernet module is connected to the server-side network via an RJ45 physical interface.

[0032] Figure 2 A block diagram of a 4K 240fps video stream processing system provided in this application. Figure 2As shown, the exemplary 4K 240fps video stream processing system 10 provided in this application includes: an image sensor 11 for outputting raw video frame data at 4K resolution and 240 frames per second; a programmable logic device 12 having an input interface and an output interface, wherein the input interface is coupled to the image sensor 11 to receive raw video frame data, and the output interface is coupled to a first processing channel 121 and a second processing channel 122 to transmit a first frame data stream and a second frame data stream respectively; a first video processing unit 13 and a second video processing unit 14, respectively coupled to the first processing channel 121 and the second processing channel 122, for performing image processing and video encoding on the first frame data stream and the second frame data stream respectively, and outputting a first encoded data stream and a second encoded data stream; a network transmission module 15 for packaging the first encoded data stream and the second encoded data stream into network data frames carrying frame sequence numbers and sending them to a server 16; and a server 16 for rearranging and interleaving the first encoded data stream and the second encoded data stream based on the frame sequence numbers, and outputting a 4K 240fps video stream. In this embodiment, the programmable logic device 12 is configured to generate frame-level identification information for each original video frame and cache the original video frame data in an external memory. The frame-level identification information includes at least a frame sequence number and a parity tag corresponding to the frame sequence number. The programmable logic device 12 allocates the original video frames to the first processing channel 121 and the second processing channel 122 in chronological order according to the parity tag, so that any original video frame is allocated to only the first processing channel 121 or the second processing channel 122, and the frame rate received by the first processing channel 121 and the second processing channel 122 is 120 frames per second. In addition, in this embodiment, the programmable logic device 12 is also configured to write the above-mentioned frame-level identification information into a preset user-defined data area of ​​the video frame data output to the first processing channel 121 and the second processing channel 122, so as to form the first frame data stream and the second frame data stream, respectively. In some embodiments, the first video processing unit 13 and the second video processing unit 14 may be two video processing system-on-a-chips, used for image processing and encoding of two channels of frame data in parallel; the network transmission module 15 may be an Ethernet module, communicating with the first video processing unit 13 and the second video processing unit 14 through the RGMII interface, and transmitting data with the server 16 through the RJ45 physical interface.

[0033] Figure 3 This is a schematic diagram of the internal functional structure of a programmable logic device provided in this application. Figure 3As shown, the programmable logic device 12 may include an input receive link, a buffer and interconnect link, a dual-output transmit link, and functional modules such as clock reset, time base, and configuration control. The input receive link may include a high-speed serial transceiver module GT_TRANS_RX, a receive alignment module GT_RX_ALIGN, and a Scalable Low Voltage Signal Embedded Clock (SLVS) receive parsing module SLVS_RX, used for transmitting data via a Scalable Low Voltage Signal Embedded Clock (SLVS) interface (e.g., ...). Figure 3 The SLVS*8 receives raw video frame data from the image sensor side and performs link access, alignment, and protocol parsing processing on the raw video frame data. It then outputs frame data and related control information that can be used for buffering to the internal AXI4 side. The buffering and interconnection links may include the AXI interconnect module AXI_INTERCONNECT, the external memory control module DDR4_CTRL, and external memory DDR4, used to achieve rate matching and data buffering between the input and output sides, and to provide on-demand data read / write access for the dual-output links; wherein, Figure 3 The connections marked with AXI4 are used to indicate the data interaction interface types between modules. The specific number of read / write channels, arbitration strategy, and cache depth can be configured according to bandwidth requirements. The dual-output transmission link may include a Scalable Low Voltage Signal Embedded Clock Transmitter Encapsulation module SLVS_TX and a high-speed serial transceiver module GT_TRANS_TX, as well as a Low Voltage Differential Signal Transmitter Encapsulation module LVDS_TX and a Low Voltage Differential Signal Interface module LVDS_INTERFACE, used to reconstruct and encapsulate frame data read from external DDR4 memory according to a preset encapsulation format and output it continuously. SLVS_TX and GT_TRANS_TX can work together to achieve transmission output via the Scalable Low Voltage Signal Embedded Clock Interface to form the first frame data stream for the first processing channel 121; LVDS_TX and LVDS_INTERFACE can work together to achieve transmission output via the Low Voltage Differential Signal Interface (e.g., ...). Figure 3 The output of lvds*16 is sent to form a second frame data stream for the second processing channel 122. It should be understood that the interface combination and parameter configuration of the above two output links are exemplary implementations, and can be adapted and set according to the interface capabilities of the subsequent processing units in actual projects.

[0034] In addition, the programmable logic device 12 may also include a CLK / RST module for providing operating clocks and performing reset control for the input receive link, buffer and interconnect links, and dual output transmit link, so that each functional module enters a preset initial state upon power-on, reconfiguration, or abnormal recovery; to highlight the main data paths, Figure 3The wiring for CLK / RST to distribute clock and reset signals to each module is not shown. The programmable logic device 12 may also include a timestamp generation module TIMER_GEN, which can receive time base signals such as 1pps / PTP and generate timestamps for frame alignment and sequence recovery; and may include a configuration register module SPI_REG for configuring the splitting strategy, user-defined data area write strategy, output interface parameters, and timestamp-related parameters. The programmable logic device 12 may also include a bridge module LOCAL_BUS_TO_AXI4 from the local bus to AXI4 to support access to internal modules or external memory from the configuration or debugging side; and may include DDR_TESTER as an optional test module for external memory read / write verification or bandwidth testing. The programmable logic device 12 may also include a sensor trigger module SENSOR_TRIGGER for outputting a trigger signal to the image sensor side to achieve frame acquisition triggering or synchronization control.

[0035] To facilitate understanding of the data pathways and functional divisions in the engineering implementation of this application, Figure 1 and Figure 2 This paper presents the overall scheme for splitting and merging 4K 240fps video streams and outputting them on the server side from the perspectives of methodology and system structure. Figure 3 Further, from the perspective of the internal functional structure of the programmable logic device 12, Figure 1 and Figure 2 The key functional modules involved, such as raw video frame input, external memory buffering, dual-channel output encapsulation and continuous transmission, time base and configuration control, are illustrated and explained in detail. It should be understood that... Figure 3 The modules shown are exemplary divisions. In actual engineering implementation, modules can be merged, split, or replaced according to device resources, interface form, and implementation complexity. As long as the functions defined in the claims of this application can be achieved, they should fall within the protection scope of this application.

[0036] The following is combined Figures 1 to 3 Exemplary method embodiments, exemplary system embodiments, and example internal structures of programmable logic devices are described in this application.

[0037] In some embodiments, to achieve the frame-level identification information generation described in step S2, the programmable logic device 12, after receiving the original video frame data, can perform frame boundary identification on the original video frames according to preset frame delimitation information, and assign an incrementing frame sequence number to each original video frame. Furthermore, the programmable logic device 12 can also generate a corresponding parity tag based on the parity attribute of the frame sequence number, thereby forming frame-level identification information containing the frame sequence number and the parity tag. The value of the parity tag can be used to characterize the odd or even attribute of the frame sequence number; for example, even frames can be marked with the first value, odd frames with the second value, or the two can be interchanged. Moreover, the correspondence between the parity tag and the meaning of the value can be configured by SPI_REG, thereby facilitating switching when adapting to different links or different processing units.

[0038] In some embodiments, to achieve the interleaving allocation described in step S3, the programmable logic device 12 can interleave the original video frames in chronological order to the first processing channel 121 and the second processing channel 122 based on parity tags, so that any original video frame enters only one of the processing channels, thereby enabling the first processing channel 121 and the second processing channel 122 to each carry a frame rate load of 120 frames per second. In one optional embodiment, the original video frames with parity tags of the first value are allocated to the first processing channel 121, and the original video frames with parity tags of the second value are allocated to the second processing channel 122; in another optional embodiment, the original video frames with parity tags of the first value are allocated to the second processing channel 122, and the original video frames with parity tags of the second value are allocated to the first processing channel 121. Through the above-described configurable interleaving allocation mechanism, the system's adaptability to different hardware channels or different downstream processing units can be improved without changing the rearrangement and interleaving merging logic on the backend server side.

[0039] In some embodiments, to implement the step S4 of writing frame-level identification information into a preset user-defined data area, the programmable logic device 12 can set up a user-defined data area in the video frame data output to the first processing channel 121 and the second processing channel 122 according to a preset encapsulation format, and write the frame sequence number and parity tag into the user-defined data area, thereby ensuring frame-level traceability of the first frame data stream and the second frame data stream during subsequent image processing, encoding, and network transmission. In an optional embodiment, the user-defined data area can be implemented by adding a preset number of user-defined data rows to the output video frame data. The user-defined data rows can be written with the frame sequence number and parity tag, and optionally with synchronization information such as timestamps. It should be understood that the specific field layout, number of rows, and writing position of the user-defined data area can be set according to the encapsulation format and bandwidth overhead adopted, and do not constitute a limitation of this application.

[0040] In some embodiments, to achieve buffering, scheduling, and continuous output under high-throughput links, the programmable logic device 12 can perform a "payload data and protocol control information separation buffering" processing method. Specifically, on the input interface side, the programmable logic device 12 can separate the payload data in the original video frame data from the protocol control information used for interface encapsulation. The payload data can be written to external memory DDR4, and the protocol control information can be recorded to a buffer queue. The protocol control information can include at least control information for frame delimitation and / or channel alignment, and can further include control information for indicating line type, line boundary, or alignment training, etc. Through the above separation buffering method, on the one hand, the difficulty of restoring the encapsulation boundary when reading data from external memory on the output side can be reduced, and on the other hand, it is convenient to perform controllable scheduling and alignment control in dual-channel output, thereby facilitating continuous output.

[0041] In the above embodiments, the buffer queue can be a first-in-first-out (FIFO) queue, and the protocol control information can be recorded in the FIFO queue row by row, thereby supporting the row-level reconstruction mechanism on the output side. Specifically, on the output interface side, the programmable logic device 12 can read the FIFO queue row by row and determine whether the current data row to be output is a user-defined data row based on the row type identifier in the protocol control information. When it is determined that the current data row to be output is a user-defined data row, the programmable logic device 12 can write the frame sequence number, parity tag, and / or timestamp into the current user-defined data row to be output before outputting it. When it is determined that the current data row to be output is not a user-defined data row, the programmable logic device 12 can read the corresponding payload data row from the external memory DDR4 row by row through the AXI bus interface, encapsulate it in combination with the protocol control information, and insert control information for frame delimitation and / or channel alignment to achieve continuous output. Through the above-mentioned "row-by-row determination - row-by-row insertion - row-by-row encapsulation" method, the continuity of the output link can be maintained while inserting user-defined data rows, and the feasibility of channel alignment and frame delimitation can be improved.

[0042] In some embodiments, to achieve the merging of network transmission and server-side data as described in steps S6 and S7, the first encoded data stream and the second encoded data stream can be packaged into network data frames and sent to server 16. The frame sequence number carried in the network data frame is used by server 16 to rearrange and interleave the two encoded data streams at the receiving end. Server 16 can restore the original time sequence according to the frame sequence number and interleave and merge the two encoded data streams to output a 4K240 frame video stream, thereby achieving backend sequence restoration and unified output based on parallel processing and encoding at the end.

[0043] In some embodiments, to improve alignment reliability in cross-channel or cross-network transmission scenarios, timestamps can be generated and used for alignment and order restoration. Specifically, the timestamp can be generated by TIMER_GEN and can be obtained based on the synchronization time provided by a precision time protocol; the timestamp can be written into a user-defined data line and transmitted with the first frame data stream and the second frame data stream, thereby serving as an auxiliary basis for frame alignment and order restoration during rearrangement and interleaving on the server side. It should be understood that whether timestamps are enabled, the granularity of timestamps, and the field length can be selected and set according to system synchronization requirements.

[0044] In some embodiments, the first processing channel 121 and the second processing channel 122 may correspond to different physical output interface links to adapt to the input interface capabilities of different subsequent video processing units. For example, the first frame data stream can be output to the first processing channel 121 through a Scalable Low Voltage Signal Embedded Clock (LVDS) interface, and the corresponding transmitting side can be implemented by SLVS_TX and GT_TRANS_TX in concert; the second frame data stream can be output to the second processing channel 122 through a Low Voltage Differential Signaling (LVDS) interface, and the corresponding transmitting side can be implemented by LVDS_TX and LVDS_INTERFACE. In an optional embodiment, the second frame data stream output by the LVDS interface may be in RAW10 format, which can be formed by reassembling data read from external memory DDR4 to adapt to the input format requirements of the second video processing unit 14. By adopting a combination of two interfaces for output, the interface adaptation capability to the subsequent processing platform can be improved while meeting the dual-channel bandwidth requirements.

[0045] In some alternative embodiments, the reception and transmission of the scalable low-voltage signal embedded clock interface are implemented by high-speed serial transceivers, and the receive-side link rate and transmit-side link rate of this interface are configured to be inconsistent to accommodate the differences in bandwidth and encapsulation overhead between the image sensor-side input and the first processing channel-side output; in this embodiment, the number of high-speed serial transceivers may be less than or equal to 8. Further, the low-voltage differential signal interface may be implemented by multiple low-voltage differential signal transmitters, and in this embodiment, the number of low-voltage differential signal transmitters may be less than or equal to 17.

[0046] In some embodiments, the sensor trigger module SENSOR_TRIGGER can be used to output a trigger signal to the image sensor 11 to achieve frame acquisition triggering or synchronization control; the configuration register module SPI_REG can be used to configure the current splitting strategy, user-defined data area writing strategy, output interface parameters, and timestamp-related parameters; the local bus to AXI bridge module LOCAL_BUS_TO_AXI4 can be used to configure side access to internal modules or external memory for debugging and status monitoring. Furthermore, DDR_TESTER can be used as an optional test module for external memory read / write verification or bandwidth testing; whether this module is enabled or not does not affect the implementation of the core processing flow of this application.

[0047] It should be understood that this application is not limited to the specific module division, interface combination or parameter configuration shown in the above embodiments; any equivalent substitutions, modifications or combinations made by those skilled in the art without departing from the spirit and scope of protection of this application shall fall within the scope of protection of this application.

[0048] In summary, this application provides a method and system for processing 4K 240fps video streams. By generating frame-level identification information on the high frame rate raw video frames output from the image sensor at the programmable logic device side, and then interleaving the raw video frames to two processing channels according to the frame-level identification information in chronological order, each channel bears a portion of the frame rate load. Simultaneously, identification information for frame-level tracking is written into the video frame data output to each processing channel, ensuring that the processed and encoded data from both channels can still be accurately rearranged and interleaved at the server side after network transmission, thereby outputting a correctly ordered 4K 240fps video stream. Therefore, parallel load sharing of high frame rate processing can be achieved at the terminal side while ensuring the restoration of the merged order, improving the stability and scalability of high frame rate video processing and transmission. In the above technical solution, by caching the original video frame data to external memory to achieve input-output rate matching, and combining the encapsulation reconstruction and continuous transmission mechanism on the output side, the risk of link interruption in high-throughput scenarios can be reduced. In addition, by further writing timestamps into the user-defined data area and introducing a time synchronization reference, the reliability of frame alignment and order recovery can be enhanced in cross-channel and cross-network transmission scenarios, thereby further improving the robustness and adaptability of the system engineering implementation.

[0049] The embodiments provided above are merely illustrative of the methods, structures, systems, and core ideas of this application. Those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the scope of protection of the claims.

Claims

1. A method for processing a 4K 240 frame video stream, characterized in that, The method comprises: acquiring 4K resolution, 240 frames per second raw video frame data of image sensor output; generating frame-level identification information for each of the raw video frames in a programmable logic device, the frame-level identification information at least including a frame sequence number and a parity tag corresponding to the frame sequence number; storing the raw video frame data to an external memory, and interleaving and distributing the raw video frames according to the parity tag to a first processing channel and a second processing channel in time sequence, so that any of the raw video frames is only distributed to the first processing channel or the second processing channel, and the frame rate received by the first processing channel and the second processing channel is both 120 frames per second; writing the frame-level identification information in a preset user-defined data area of the video frame data output to the first processing channel and the second processing channel to form a first frame data stream and a second frame data stream respectively; performing image processing and video encoding on the first frame data stream and the second frame data stream respectively to obtain a first encoded data stream and a second encoded data stream; packing the first encoded data stream and the second encoded data stream into network data frames carrying the frame sequence number and sending them to a server; rearranging and interleaving the first encoded data stream and the second encoded data stream based on the frame sequence number, and outputting a 4K240 frame video stream.

2. The method of claim 1, wherein the 4K 240 frame video stream is processed by the video processing device, and the video processing device is a video decoder. The parity tag is used to represent the parity attribute of the frame sequence number, and the interleaving and distributing the raw video frames to the first processing channel and the second processing channel according to the parity tag comprises: distributing raw video frames with a first value of the parity tag to the first processing channel and raw video frames with a second value of the parity tag to the second processing channel; or distributing raw video frames with the first value of the parity tag to the second processing channel and raw video frames with the second value of the parity tag to the first processing channel. The writing of the frame-level identification information in the preset user-defined data area comprises: adding a preset number of user-defined data rows in a preset packaging format in the video frame data output to the first processing channel and the second processing channel, and writing the frame sequence number, the parity tag and a timestamp in the user-defined data rows.

3. The method of claim 1, wherein the 4K 240 frame video stream is processed by the video processing device, and the video processing device is a video decoder. The storing of the raw video frame data to the external memory and the interleaving and distributing comprise:

4. The method of claim 1, wherein the 4K 240 frame video stream is a 4K 60P video stream. separating payload data in the raw video frame data from protocol control information used for interface protocol packaging on the input interface side of the programmable logic device, writing the payload data to the external memory, and recording the protocol control information to a cache queue; reconstructing the interface protocol packaging of the payload data read from the external memory according to the protocol control information recorded by the cache queue on the output interface side of the programmable logic device, and continuously outputting the payload data corresponding to the first processing channel to the first processing channel and the payload data corresponding to the second processing channel to the second processing channel; wherein the protocol control information at least includes control information used for frame definition and / or channel alignment. ​ 5. The method for processing a 4K 240 frame video stream as described in claim 4, characterized in that, The cache queue is a first-in first-out queue, and the protocol control information is recorded in the first-in first-out queue by rows; The reconstruction interface protocol package comprises: The first-in first-out queue is read by rows, and it is determined whether the current data row to be output is a user-defined data row according to the row type identifier in the protocol control information; When it is determined that the current data row to be output is a user-defined data row, the output interface side writes the frame number, the parity tag and / or the timestamp in the current user-defined data row to be output; When it is determined that the current data row to be output is a non-user-defined data row, the output interface side reads the corresponding payload data row from the external memory by rows through a bus interface and packages the protocol control information, while inserting control information for frame definition and / or channel alignment to realize continuous output.

6. The method for processing a 4K 240fps video stream as described in claim 3, characterized in that, The timestamp is generated based on the synchronization time provided by the precision time protocol and is used for frame alignment and sequence recovery in the rearrangement and interlacing combination.

7. The method of claim 1, wherein the 4K 240 frame video stream is processed by a video decoder, and the video decoder is configured to perform the method of claim 1. The first frame data stream is output to the first processing channel through an extensible low-voltage signal embedded clock interface, and the second frame data stream is output to the second processing channel through a low-voltage differential signal interface. The second frame data stream output by the low-voltage differential signal interface is in RAW10 format, which is reorganized from the data read from the external memory.

8. A processing system of a 4K 240 frame video stream, characterized in that, Comprise: An image sensor for outputting raw video frame data with a resolution of 4K and a frame rate of 240 frames per second; A programmable logic device having an input interface and an output interface, the input interface being coupled to the image sensor to receive the raw video frame data, and the output interface being coupled to a first processing channel and a second processing channel to transmit a first frame data stream and a second frame data stream, respectively; A first video processing unit and a second video processing unit, respectively coupled to the first processing channel and the second processing channel, for performing image processing and video encoding on the first frame data stream and the second frame data stream, respectively, to obtain a first encoded data stream and a second encoded data stream; A network transmission module for packaging the first encoded data stream and the second encoded data stream into network data frames carrying frame numbers and sending them to a server; A server for rearranging and interlacing the first encoded data stream and the second encoded data stream based on the frame numbers and outputting a 4K240 frame video stream; The programmable logic device is configured to generate frame-level identification information for each raw video frame and cache the raw video frame data to an external memory. The frame-level identification information at least includes a frame number and a parity tag corresponding to the frame number, and the programmable logic device interlaces and distributes the raw video frames to the first processing channel and the second processing channel in time sequence according to the parity tag, so that any raw video frame is only distributed to the first processing channel or the second processing channel, and the frame rate received by the first processing channel and the second processing channel is both 120 frames per second. The programmable logic device is further configured to write the frame-level identification information in a preset user-defined data area of video frame data output to the first processing channel and the second processing channel, to form the first frame data stream and the second frame data stream respectively.

9. The processing system of 4K240 frame video stream according to claim 8, characterized in that, The preset user-defined data area comprises preset number of user-defined data lines added in preset packaging format; The programmable logic device is configured to write the frame sequence number, the odd-even label and a timestamp in the user-defined data line, the timestamp being generated based on a synchronization time provided by a precision time protocol; The programmable logic device receives the original video frame data through an extensible low-voltage signal embedded clock interface and outputs the first frame data stream, and outputs the second frame data stream through a low-voltage differential signal interface.

10. The system for processing 4K240 frame video streams of claim 8, wherein, The programmable logic device is configured to separate payload data in the original video frame data from protocol control information for interface protocol packaging on the input interface side, write the payload data to the external memory, and record the protocol control information to a cache queue; On the output interface side, the payload data read from the external memory is reconstructed for interface protocol packaging according to the protocol control information recorded by the cache queue, and the payload data corresponding to the first processing channel is continuously output to the first processing channel, and the payload data corresponding to the second processing channel is continuously output to the second processing channel; wherein the protocol control information at least comprises control information for frame definition and / or channel alignment.

Images