Video streaming control system architecture

Abstract

The present application discloses a video stream transmission control system architecture, which relates to the field of video stream processing. The detector stores and sends video streams to the outside. The cache unit is connected to the detector to cache the video stream, and caches and outputs the video stream according to instructions and timing control. The output of the cache unit is connected to several pipelined video processing modules, and the video processing module receives the video stream, performs data processing and transmits the video stream backward step by step according to instructions and control timing. The NIOS processor communicates with the cache unit and all pipelined video processing modules respectively, and performs instruction and data interaction control between the cache unit and the video processing module. The video processing modules feedback the output back pressure signal to the superior step by step in the reverse flow direction, and the cache unit generates the video stream output sequence based on the back pressure signal. This scheme can be applied to a video stream transmission system with multi-module joint adjustment, maintains the synchronization and consistency of system transmission, and increases or decreases system modules, and has higher portability.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of video stream processing, and in particular to a video stream transmission control system architecture. Background Art

[0002] In the past, when developing new projects, changes in project requirements and hardware conditions would make it difficult to migrate the original system to the new project. Although modules could be borrowed, the system itself needed to be reconstructed and would be limited by the original modules, which required a lot of time to rewrite and debug. To achieve the reusability of the system, the system needs to unify the design of the video interface and unify the management of all modules involved, including clock, parameter control, etc.

[0003] In the hardware development model, there is no unified and standardized interface between video stream modules. When data is exchanged between modules, the developer is responsible for formulating a suitable interface for upstream and downstream communication and video transmission. From the perspective of single system development, this is an efficient and flexible method that can be adjusted according to the processing points of each module during the video transmission process.

[0004] From the macro perspective of the development team or company, the design architecture of the above video streaming module will bring great difficulties to transplantation and debugging. When the data interaction of the video module is not limited to upstream and downstream, but extends to the synchronization and communication of multiple modules, it is necessary to adjust the module or design a complex mechanism to achieve synchronization. When multiple people jointly develop and debug, it is impossible to grasp all module interface information, and when problems occur, it is difficult to quickly locate the crux of the problem. Summary of the invention

[0005] The embodiment of the present application provides a video stream transmission control system architecture to solve the problem of video stream transmission and interconnection debugging interface data synchronization.

[0006] The video stream transmission control system architecture includes:

[0007] Detector, used to store and send video streams externally;

[0008] A cache unit, connected to the detector, caches the video stream, and caches and outputs the video stream according to instructions and timing control;

[0009] A video processing module, wherein the output of the cache unit is connected to a plurality of pipelined cascaded video processing modules, wherein the video processing module receives a video stream, performs data processing according to instructions and control timing, and transmits the video stream backward step by step;

[0010] The NIOS processor is respectively connected to the cache unit and all the pipelined video processing modules for communication, and performs instruction and data interaction control between the cache unit and the video processing module;

[0011] The video processing modules output a back pressure signal in a reverse flow direction to the upper level step by step, and the cache unit adjusts and generates a video stream output sequence based on the back pressure signal.

[0012] Specifically, the cache unit and the pipelined cascaded video processing modules transmit video streams via an Avalon-ST interface and bus;

[0013] The NIOS processor communicates and interacts with each video processing module and the cache unit through the Avalon-MM interface and the control bus.

[0014] Specifically, the cache unit is a DDR cache or a FIFO cache;

[0015] When the cache unit is a DDR cache, the detector outputs the original video stream into the QSYS platform, and transmits the video frame to the mapping address allocated by the DDR through the Avalon-MM interface and bus;

[0016] When the cache unit is a FIFO cache, the detector outputs the original video stream and transfers the video frames into the FIFO through the Avalon-ST interface and bus.

[0017] Specifically, during the transmission of the video stream in the packet, the source end video processing module sends the data stream to the sink end video processing module; a packet header signal, a packet tail signal, a data signal, a valid signal and a back pressure ready signal are connected between the source end and the sink end;

[0018] The source sends a packet header signal, a packet tail signal, a data signal, and a valid signal to the sink, and the sink feeds back a back pressure signal to the source.

[0019] When the source sets the valid signal, it sends the video stream data packet to the sink through the data signal. When the source sends the cleared valid signal, the adjacent source and sink enter the idle state. When the sink sets the ready signal, it receives the video stream data packet transmitted by the source.

[0020] Specifically, the time interval between two times when the source end sets the packet header signal is the frame processing time, and the sink end processes and caches the received data packets within the frame processing time;

[0021] When the data processing rate of the sink is lower than the data sending rate of the source, a cleared ready signal is sent to the source.

[0022] Specifically, the video processing module as the source end is provided with a back pressure delay parameter ready latency. When the source end receives a cleared ready signal fed back by the sink end, the valid signal is set based on the back pressure delay parameter.

[0023] Specifically, when ready latency is not zero, the source calculates the ready cycle based on the set ready signal and readylatency sent by the sink, and sets the valid cycle according to the length of the ready cycle; the ready cycle is the clock cycle of the sink receiving data sent by the source, and the valid cycle is the clock cycle of the source sending a valid signal to the sink.

[0024] Specifically, when the ready latency is zero, the source sets the valid signal when the data is not empty and maintains the valid state; when the source detects that the sink sets the ready signal, the video stream data packet is sent through the data signal; the sink receives the video stream data packet within the clock cycle when the ready signal and the valid signal are both valid.

[0025] Specifically, the ready signal is fed back to the front stage step by step in the reverse flow, and the cache unit adjusts the transmission rate of the video stream and sets the timing according to the ready signal fed back by the first-stage video processing module.

[0026] Specifically, when there is no ready signal on the interface between the source and the sink, the source assumes that the sink is always ready to receive data, and the source sends the video stream according to its own cached data status; when there is no valid signal on the interface between the source and the sink, the sink receives the video stream outside the period of the ready signal based on its own cached data.

[0027] The beneficial effects brought by the technical solution provided in the embodiment of the present application include at least: the original video stream of the detector is processed through the DDR cache, the front-end video frame rate and format are processed, and then the process of being processed and finally output by each module in turn. The back pressure signal between the video processing modules is negatively fed back step by step, and each module transmits the video stream to the downstream according to the back pressure signal of the lower level, and also feeds back to the previous level module according to its own processing situation, and finally feeds back to the DDR cache, and the DDR cache generates a suitable video output timing to cooperate with the back-end module to process the video stream. When the module data processing is not timely at any stage, it only needs to simply control the ready signal to output the video stream according to the timing that meets its own requirements, ensure the synchronization and consistency of the system, and be more convenient to configure in the joint debugging scenario, reduce the lateral changes of other modules, and be of great help to the reuse and secondary development of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a structural diagram of the video stream transmission control system architecture provided by an embodiment of the present application;

[0029] Figure 2shows a schematic diagram of bus connection between two adjacent video processing modules;

[0030] Figure 3 Figure 2 shows a signal timing diagram for a possible ready latency parameter. DETAILED DESCRIPTION

[0031] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0032] The term "multiple" as used herein refers to two or more than two. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the related objects are in an "or" relationship.

[0033] Figure 1 This is a schematic diagram of the structure of the video stream transmission control system architecture provided by the embodiment of the present application. The system architecture includes a detector, a cache unit, several video processing modules and a NIOS processor. The specific introduction is as follows:

[0034] The detector serves as a data source, which is connected to an external video acquisition terminal or other external input for storing and sending video streams to this system.

[0035] The cache unit is connected to the detector, caches the video stream, and controls the cache and output of the video stream according to instructions and timing.

[0036] Several video processing modules are in a cascade state, especially the scenario of multiple devices (modules) interoperating with each other proposed in this application. The output of the cache unit is connected to the first-level video processing module, and the video stream is transmitted backward step by step along the pipeline cascade form. The video processing module receives the video stream, performs data processing and transmits the video stream backward step by step according to the instructions and control timing;

[0037] The NIOS processor communicates with the cache unit and all the pipelined video processing modules, and performs instruction and data interaction control between the cache unit and the video processing module;

[0038] The video processing modules output back pressure signals in a countercurrent manner and feed them back to the upper level step by step. The cache unit generates a video stream output sequence based on the back pressure signal.

[0039] In this system architecture, the cache unit and the pipelined video processing modules transmit video streams through the Avalon-ST interface and bus. The NIOS processor communicates and interacts with each video processing module and cache unit through the Avalon-MM interface and control bus.

[0040] The Avalon-ST interface is very suitable for processing video data streams due to its high bandwidth, low latency, and unidirectional master-slave characteristics. Its unique video data back pressure concept supports a flexible data transmission structure. The downstream (destination) can notify the upstream (source) to stop sending data at any time when the video data load exceeds the module processing capacity or the internal cache is about to overflow, and realize the step-by-step negative feedback between modules, and finally inform the video source. The video source can reasonably and meticulously control the video transmission rate based on this signal, so as to achieve a dynamic balance of video stream transmission. In this way, the video rate is controlled from the source as a whole, which is undoubtedly more scientific and reasonable than the traditional video stream modules "fighting each other" and adjusting the video rate according to their own needs.

[0041] The Avalon-MM interface supports a top-down mesh structure with the NIOS processor as the core, which can well transfer complex operations with relatively low real-time requirements but strong inter-module correlation to the NIOS processor. NIOS directly receives instructions and data from various sensors and modules, and then issues control parameters after centralized processing, which greatly enhances the overall correlation and coordination of the system. When the system needs to add or remove video processing modules and make adjustments, since a single video processing module only involves an interface to the NIOS processor, it can minimize the horizontal changes of other video processing modules, which is very helpful for system reuse and secondary development.

[0042] In some embodiments, the amount of video stored in the detector is large, and the pipeline video processing module usually has limited data processing capabilities, so a cache unit is required to perform video frame cache regulation. In this application, the cache unit can be a DDR cache or a FIFO cache.

[0043] When the cache unit is a DDR cache, such as Figure 1 In the demonstration, the detector outputs the original video stream into the QSYS platform, and the video frame is transmitted to the bus through the Avalon-MM interface and bus provided by the IP core to the pre-allocated mapping address of the DDR, and stored in the DDR to complete the video frame buffer. This method can save on-chip storage space through the use of DDR, especially when the video stream itself needs to use DDR for front-end processing, the advantage of saving resources is particularly obvious.

[0044] When the cache unit is a FIFO cache, the detector outputs the original video stream and enters the QSYS platform through a dedicated original video stream to the Avalon-ST interface IP core. Such an IP generally provides a certain scale of data cache FIFO and bus to provide buffer space for the back-end Avalon-ST interface control. This method is more dependent on on-chip resources.

[0045] This application uses DDR cache for subsequent design description. Figure 1 The original video stream of the detector passes through the DDR cache, completes the processing of the front-end video frame rate and format, and then passes through the processing of each module in turn for the final output. Note that in the process of forward video transmission, the back pressure signal is also negatively fed back step by step. Each module not only receives the back pressure signal of the lower level and transmits the video stream downstream accordingly, but also feeds back to the previous level module according to its own processing situation, and finally feeds back to the DDR cache. Here, it is required to generate the appropriate video output timing through the back pressure signal and cooperate with the back-end module to process the video stream. All these modules, including DDR itself, are controlled by the NIOS processor, and the control parameters can be adjusted according to the actual situation and user needs.

[0046] Under this mechanism, the buffering pressure and rate matching requirements for data within the frame will be fed back step by step through the back pressure signal and finally accumulated to the source of the video stream, that is, the data buffering and video stream timing generation work are completely completed by the DDR controller based on the state machine to identify the back pressure signal. Therefore, in this system, the front end first buffers the video stream through DDR, and this detailed adjustment of video data output is undoubtedly the most appropriate.

[0047] Figure 2 The bus connection diagram between two adjacent video processing modules is shown. During the DDR back-end video stream processing and transmission process, the video stream always flows unidirectionally from the source end to the sink end through the Avalon-ST interface, that is, between any two cascaded video processing modules, the sender is the source end device and the receiver is the sink end device. The source end sends the entire frame of video to the sink end in the form of a data packet, and the video stream is transmitted in the packet. The source end and the sink end are connected with the packet header start of packet signal, the packet tail end of packet signal, the data signal, the valid signal and the back pressure ready signal. The packet header and packet tail signals are also called frame start and frame end signals. The start and end of a frame image are of great significance to the image processing module. The image processing algorithm needs to identify the start pixel and end pixel of a frame to ensure the correct processing of each frame image. Naturally, the Avalon-ST packet transmission mode is used to define the frame start and frame end of the data packet. This is a set of additional signal lines. During the transmission process, both upstream and downstream must include these signals.

[0048] After determining the start and end signals of the data packet, the time from the start of the frame of the current data packet to the arrival of the next data packet (the start of the next frame) can generally be used as the frame processing time of the subsequent modules. When all modules through which the video stream flows can complete processing within this time period, the video stream transmission control system of this solution can be used to solve the problem of in-frame caching.

[0049] In traditional video transmission systems, most processing modules need to cache video images. A portion of the cache is used to meet the needs of the algorithm inside the module. The usage limit of this portion of the cache is independent of the characteristics of the algorithm itself and the interface mechanism. Another portion is to balance the video stream transmission between modules. This portion of the cache can be optimized under the Avalon-ST interface mechanism. When the upstream video stream transmission rate exceeds the processing speed of this module, under the traditional video transmission mechanism, the downstream module will passively use the FIFO to cache the data that exceeds the processing capacity. This application is different from the traditional relatively rigid video transmission mechanism. Because each module of the Avalon-ST interface mechanism can autonomously adjust the transmission of the video stream within the frame through the feedback ready signal to match its own processing speed, it can relatively significantly reduce the pressure on the internal data buffer of the module, reduce the demand for FIFO inside a single module, and save the use of on-chip storage space.

[0050] The valid signal and the ready signal are signals set by the present application to improve the efficiency of data transmission and adaptively adjust the video stream transmission. In the definition of the present application, when the source sets (pulls high) the valid signal, it will send a video stream data packet to the destination through the data signal. When the source sends a clear (pulls low) valid signal, the adjacent source and destination will enter an idle state. When the destination sets the ready signal, it will receive the video stream data packet transmitted by the source. In fact, during the transmission of the video stream in the packet, it will be affected by the source's intention to send data and the destination's intention to receive data. The source can learn the destination's intention to receive data through the ready signal, and the transmission of the video stream will only occur when both the source and the destination agree.

[0051] It should be noted that the ready signal and the valid signal are not necessary in the Avalon-ST interface. In some embodiments, when there is no ready signal on the interface, the source will assume that the sink is always ready to receive data and only send the video stream according to its own will, specifically when there is still data left / cached. Similarly, when there is no valid signal on the interface between the source and the sink, the sink will also receive the video stream outside the period of the ready signal according to its own will.

[0052] Under the premise of using the ready signal and the valid signal, the mechanism for realizing adaptive adjustment of video frame transmission is: the sink processes and caches the received data packets within the frame processing time. When the data processing rate of the sink is lower than the data sending rate of the source, a cleared ready signal is sent to the source to start the adjustment mechanism.

[0053] Regarding the specific operation mode and data response timing of the adjustment mechanism, this application describes it in detail through the back pressure delay parameter readylatency. The video processing module at the source end is provided with a back pressure delay parameter ready latency. When the source end receives the cleared ready signal fed back by the sink end, the valid signal is set based on the back pressure delay parameter. The interface that supports back pressure uses the defined ready latency parameter to indicate how many clock cycles after the ready signal at the sink end takes effect before the valid signal can be set.

[0054] 1. When the ready latency is not zero, the source calculates the ready cycle based on the set ready signal sent by the sink and the saved ready latency, and sets the valid cycle according to the length of the ready cycle.

[0055] The ready cycle here is the clock cycle of the sink receiving data sent by the source, and the valid cycle is the clock cycle of the source sending a valid signal to the sink. The sink module will set the ready signal before the start of the ready cycle. After detecting that the ready signal has become high, the source module calculates the ready cycle of the sink based on the ready latency parameter, and then sets the valid cycle. However, if the sink is not in the ready cycle, the source cannot set the valid signal.

[0056] Figure 3 The figure shows a possible signal timing diagram under the ready latency parameter, taking ready latency = 1 clock as an example. After the ready signal at the sink end becomes high in the 0th cycle, the source end delays one clock according to the ready latency, and then switches the valid signal to a high level in the 1st cycle. In the 1st and 2nd cycles, the valid signal and ready are both high, and the data signal at this time transmits video frame data normally. Figure 3 The ready cycle is 3 clocks, so the valid cycle is also set to 3 clocks. Since the ready latency = 1 clock, the valid signal is cleared in the 4th cycle. However, it should be noted that since the ready signal represents the willingness of the sink to receive data, the data will no longer be received after the ready signal is cleared in the 3rd cycle. Therefore, data D6, D7 and D8 are not received. Although the source sends, the sink may not receive. Therefore, it can be seen that the valid cycle can be set to Tready-Tlatency to keep it synchronized with the ready signal.

[0057] 2. When the ready latency is zero, it is quite special. The source module will not sample the ready signal before sending valid data. Instead, it will set the valid signal as long as the upstream can provide data stream, and then wait for the sink to set the ready signal, indicating that the data within the clock cycle has been received, and send the video stream data packet through the data signal. In this state, the source can change the provided data at any time, and the sink will only sample data when both ready and valid are valid.

[0058] In the pipeline cascade video processing module, the above-mentioned video stream transmission mechanism can be independently performed between any adjacent source and sink ends, and the ready signal can be fed back to the previous stage step by step in the reverse flow until the cache unit adjusts the transmission rate of the video stream and sets the timing according to the ready signal fed back by the first-level video processing module. In this way, the cache unit can coordinate the timing and video stream speed of all video processing modules in the entire system. When the output video format frame rate is basically consistent with the video stream transmitted by the Avalon-ST interface, after completing the processing of all modules, it is only necessary to simply control the ready signal in any video processing module to output the video stream according to the timing that meets its own requirements. This is also a major advantage of this system.

[0059] In addition, based on the above, after the initial system architecture is completed, the subsequent development work is quite flexible. The improvement of existing functional modules can be controlled by the NIOS processor Avalon-MM interface to adjust the parameters of the video processing module to meet the various needs of users. The development of new functional modules under the framework of the Avalon-ST interface can also enable the new video processing module to be flexibly inserted into the required position in the system, and the step-by-step feedback mechanism of the back pressure signal can make the buffer storage of the video stream no longer face the dilemma of a single move affecting the whole body. The timing control can still be uniformly arranged at the source of the video stream.

[0060] This specific embodiment is merely an explanation of the present invention and is not a limitation of the present invention. After reading this specification, those skilled in the art may make non-creative modifications to the present embodiment as needed. However, as long as they are within the scope of the claims of the present invention, they are protected by the patent law.

Claims

1. A video stream transmission control system architecture, characterized in that: include: Detector, used to store and send video streams externally; A cache unit, connected to the detector, caches the video stream, and caches and outputs the video stream according to instructions and timing control; Video processing module, the output of the cache unit is connected to a plurality of pipelined cascaded video processing modules, the video processing module receives the video stream, performs data processing and transmits the video stream backward step by step according to the instructions and control timing; the cache unit and the pipelined cascaded video processing modules transmit the video stream through the Avalon-ST interface and bus; The NIOS processor is respectively connected to the cache unit and all the pipelined video processing modules for communication, and performs instruction and data interaction control with the cache unit and the video processing module; the NIOS processor performs communication and interaction control with each video processing module and the cache unit through the Avalon-MM interface and the control bus; The video processing modules output a back pressure signal in a reverse flow direction to the upper level step by step, and the cache unit generates a video stream output sequence based on the back pressure signal. The cache unit is a DDR cache or a FIFO cache; When the cache unit is a DDR cache, the detector outputs the original video stream into the Q-SYS platform, and transmits the video frame to the mapping address allocated by the DDR through the Avalon-MM interface and bus; When the cache unit is a FIFO cache, the detector outputs the original video stream and transfers the video frames into the FIFO through the Avalon-ST interface and bus; During the transmission of the video stream in the packet, the source end video processing module sends the data stream to the sink end video processing module; the source end and the sink end are connected with a packet header signal, a packet tail signal, a data signal, a valid signal and a back pressure ready signal; The source sends a packet header signal, a packet tail signal, a data signal, and a valid signal to the sink, and the sink feeds back a back pressure signal to the source. When the source sets the valid signal, it sends the video stream data packet to the sink through the data signal. When the source sends the cleared valid signal, the adjacent source and sink enter the idle state. When the sink sets the ready signal, it receives the video stream data packet transmitted by the source.

2. The video stream transmission control system architecture according to claim 1, characterized in that: The time interval between two packet header signal settings at the source end is the frame processing time, and the sink end processes and caches the received data packets within the frame processing time; When the data processing rate of the sink is lower than the data sending rate of the source, a cleared ready signal is sent to the source.

3. The video stream transmission control system architecture according to claim 2, characterized in that: The video processing module at the source end is provided with a back pressure delay parameter ready latency. When the source end receives the cleared ready signal fed back by the sink end, the valid signal is set based on the back pressure delay parameter.

4. The video stream transmission control system architecture according to claim 3, characterized in that: When the ready latency is not zero, the source calculates the ready cycle based on the set ready signal and ready latency sent by the sink, and sets the valid cycle according to the length of the ready cycle; the ready cycle is the clock cycle of the sink receiving the data sent by the source, and the valid cycle is the clock cycle of the source sending a valid signal to the sink.

5. The video stream transmission control system architecture according to claim 3, characterized in that: When the ready latency is zero, the source sets the valid signal when the data is not empty and maintains the valid state; when the source detects that the sink sets the ready signal, it sends the video stream data packet through the data signal; the sink receives the video stream data packet within the clock cycle when the ready signal and the valid signal are both valid.

6. The video stream transmission control system architecture according to claim 1 or 2, characterized in that: The ready signal is fed back to the previous stage in a reverse flow, and the cache unit adjusts the transmission rate and sets the timing of the video stream according to the ready signal fed back by the first-stage video processing module.

7. The video stream transmission control system architecture according to claim 1, characterized in that: When there is no ready signal on the interface between the source and the sink, the source assumes that the sink is always ready to receive data, and the source sends the video stream according to its own cached data status; when there is no valid signal on the interface between the source and the sink, the sink receives the video stream outside the period of the ready signal based on its own cached data.

Images