A frame-level synchronization access processing method for multi-source heterogeneous video streams

Abstract

The application relates to the technical field of video processing, in particular to a frame-level synchronous access processing method for multi-source heterogeneous video streams. In the application, a channel frame boundary sequence is generated, a cross-channel frame period pairing relationship is established, a frame arrival deviation relationship is formed, a cache reading state is generated according to a read-write pointer of a cyclic frame cache and an unread data amount of a read cache area, a frame cache synchronous handover boundary is determined, a first video frame is kept to the boundary, a cache role conversion permission is generated after each paired frame is written, the write cache area is converted into the read cache area, and a synchronous video frame is output, a subsequent handover boundary correction amount is generated according to the synchronous frame handover state to correct the frame cache synchronous handover boundary of the next frame period, so that the incomplete video frame output, the video frames of different frame periods being correspondingly output and the video data reading gap are reduced.

Description

Technical Field

[0001] This invention relates to the field of video processing technology, specifically to a frame-level synchronous access processing method for multi-source heterogeneous video streams. Background Technology

[0002] Multi-source video access systems typically require sending multiple video streams from different image acquisition devices, video interfaces, or transmission links to the same processing platform for subsequent video fusion, stitching, analysis, or display. Due to factors such as video interface type, transmission path, frame format, and clock phase, corresponding video frames in different video channels may arrive in different orders. To ensure that subsequent processing links can read video data from each channel according to the corresponding frame period, it is usually necessary to first write the received video frames to a buffer, and then provide video frames to subsequent stages through buffer switching.

[0003] In existing technologies, one approach is to set up multiple frame buffers for each video channel, write video data to the corresponding buffer based on the frame synchronization signal, and switch the buffer with written video data to the read buffer after a preset waiting time. Another approach is to perform buffer switching after detecting a video frame write completion signal, or to uniformly update the read buffers of each channel when the write completion signals of each channel meet preset logical conditions. These methods can, to some extent, absorb timing differences during video stream transmission and separate the video data writing and reading processes.

[0004] However, the arrival interval between video frames may change depending on the transmission status, and the reading progress of the current read buffer may also change. When switching buffers using a preset waiting time, the buffer switching time may be earlier than the completion time of writing some channel video frames, causing the subsequent stage to read video frames that have not been fully written; or the switching may be performed only after all valid data in the current read buffer has been read, resulting in periods where no valid video data can be read by the subsequent stage. When using a write completion signal to trigger buffer switching, if each channel responds to its own completion signal, each channel may update the read buffer at different times, causing multiple video frames output in the same time to correspond to different frame periods; if the switching is performed only after all channels have completed writing, the current read buffer may be read before the switching is completed when the arrival interval between channels increases. Therefore, in the existing buffer switching method, the buffer role change time may fall before the completion of writing of video frames in the same period, or after the completion of reading valid data from the current read buffer. Summary of the Invention

[0005] The purpose of this invention is to provide a frame-level synchronous access processing method for multi-source heterogeneous video streams to solve the problems mentioned in the background art. Specifically, it addresses the problem that in the prior art, the buffer role switching time may be earlier than the complete writing time of each video frame in the same period or later than the effective reading end time of the current read buffer, resulting in incomplete video frames being read by subsequent stages, video frames of different frame periods being output accordingly, or gaps in video data reading.

[0006] To achieve the above objectives, the present invention aims to provide a frame-level synchronization access processing method for multi-source heterogeneous video streams, specifically including the following method steps:

[0007] S1. The FPGA parses multiple video streams according to interface type, detects the start edge of frames, latches a shared counter value at the start edge of each frame, and increments the uniformly reset channel frame number to generate a channel frame boundary sequence; the generation process of the channel frame boundary sequence specifically includes:

[0008] Synchronize the frame synchronization signal for each video channel and detect the frame start edge. When a valid frame start edge is detected, latch the current value of the free run counter, determine the current value as the frame start count value, and associate and save the incremented channel frame number with the frame start count value.

[0009] The video channel identifier, channel frame number, frame start count value, channel format parameters, write buffer index, and boundary validity status are used to form a frame boundary record. The frame boundary records of each video channel are arranged according to the shared counter value to generate a channel frame boundary sequence.

[0010] Step S1 uses the FPGA to parse multiple video streams according to their interface types and records the frame start count value and channel frame number of each video channel under a unified time base formed by a shared counter. This converts multiple video streams with different interface types and different frame start times into channel frame boundary sequences that can be compared by time and frame number. The channel frame boundary sequence can characterize the channel affiliation, frame period, start time, and corresponding write buffer index of each video frame, providing a unified data basis for subsequent identification of video frames with the same frame period and reducing the possibility of establishing a correspondence between video frames with different frame periods due to differences in interface timing.

[0011] S2. Based on the channel frame boundary sequence, establish cross-channel frame period pairing relationships for video frames with the same frame number, compare the count values ​​of each paired frame, and generate frame arrival deviation relationships.

[0012] The process of establishing cross-channel frame period pairing relationships specifically includes:

[0013] Read the channel frame boundary sequence, group the frame boundary records according to the channel frame number, and when there is a frame boundary record for each valid video channel in a group and the channel frame number of each frame boundary record is the same, establish a cross-channel frame period pairing relationship between the video frames in the group.

[0014] The pairing frame number, the identifier of the video channel involved in the pairing, the frame start count value of each video channel, the write buffer index, and the pairing validity status are used to form a cross-channel frame period pairing relationship;

[0015] When the pairing wait time reaches the pairing wait threshold and the group is still incomplete, cancel the pairing validity status of the current group and mark the data that has been written as non-transferable data; after pairing is completed, change the corresponding frame boundary record to the paired status.

[0016] The process of generating the frame arrival deviation relationship specifically includes:

[0017] Read the frame start count value of each paired frame in the cross-channel frame period pairing relationship, and take the minimum and maximum values ​​of each frame start count value to obtain the earliest paired frame arrival count value and the latest paired frame arrival count value;

[0018] The arrival deviation of each video channel is determined by the difference between the frame start count value of each paired frame and the arrival count value of the earliest paired frame. The frame arrival span is determined by the difference between the arrival count value of the latest paired frame and the arrival count value of the earliest paired frame, and the video channels are arranged in order of arrival.

[0019] When the frame reaches a span exceeding the pairing deviation limit, the pairing valid state is changed to the out-of-tolerance state.

[0020] Step S2 establishes a cross-channel frame cycle pairing relationship by grouping the frame boundary records in the channel frame boundary sequence according to the channel frame number, so that each video frame participating in the subsequent buffer role conversion has the same paired frame number; then, a frame arrival deviation relationship is generated according to the frame start count value of each paired frame, obtaining the earliest paired frame arrival count value, the latest paired frame arrival count value, the frame arrival span, and the video channel order arranged in the order of arrival. This step can clarify the arrival range of each video frame in the same frame cycle, and impose state restrictions on paired frames with incomplete grouping or frame arrival span exceeding the pairing deviation upper limit, providing a basis for constraining the buffer role conversion time with the arrival time of the latest paired frame, and reducing the situation where video frames of different frame cycles are output accordingly.

[0021] S3. A circular frame buffer containing a write buffer and a read buffer is used. Each paired frame is written to the write buffer and marked as written. The duration of continuous read supply is determined based on the read / write pointer and the amount of unread data in the read buffer, and a cache read supply status is generated. The specific process of generating the cache read supply status includes:

[0022] Read the read pointer, write pointer, total number of bytes in the read buffer, amount of unread data in the read buffer, and average read rate of each video channel;

[0023] The continuous reading duration for each video channel is determined based on the amount of unread data in the read buffer minus the amount of data reserved for safety and the average read rate of the subsequent stage. The continuous reading duration is set to zero when the amount of unread data in the read buffer is less than or equal to the amount of data reserved for safety.

[0024] The minimum value among the continuous reading durations of each video channel is determined as the effective continuous reading duration, and the read pointer, write pointer, unread data in the read buffer, continuous reading duration, effective continuous reading duration, write completion flag of each paired frame, and pointer occupation conflict status are used to form the buffer reading status.

[0025] When the write buffer index of any paired frame is the same as the current read buffer index of the corresponding video channel, or when the buffer corresponding to the write buffer index has been occupied by different paired frames, the pointer occupancy conflict state is determined to be a conflict state; otherwise, it is determined to be a non-conflict state. When the read pointer or write pointer exceeds the range of the number of circular frame buffers, the pointer occupancy conflict state is a conflict state, or the amount of unread data in the read buffer is greater than the total number of bytes in the read buffer, the buffer read state is determined to be an invalid state.

[0026] Step S3 uses a circular frame buffer to carry the writing process of each paired frame and the reading process of the current video frame. It determines the continuous reading duration based on the amount of unread data in the read buffer, the amount of data safely reserved, and the average reading rate of the subsequent stage. Then, it determines the effective continuous reading duration from the continuous reading duration of each video channel and generates a buffer reading status that includes read / write pointers, write completion flags, and pointer occupancy conflict states. This step converts the time when the current read buffer can still maintain effective reading into state information that can participate in the handover boundary calculation. This provides a basis for judging whether the buffer role transition time may be later than the effective reading end time of the current read buffer, thereby reducing the possibility of gaps in video data reading.

[0027] S4. Determine the frame buffer synchronization handover boundary based on the frame arrival deviation relationship and the buffer read / write status, ensuring that the frame buffer synchronization handover boundary is no earlier than the arrival time of the latest paired frame and no later than the end time of reading from the read buffer. The process of determining the frame buffer synchronization handover boundary specifically includes:

[0028] Read the latest paired frame arrival count and frame arrival span, and read the valid continuous read duration and pointer occupancy conflict status in the cache read status;

[0029] The common counter value corresponding to the snapshot of the unified latched buffer state when the latest pairing frame arrives is determined as the boundary calculation time; the sum of the latest pairing frame arrival count value and the protection waiting period is determined as the lower boundary of handover; and the sum of the boundary calculation time and the common count clock period converted from the effective continuous read duration is determined as the upper boundary of handover.

[0030] The sum of the lower boundary of handover, the reference handover waiting period, and the correction amount of the previous valid subsequent handover boundary is determined as the boundary candidate value, and the boundary candidate value is restricted to the lower boundary of handover and the upper boundary of handover to obtain the frame buffer synchronization handover boundary;

[0031] The frame buffer synchronization handover boundary is determined to be valid only when the frame arrival deviation relationship is valid, the frame arrival span does not exceed the pairing deviation upper limit, the buffer read state is valid, the handover upper boundary is not less than the handover lower boundary, and the pointer occupation conflict state is a conflict-free state.

[0032] Step S4 determines the lower boundary of the handover by the latest paired frame arrival count in the frame arrival deviation relationship, and determines the upper boundary of the handover by the effective continuous reading duration in the buffer reading state. Then, the candidate boundary values ​​are restricted between the lower boundary and the upper boundary of the handover to obtain the frame buffer synchronization handover boundary. Thus, the frame buffer synchronization handover boundary is jointly constrained by the arrival status of each video frame in the same period and the remaining reading capacity of the current read buffer, so that the buffer role switching time is not earlier than the arrival time of the latest paired frame and not later than the end time of the read buffer reading. This helps to avoid the buffer role switching when the paired video frame has not yet met the conditions for complete writing, and at the same time reduces the situation where the buffer role switching has not been performed after the current read buffer has ended its effective reading.

[0033] S5. In the control frame arrival deviation relationship, the first arriving video frame is held to the frame buffer synchronization handover boundary, generating the first arriving frame holding state, and generating a buffer role conversion permission when the frame buffer synchronization handover boundary is reached and all paired frames have been written.

[0034] The process of generating the frame hold state specifically includes:

[0035] Read the video channel order, arrival deviation, frame arrival span, frame start count value and latest paired frame arrival count value in the frame arrival deviation relationship. Check the arrival deviation and frame start count value of each video channel in order. Determine the video channel whose arrival deviation is less than the frame arrival span and whose frame start count value is less than the latest paired frame arrival count value as the first arriving video frame, and read the frame buffer synchronization handover boundary.

[0036] The hold period of each video channel is determined based on the difference between the frame buffer synchronization handover boundary and the frame start count value of each paired frame. The write buffer index is locked and the write pointer is prohibited from overwriting the corresponding write buffer from the start of writing to the write buffer of the corresponding video frame until the frame buffer synchronization handover boundary is reached.

[0037] When the frame buffer synchronization handover boundary is reached, the write buffer index, paired frame number and write completion flag are verified. The paired frame number, video channel identifier, write buffer index, hold start count value, hold end count value, write completion flag and hold valid status are used to form the hold status of the first arriving frame. When the hold valid status of all first arriving video frames is complete, the hold status of the first arriving frame is determined to be valid.

[0038] The process of generating a cached role conversion license specifically includes:

[0039] When the shared counter reaches the frame buffer synchronization handover boundary, read the paired frame number, video channel identifier and write buffer index in the cross-channel frame period pairing relationship, and read the write completion flag of each paired frame;

[0040] Based on the video channel identifier, each write buffer index is determined as a candidate index for the new read buffer of the corresponding video channel;

[0041] A cache role conversion permission is generated only when the frame buffer synchronization handover boundary is valid, the first-arriving frame is kept in the state, each paired frame has a write completion mark, each new read buffer candidate index does not overlap with the current read buffer index of the corresponding video channel, and the cache read status is valid.

[0042] The paired frame number, license generation count value, old read buffer index of each video channel, new read buffer candidate index and license validity status are used to form a cache role conversion license, and the license validity status is latched until the conversion is completed or the cache role conversion timeout occurs.

[0043] Step S5 identifies the first arriving video frame based on the frame arrival deviation relationship and locks the write buffer index corresponding to the first arriving video frame to the frame buffer synchronization handover boundary, forming a first arriving frame hold state. This ensures that the paired frame that has arrived and been written is not overwritten by the write pointer or used for other frame cycles while waiting for other paired frames. After the shared counter reaches the frame buffer synchronization handover boundary, a buffer role conversion permission is generated only when the first arriving frame hold state is established, each paired frame has a write completion mark, the write buffer index does not overlap with the current read buffer, and the buffer read status is valid. This allows all video frames corresponding to the same paired frame number to jointly obtain the buffer role conversion condition, reducing the possibility that incompletely written video frames are read by subsequent stages or that each video channel converts its buffer role separately.

[0044] S6. When the first-arrival frame hold state and buffer role transition permission are established, the write buffer is converted to the read buffer, a synchronization frame handover state is generated, and a synchronization video frame is output; the generation process of the synchronization frame handover state specifically includes:

[0045] Verify that the paired frame number in the first-arriving frame hold state matches the paired frame number in the cache role conversion permission, and verify that the write buffer index in the first-arriving frame hold state matches the new read buffer candidate index in the cache role conversion permission according to the video channel identifier;

[0046] After verification, within the same cache role conversion control cycle, the original write cache role is changed to the read cache role in the order of locking the new read cache, updating the read pointer, and releasing the old read cache.

[0047] It provides the paired frame number, new read buffer index and buffer base address to the subsequent read link, and outputs the synchronous video frame associated with the same paired frame number after the first valid burst read is completed in all video channels;

[0048] The actual conversion count value is latched, and the amount of unread data in the post-conversion read buffer of each video channel is obtained in the handover feedback window. The paired frame number, actual conversion count value, new read buffer index, old read buffer index, amount of unread data in the post-conversion read buffer, and handover validity status are used to form the synchronous frame handover status.

[0049] Step S6 verifies that the paired frame number in the first-arriving frame holding state matches the paired frame number in the cache role conversion permission, and verifies that the write buffer index in the first-arriving frame holding state matches the candidate index of the new read buffer in the cache role conversion permission according to the video channel identifier. Within the same cache role conversion control cycle, the new read buffer is locked, the read pointer is updated, and the old read buffer is released in sequence, so that each write buffer corresponding to the same paired frame number is converted into a read buffer according to a unified control timing. After all video channels complete the first valid burst read, a synchronous video frame is output, which allows the subsequent read links to obtain the cache address and video data associated with the same paired frame number, reducing the output of video frames with different frame periods due to different cache role conversion times of each video channel, and reducing the possibility of the subsequent reading of video frames that have not completed the cache role conversion. At the same time, a synchronous frame handover state is formed, providing a feedback basis for obtaining the actual conversion status later.

[0050] S7. Based on the frame arrival span corresponding to the synchronization frame handover state, the deviation of the transition time relative to the frame buffer synchronization handover boundary, and the amount of unread data in the read buffer after the transition, a subsequent handover boundary correction amount is generated. The frame buffer synchronization handover boundary of the next frame period is corrected using the subsequent handover boundary correction amount. The specific process of generating the subsequent handover boundary correction amount includes:

[0051] Only when the synchronization frame handover state is valid, read the frame arrival span, frame buffer synchronization handover boundary, actual conversion count value, and the amount of unread data in the post-conversion read buffer of each video channel;

[0052] The frame arrival span deviation is determined based on the difference between the current frame arrival span and the frame arrival span reference value. The conversion time deviation is determined based on the difference between the frame buffer synchronization handover boundary and the actual conversion count value. The buffer balance deviation is determined based on the difference between the minimum value of the unread data in the conversion read buffer of each video channel and the target value of the unread data in the conversion read buffer, and converted into a common counting clock cycle according to the reference read rate.

[0053] The frame arrival span deviation, transition time deviation and buffer balance deviation are synthesized according to arrival weight, transition weight and buffer weight, and the synthesis result is limited to the upper limit of the correction period to generate the subsequent handover boundary correction amount.

[0054] When determining the frame buffer synchronization handover boundary in the next frame period, the correction amount of the previous valid subsequent handover boundary is used to correct the boundary candidate value, and the boundary candidate value is restricted to the handover lower boundary determined by the arrival time of the latest paired frame and the handover upper boundary determined by the end time of reading from the read buffer.

[0055] Step S7 obtains the frame arrival span, actual conversion count, and unread data amount in the read buffer after conversion by synchronizing the frame handover status, and forms the frame arrival span deviation, conversion time deviation, and buffer remaining deviation respectively. Then, it generates the subsequent handover boundary correction amount according to the corresponding weights, so that the boundary candidate value of the next frame period can be adjusted according to the actual frame arrival situation, buffer role conversion execution situation, and buffer remaining amount after conversion in the previous frame period. The subsequent handover boundary correction amount is still limited to the lower handover boundary determined by the arrival time of the latest paired frame and the upper handover boundary determined by the end time of reading in the read buffer. Therefore, when the frame arrival interval or the subsequent reading status changes, the frame buffer synchronization handover boundary of the next frame period can be corrected, reducing the situation where the buffer role conversion time is earlier than the complete writing time of each video frame in the same period or later than the effective reading end time of the current read buffer.

[0056] Compared with the prior art, the beneficial effects of the present invention are:

[0057] 1. Based on further analysis and research of existing technical problems, this invention recognizes that the timing of buffer role switching needs to be constrained by both the actual arrival status of video frames in the same period and the remaining reading time of the current read buffer. To this end, this invention establishes cross-channel frame period pairing relationships and generates frame arrival deviation relationships through channel frame boundary sequences. At the same time, it generates buffer reading status based on the amount of unread data in the read buffer, thereby determining the frame buffer synchronization handover boundary no earlier than the arrival time of the latest paired frame and no later than the end time of reading in the read buffer. Then, the first arriving video frame is kept at the frame buffer synchronization handover boundary, and the buffer role switching is performed after all paired frames have been written. This can reduce the situation where incomplete video frames are read in the later stage, video frames of different frame periods are output accordingly, and gaps occur in video data reading.

[0058] 2. The present invention generates a subsequent handover boundary correction amount based on the frame arrival span, actual conversion count value and unread data amount in the read buffer after conversion in the synchronous frame handover state, and uses it to correct the frame buffer synchronous handover boundary of the next frame period, so that the subsequent buffer role conversion time can be adjusted according to the frame arrival difference and the change of the remaining read capacity of the buffer, further reducing the possibility that the buffer role conversion time exceeds the allowed handover period. Attached Figure Description

[0059] Figure 1 This is a schematic diagram of the overall method steps of the present invention;

[0060] Figure 2 This is a schematic diagram of the core process of the overall method of the present invention;

[0061] Figure 3 This is a schematic diagram of the core process for generating the channel frame boundary sequence in step S1 of the present invention.

[0062] Figure 4 This is a schematic diagram of the core process for generating the cross-channel frame period pairing and frame arrival deviation relationship in step S2 of the present invention.

[0063] Figure 5 This is a schematic diagram of the core process of writing the paired frame cache and generating the cache read state in step S3 of the present invention.

[0064] Figure 6 This is a schematic diagram of the core process for determining the frame buffer synchronization handover boundary in step S4 of the present invention.

[0065] Figure 7 This is a schematic diagram of the core process of maintaining the state of the first-arriving frame and generating the cache role conversion license in step S5 of the present invention.

[0066] Figure 8 This is a schematic diagram of the core process of the cyclic frame buffer role switching and synchronization frame handover in step S6 of the present invention;

[0067] Figure 9 This is a schematic diagram of the core process for generating the subsequent handover boundary correction amount in step S7 of the present invention. Detailed Implementation

[0068] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0069] Next, please refer to Figure 1-2 The purpose of this embodiment is to provide a frame-level synchronization access processing method for multi-source heterogeneous video streams, including steps S1 to S7, wherein:

[0070] S1. The FPGA parses multiple video streams according to the interface type, detects the frame start edge, latches the shared counter value at the start edge of each frame and increments the channel frame number that is uniformly reset, and generates a channel frame boundary sequence.

[0071] S2. Based on the channel frame boundary sequence, establish cross-channel frame period pairing relationships for video frames with the same frame number, compare the count values ​​of each paired frame, and generate frame arrival deviation relationships; where the same frame number means that the channel frame number of each video channel is the same.

[0072] S3. Use a circular frame buffer containing a write buffer and a read buffer. Write each paired frame to the write buffer and mark the write as complete. Determine the continuous read duration based on the read / write pointer and the amount of unread data in the read buffer, and generate the cache read status.

[0073] S4. Determine the frame buffer synchronization handover boundary based on the frame arrival deviation relationship and the buffer read status, so that the frame buffer synchronization handover boundary is not earlier than the arrival time of the latest paired frame and not later than the end time of reading the buffer area.

[0074] S5. In the control frame arrival deviation relationship, the first arriving video frame is held to the frame buffer synchronization handover boundary, generating the first arriving frame holding state, and generating a buffer role conversion permission when the frame buffer synchronization handover boundary is reached and all paired frames have been written.

[0075] S6. When the first-arrival frame hold state and the buffer role conversion permission are established, the write buffer is converted to the read buffer, a synchronous frame handover state is generated, and a synchronous video frame is output.

[0076] S7. Based on the frame arrival span corresponding to the synchronous frame handover state, the deviation of the conversion time relative to the frame buffer synchronous handover boundary, and the amount of unread data in the read buffer after conversion, generate the subsequent handover boundary correction amount, and use the subsequent handover boundary correction amount to correct the frame buffer synchronous handover boundary of the next frame period.

[0077] This embodiment uses an FPGA-based multi-channel video acquisition platform as its application foundation. FPGA stands for Field Programmable Gate Array, which serves as the execution entity for multi-channel video acquisition and frame buffer control in this embodiment. Video inputs include HDMI video streams, SDI video streams, and Cameralink video streams. HDMI is a high-definition multimedia interface; SDI is a serial digital interface; and Cameralink is the name of a digital camera interface standard. Video data is output after interface parsing, format processing, frame boundary detection, cyclic frame buffering, and PCIe transmission. PCIe is a high-speed peripheral component interconnection interface. This embodiment illustrates a frame-level synchronous access processing method for multi-source heterogeneous video streams, without limiting the number of video interfaces, pixel format, resolution, the specific value of the target frame rate used by each video channel, the number of cyclic frame buffers, or the PCIe link level. Each valid video channel participating in the same cross-channel frame period pairing relationship uses the same nominal frame rate.

[0078] In this embodiment, the FPGA uses a shared counting clock to drive a free-running counter, providing a unified timing reference for different video channels. Each video channel corresponds to a channel frame number and a set of cyclic frame buffers. The cyclic frame buffer includes a write buffer and a read buffer, with two or more buffers. The write buffer receives the current frame, and the read buffer provides the video frames from the previous frame period that have been handed over to the subsequent read link. Read and write pointers are used to identify the write buffer and the read buffer, and the unread data amount is used to indicate the number of data bytes in the read buffer that have not yet been read by the subsequent link.

[0079] The specific steps are as follows.

[0080] Please see Figure 3 S1. The FPGA parses multiple video streams according to interface type, detects the start edge of frames, latches a common counter value at the start edge of each frame, and increments the uniformly reset channel frame number to generate a channel frame boundary sequence. Step S1 includes the video stream access and verification process and the generation process of the channel frame boundary sequence.

[0081] The video stream access and verification process specifically includes:

[0082] HDMI video streams are obtained from the HDMI receiving interface, SDI video streams from the SDI receiving interface, and Cameralink video streams from the Cameralink receiving interface. Each interface outputs a pixel clock, frame synchronization signal, data validity signal, and pixel data. The interface parser organizes the pixel data into a unified internal pixel format according to the current interface configuration, and uses resolution, frame rate, effective pixel bit width, and effective line length as channel format parameters. YUV422 is a 4:2:2 pixel format, where Y represents the luminance component, U and V represent the two chrominance components, and 4:2:2 means that the luminance component is sampled at each pixel, and the two chrominance components are sampled at a horizontal ratio of half. This embodiment uses the YUV422 format, with each pixel occupying an average of 2 bytes, a resolution of 1920×1080, and a frame rate of 30 frames per second. When using other formats, the amount of data per frame is calculated based on the pixel bit width without changing the subsequent processing relationships.

[0083] After the FPGA locks the video interface, it begins receiving data. When two complete frames are detected consecutively, and the number of valid lines, the number of valid pixels per line, and the frame interval all fall within the allowable range of the channel format parameters, the video channel is determined to be in a valid access state. The allowable frame interval range is formed by a fluctuation of 5% above and below the nominal frame period. Before determining that multiple video channels are in a valid access state, it is also verified that each video channel participating in the same cross-channel frame period pairing relationship has the same nominal frame rate and the same nominal frame period. Video channels with inconsistent nominal frame rates or nominal frame periods are not included in the same set of valid video channels, nor do they participate in subsequent pairing of frame numbers within the same channel. When the video interface is not locked, the number of valid lines is zero, the number of valid pixels per line is inconsistent with the configuration, or the interval between two consecutive frames exceeds the allowable range, the channel frame number is not incremented, the corresponding video frame is not written to the circular frame buffer, and the system continues to wait for a new complete frame. If a complete frame cannot be obtained after three consecutive nominal frame periods, the valid access state of the corresponding video channel is cleared, and the channel frame number is triggered to be reset uniformly.

[0084] The shared counting clock comes from the FPGA's internal stable clock; this embodiment uses a 200 MHz shared counting clock, and the free-running counter uses 64-bit unsigned counting; the counting interval is calculated using the modulo-2 power 64 difference value when the counter wraps around; 200 MHz is only a value for this embodiment, and the shared counting clock can be in the range of 100 MHz to 300 MHz; when the clock frequency changes, all subsequent durations expressed in counting periods are converted according to the actual clock frequency;

[0085] Unified reset is performed when the system is powered on, all video interfaces reach a valid access state for the first time, any video interface is relocked, or no cross-channel frame period pairing relationship is formed for three consecutive frame periods. Unified reset clears the channel frame number of each video channel to zero and records the unified reset time. After unified reset, the first complete frame detected by each video channel is assigned channel frame number 1. When a video channel arrives at the first complete frame before other channels, only the frame start edge is recorded, and the channel is not allowed to enter the next channel frame number in advance until the pairing timeout or other video channels arrive.

[0086] The process of generating the channel frame boundary sequence specifically includes:

[0087] The FPGA performs dual-trigger synchronization on the frame synchronization signal of each video channel and detects the frame start edge that transitions from an invalid level to an effective level. To transmit the frame start edge across clock domains, the edge pulse can be extended to multiple consecutive shared counting clock cycles. In this embodiment, the edge pulse is extended by 8 shared counting clock cycles. The extension value is only used to ensure that the edge event is reliably sampled by the shared counting clock and is not used as the arrival time in the frame arrival deviation relationship.

[0088] In the same common counting clock cycle when a valid frame start edge is detected, the FPGA latches the current value of the free-running counter and determines the latched value as the frame start count value of the corresponding video frame; then, the corresponding channel frame number is incremented by 1, and the incremented channel frame number is associated with and saved with the frame start count value; the first complete frame after unified reset corresponds to channel frame number 1; a frame boundary record includes video channel identifier, channel frame number, frame start count value, channel format parameters, write buffer index, and boundary valid status; the video channel identifier is represented by a combination of interface type and channel number; the write buffer index comes from the write pointer maintained in step S3, and step S1 only reads this index and does not change the role of the circular frame buffer in step S1;

[0089] When multiple frame start edges are detected in the same video channel within a nominal frame period, only the earliest frame start edge that passes the format verification is retained, and subsequent edges are excluded as jitter edges; when the interval between two adjacent valid frame start edges is less than 50% of the nominal frame period, the latter edge is excluded; when the interval is greater than 150% of the nominal frame period, it is determined that there are missing frames in the middle, and the channel frame number is incremented according to the number of missing frames; when the number of missing frames cannot be determined, the current channel frame boundary sequence is terminated and a unified reset is performed.

[0090] When the shared counter does not wrap around, the frame boundary records of each video channel are arranged in ascending order of the shared counter value. When the shared counter wraps around, the frame start count value of the first frame boundary record formed in the same group is used as the reference count value. The modulo 2 power 64 positive difference between the other frame start count values ​​and the reference count value is calculated, and the frame boundary records of each video channel are arranged in ascending order of the positive difference to generate a channel frame boundary sequence. The channel frame boundary sequence is stored in an on-chip register group or on-chip memory and is updated each time a new frame boundary record is formed. When the channel frame boundary sequence does not contain all valid video channels, it remains in a waiting state. If it is still incomplete after reaching the pairing waiting threshold, the timeout processing in step S2 is executed.

[0091] The process of generating the channel frame boundary sequence obtains the channel frame number and frame start count value of each video channel under a unified timing reference, providing input for step S2 to establish cross-channel frame period pairing relationship and generate frame arrival deviation relationship; step S1 obtains the channel frame boundary sequence and passes the channel format parameters and write buffer index to steps S2 and S3.

[0092] Please see Figure 4 S2. Based on the channel frame boundary sequence, establish cross-channel frame period pairing relationships for video frames with the same frame number, compare the count values ​​of each paired frame, and generate frame arrival deviation relationships. Step S2 includes the process of establishing cross-channel frame period pairing relationships and the process of generating frame arrival deviation relationships.

[0093] The process of establishing cross-channel frame period pairing relationships specifically includes:

[0094] Read the channel frame boundary sequence obtained in step S1, and group the frame boundary records according to the channel frame number; each group must contain one frame boundary record for each valid video channel, and the channel frame numbers of each frame boundary record must be exactly the same; a valid video channel is a video channel that has the same nominal frame rate and the same nominal frame period as verified in step S1; when the above conditions are met, establish a cross-channel frame period pairing relationship for each video frame in the group; in this embodiment, each video frame that establishes a cross-channel frame period pairing relationship is uniformly referred to as a paired frame; the cross-channel frame period pairing relationship includes the paired frame number, the identifier of the video channel participating in the pairing, the frame start count value of each video channel, the write buffer index, and the pairing valid status;

[0095] The pairing wait threshold is set to 120% to 180% of the nominal frame period; in this embodiment, it is set to 150%. For a video stream of 30 frames per second, the nominal frame period is 33.333 milliseconds, and the pairing wait threshold is 50 milliseconds. The pairing wait time is calculated from the start edge of the first frame corresponding to the frame number of the same channel. If the wait time does not reach the pairing wait threshold, the video frames that have arrived and their write buffer indexes are retained, and incomplete cross-channel frame period pairing relationships are not established.

[0096] When the pairing wait time reaches the pairing wait threshold and there is still a missing frame boundary record for a video channel, the pairing validity status of the current group is cancelled, the data that has been written is marked as non-transferable data, and the corresponding write buffer is reused after being cleared in step S3; when one group fails to pair, subsequent groups continue to try to pair according to the channel frame number; when three groups fail to pair, the generation of new cross-channel frame period pairing relationships is stopped, and the unified reset in step S1 is triggered;

[0097] When a video channel has a duplicate channel frame number, the frame boundary record with the smaller shared counter value and passed format verification is retained; when the channel frame number of a video channel is more than one frame number ahead of other video channels, the unpaired record corresponding to the smallest channel frame number is identified as an expired record and excluded, until each video channel has the same channel frame number again; the exclusion process does not change the read buffer area that has already been handed over, to avoid abnormal pairing affecting the synchronous video frames being output;

[0098] Cross-channel frame period pairing relationships are stored using the paired frame number as an index, and the write buffer index of all paired video frames is read in step S3; one paired frame number corresponds to only one valid cross-channel frame period pairing relationship; after pairing is completed, the corresponding frame boundary record in the channel frame boundary sequence is changed to the paired state and no longer participates in other groupings;

[0099] The process of establishing the cross-channel frame period pairing relationship obtains the correspondence between video frames in the same frame period, providing input for the generation of frame arrival deviation relationship and the sub-channel buffer writing in step S3.

[0100] The process of generating the frame arrival deviation relationship specifically includes:

[0101] Read the starting count value of each frame in the cross-channel frame period pairing relationship; Indicates the number of valid video channels; This represents the video channel number and is used as the index for all video channel-related variables, with a value range of 1 to 1. ; This indicates the paired frame number and is uniformly used as a superscript for frame period-related variables; The video channel is in the first The frame start count value under each paired frame number is represented as follows: The shared counting clock frequency is expressed as The unit is Hertz; for all valid video channels Take the minimum value to get and will The earliest paired frame arrival count was determined; for all valid video channels... Take the maximum value to get and will The latest pairing frame arrival count value has been determined. This only represents the minimum frame start count value. This only indicates the maximum frame start count value; neither of these terms has any other technical meaning. and Used only as an operator for finding the minimum and maximum values; Indicated by To obtain a non-negative remainder modulo, used to handle the wrap-around of the 64-bit shared counter; The video channel is in the first The arrival deviation counting period under each paired frame number is represented as follows: The corresponding arrival deviation time is expressed as , No. The frame arrival span of a pair of frame numbers is represented as follows: It is determined according to the following relationship:

[0102] ;

[0103] ;

[0104] ;

[0105] ;

[0106] ;

[0107] Before performing minimum, maximum, and difference calculations, the frame start count values ​​within the same paired group are first unwound. If no 64-bit shared counter unwound occurs within the same paired group, the original frame start count value is used directly. If unwound occurs, the frame start count value of the earliest formed frame boundary record in the group is used as the reference count value. The modulo 2 power 64 positive difference between the other frame start count values ​​and the reference count value is calculated, and the arrival order is determined according to the positive difference. Subsequent minimum, maximum, and difference calculations are all performed based on the arrival order relationship after unwound unwinding.

[0108] Arrival deviation is represented by the shared counting clock cycle, and when converted to time, it is divided by the shared counting clock frequency. The frame arrival deviation relationship includes the arrival count value of the earliest paired frame, the arrival count value of the latest paired frame, the arrival deviation of each video channel, the frame arrival span, and the order of video channels in order of arrival. The video channel with an arrival deviation of zero is the earliest arriving video channel, and the video channel with an arrival deviation greater than zero is the later arriving video channel. When the shared counting clock frequency is missing or less than or equal to zero, the arrival deviation time conversion is not performed, a valid frame arrival deviation relationship is not generated, and the shared counting clock verification and unified reset of the channel frame number are triggered.

[0109] When the frame arrival span exceeds the pairing deviation limit, the cross-channel frame period pairing relationship is still retained, but the pairing valid state is changed to the out-of-tolerance state, and step S5 is not allowed to generate a normal buffer role conversion permission; the pairing deviation limit is set jointly based on the maximum jitter of the interface and the waiting time that the cyclic frame buffer can withstand, and the value range is 10% to 50% of the nominal frame period. In this embodiment, it is set to 10 milliseconds; when the frame arrival span is equal to the pairing deviation limit, it is processed according to the valid limit; when it exceeds the pairing deviation limit, it enters the out-of-tolerance branch;

[0110] When using the above-mentioned wrap-around unwinding process, the actual arrival span of each paired frame within the same pairing group is less than 2^63 shared counting clock cycles to ensure that the modulo 2^64 positive difference uniquely corresponds to the actual arrival order. If this condition is not met, the current cross-channel frame cycle pairing relationship is canceled, no frame arrival deviation relationship is generated, and it is processed as a pairing failure branch. If the starting count value of a frame is missing, is not a valid unsigned count, or the corresponding boundary valid state is invalid, the current cross-channel frame cycle pairing relationship is also canceled, no frame arrival deviation relationship is generated, and it is processed as a pairing failure branch.

[0111] The process of generating the frame arrival deviation relationship obtains the arrival order and frame arrival span of each paired video frame under a unified timing reference, providing input for step S4 to determine the frame buffer synchronization handover boundary and step S5 to identify the first arriving video frame; step S2 obtains the cross-channel frame period pairing relationship and frame arrival deviation relationship, and passes the write buffer index of each paired frame to step S3.

[0112] Please see Figure 5 S3. Using a circular frame buffer containing a write buffer and a read buffer, each paired frame is written to the write buffer and marked as written. The duration of continuous read supply is determined based on the read / write pointer and the amount of unread data in the read buffer, and a cache read supply state is generated. Step S3 includes the paired frame writing process and the generation process of the cache read supply state.

[0113] The pairing frame writing process specifically includes:

[0114] Each video channel is configured with a set of cyclic frame buffers, with the number of cyclic frame buffers ranging from 3 to 16. In this embodiment, the HDMI video channel is configured with 10 buffers, and the SDI video channel and Cameralink video channel are each configured with 3 buffers. The capacity of each buffer is not less than the data size of a complete video frame. For a 1920×1080 YUV422 video frame, the data size of each frame is 4,147,200 bytes. DDR stands for Double Data Rate, which in this embodiment refers to the external memory used to store video frames. Different video channels use non-overlapping DDR address ranges, and the write buffer base address is determined by the video channel address base value, the write pointer, and the single-frame address step size.

[0115] After pixel-width conversion and first-in-first-out (FIFO) buffering, video frames are written to DDR via AXI burst. AXI is an advanced scalable interface, a type of on-chip bus interface used in this embodiment to transmit video data between the video input clock domain and the DDR control clock domain. A write burst is initiated when the readable data in the FIFO buffer reaches the length of one burst. When the video frame ends and all burst write responses are completed, the corresponding write buffer is marked as write complete. The write complete marker is associated with and stored with the paired frame number, video channel identifier, and write buffer index.

[0116] When the number of bytes written equals the single-frame data size calculated by the channel format parameters, the length is validated. When the number of bytes written is less than the single-frame data size, an AXI write response error occurs, the first-in-first-out buffer overflows, or there is still a burst of unresponsive events after the video frame ends, the write is not marked as complete, and the corresponding paired frame is identified as a write error frame. Write error frames do not participate in the buffer role conversion permission judgment in step S5. When two consecutive paired frame numbers experience write errors, the write pointer of the corresponding video channel is stopped from advancing, and the interface is requested to be relocked.

[0117] After the corresponding video frame is written, the current write buffer enters the pending handover state; the write pointer only advances to the next buffer if the next buffer is free; the write pointer returns to the first buffer after advancing to the end of the circular frame buffer; the write pointer must not advance to the read buffer currently pointed to by the read pointer, nor to the write buffer that is in the pending handover state or has reached the frame hold state; when there is no free buffer, the external video source is not paused, nor is writing to the already occupied buffer continued, but the pixel data of the next complete video frame is excluded and the unavailable branch of the buffer is recorded; the frame boundary record and pending handover state of the current paired frame remain unchanged, and step S4 determines whether there is a valid frame buffer synchronization handover boundary based on the buffer read status.

[0118] The process of generating a cache read-read state specifically includes:

[0119] Read the read pointer, total number of bytes in the read buffer, subsequent read address, and subsequent average read rate for each video channel. The subsequent average read rate is determined by dividing the number of bytes read in the most recent complete frame period by the corresponding read duration. If the corresponding read duration is less than or equal to zero, the division is not performed, and the nominal output rate corresponding to the channel format parameters is used. If the nominal output rate is missing or less than or equal to zero, the buffer read status is determined to be invalid. If there is no effective read rate in the current frame period, the nominal output rate corresponding to the channel format parameters is used. The nominal output rate is equal to the product of the single frame data volume and the frame rate.

[0120] For the current paired frame, in step S2, the same common counting clock cycle or the adjacent buffer state sampling cycle for the latest paired frame to reach the count value is determined, and the amount of unread data in the read buffer of each video channel and the average read rate of the subsequent stage are latched. This unified latching time is used as the boundary calculation time in step S4, and the continuous reading duration and the effective continuous reading duration are both calculated based on the same set of buffer state snapshots at this time.

[0121] No. The amount of unread data in the read buffer of each video channel is represented as follows: The unit is bytes; the first The amount of securely reserved data for each video channel is represented as follows: The unit is bytes; the first The average read rate of the subsequent stage for each video channel is expressed as: The unit is bytes per second; the first The continuous reading duration of each video channel is expressed as: The unit is seconds; the first The effective continuous read duration corresponding to each paired frame number is represented as follows: The unit is seconds; the amount of data reserved for security purposes is used to cover PCIe read request latency, AXI burst interval, and rate calculation errors, and its value ranges from 5% to 20% of the data volume of a single frame; in this embodiment, it is 10%; in the formula... Indicates that within the same video channel, for minus The obtained value is the maximum of zero; the formula contains... This indicates that the results obtained from each of all valid video channels are... Take the minimum value; the continuous reading duration and the effective continuous reading duration are determined according to the following relationship:

[0122] ;

[0123] ;

[0124] When the amount of unread data in the read buffer is less than or equal to the amount of data to be safely reserved, the continuous read duration is set to zero. When the average read rate of the subsequent stage is missing or less than or equal to zero, the division in the continuous read duration formula is not performed, the continuous read duration is set to zero, and the abnormal read branch is entered. The minimum value of the continuous read duration of each video channel is used as the effective continuous read duration of the current cross-channel pairing frame, ensuring that no video channel will exhaust the read buffer data before other channels during the handover waiting period.

[0125] The buffer read status includes the read pointer, write pointer, unread data in the read buffer, continuous read duration, effective continuous read duration, write completion flag of each paired frame, and pointer occupancy conflict status for each video channel. When the write buffer index of any paired frame is the same as the current read buffer index of the corresponding video channel, or when the buffer corresponding to the write buffer index is occupied by different paired frames, the pointer occupancy conflict status is determined to be a conflict status; otherwise, it is determined to be a non-conflict status. When the read pointer or write pointer exceeds the range of the number of cyclic frame buffers, the pointer occupancy conflict status is a conflict status, or the unread data is greater than the total number of bytes in the read buffer, the buffer read status is determined to be an invalid status, the normal boundary determination in step S4 is stopped, and the original read buffer output is maintained.

[0126] When transmitting using the XDMA link based on the PCIe interface, the host reads the paired frame number and cache index through the register, and then reads the video data from the corresponding DDR address. XDMA is the interface name for the PCIe Direct Memory Access Link. In this embodiment, it is only used as a downstream reading link and does not change the rules for determining the frame buffer synchronization handover boundary. When the PCIe link is paused for a short time, the amount of unread data remains unchanged, and the continuous reading duration is recalculated according to the effective reading rate after recovery.

[0127] The process of generating the cache read status obtains the time boundary that the current read cache area can maintain synchronous waiting and the write completion status of each write cache area, providing input for step S4 to determine the frame cache synchronization handover boundary and step S5 to generate cache role conversion permission; step S3 obtains the cache read status and continues to update it until the handover is completed in the current frame period.

[0128] Please see Figure 6 S4. Determine the frame buffer synchronization handover boundary based on the frame arrival deviation relationship and the buffer read / write status, ensuring that the frame buffer synchronization handover boundary is no earlier than the arrival time of the latest paired frame and no later than the end time of the read buffer. Step S4 includes the process of determining the frame buffer synchronization handover boundary.

[0129] The process of determining the frame buffer synchronization handover boundary specifically includes:

[0130] Read the latest paired frame arrival count and frame arrival span obtained in step S2, and read the effective continuous read duration and pointer occupancy conflict status obtained in step S3; the protection wait period is expressed as... The unit is the common counting clock cycle, which is used to cover frame start edge sampling, cross-clock domain synchronization and the first AXI write burst start time; The value range corresponds to 0.2 milliseconds to 2 milliseconds. In this embodiment, 200,000 cycles corresponding to 1 millisecond are used.

[0131] The sum of the latest paired frame arrival count and the protection wait period is determined as the lower boundary of the handover. The unit is the count value of the shared counter; Ensure that the frame buffer synchronization handover boundary is not earlier than the arrival time of the latest paired frame; determine the first The shared counter value corresponding to the snapshot of the cached state of each paired frame number is uniformly latched when the latest paired frame arrives, representing the boundary calculation time. The effective continuous reading duration is converted into a shared counting clock cycle and then synchronized with the data. Add them together to get the upper boundary of the intersection. The unit is the count value of the shared counter; Ensure that the frame buffer synchronization handover boundary is no later than the end time of reading from the read buffer; symbol This indicates that the calculation result within the symbol is rounded down, that is, the largest integer not greater than the calculation result is taken; and Determined according to the following relationship:

[0132] ;

[0133] ;

[0134] The benchmark handover waiting period is expressed as The unit is the shared counting clock cycle; The count value of the last paired frame in the most recent 32 valid frame periods is used to complete the write operation, along with the corresponding... The median of the non-negative differences between them; when there are fewer than 8 valid samples, the number of shared counting clock cycles corresponding to 0.5 milliseconds is used, which is 100,000 cycles in this embodiment; the subsequent handover boundary correction amount obtained in step S7 for the previous valid frame cycle is expressed as: The unit is the shared counting clock cycle; when the system is running for the first time, after a unified reset there is no effective correction amount, or when the synchronization frame handover has not been completed for three consecutive frame cycles, Initialize to zero; , and The sum is used as a boundary candidate value, and the boundary candidate value is restricted to... and Between, we obtain the first The frame buffer synchronization handover boundary of each paired frame number The unit is the count value of the shared counter, and the formula is expressed as follows:

[0135] ;

[0136] Boundary constraint process adopts first AND Compare, and then with The order of comparison; the boundary candidate value is less than hour, Pick Boundary candidate values ​​are greater than hour, Pick Boundary candidate values ​​are located at and In between, Take the boundary candidate values;

[0137] A valid handover interval exists when the upper handover boundary is greater than or equal to the lower handover boundary; when the upper handover boundary is less than the lower handover boundary, the read buffer can no longer continuously provide reading before the latest paired frame arrives and the protection waiting period has passed, and there is no frame buffer synchronization handover boundary that satisfies the double boundary constraints in the current frame period; at this time, a normal boundary is not generated, step S5 is not allowed to generate buffer role conversion permission, the last complete synchronized video frame of the previous read buffer is maintained for output, and the current cross-channel frame period pairing relationship is determined as a read-supply conflict state; when two consecutive frame periods have read-supply conflict states, the channel frame number is uniformly reset and the cross-channel frame period pairing relationship is re-established.

[0138] When the frame arrival span exceeds the pairing deviation limit, the buffer read state is invalid, or the pointer occupancy conflict state is in a conflict state, a normal frame buffer synchronization handover boundary is not generated; the frame buffer synchronization handover boundary is valid only when the frame arrival deviation relationship is valid, the buffer read state is valid, the pointer occupancy conflict state is in a conflict-free state, and the handover upper boundary is not less than the handover lower boundary; the valid boundary is associated with the paired frame number and saved for reading in steps S5 and S7;

[0139] The process of determining the frame buffer synchronization handover boundary yields a frame buffer synchronization handover boundary that is simultaneously limited by the arrival time of the latest paired frame and the end time of reading from the read buffer area; the frame buffer synchronization handover boundary is used in step S5 to control both the hold-off time of the first arriving video frame and the generation time of the buffer role switching permission.

[0140] Please see Figure 7S5. The first arriving video frame in the control frame arrival deviation relationship is held up to the frame buffer synchronization handover boundary, generating a first arriving frame holding state. When the frame buffer synchronization handover boundary is reached and all paired frames have been written, a buffer role conversion permission is generated. Step S5 includes the process of generating the first arriving frame holding state and the process of generating the buffer role conversion permission.

[0141] The process of generating the frame hold state specifically includes:

[0142] Read the video channel order and arrival deviation of each video channel obtained in step S2, and read the video channel order obtained in step S4. ;No. The video channel is in the first The hold period under each paired frame number is represented as follows: The unit is the common counting clock cycle. equal and difference; A value less than zero indicates an anomaly in the boundary calculation. Determined to be invalid; When the value is greater than or equal to zero, starting from the beginning of writing to the write buffer for the corresponding video frame, the write buffer index is locked and the write pointer is prohibited from overwriting the buffer, until the value is reached. :

[0143] ;

[0144] The hold status of the first arriving frame includes the paired frame number, video channel identifier, write buffer index, hold start count value, hold end count value, write completion flag, and hold valid status. According to the order of arrival of video channels, the arrival deviation and frame start count value of each video channel are checked in turn. The video channel whose arrival deviation is less than the frame arrival span and whose frame start count value is less than the arrival count value of the latest paired frame is determined as the first arriving video frame. The hold period of the latest arriving video frame still includes a protection wait period, which is used to wait for its first batch of data to enter the write buffer stably.

[0145] During the hold period, video frame data continues to be written to the corresponding write buffer according to the normal AXI write process; the hold action does not pause video interface reception, nor does it repeat the writing of pixel data that has already been completed; the technical meaning of hold is to lock the occupancy relationship between the current paired frame and the write buffer, and prevent the write pointer from releasing or overwriting the write buffer before the handover; if a write exception occurs before a certain first-arriving video frame reaches the hold cutoff count value, the hold status of the first-arriving frame is changed to an invalid state, and the buffer role conversion permission for this frame period is canceled;

[0146] When the shared counter reaches the hold cutoff count value, check whether the corresponding write buffer index is still consistent with the cross-channel frame period pairing relationship; if the index is consistent and the write completion flag is valid, the hold valid state of the corresponding video channel is determined to be complete; if the index is inconsistent, the frame number is inconsistent, or the write completion flag is invalid, the hold valid state is determined to be failed; if all first-arriving video frames are complete, the first-arriving frame hold state is established; if there is a failure state, the first-arriving frame hold state is not established.

[0147] The process of generating the hold state of the first arriving frames obtains the occupancy status of each first arriving video frame held to the frame buffer synchronization handover boundary, providing a prerequisite for the generation of buffer role conversion permission and the execution of the write buffer and read buffer role conversion in step S6.

[0148] The process of generating a cached role conversion license specifically includes:

[0149] When the shared counter reaches the frame buffer synchronization handover boundary, all video channel identifiers and write buffer indices in the cross-channel frame period pairing relationship are read, and the write completion flag continuously updated in step S3 is read; according to the video channel identifier, each write buffer index is determined as the candidate index of the new read buffer for the corresponding video channel; a buffer role conversion permission is generated only when the frame buffer synchronization handover boundary is valid, the first-arriving frame is kept in the state, each paired video frame has a write completion flag, each candidate index of the new read buffer does not overlap with the current read buffer index of the corresponding video channel, and the buffer read status is still valid.

[0150] The cache role conversion permission includes the paired frame number, permission generation count value, old read buffer index of each video channel, new read buffer candidate index, and permission validity status; the new read buffer candidate index is equal to the write buffer index of the corresponding paired video frame in the cross-channel frame period pairing relationship; after the permission validity status is formed, it is latched until step S6 to complete the conversion or when the conversion timeout occurs;

[0151] If a paired video frame has not been written when the frame buffer synchronization handover boundary is reached, no buffer role conversion permission is generated; if the read buffer still has continuous reading capacity, the original read buffer continues to output, but the already determined frame buffer synchronization handover boundary is not extended; paired frames that have not been written are excluded as timeout frames; if there is still no buffer role conversion permission when the read buffer stops reading, the previous complete synchronized video frame is output and the re-pairing process begins.

[0152] If, after a permit is generated, it is found that any candidate index of the new read buffer has changed, the write completion mark has been revoked, or the paired frame number is inconsistent, the cache role conversion permit shall be revoked immediately; the permit revocation shall not change the read pointer of the old read buffer, and the role conversion shall not be performed in step S6.

[0153] The generation process of the cache role conversion permission obtains an action permission that is consistent with the frame buffer synchronization handover boundary, the first-arriving frame hold state, and the state of all paired frames written to complete, providing direct input for step S6 to execute the cyclic frame buffer role conversion; step S5 obtains the first-arriving frame hold state and the cache role conversion permission.

[0154] Please see Figure 8 S6. When the first-arrival frame hold state and buffer role transition permission are established, the write buffer is converted to the read buffer, a synchronization frame handover state is generated, and a synchronization video frame is output. Step S6 includes the process of generating the synchronization frame handover state.

[0155] The process of generating the synchronization frame handover state specifically includes:

[0156] Read the first-arrival frame hold state and cache role conversion permission obtained in step S5, verify that the paired frame number in the first-arrival frame hold state matches the paired frame number in the cache role conversion permission, and verify that the write buffer index in the first-arrival frame hold state matches the new read buffer candidate index in the cache role conversion permission according to the video channel identifier; after the verification is successful, within the same cache role conversion control cycle, save the old read pointer of each video channel as the index to be released, write the new read buffer candidate index to the read pointer, and change the original write buffer role to the read buffer role; the role conversion adopts the order of locking the new read buffer first, then updating the read pointer, and finally releasing the old read buffer to avoid the read pointer pointing to the buffer that is being written or has already been released during the conversion;

[0157] After the role switch, the AXI register provides the paired frame number, new read buffer index, and buffer base address to the subsequent read link. After receiving the same paired frame number, the subsequent read link reads the complete video frame from the new read buffer of each video channel. When using the XDMA link output based on the PCIe interface, the host first reads the paired frame number and new read buffer index, and then initiates a DMA read from the device to the host based on the buffer base address. DMA stands for Direct Memory Access, which in this embodiment means that video data is transferred between the device memory and the host memory by the data transfer link. When all video channels have completed the first valid burst read, the output of the synchronous video frame associated with the same paired frame number begins.

[0158] Synchronization frame handover status includes paired frame number, actual conversion count value, new read buffer index for each video channel, old read buffer index, amount of unread data in the post-conversion read buffer, first valid read count value, and handover validity status; The actual conversion count value for each paired frame number is represented as follows: And latch it when the read pointer is updated for the last video channel; The video channel is in the first The amount of unread data in the post-translation read buffer under each paired frame number is represented as follows: The unit is bytes; After all video channels complete their first valid burst read, the data is acquired through a handover feedback window; the handover feedback window is between 0.2 milliseconds and 1 millisecond, and in this embodiment, it is 0.5 milliseconds; the same handover feedback window is used to acquire the data from each video channel. This ensures that subsequent comparisons have the same time reference.

[0159] All video channels' read pointers were successfully updated, paired frame numbers matched, and no read errors occurred in the handover feedback window for each new read buffer. When the value is greater than zero, the synchronization frame handover state is valid; if the read pointer update of any video channel fails, the updated read pointer is rolled back within the buffer role conversion timeout threshold, so that all video channels are restored to the old read buffer area; the buffer role conversion timeout threshold is 0.1 milliseconds to 1 millisecond, and is 0.5 milliseconds in this embodiment; if the rollback is successful, no valid synchronization frame handover state is generated, and the old read buffer area output is maintained; if the rollback fails, DMA reading is stopped and a unified reset is requested.

[0160] If, during the output process, it is detected that the length of the new read buffer data of a certain video channel is less than a complete video frame, the data group will not be output as a synchronous video frame. Instead, the handover valid status will be changed to failure, and the previous complete synchronous video frame will be used to fill the current output cycle. After the output link is restored, steps S4 to S6 will be re-executed from the next valid cross-channel frame cycle pairing relationship.

[0161] The old read buffer is released as an idle buffer only after the first valid read has been completed for all video channels; after release, the write pointer can reuse the buffer in subsequent frame cycles; in this way, the buffer role change not only changes the read pointer, but also completes the synchronous transfer from the old read buffer to the new read buffer.

[0162] The process of generating the synchronization frame handover state obtains the count value of the actual completed buffer role conversion, the amount of unread data in each read buffer after conversion, and the output synchronization video frames, providing input for step S7 to generate the subsequent handover boundary correction amount; step S6 obtains the synchronization frame handover state and outputs the synchronization video frames associated with the state.

[0163] Please see Figure 9 S7. Based on the frame arrival span corresponding to the synchronization frame handover state, the deviation of the transition time relative to the frame buffer synchronization handover boundary, and the amount of unread data in the read buffer after the transition, a subsequent handover boundary correction amount is generated. The frame buffer synchronization handover boundary of the next frame period is corrected using the subsequent handover boundary correction amount. Step S7 includes the process of generating the subsequent handover boundary correction amount.

[0164] The process of generating subsequent handover boundary corrections specifically includes:

[0165] Read the frame arrival span obtained in step S2, read the frame buffer synchronization handover boundary obtained in step S4, and read the actual conversion count value and the amount of unread data in each read buffer area after conversion obtained in step S6; update the subsequent handover boundary correction only when the synchronization frame handover state is valid; keep the previous valid subsequent handover boundary correction unchanged when the synchronization frame handover state is invalid, a read pointer rollback occurs, or the current frame period does not form a buffer role conversion permission; reset the subsequent handover boundary correction to zero when there is no valid synchronization frame handover state for three consecutive frame periods.

[0166] No. The frame arrival span reference value for each paired frame number is represented as follows: The median of the frame arrival span corresponding to the handover status of the most recent 32 valid synchronization frames is used; when there are fewer than 8 valid samples, the initial frame arrival span reference value in the interface configuration is used, which in this embodiment is 1.8 milliseconds corresponding to 360,000 shared counting clock cycles; the first The frame arrival span deviation for each paired frame number is expressed as: The unit is the common counting clock cycle. equal minus ;

[0167] No. The conversion time deviation of each paired frame number is expressed as: The unit is the common counting clock cycle. equal minus ; A negative value indicates that the role transition occurred later than This is used to shift the boundary candidate values ​​of the next frame period forward; A positive value indicates that the role transition occurred earlier than This is used to shift the boundary candidate values ​​of the next frame period to the right; deviations in absolute value generated by the register update pipeline that do not exceed one common counting clock cycle are treated as zero.

[0168] The target value for the amount of unread data in the read buffer after conversion is expressed as follows: The unit is bytes, and it represents 90% to 98% of the data volume of a single frame; in this embodiment, it is 92%. (Each video channel...) The minimum value in is represented as the first Unread data volume after valid conversion of each paired frame number The reference read rate used to convert byte differences into count cycles is expressed as: The unit is bytes per second, and it is the minimum value among the nominal output rates of all valid video channels; The buffer clearance deviation for each paired frame number is expressed as: The unit is the shared counting clock cycle; Higher than hour, A positive value indicates that the read buffer has ample read margin, allowing the boundary candidate value to be shifted appropriately in the next frame period; Below hour, A negative value indicates that the read buffer has insufficient read capacity, and the boundary candidate value needs to be shifted forward appropriately in the next frame period.

[0169] The arrival weight is represented as The transformation weights are expressed as Cache weight is represented as All three are dimensionless values ​​and their sum is 1; This indicates the proportion of the impact of inter-frame temporal changes on the next boundary. This indicates the proportion of the impact of the cache role transition execution deviation on the next boundary. This indicates the proportion of the converted read margin to the next boundary; in this embodiment... , and They are 0.50, 0.30, and 0.20 respectively; the upper limit of the correction period is expressed as... The unit is the shared counting clock cycle, which is 400,000 cycles in this embodiment; The subsequent handover boundary correction amount corresponding to each paired frame number is expressed as follows: ; This means that the first calculation result is limited to between the last two boundary values; the result is negative when it is lower than the upper limit of the negative correction period. When it exceeds the upper limit of the positive correction period, take ; Determined according to the following relationship:

[0170] ;

[0171] ;

[0172] ;

[0173] ;

[0174] Restricted to - and Between; in this embodiment This is 400,000 shared counting clock cycles, corresponding to 2 milliseconds; It is represented by a shared counting clock cycle and is associated with and stored in relation to the current paired frame number, generation time, and the validity of the synchronization frame handover status.

[0175] When step S4 is executed in the next frame cycle, the corresponding , Same as the previous one The sum is used as a candidate boundary value; positive values Shift the boundary candidate values ​​to the right, negative values This shifts the boundary candidate values ​​forward; the boundary candidate values ​​continue to be affected by the next frame period. and Limitations, therefore the correction will not make Even if the arrival time is earlier than the latest paired frame, it will not cause Later than the end time of reading from the buffer;

[0176] When a weight input or reference value is missing, no new subsequent handover boundary correction is calculated, and the previous valid value is retained. When the reference read rate is missing or less than or equal to zero, the buffer margin deviation calculation is not performed, no new subsequent handover boundary correction is generated, and the previous valid subsequent handover boundary correction is retained. When a reference read rate greater than zero cannot be obtained for three consecutive frame periods, the subsequent handover boundary correction is reset to zero. When the sum of the three weights deviates from 1 by more than 0.001, normalization is performed according to the sum of the three weights. When the sum of the weights is zero, the default weight of this embodiment is used. When the calculation result is not a finite value, the subsequent handover boundary correction is reset to zero, and the current frame period is recorded as not participating in the subsequent reference value update.

[0177] The process of generating the subsequent handover boundary correction amount yields a subsequent handover boundary correction amount that reflects the frame arrival time, the execution lag of the buffer role conversion, and the read margin after the conversion, providing input for determining the frame buffer synchronization handover boundary in step S4 of the next frame cycle; step S7 completes the feedback of the execution status of the current frame cycle to the process of determining the boundary of the next frame cycle.

[0178] In this embodiment, the arrival weight, transformation weight, and cache weight are calibrated using records of no less than 300 effective frame periods. The selection condition is to minimize the verification error consisting of the number of handover failures, the number of read conflicts, and the average hold time. When the sample is insufficient, 0.50, 0.30, and 0.20 are used, and the sum of the three weights is guaranteed to be 1.

[0179] In this embodiment, the pairing wait threshold, pairing deviation upper limit, security retention data volume ratio, protection wait period, and cache role conversion timeout threshold are determined based on the interface nominal frame period, maximum observation jitter, PCIe read latency, and DDR burst response time. When the parameters exceed the limit range, they are corrected to the nearest boundary value. If a valid frame cache synchronization handover boundary cannot be formed after correction, the cache role conversion of the current frame period is stopped.

[0180] To further illustrate the working process of this embodiment, a specific operation process is given below:

[0181] Three 1920×1080 YUV422 video streams at 30 frames per second are used, with each pixel occupying an average of 2 bytes and a single frame data volume of 4,147,200 bytes. The three video streams are connected via HDMI, SDI and Cameralink interfaces respectively, sharing a common counting clock frequency of 200 MHz and a common counting clock cycle of 5 nanoseconds. After a unified reset, the video frame corresponding to channel frame number 100 is used as the object of this numerical implementation.

[0182] S1: All three video interfaces are in a valid access state. The number of valid lines in two consecutive complete frames is 1080, the number of valid pixels in each line is 1920, and the frame interval is within the allowable range of 5% fluctuation above and below the nominal frame period. The FPGA detects the frame start edge corresponding to channel frame number 100. The frame start count values ​​of HDMI video frame, SDI video frame and Cameralink video frame are 2000000000, 2000180000 and 2000420000, respectively. The FPGA associates and saves the video channel identifier, channel frame number, frame start count value, channel format parameters and write buffer index, and generates the channel frame boundary sequence according to the frame start count value from small to large. If any interface is not locked, the number of valid lines is zero, or a complete frame is not formed in three consecutive nominal frame periods, the corresponding record is not added to the channel frame boundary sequence, and a unified reset is performed.

[0183] S2. The three frame boundary records generated in step S1 have the same channel frame number 100, so a cross-channel frame cycle pairing relationship is established. The minimum value of the three frame start count values ​​is 2000000000, and the maximum value is 2000420000. The arrival deviations of HDMI video frames, SDI video frames, and Cameralink video frames are 0, 180000, and 420000 shared counting clock cycles, respectively, corresponding to 0 milliseconds, 0.90 milliseconds, and 2.10 milliseconds. The frame arrival span is 420000 cycles. The pairing deviation limit in this embodiment is 10 milliseconds. The current frame arrival span does not exceed the pairing deviation limit, so a valid frame arrival deviation relationship is generated. When the pairing wait time reaches 50 milliseconds and there is still no frame boundary record for any video channel, the cross-channel frame cycle pairing relationship corresponding to channel frame number 100 is canceled, and the re-pairing process is initiated.

[0184] S3: The three paired video frames are written to their respective write buffers in their circular frame buffers. The HDMI, SDI, and Cameralink video frames complete their AXI write bursts at counts of 2000510000, 2000630000, and 2000680000, respectively, with each frame writing 4,147,200 bytes. Therefore, the write completion markers for all three channels are valid. The unread data amounts in the read buffers of the three video channels are 3,360,000 bytes, 3,240,000 bytes, and 3,300,000 bytes, respectively. The total amount of data to be retained is 414,720 bytes, and the average read rate of the subsequent stage is 124,416,000 bytes per second. The calculated continuous read durations for the three paths are 23.673 milliseconds, 22.708 milliseconds, and 23.191 milliseconds, respectively. The minimum value of 22.708 milliseconds is determined as the effective continuous read duration, and an effective cache read state is generated. If the number of bytes written on any path is less than 4,147,200 bytes, an AXI write response error occurs, or the average read rate of the subsequent stage is zero, the cache read state is determined to be invalid, and the original read buffer output is maintained.

[0185] S4. When the latest paired frame arrives, the latch boundary calculation time is 2000420000. The protection wait period is 200000 shared counting clock cycles, corresponding to 1 millisecond. Therefore, the lower handover boundary is 2000620000. The effective continuous read duration of 22.708 milliseconds is converted into 4541666 shared counting clock cycles, resulting in the upper handover boundary of 2004961666. The reference handover wait period is 100000 cycles, and the subsequent handover boundary correction amount of the previous effective frame cycle is 0. The boundary candidate value is 2000720000. This boundary candidate value is greater than the lower handover boundary and less than the upper handover boundary. Therefore, 2000720000 is determined as the frame buffer synchronization handover boundary. When the upper handover boundary is less than the lower handover boundary, a valid frame buffer synchronization handover boundary is not generated, and the current paired frame is not allowed to perform buffer role conversion.

[0186] S5. Based on the frame buffer synchronization handover boundary determined in step S4, the hold periods for HDMI video frames, SDI video frames, and Cameralink video frames are 720,000, 540,000, and 300,000 shared count clock cycles, respectively. Within each hold period, the occupancy relationship between the paired video frame and the corresponding write buffer is locked, and the write pointer is prohibited from overwriting the corresponding write buffer. When the shared counter reaches 2000720000, all three video frames have been written, and the write buffer index is consistent with the cross-channel frame cycle pairing relationship. Therefore, the first-to-arrive frame hold state is established, and a buffer role conversion permission is generated. If any video frame has not been written or the write buffer index has changed at the frame buffer synchronization handover boundary, no buffer role conversion permission is generated, and the current paired frame is excluded as a timeout frame.

[0187] S6. After the first-arrival frame hold state and buffer role conversion permission are both established, first lock the three new read buffers, then update the three read pointers, and finally release the three old read buffers. When the last video channel completes the read pointer update, the latched actual conversion count value is 2000760000, and the actual conversion time is 40000 shared count clock cycles later than the frame buffer synchronization handover boundary. After a 0.5-millisecond handover feedback window, the unread data amounts in the three new read buffers are 4100000 bytes, 4080000 bytes, and 4090000 bytes, respectively. The three read pointers are successfully updated and the paired frame numbers are consistent. Therefore, a valid synchronization frame handover state is generated, and the synchronous video frame corresponding to the channel frame number 100 is output using the XDMA link based on the PCIe interface. If any read pointer fails to update within 0.5 milliseconds, the already updated read pointer is rolled back and the old read buffer continues to be used for output.

[0188] S7. The frame arrival span reference value is taken as 360,000 shared counting clock cycles. The current frame arrival span is 420,000 cycles, therefore the frame arrival span deviation is 60,000 cycles. The actual conversion count value is 20,007,600,000, and the frame buffer synchronization handover boundary is 20,007,200,000. The conversion time deviation calculated by subtracting the actual conversion count value from the frame buffer synchronization handover boundary is -40,000 cycles. The minimum unread data amount in the three-way conversion read buffer is 4,080,000 bytes, the target value is 3,815,424 bytes, and the reference read rate is 124,416,000 bytes per second. The calculated buffer remaining deviation is approximately 425. 309 cycles; using an arrival weight of 0.50, a transformation weight of 0.30, and a buffer weight of 0.20, the subsequent handover boundary correction amount is approximately 103,062 cycles, which does not exceed the upper limit of 400,000 cycles for correction cycles; the next frame cycle adds 103,062 cycles to the sum of the lower handover boundary and the baseline handover waiting cycle, shifting the boundary candidate value approximately 0.515 milliseconds backward relative to the uncorrected state, and continuing to be constrained by the lower and upper handover boundaries of the next frame cycle; the subsequent handover boundary correction amount is not updated when the synchronization frame handover state is invalid, and the subsequent handover boundary correction amount is reset to zero when no valid synchronization frame handover state is obtained for three consecutive frame cycles.

[0189] As can be seen from the above description, the frame-level synchronization access processing method for multi-source heterogeneous video streams provided in this embodiment has the following technical effects:

[0190] By establishing cross-channel frame cycle pairing relationships and generating frame arrival deviation relationships through channel frame boundary sequences, and generating buffer read status based on the amount of unread data in the read buffer, the frame buffer synchronization handover boundary is determined to be no earlier than the arrival time of the latest paired frame and no later than the end time of read buffer read. By keeping the first arriving video frame to the frame buffer synchronization handover boundary and generating buffer role conversion permission after all paired frames have been written, the write buffer is converted to the read buffer and synchronous video frames are output. This reduces the occurrence of incomplete video frames read by subsequent stages, video frames of different frame cycles being output accordingly, and gaps in video data reading. In addition, based on the frame arrival span, actual conversion count value, and amount of unread data in the read buffer after conversion in the synchronous frame handover status, a subsequent handover boundary correction amount is generated to correct the frame buffer synchronization handover boundary of the next frame cycle. This allows the buffer role conversion time to be adjusted according to the frame arrival difference, actual conversion deviation, and remaining buffer read capacity, reducing the possibility that the buffer role conversion time exceeds the handover period limited by the lower and upper handover boundaries.

[0191] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A frame-level synchronization access processing method for multi-source heterogeneous video streams, characterized in that, The methods and steps include the following: S1. The FPGA parses multiple video streams according to the interface type, detects the frame start edge, latches the shared counter value at the start edge of each frame and increments the channel frame number that is uniformly reset, and generates a channel frame boundary sequence. S2. Based on the channel frame boundary sequence, establish a cross-channel frame period pairing relationship for video frames with the same frame number, compare the count values ​​of each pairing frame, and generate a frame arrival deviation relationship. S3. Use a circular frame buffer containing a write buffer and a read buffer. Write each paired frame to the write buffer and mark the write as complete. Determine the continuous read duration based on the read / write pointer and the amount of unread data in the read buffer, and generate the cache read status. S4. Determine the frame buffer synchronization handover boundary based on the frame arrival deviation relationship and the buffer read status, so that the frame buffer synchronization handover boundary is not earlier than the arrival time of the latest paired frame and not later than the end time of reading the buffer area. S5. Control the first arriving video frame in the frame arrival deviation relationship to remain at the frame buffer synchronization handover boundary, generate the first arriving frame holding state, and generate a buffer role conversion permission when the frame buffer synchronization handover boundary is reached and all paired frames have been written. S6. When the first-arrival frame hold state and the cache role conversion permission are established, the write buffer is converted into a read buffer, a synchronization frame handover state is generated, and a synchronization video frame is output. S7. Based on the frame arrival span corresponding to the synchronization frame handover state, the deviation of the conversion time relative to the frame buffer synchronization handover boundary, and the amount of unread data in the read buffer after conversion, a subsequent handover boundary correction amount is generated, and the frame buffer synchronization handover boundary of the next frame period is corrected with the subsequent handover boundary correction amount.

2. The frame-level synchronization access processing method for multi-source heterogeneous video streams according to claim 1, characterized in that, The process of generating the channel frame boundary sequence specifically includes: The frame synchronization signal of each video channel is synchronized and the frame start edge is detected. When a valid frame start edge is detected, the current value of the free run counter is latched, the current value is determined as the frame start count value, and the incremented channel frame number is associated with and saved with the frame start count value. The video channel identifier, channel frame number, frame start count value, channel format parameters, write buffer index, and boundary validity status are used to form a frame boundary record. The frame boundary records of each video channel are arranged according to the shared counter value to generate the channel frame boundary sequence.

3. The frame-level synchronization access processing method for multi-source heterogeneous video streams according to claim 1, characterized in that, The process of establishing the cross-channel frame period pairing relationship specifically includes: Read the channel frame boundary sequence, group the frame boundary records according to the channel frame number, and when there is a frame boundary record for each valid video channel in a group and the channel frame number of each frame boundary record is the same, establish the cross-channel frame period pairing relationship between the video frames in the group. The cross-channel frame period pairing relationship is formed by the pairing frame number, the video channel identifier involved in the pairing, the frame start count value of each video channel, the write buffer index, and the pairing validity status. When the pairing wait time reaches the pairing wait threshold and the group is still incomplete, cancel the pairing validity status of the current group and mark the data that has been written as non-transferable data; after pairing is completed, change the corresponding frame boundary record to the paired status.

4. The frame-level synchronization access processing method for multi-source heterogeneous video streams according to claim 3, characterized in that, The process of generating the frame arrival deviation relationship specifically includes: Read the frame start count value of each paired frame in the cross-channel frame period pairing relationship, and take the minimum and maximum values ​​of each frame start count value to obtain the earliest paired frame arrival count value and the latest paired frame arrival count value; The arrival deviation of each video channel is determined based on the difference between the frame start count value of each paired frame and the arrival count value of the earliest paired frame. The frame arrival span is determined based on the difference between the arrival count value of the latest paired frame and the arrival count value of the earliest paired frame, and the video channel order is formed according to the arrival order. When the span of the frame exceeds the upper limit of the pairing deviation, the pairing valid state is changed to the out-of-tolerance state.

5. The frame-level synchronization access processing method for multi-source heterogeneous video streams according to claim 1, characterized in that, The process of generating the cache read-read state specifically includes: Read the read pointer, write pointer, total number of bytes in the read buffer, amount of unread data in the read buffer, and average read rate of each video channel; The continuous reading duration for each video channel is determined based on the amount of unread data in the read buffer minus the amount of data reserved for security purposes and the average read rate of the subsequent stage. The continuous reading duration is set to zero when the amount of unread data in the read buffer is less than or equal to the amount of data reserved for security purposes. The minimum value among the continuous reading durations of each video channel is determined as the effective continuous reading duration, and the read pointer, write pointer, unread data in the read buffer, continuous reading duration, effective continuous reading duration, write completion flag of each paired frame, and pointer occupation conflict status are used to form the buffer reading status. When the write buffer index of any paired frame is the same as the current read buffer index of the corresponding video channel, or when the buffer corresponding to the write buffer index has been occupied by different paired frames, the pointer occupancy conflict state is determined to be a conflict state; otherwise, it is determined to be a non-conflict state. When the read pointer or write pointer exceeds the range of the number of cyclic frame buffers, the pointer occupancy conflict state is a conflict state, or the amount of unread data in the read buffer is greater than the total number of bytes in the read buffer, the buffer read state is determined to be an invalid state.

6. The frame-level synchronization access processing method for multi-source heterogeneous video streams according to claim 4, characterized in that, The process of determining the frame buffer synchronization handover boundary specifically includes: Read the latest paired frame arrival count and the frame arrival span, and read the valid continuous read duration and pointer occupancy conflict status in the cache read status; The common counter value corresponding to the snapshot of the unified latched cache state when the latest pairing frame arrives is determined as the boundary calculation time; the sum of the count value of the latest pairing frame arrival and the protection waiting period is determined as the lower boundary of handover; and the sum of the boundary calculation time and the common count clock period converted from the effective continuous read duration is determined as the upper boundary of handover. The sum of the lower handover boundary, the reference handover waiting period, and the correction amount of the previous valid subsequent handover boundary is determined as the boundary candidate value, and the boundary candidate value is restricted between the lower handover boundary and the upper handover boundary to obtain the frame buffer synchronization handover boundary; The frame buffer synchronization handover boundary is determined to be valid only when the frame arrival deviation relationship is valid, the frame arrival span does not exceed the pairing deviation upper limit, the buffer read state is valid, the handover upper boundary is not less than the handover lower boundary, and the pointer occupation conflict state is a conflict-free state.

7. The frame-level synchronization access processing method for multi-source heterogeneous video streams according to claim 1, characterized in that, The process of generating the first-arrival frame hold state specifically includes: Read the video channel order, arrival deviation of each video channel, frame arrival span, frame start count value of each paired frame, and arrival count value of the latest paired frame in the frame arrival deviation relationship. Check the arrival deviation and frame start count value of each video channel in the order of video channels. Determine the video channel whose arrival deviation is less than the frame arrival span and whose frame start count value is less than the arrival count value of the latest paired frame as the first arriving video frame, and read the frame buffer synchronization handover boundary. The hold period of each video channel is determined based on the difference between the frame buffer synchronization handover boundary and the frame start count value of each paired frame. The write buffer index is locked and the write pointer is prohibited from overwriting the corresponding write buffer from the start of writing to the write buffer of the corresponding video frame until the frame buffer synchronization handover boundary is reached. When the frame buffer synchronization handover boundary is reached, the write buffer index, paired frame number and write completion flag are verified. The paired frame number, video channel identifier, write buffer index, hold start count value, hold end count value, write completion flag and hold valid state are used to form the first-arriving frame hold state. When the hold valid state of all first-arriving video frames is complete, the first-arriving frame hold state is determined to be established.

8. The frame-level synchronization access processing method for multi-source heterogeneous video streams according to claim 7, characterized in that, The process of generating the cached role conversion permission specifically includes: When the shared counter reaches the frame buffer synchronization handover boundary, the paired frame number, video channel identifier and write buffer index in the cross-channel frame period pairing relationship are read, and the write completion flag of each paired frame is read. Based on the video channel identifier, each write buffer index is determined as a candidate index for the new read buffer of the corresponding video channel; The cache role conversion permission is generated only when the frame buffer synchronization handover boundary is valid, the first-arriving frame hold state is established, each paired frame has a write completion mark, each new read buffer candidate index does not overlap with the current read buffer index of the corresponding video channel, and the cache read state is valid. The paired frame number, the license generation count value, the old read buffer index of each video channel, the new read buffer candidate index, and the license validity status are used to form the cache role conversion license, and the license validity status is latched until the conversion is completed or the cache role conversion timeout occurs.

9. The frame-level synchronization access processing method for multi-source heterogeneous video streams according to claim 8, characterized in that, The process of generating the synchronization frame handover state specifically includes: Verify that the paired frame number in the first-arriving frame hold state is consistent with the paired frame number in the cache role conversion permission, and verify that the write buffer index in the first-arriving frame hold state is consistent with the new read buffer candidate index in the cache role conversion permission according to the video channel identifier; After verification, within the same cache role conversion control cycle, the original write cache role is changed to the read cache role in the order of locking the new read cache, updating the read pointer, and releasing the old read cache. Provide the paired frame number, new read buffer index and buffer base address to the subsequent read link, and output the synchronous video frame associated with the same paired frame number after the first valid burst read is completed in all video channels; The actual conversion count value is latched, and the amount of unread data in the post-conversion read buffer of each video channel is obtained in the handover feedback window. The pairing frame number, actual conversion count value, new read buffer index, old read buffer index, amount of unread data in the post-conversion read buffer, and handover validity status are used to form the synchronization frame handover status.

10. The frame-level synchronization access processing method for multi-source heterogeneous video streams according to claim 1, characterized in that, The process of generating the subsequent handover boundary correction amount specifically includes: Only when the synchronization frame handover state is valid, read the frame arrival span, the frame buffer synchronization handover boundary, the actual conversion count value, and the amount of unread data in the post-conversion read buffer of each video channel; The frame arrival span deviation is determined based on the difference between the current frame arrival span and the frame arrival span reference value. The conversion time deviation is determined based on the difference between the frame buffer synchronization handover boundary and the actual conversion count value. The buffer balance deviation is determined based on the difference between the minimum value of the unread data in the conversion read buffer of each video channel and the target value of the unread data in the conversion read buffer, and converted into a common counting clock cycle according to the reference read rate. The frame arrival span deviation, the conversion time deviation, and the buffer balance deviation are synthesized according to the arrival weight, the conversion weight, and the buffer balance deviation, and the synthesis result is limited to the upper limit of the correction period to generate the subsequent handover boundary correction amount. When determining the frame buffer synchronization handover boundary in the next frame period, the previous valid subsequent handover boundary correction amount is used to correct the boundary candidate value, and the boundary candidate value is restricted to the handover lower boundary determined by the arrival time of the latest paired frame and the handover upper boundary determined by the read buffer end time.

Images