Multi-channel video processing method

Abstract

The invention relates to the technical field of multi-channel video stitching, in particular to a multi-channel video processing method. The method comprises the following steps: decoding n paths of videos to be subjected to panoramic stitching to obtain n paths of video frame sequences; video frames included in each video frame sequence are stored in corresponding queues according to the sequence of acquisition time from morning to evening; obtaining the number of video frames stored in each queue in the n queues at the current moment; if qi is smaller than a first preset value, frame loss processing is not carried out on the queue of the ith path; if qi is greater than or equal to a first preset value and less than a second preset value, performing first preset frame loss processing on the ith queue; if qi is greater than or equal to a second preset value and less than a third preset value, performing second preset frame loss processing on the ith queue; and if qi is greater than or equal to a third preset value, carrying out third preset frame loss processing on the ith queue. According to the invention, real-time synchronous processing of multiple paths of videos can be carried out before panoramic stitching.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-channel video splicing, and in particular to a multi-channel video processing method. Background Art

[0002] With the rapid development of my country's civil aviation industry, the scale of medium and large busy airports continues to expand and the layout becomes more complex. Large ground traffic flow and high density have become normalized. The apron control has problems such as visual difficulties and heavy control workload. Unsafe incidents / accidents occur frequently, affecting the efficiency of airport operations. Multiple groups of high-definition cameras are deployed on the top of the terminal building and the top of the control tower at the airport to collect visual data and surveillance data of nearby aircraft positions. The data is transmitted to the airport center computer room through the computer room security network for splicing and fusion processing, and an entire large-scene ultra-high-definition visual presentation screen is output, which is then transmitted to the control seat of the apron tower control center to replace the visual command of the commander.

[0003] In actual airport applications, the video captured by traditional mainstream cameras is 2K (i.e., the resolution is 1920×1080 pixels). When encoding and decoding 2K videos, the processing efficiency requirements are low, and there is no need to synchronize the multi-channel 2K videos to be spliced; while the video captured by the target mainstream camera has been converted to high-definition 4K (i.e., the resolution is 3840×2160 pixels). When encoding and decoding 4K videos, the processing efficiency requirements are high, and it is necessary to synchronize the multi-channel 4K videos to be spliced. In addition, multiple groups of front-end camera equipment are usually installed on outdoor commanding heights such as the top of the terminal building and the top of the tower building, and the splicing server is often placed in the airport central computer room. Due to the complex airport network environment, the video data of the front-end camera equipment is often transmitted from different installation points to the airport central computer room. It often jumps several routes, resulting in different degrees of delay between the video data of the multi-channel front-end cameras obtained by the splicing server in the central computer room. If the real-time synchronization of multiple videos is not performed before the panoramic stitching stage, the aircraft smear and crossing phenomena will appear in the output panoramic video screen. How to perform real-time synchronous processing of multiple videos before panoramic stitching is an urgent problem to be solved. Summary of the invention

[0004] The present invention aims to provide a method for processing multi-channel videos, so as to perform real-time synchronous processing of multi-channel videos before panoramic stitching, so as to avoid the phenomenon of aircraft smear and crossing in the output panoramic video picture.

[0005] According to the present invention, a method for processing multi-channel videos is provided, the method comprising the following steps:

[0006] S100, decoding n channels of videos to be panoramically stitched to obtain n channels of video frame sequences, where n is the number of channels of videos to be panoramically stitched, and n≥2.

[0007] S200, storing the video frames included in each video frame sequence in a corresponding queue in the order of acquisition time from earliest to latest; the queue has a first-in-first-out feature.

[0008] S300, obtaining the number of video frames stored in each of the n queues at the current moment.

[0009] S400, if q i If q is less than the first preset value, no frame drop processing is performed on the queue corresponding to the i-th video channel; i is greater than or equal to the first preset value and less than the second preset value, the queue corresponding to the i-th video channel is subjected to the first preset frame loss processing; if q i is greater than or equal to the second preset value and less than the third preset value, the queue corresponding to the i-th video channel is subjected to the second preset frame loss processing; if q i is greater than or equal to a third preset value, the queue corresponding to the i-th video channel is subjected to a third preset frame loss process; the first preset value is less than the second preset value, the second preset value is less than the third preset value, the frame loss rate of the first preset frame loss process is less than the frame loss rate of the second preset frame loss process, and the frame loss rate of the second preset frame loss process is less than the frame loss rate of the third preset frame loss process; q i is the number of video frames stored in the queue corresponding to the i-th video at the current moment, and the value of i ranges from 1 to n.

[0010] Compared with the prior art, the present invention has at least the following beneficial effects:

[0011] The present invention stores the video frames corresponding to each of the n-channel videos to be panoramically stitched in a corresponding queue in the order of acquisition time from earliest to latest. Since the queue has a first-in-first-out feature, the video frames corresponding to each channel of video with earlier acquisition time are transmitted to the stitching server for stitching at the earliest. The present invention obtains the number of video frames in the queue corresponding to each channel of video. If the number of video frames in the queue corresponding to a certain channel of video is greater, it means that the delay phenomenon of the channel of video in the process of being transmitted to the stitching server is more serious. The present invention determines the number of video frames in the queue corresponding to any channel of video according to the number of video frames in the queue corresponding to any channel of video. For the frame loss processing method of the video frames in the queue corresponding to the video, by dividing the number of video frames in the queue into 4 levels, 4 different frame loss processing methods are determined, including no frame loss and 3 frame loss processing methods with different frame loss rates; wherein, the frame loss processing method with a larger number of video frames in the corresponding queue has a larger frame loss rate, which is conducive to quickly realizing the synchronization of video frames of n-channel videos before panoramic stitching; thus, the present invention realizes the real-time synchronization of n-channel video frames to be panoramic stitching before panoramic stitching, which can avoid the phenomenon of aircraft dragging and crossing in the output panoramic video picture. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0013] Figure 1 A flowchart of a method for processing multiple channels of video provided by an embodiment of the present invention;

[0014] Figure 2 A schematic diagram of video frame delay when synchronization is not performed provided by an embodiment of the present invention;

[0015] Figure 3 A schematic diagram of video frame delay when the frame loss rate is 1 / 2 provided by an embodiment of the present invention;

[0016] Figure 4 A schematic diagram of video frame delay when the frame loss rate is 1 / 4 provided in an embodiment of the present invention;

[0017] Figure 5 A schematic diagram of video frame delay when the frame loss rate is 1 / 3 provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0018] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0019] According to this embodiment, Figure 1 As shown, a method for processing multi-channel videos is provided, the method comprising the following steps:

[0020] S100, decoding n channels of videos to be panoramically stitched to obtain n channels of video frame sequences, where n is the number of channels of videos to be panoramically stitched, and n≥2.

[0021] As a specific implementation, each of the n videos to be panoramically stitched is a 4K video, and the video obtained after panoramic stitching is an 8K video.

[0022] Those skilled in the art know that any decoding method in the prior art falls within the protection scope of the present invention.

[0023] S200, storing the video frames included in each video frame sequence in a corresponding queue in the order of acquisition time from earliest to latest; the queue has a first-in-first-out feature.

[0024] In this embodiment, the number of queues is equal to the number of videos to be panoramically stitched, each queue corresponds to one video to be panoramically stitched, and different queues are used to store video frames obtained after decoding different videos to be panoramically stitched; if n=4, then the number of queues is also 4; if n=8, then the number of queues is also 8.

[0025] S300, obtaining the number of video frames stored in each of the n queues at the current moment.

[0026] In this embodiment, the frame rates of any two of the n videos are the same, and the number of video frames stored in any queue is related to the speed at which the video frames stored in the queue are transmitted to the stitching server. If there is no delay in the transmission of the video frames obtained after the n videos are decoded to the stitching server, then there will be no retention of video frames in the n queues corresponding to the n videos; if the number of video frames stored in a queue is large, it is determined that the speed at which the video frames stored in the queue are transmitted to the stitching server is slow, and the delay in the video frames stored in the queue being transmitted to the stitching server is large; if the number of video frames stored in a queue is small, it is determined that the speed at which the video frames stored in the queue are transmitted to the stitching server is fast, and the delay in the video frames stored in the queue being transmitted to the stitching server is small.

[0027] S400, if q i If q is less than the first preset value, no frame drop processing is performed on the queue corresponding to the i-th video channel; i is greater than or equal to the first preset value and less than the second preset value, the queue corresponding to the i-th video channel is subjected to the first preset frame loss processing; if q i is greater than or equal to the second preset value and less than the third preset value, the queue corresponding to the i-th video channel is subjected to the second preset frame loss processing; if q i is greater than or equal to a third preset value, the queue corresponding to the i-th video channel is subjected to a third preset frame loss process; the first preset value is less than the second preset value, the second preset value is less than the third preset value, the frame loss rate of the first preset frame loss process is less than the frame loss rate of the second preset frame loss process, and the frame loss rate of the second preset frame loss process is less than the frame loss rate of the third preset frame loss process; q i is the number of video frames stored in the queue corresponding to the i-th video at the current moment, and the value of i ranges from 1 to n.

[0028] As a preferred specific implementation, the xth preset value is m x , the frame loss rate of the xth preset frame loss process is d x , d x Satisfies the following relationship: d x ≥m x ×T / (z×t0), z is the number of image frames generated per unit time by any video to be stitched in panorama, t0 is the preset synchronization time, T is the unit time, T=1s, x=1, 2, 3. Based on this preferred specific implementation, it is beneficial to achieve synchronization of n videos within t0. Optionally, t0 is an empirical value.

[0029] As a more preferred specific implementation, when the frame rate of any video to be panoramically stitched is 25fps (i.e., the number of image frames generated within 1s is 25), the first preset value is 3, the second preset value is 6, and the third preset value is 9. The frame loss rate of the first preset frame loss processing is 1 / 3, the frame loss rate of the second preset frame loss processing is 1 / 2, and the frame loss rate of the third preset frame loss processing is 2 / 3. It should be understood that a frame loss rate of 1 / 3 means that 1 frame is lost in every 3 frames and 2 frames are retained; a frame loss rate of 1 / 2 means that 1 frame is lost in every 2 frames and 1 frame is retained; a frame loss rate of 2 / 3 means that 2 frames are lost in every 3 frames and 1 frame is retained.

[0030] As a specific implementation, when the frame loss rate of the first preset frame loss processing is 1 / 3, the process of performing the first preset frame loss processing on the queue corresponding to the i-th video channel includes:

[0031] S410, obtaining the frame number corresponding to each video frame in the queue corresponding to the i-th video.

[0032] In this embodiment, if a video frame in the queue corresponding to the i-th video is the a-th frame captured by the i-th video acquisition device, then the frame number corresponding to the video frame is a; for example, if a video frame in the queue corresponding to the i-th video is the 10th frame captured by the i-th video acquisition device, then the frame number corresponding to the video frame is 10; if a video frame in the queue corresponding to the i-th video is the 100th frame captured by the i-th video acquisition device, then the frame number corresponding to the video frame is 100.

[0033] S411, if the number of frames corresponding to the jth video frame in the queue corresponding to the i-th video is c i,j , and c i,j %3=0, then the j-th video frame in the queue corresponding to the i-th video is discarded; otherwise, the j-th video frame in the queue corresponding to the i-th video is retained; % is the remainder.

[0034] Based on S410 - S411 , the first preset frame loss processing with a frame loss rate of 1 / 3 can be performed on the video frames in the queue corresponding to the i-th video.

[0035] As a specific implementation, when the frame loss rate of the second preset frame loss processing is 1 / 2, the process of performing the second preset frame loss processing on the queue corresponding to the i-th video channel includes:

[0036] S420, obtaining the frame number corresponding to each video frame in the queue corresponding to the i-th video.

[0037] S421, if the number of frames corresponding to the jth video frame in the queue corresponding to the i-th video is c i,j , and c i,j %2=0, the j-th video frame in the queue corresponding to the i-th video is discarded; otherwise, the j-th video frame in the queue corresponding to the i-th video is retained; % is the remainder.

[0038] Based on S420-S421, the second preset frame loss processing with a frame loss rate of 1 / 2 can be performed on the video frames in the queue corresponding to the i-th video.

[0039] As a specific implementation, when the frame loss rate of the third preset frame loss processing is 2 / 3, the process of performing the third preset frame loss processing on the queue corresponding to the i-th video channel includes:

[0040] S430, obtaining the frame number corresponding to each video frame in the queue corresponding to the i-th video.

[0041] S431, if the number of frames corresponding to the jth video frame in the queue corresponding to the i-th video is c i,j , and c i,j%3=0, then the j-th video frame in the queue corresponding to the i-th video is retained; otherwise, the j-th video frame in the queue corresponding to the i-th video is discarded; % is the remainder.

[0042] Based on S430-S431, the third preset frame loss processing with a frame loss rate of 2 / 3 can be performed on the video frames in the queue corresponding to the i-th video.

[0043] As a specific implementation, n=3, each video ID is 1000, 1001 and 1002 respectively, each video is a 4K video, and the frame rate of each video is 25fps. When the three videos are not synchronized, the video frame delay between the videos can reach 3.2 seconds (i.e., 80 frames of timestamp difference). Figure 2 As shown; compared with the case where the three-way video is not synchronized, the processing method based on this implementation can effectively reduce the video frame delay amplitude between the three-way video and improve the synchronization between the video frames. Among them, when the frame loss rate of the frame loss processing is 1 / 2, the frame loss speed is relatively fast, the frame rate of the video output after splicing changes relatively greatly (up to 23-27fps), and the timestamp difference of the video frames of different channels is relatively small (1-2 frames of timestamp difference), as shown Figure 3 As shown in the figure, when the frame loss rate of the frame loss processing is 1 / 4, the frame loss speed becomes relatively slow, the timestamp difference of video frames from different channels is relatively large (up to 5 frames of timestamp difference), and the synchronization of video frames from different channels is relatively poor, such as Figure 4 As shown in the figure, when the frame loss rate of the frame loss processing is 1 / 3, the frame loss speed is moderate, the frame rate of the video output after splicing changes relatively stably (24-26fps), and the timestamp difference of the video frames of different channels is moderate (1-3 frames). Figure 5 shown.

[0044] In this embodiment, the video frames corresponding to each of the n videos to be panoramically stitched are stored in the corresponding queues in the order of acquisition time from earliest to latest. Since the queue has a first-in-first-out feature, the video frames corresponding to each video with earlier acquisition time are transmitted to the stitching server for stitching at the earliest. In this embodiment, the number of video frames in the queue corresponding to each video is obtained. If the number of video frames in the queue corresponding to a certain video is larger, it means that the delay of the video in the process of being transmitted to the stitching server is more serious. In this embodiment, the number of video frames in the queue corresponding to any video is determined according to the number of video frames in the queue corresponding to any video. A frame loss processing method for the video frames in the queue corresponding to the video is determined. By dividing the number of video frames in the queue into four levels, four different frame loss processing methods are determined, including no frame loss and three frame loss processing methods with different frame loss rates. Among them, a frame loss processing method with a larger number of video frames in the corresponding queue has a larger frame loss rate, which is conducive to quickly realizing video frame synchronization of n videos before panoramic stitching. Therefore, this embodiment realizes real-time synchronization of n video frames to be panoramic stitched before panoramic stitching, which can avoid the phenomenon of aircraft dragging and crossing in the output panoramic video picture.

[0045] As a specific implementation, after S400, the method further includes the following steps:

[0046] S500: Determine whether the splicing server has received n video frames transmitted by the queues in the current splicing cycle.

[0047] In this embodiment, the stitching server is used to implement the panoramic stitching process, which performs stitching according to the video frames received in each stitching cycle; if the frame rate of each of the n videos is 25fps, then the stitching cycle is 0.04s, and the frame rate of the panoramic video obtained after stitching is also 25fps.

[0048] S600, if the stitching server receives video frames transmitted by n queues in the current stitching cycle, panoramic stitching is performed according to the received video frames transmitted by the n queues; otherwise, enter S700.

[0049] Those skilled in the art know that any panoramic stitching method in the prior art falls within the protection scope of the present invention.

[0050] S700, obtaining video frames to be transmitted by the queue of untransmitted video frames in a current splicing cycle according to video frames transmitted by the queue of untransmitted video frames in a historical splicing cycle.

[0051] In this embodiment, the historical splicing cycle refers to the splicing cycle before the current splicing cycle.

[0052] Those skilled in the art are aware that any compensation frame prediction method in the prior art falls within the protection scope of the present invention.

[0053] S800: Perform panoramic stitching according to video frames to be transmitted in a current stitching cycle by a queue of non-transmitted video frames and video frames transmitted in a current stitching cycle by a queue of transmitted video frames.

[0054] Based on S500-S800, the splicing process of n-channel videos can be realized.

[0055] Although some specific embodiments of the present invention have been described in detail by way of example, it will be appreciated by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the present invention. It will also be appreciated by those skilled in the art that various modifications may be made to the embodiments without departing from the scope and spirit of the present invention. The scope of the present invention is defined by the appended claims.

Claims

1. A method for processing multi-channel videos, characterized in that: The method comprises the following steps: S100, decoding n channels of videos to be panoramically stitched to obtain n channels of video frame sequences, where n is the number of channels of videos to be panoramically stitched, and n≥2; S200, storing the video frames included in each video frame sequence in a corresponding queue in the order of acquisition time from earliest to latest; the queue has a first-in-first-out feature; S300, obtaining the number of video frames stored in each of the n queues at the current moment; S400, if q i If q is less than the first preset value, no frame drop processing is performed on the queue corresponding to the i-th video channel; i is greater than or equal to the first preset value and less than the second preset value, the queue corresponding to the i-th video channel is subjected to the first preset frame loss processing; if q i is greater than or equal to the second preset value and less than the third preset value, the queue corresponding to the i-th video channel is subjected to the second preset frame loss processing; if q i is greater than or equal to a third preset value, the queue corresponding to the i-th video channel is subjected to a third preset frame loss process; the first preset value is less than the second preset value, the second preset value is less than the third preset value, the frame loss rate of the first preset frame loss process is less than the frame loss rate of the second preset frame loss process, and the frame loss rate of the second preset frame loss process is less than the frame loss rate of the third preset frame loss process; q i is the number of video frames stored in the queue corresponding to the i-th video at the current moment, and the value of i ranges from 1 to n.

2. The method for processing multi-channel videos according to claim 1, characterized in that: The xth preset value is m x , the frame loss rate of the xth preset frame loss process is d x , d x Satisfies the following relationship: d x ≥m x ×T / (z×t0), z is the number of image frames generated per unit time by any video to be panoramically stitched, t0 is the preset synchronization time, T is the unit time, T=1s, x=1,2,3.

3. The method for processing multi-channel videos according to claim 2, characterized in that: When the frame rate of any video to be panoramic stitched is 25fps, the first preset value is 3, the second preset value is 6, and the third preset value is 9. The frame loss rate of the first preset frame loss processing is 1 / 3, the frame loss rate of the second preset frame loss processing is 1 / 2, and the frame loss rate of the third preset frame loss processing is 2 / 3.

4. The method for processing multi-channel videos according to claim 3, characterized in that: The process of performing the first preset frame loss processing on the queue corresponding to the i-th video channel includes: S410, obtaining the frame number corresponding to each video frame in the queue corresponding to the i-th video; S411, if the number of frames corresponding to the jth video frame in the queue corresponding to the i-th video is c i,j , and c i,j %3=0, then the j-th video frame in the queue corresponding to the i-th video is discarded; otherwise, the j-th video frame in the queue corresponding to the i-th video is retained; % is the remainder.

5. The method for processing multi-channel videos according to claim 3, characterized in that: The process of performing the second preset frame loss processing on the queue corresponding to the i-th video channel includes: S420, obtaining the frame number corresponding to each video frame in the queue corresponding to the i-th video; S421, if the number of frames corresponding to the jth video frame in the queue corresponding to the i-th video is c i,j , and c i,j %2=0, the j-th video frame in the queue corresponding to the i-th video is discarded; otherwise, the j-th video frame in the queue corresponding to the i-th video is retained; % is the remainder.

6. The method for processing multi-channel videos according to claim 3, characterized in that: The process of performing the third preset frame loss processing on the queue corresponding to the i-th video channel includes: S430, obtaining the frame number corresponding to each video frame in the queue corresponding to the i-th video; S431, if the number of frames corresponding to the jth video frame in the queue corresponding to the i-th video is c i,j , and c i,j %3=0, then the j-th video frame in the queue corresponding to the i-th video is retained; otherwise, the j-th video frame in the queue corresponding to the i-th video is discarded; % is the remainder.

7. The method for processing multi-channel videos according to claim 1, characterized in that: After S400, the method further includes the following steps: S500, determining whether the splicing server has received n video frames transmitted by the queues within the current splicing cycle; S600, if the stitching server receives video frames transmitted by n queues in the current stitching cycle, perform panoramic stitching according to the received video frames transmitted by the n queues; otherwise, enter S700; S700, obtaining video frames to be transmitted by the queue of untransmitted video frames in the current splicing cycle according to video frames transmitted by the queue of untransmitted video frames in the historical splicing cycle; S800: Perform panoramic stitching according to video frames to be transmitted in a current stitching cycle by a queue of non-transmitted video frames and video frames transmitted in a current stitching cycle by a queue of transmitted video frames.

Images