Multi-path video splicing method and device and storage medium

Abstract

The application provides a multi-path video splicing method and device and a storage medium, and belongs to the technical field of video processing. The method comprises the following steps: receiving multi-path video streams in real time through a WebRTC interface, and continuously performing timestamp synchronization on the received multi-path video streams; performing a video splicing operation on a first group of video frames after synchronization by using preset initial splicing strategy parameters to obtain a first splicing frame; for each group of video frames to be spliced subsequently, the following steps are sequentially performed: inputting a splicing quality index and a time delay index of the current spliced video frame into a reinforcement learning model, and obtaining adjusted target splicing parameters output by the reinforcement learning model; applying the adjusted target splicing parameters to the next group of video frames for splicing, and calculating the splicing quality index and the time delay index after splicing, and repeating the above steps until no new multi-path video stream is input or a stop instruction is received.

Description

Technical Field

[0001] This application relates to the field of video processing technology, specifically to a method, apparatus, and storage medium for multi-channel video splicing. Background Technology

[0002] With the development of internet video communication technology, WebRTC technology has been widely used in scenarios such as multi-party video conferencing, security monitoring, and remote collaboration. To meet the needs of multi-view viewing, it is often necessary to stitch multiple video streams into a single composite image. However, traditional video stitching methods are mostly based on feature point matching such as SIFT and SURF, typically completing feature extraction, registration, and fusion within a preset overlapping area (or a fixed scale / window). Affected by factors such as lighting changes, camera shake, resolution differences, and dynamic scene changes, obvious stitching seams, registration deviations, misalignments, and uneven fusion are prone to occur, and it is difficult to simultaneously ensure stitching quality and real-time performance with limited computing resources. Therefore, there is an urgent need for a video stitching method that guarantees both video stitching quality and real-time performance. Summary of the Invention

[0003] The purpose of this application is to provide a method, apparatus, and storage medium for multi-channel video splicing.

[0004] To achieve the above objectives, the first aspect of this application provides a multi-channel video stitching method, comprising: Multiple video streams are received in real time via the WebRTC interface, and the received video streams are continuously timestamped. The video splicing operation is performed on the first group of video frames in the synchronized multi-channel video frames using the preset initial splicing strategy parameters to obtain the first spliced ​​frame. The splicing quality index and latency index of the first spliced ​​frame are calculated. The latency index includes at least the end-to-end latency estimate, which is the total time taken for the video frame to travel from the transmitting end to the receiving end. For each subsequent group of video frames to be stitched, perform the following steps sequentially until no new multi-channel video frames are input or a stop command is received: The stitching quality and latency metrics of the current stitched video frame are input into the reinforcement learning model, and the adjusted target stitching parameters output by the reinforcement learning model are obtained. Apply the adjusted target stitching parameters to the next set of video frames for stitching, and calculate the stitching quality index and latency index after stitching. Repeat the above steps until there are no new multi-channel video stream inputs or a stop command is received.

[0005] In this embodiment, the reinforcement learning model is trained using a joint reward function. This joint reward function is used to calculate a stitching error penalty term based on stitching quality metrics, and a timeout penalty term based on the comparison between the end-to-end latency estimate and a preset target latency threshold. The stitching quality metrics include at least edge discontinuity, brightness difference, inlier rate, and geometric reprojection error. The joint reward function is:

[0006] in, For the joint reward function, For edge discontinuity, For brightness difference, This is for geometric reprojection error. For interior point ratio, This is the estimated end-to-end delay. To preset the target latency threshold, These are the weighting parameters for edge discontinuity, brightness difference, geometric reprojection error, inlier rate, and end-to-end delay estimate, respectively.

[0007] In this embodiment, the reinforcement learning model includes a policy network and a value network. The stitching quality and latency metrics of the currently stitched video frames are input into the reinforcement learning model, and the adjusted target stitching parameters output by the reinforcement learning model are obtained. This includes: obtaining the video stitching quality, latency, stitching parameters, video frame scene change metrics, and policy parameters for the decision period corresponding to the video frames within a preset decision period; for each group of video frames, constructing a state vector for the video frames based on the stitching quality, video frame scene change, and latency metrics; constructing a continuous parameter vector corresponding to the decision period using the policy network based on the stitching parameters; determining the advantage function corresponding to the policy parameters using the value network based on the state vector, continuous parameter vector, and a preset joint reward function; constructing a truncated objective function based on the advantage function and updating the policy parameters based on the truncated objective function; and determining the target stitching parameters based on the updated policy parameters.

[0008] In this embodiment of the application, the truncation objective function is:

[0009]

[0010] in, To truncate the objective function, The probability ratio between the old and new strategies. For the dominant function, This is a truncation function. The cutoff factor is , For strategy parameters, For the updated policy network, For the policy network before the update, It is a continuous parameter vector. This is the state vector.

[0011] In this embodiment, applying the adjusted target stitching parameters to the next set of video frames for stitching, and calculating the stitching quality index and latency index after stitching, includes: performing feature extraction on the next set of video frames to obtain the keypoint set and the descriptor corresponding to the keypoint set for each video frame in the next set of video frames, and determining the feature extraction time; for each group of adjacent video frames in the next set of video frames, performing feature matching and filtering on the keypoint set based on the descriptor to obtain the inlier set between adjacent video frames, and determining the inlier rate and feature matching time, wherein the inlier is the keypoint that has been successfully matched; and calculating the latency index between each group of adjacent video frames based on the inlier set. Homography matrix; Based on the homography matrix, geometric alignment is performed on all adjacent video frames to obtain the geometrically aligned overlapping region image, and the homography estimation time and geometric reprojection error of the overlapping region image are determined; Exposure difference compensation, stitching line selection and smoothing are performed on the overlapping region image, and the processed overlapping region image is fused to obtain the current stitched frame, and the fusion time is determined; The stitching quality index and latency index of the current stitched frame are calculated, wherein the stitching quality index includes at least edge discontinuity, brightness difference, inlier rate and geometric reprojection error, and the latency index includes at least feature extraction time, feature matching time, homography estimation time and fusion time.

[0012] In this embodiment, the feature extraction operation on the next set of video frames to obtain the keypoint set and the descriptor corresponding to the keypoint set for each video frame in the next set of video frames includes: obtaining the overlapping region image corresponding to the previous set of video frames and using the region where it is located as the candidate overlapping region for the next set of video frames; for each video frame, performing a scale-invariant feature transformation algorithm within the candidate overlapping region to obtain the candidate keypoint set and the candidate descriptor corresponding to the candidate keypoint set; calculating the contrast and edge response value of each candidate keypoint; and filtering the candidate keypoint set based on preset contrast thresholds and edge response thresholds to remove candidate keypoints whose contrast is less than the contrast threshold or whose edge response value is less than the edge response threshold, thereby obtaining the keypoint set and the descriptor corresponding to the keypoint set.

[0013] In this embodiment of the application, for each group of adjacent video frames in the next group of video frames, feature matching and filtering of the key point set based on the descriptor are performed to obtain the inner point set between adjacent video frames. This includes: for each group of adjacent video frames, using the K-nearest neighbor matching algorithm based on the descriptor to determine the initial matching point set between adjacent video frames; and using the random sampling consensus algorithm to filter the initial matching point set to remove mismatched points in the initial matching point set to obtain the inner point set.

[0014] In this embodiment of the application, the method further includes: determining the ratio of inliers between adjacent video frames based on the ratio of the number of inliers to the number of initial matching points; if the ratio of inliers is less than a preset lower limit of the ratio of inliers, triggering a degradation strategy and re-performing feature extraction on the next set of video frames, wherein the degradation strategy includes at least expanding the candidate overlapping region.

[0015] In this embodiment of the application, the method further includes: after calculating the homography matrix between each group of adjacent video frames based on the inlier set, applying cross-frame smoothing constraints to the homography matrix to obtain a transformation matrix; and performing geometric alignment on all adjacent video frames based on the transformation matrix to obtain a geometrically aligned overlapping region image, and obtaining the homography estimation time and the geometric reprojection error of the overlapping region image.

[0016] In this embodiment, the homography matrix is ​​subjected to cross-frame smoothing constraints based on the following formula:

[0017] in, The transformation matrix is... The homography matrix has cross-frame smoothing coefficients. It is a homography matrix. This is the homography matrix corresponding to the previous set of video frames.

[0018] A second aspect of this application provides a multi-channel video splicing device, comprising: The memory is configured to store instructions; The processor is configured to retrieve instructions from memory and, when executing instructions, to implement the aforementioned multi-channel video stitching method.

[0019] A third aspect of this application provides a machine-readable storage medium storing instructions that, when executed by a processor, configure the processor to perform the aforementioned multi-channel video stitching method.

[0020] The above technical solution receives multiple video streams in real time via a WebRTC interface and continuously timestamps the received video streams. Using preset initial strategy stitching parameters, it performs video stitching on the first group of video frames from the synchronized streams to obtain the first stitched frame. It then calculates the stitching quality and latency metrics of the first stitched frame. For each subsequent group of video frames, the stitching quality and latency metrics corresponding to the previous group are input into a reinforcement learning model, and the target stitching parameters output by the reinforcement learning model are obtained. The reinforcement learning model is trained using a joint reward function, which is used to calculate a stitching error penalty term based on the stitching quality metrics and a timeout penalty term based on the comparison between the end-to-end latency estimate and the preset target latency threshold. Finally, it performs video stitching on the next group of video frames using the target stitching parameters to obtain the current stitched frame. By simultaneously incorporating latency and stitching quality into the reward function, it ensures both stitching quality and real-time performance of the video stitching, thus optimizing the end-to-end experience.

[0021] Other features and advantages of the embodiments of this application will be described in detail in the following detailed description section. Attached Figure Description

[0022] The accompanying drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the following detailed description to explain the embodiments of this application, but do not constitute a limitation on the embodiments of this application. In the drawings: Figure 1 The schematic diagram illustrates a flowchart of a multi-channel video stitching method according to an embodiment of this application; Figure 2 This illustration schematically shows a flowchart of a reinforcement learning process according to an embodiment of this application; Figure 3 This illustration schematically shows a video stitching process according to an embodiment of the present application; Figure 4 This schematic diagram illustrates a structural block diagram of a multi-channel video splicing device according to an embodiment of the present application; Figure 5 The diagram illustrates the internal structure of a computer device according to an embodiment of this application. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for illustration and explanation of the embodiments of this application and are not intended to limit the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0024] Figure 1 A schematic flowchart of a multi-channel video stitching method according to an embodiment of this application is shown. Figure 1 As shown in one embodiment of this application, a multi-channel video stitching method is provided, including the following steps: Step 101: Receive multiple video streams in real time via the WebRTC interface and continuously synchronize the timestamps of the received video streams.

[0025] Step 102: Perform video splicing operation on the first group of video frames in the synchronized multi-channel video frames using preset initial splicing strategy parameters to obtain the first spliced ​​frame, and calculate the splicing quality index and latency index of the first spliced ​​frame. The latency index includes at least the end-to-end latency estimate, which is the total time taken for the video frame to travel from the transmitting end to the receiving end.

[0026] Step 103: For each subsequent group of video frames to be stitched, perform the following steps sequentially until no new multi-channel video frames are input or a stop command is received: Step 104: Input the stitching quality index and latency index of the current stitched video frame into the reinforcement learning model, and obtain the adjusted target stitching parameters output by the reinforcement learning model.

[0027] Step 105: Apply the adjusted target stitching parameters to the next set of video frames for stitching, and calculate the stitching quality index and latency index after stitching. Repeat the above steps until there are no new multi-channel video stream inputs or a stop command is received.

[0028] WebRTC is a real-time communication technology that enables the transmission of data such as video and / or audio streams between browsers without the need for an intermediary. A processor can receive multiple video streams in real time through the WebRTC interface and continuously timestamp the received streams. Specifically, frame alignment is performed within the jitter buffer. For each video frame, if the time difference exceeds the maximum allowed frame time difference, the processor can drop frames according to a frame dropping policy threshold (θ). dropThe processor discards the video frame or compensates for it with the previous frame, ultimately outputting a synchronized multi-channel video frame group and real-time link metrics. Then, the processor can use preset initial stitching strategy parameters to perform video stitching on the first group of video frames from the synchronized multi-channel video frames, obtaining the first stitched frame, and calculating the stitching quality metrics and latency metrics of the first stitched frame. The stitching quality metrics include at least edge discontinuity. Brightness difference Interior point ratio Geometric reprojection error The latency metric should include at least the end-to-end latency estimate. Feature extraction time Feature matching time Homography estimation time Integration time Shaking Furthermore, for each subsequent group of video frames to be stitched, the processor can sequentially execute the following steps until no new multi-channel video frames are input or a stop command is received: input the stitching quality index and latency index of the currently stitched video frame into the reinforcement learning model, obtain the adjusted target stitching parameters output by the reinforcement learning model, then apply the adjusted target stitching parameters to the next group of video frames for stitching, and calculate the stitching quality index and latency index after stitching, repeating the above steps until no new multi-channel video streams are input or a stop command is received.

[0029] In one embodiment, the reinforcement learning model can be a proximal policy optimization algorithm model, a SAC model, a deep Q-network, etc. Taking the proximal policy optimization algorithm model as an example, the proximal policy optimization algorithm model includes a policy network and a value network, which can be trained using a joint reward function. The joint reward function is used to calculate a splicing error penalty term based on the splicing quality index, and to calculate a timeout penalty term based on the comparison between the end-to-end delay estimate and the preset target delay threshold. Figure 2 As shown, Figure 2 The illustration shows a reinforcement learning process according to an embodiment of this application. The process involves inputting the stitching quality index and latency index of the currently stitched video frame into the reinforcement learning model, and obtaining the adjusted target stitching parameters output by the reinforcement learning model. This includes obtaining the video stitching quality index, latency index, stitching parameters, video frame scene change index, and policy parameters for the decision period corresponding to the video frames within a preset decision period. The decision period can be defined as every K frames or each frame. The stitching quality index includes at least edge discontinuity. Brightness difference Interior point ratio Geometric reprojection error Overall processing frame rate and packet loss rate The latency metric should include at least the end-to-end latency estimate. Feature extraction time Feature matching time Homography estimation time Integration time and shaking The video frame scene change indicators should include at least the brightness change magnitude and motion intensity, and the stitching parameters should include at least the scale of the candidate overlapping region. Maximum number of key points per frame Feature matching ratio test threshold Scale-Invariant Feature Transform (SIFT) Contrast Threshold SIFT edge response threshold Maximum number of iterations for Random Sampling Consensus (RANSAC) RANSAC reprojection error threshold Homography matrix cross-frame smoothing coefficient Single-response update frequency Exposure / brightness compensation intensity levels 1. Suture smoothness strength setting Overlapping region fusion weight Multi-band fusion layers Output resolution settings Target encoding bitrate level Keyframe interval levels To ensure feasibility, the splicing parameters can be discretized into several levels, or continuous actions with boundary trimming can be used. Specifically, in one embodiment, the splicing parameters and their value ranges are shown in Table 1 below: Table 1

[0030] For each group of video frames, a state vector is constructed based on stitching quality metrics, video frame scene change metrics, and latency metrics. For example, the state vector... It can be:

[0031] in, The magnitude of brightness change can be obtained from the brightness statistics of adjacent frames and is used to characterize sudden changes in illumination.

[0032] Then, based on the concatenation parameters, a continuous parameter vector corresponding to the decision cycle can be constructed using the policy network. For example, in one embodiment, the continuous parameter vector... It can be:

[0033] Then, based on the state vector, continuous parameter vector, and joint reward function, the advantage function corresponding to the policy parameters can be determined using the value network. Specifically, the processor can generate collected trajectory samples based on the state vector, continuous parameter vector, and joint reward function collected during the decision-making cycle. Based on the collected trajectory samples, the dominance function is calculated using the generalized dominance estimation (GAE) method. This is the state vector at the start of the decision cycle. It is a continuous parameter vector. For the reward function, Given the state vector at the end of the decision cycle, the joint reward function is:

[0034] in, For the joint reward function, For edge discontinuity, For brightness difference, This is for geometric reprojection error. For interior point ratio, This is the estimated end-to-end delay. To preset the target latency threshold, These are weight parameters for edge discontinuity, brightness difference, geometric reprojection error, interior point rate, and end-to-end delay estimation, respectively. Then, the processor can construct a truncation objective function for the near-end policy optimization algorithm based on the advantage function, and update the policy parameters according to the truncation objective function; based on the updated policy parameters, the target splicing parameters are determined. Specifically, in one embodiment, a "policy-value (actor-critic)" structure is adopted, and the policy network... Output action distribution, value network Estimate state value. Collect trajectory samples in each decision cycle. The dominance function is calculated using GAE. And update the policy parameters according to the truncation objective function of PPO. This limits the magnitude of a single update and improves training stability, while updating the value network parameters by minimizing the value error. The objective function for truncation is:

[0035]

[0036] in, To truncate the objective function, The probability ratio between the old and new strategies. For the dominant function, This is a truncation function. The cutoff factor is , For strategy parameters, For the updated policy network, For the policy network before the update, It is a continuous parameter vector. Let be the state vector. Preferably, in terms of deployment, an "offline pre-training + online adaptation" strategy can be adopted: first, the initial strategy is pre-trained on multi-scene video data for playback, and then, during the actual WebRTC operation, small-step updates are performed using real-time feedback every K frames, or only inference is performed, so as to achieve rapid adaptation to environmental changes and network fluctuations.

[0037] In one embodiment, such as Figure 3 As shown, applying the adjusted target stitching parameters to the next set of video frames for stitching, and calculating the stitching quality and latency metrics after stitching, includes: performing feature extraction on the next set of video frames to obtain the keypoint set and the corresponding descriptor for each video frame in the next set of video frames, and determining the feature extraction time. Specifically, the processor can acquire the overlapping region image corresponding to the previous set of video frames and use its region as the candidate overlapping region for the next set of video frames; for each video frame, a scale-invariant feature transformation algorithm is executed within the candidate overlapping region to obtain the candidate keypoint set and the candidate descriptor corresponding to the candidate keypoint set; the contrast and edge response values ​​of each candidate keypoint are calculated; based on preset contrast thresholds and edge response thresholds, the candidate keypoint set is filtered to remove candidate keypoints with contrast values ​​less than the contrast threshold or edge response values ​​less than the edge response threshold, thus obtaining the keypoint set and the descriptor corresponding to the keypoint set. Furthermore, the processor can also adjust the maximum number of keypoints per frame. Control the number of key points to limit the amount of computation.

[0038] Then, for each group of adjacent video frames in the next set of video frames, feature matching and filtering of the keypoint set are performed based on the descriptor to obtain the inlier set between adjacent video frames, and the inlier ratio and feature matching time are determined, where inliers are keypoints that have been successfully matched. Specifically, this includes: for each group of adjacent video frames, using the K-nearest neighbor matching algorithm based on the descriptor, and checking the threshold based on the feature matching ratio. An initial screening is performed to determine the initial set of matching points between adjacent video frames; a random sampling consensus algorithm is used, based on the RANSAC reprojection error threshold. and the maximum number of iterations in RANSAC The initial set of matching points is filtered to remove mismatched points, resulting in a set of inner points. Further, the ratio of inner points between adjacent video frames is determined based on the ratio of the number of inner points to the number of initial matching points. If the ratio of inner points is less than a preset lower limit, a degradation strategy is triggered, and feature extraction is performed again on the next set of video frames. This degradation strategy may include expanding the candidate overlapping region. Increase the maximum number of key points per frame. Alternatively, temporarily reduce the output resolution to ensure real-time performance.

[0039] Based on the set of interior points, the homography matrix between each group of adjacent video frames is calculated; cross-frame smoothing constraints are applied to the homography matrix to suppress image jitter, resulting in the transformation matrix:

[0040] in, The transformation matrix is... The homography matrix has cross-frame smoothing coefficients. It is a homography matrix. Let be the homography matrix corresponding to the previous set of video frames. Based on the transformation matrix, geometric alignment is performed on all adjacent video frames to obtain the geometrically aligned overlapping region image, and the homography estimation time and the geometric reprojection error of the overlapping region image are determined.

[0041] Exposure difference compensation, stitching line selection, and smoothing are performed on the overlapping area images. The processed overlapping area images are then fused to obtain the current stitched frame, and the fusion time is determined. The stitching quality metrics and latency metrics of the current stitched frame are calculated. The stitching quality metrics include at least edge discontinuity, brightness difference, inlier rate, and geometric reprojection error. The latency metrics include at least feature extraction time, feature matching time, homography estimation time, and fusion time. Then, the processor can scale and encode the current stitched frame according to the output resolution level, and reduce the target encoding bitrate level or output resolution level when the network bandwidth increases. Finally, it is transmitted and displayed in real time via WebRTC. Further, when the inlier rate is detected... Below the threshold or end-to-end delay estimate Continuously exceeding the preset target delay threshold At this time, the system triggers a real-time safety net mechanism: prioritizing a reduction in the output resolution level. Limit the maximum number of key points per frame. Maximum number of iterations with RANSAC Increase the frequency of single-response updates Alternatively, switch to a fusion method with lower computational requirements to ensure the continuity and low latency of the spliced ​​output.

[0042] Through the above embodiments, multiple video streams are received in real time via the WebRTC interface, and the received video streams are continuously timestamped. A video stitching operation is performed on the first group of video frames in the synchronized multi-stream video frames using preset initial stitching parameters to obtain the first stitched frame. The stitching quality index and latency index of the first stitched frame are calculated. For each subsequent group of video frames, the stitching quality index and latency index corresponding to the previous group of video frames are input into the reinforcement learning model, and the target stitching parameters output by the reinforcement learning model are obtained. The reinforcement learning model is trained using a joint reward function, which is used to calculate a stitching error penalty term based on the stitching quality index and a timeout penalty term based on the comparison between the end-to-end latency estimate and the preset target latency threshold. The target stitching parameters are then used to perform a video stitching operation on the next group of video frames to obtain the current stitched frame. By incorporating WebRTC metrics such as network latency and jitter, along with algorithm execution time, into the state and reward system, the end-to-end experience is directly optimized, rather than just the offline stitching quality. It doesn't just adjust a single stitching parameter, but makes joint decisions on cross-module parameters such as feature quantity / threshold, RANSAC iteration, homography smoothing, fusion method, and output resolution. Timeout penalties and safety net strategies are introduced to ensure stable operation on computing-constrained terminals / real-time conferences. Real-time responses are provided to dynamic factors such as sudden changes in illumination, jitter, and resolution inconsistencies to prevent fixed parameters from failing when the scene changes.

[0043] Figure 1 This is a flowchart illustrating a multi-channel video stitching method in one embodiment. It should be understood that, although... Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise explicitly stated herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0044] In one embodiment, a multi-channel video splicing device (not shown in the figure) is provided, comprising: The memory is configured to store instructions; The processor is configured to retrieve instructions from memory and, when executing instructions, to implement the aforementioned multi-channel video stitching method.

[0045] The multi-channel video splicing device includes a processor and a memory. The processor contains a kernel, which retrieves the corresponding program units from the memory. One or more kernels can be configured, and the multi-channel video splicing method can be implemented by adjusting the kernel parameters.

[0046] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0047] In one embodiment, such as Figure 4 As shown, the multi-channel video splicing device includes: The video capture module is used to receive multiple video streams in real time via the WebRTC interface, and continuously perform jitter buffering and timestamp synchronization on the received multiple video streams; The feature extraction module is used to perform feature extraction operations on the next set of video frames to obtain the key point set of each video frame in the next set of video frames and the descriptor corresponding to the key point set. The feature matching module is used to perform feature matching and filtering on the key point set based on descriptors for each group of adjacent video frames in the next group of video frames, so as to obtain the set of interior points between adjacent video frames. The fusion processing module is used to calculate the homography matrix between each group of adjacent video frames based on the set of interior points, perform geometric alignment on all adjacent video frames based on the homography matrix to obtain the geometrically aligned overlapping area image, perform exposure difference compensation, stitching line selection and smoothing on the overlapping area image, and fuse the processed overlapping area image to obtain the current stitched frame. The reinforcement learning optimization module is used to dynamically select and update the stitching parameters based on the stitching quality and latency feedback of the next set of video frames, so that the system can maintain stable stitching under different lighting, motion, resolution differences and network jitter, and meet the real-time constraints of WebRTC. The video transmission and display module is used to encode the spliced ​​frames and send them to the client via the WebRTC channel.

[0048] This application provides a storage medium storing a program that, when executed by a processor, implements the above-described multi-channel video stitching method.

[0049] This application provides a processor for running a program, wherein the program executes the above-described multi-channel video stitching method during runtime.

[0050] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 5As shown in the figure, the computer device includes a processor A01, a network interface A02, and a memory (not shown) connected via a system bus. The processor A01 provides computing and control capabilities. The memory includes internal memory A03 and a non-volatile storage medium A04. The non-volatile storage medium A04 stores an operating system B01 and a computer program B02. The internal memory A03 provides an environment for the operation of the operating system B01 and the computer program B02 stored in the non-volatile storage medium A04. The network interface A02 is used for communication with external terminals via a network connection. When executed by the processor A01, the computer program B02 implements a multi-channel video stitching method.

[0051] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0052] This application provides a computer (electronic) device, which includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of any of the above multi-channel video stitching methods.

[0053] This application also provides a computer program product that, when executed on a data processing device, is adapted to execute a program that initializes a multi-channel video stitching method step.

[0054] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0055] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0056] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0057] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0058] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0059] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0060] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0061] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0062] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for splicing multiple video streams, characterized in that, The method includes: Multiple video streams are received in real time via the WebRTC interface, and the received video streams are continuously timestamped. A video splicing operation is performed on the first group of video frames in the synchronized multi-channel video frames using preset initial splicing strategy parameters to obtain the first spliced ​​frame. The splicing quality index and latency index of the first spliced ​​frame are calculated. The latency index includes at least the end-to-end latency estimate, which is the total time taken for the video frame to travel from the transmitting end to the receiving end. For each subsequent set of video frames to be stitched together, perform the following steps in sequence: The stitching quality index and latency index of the current stitched video frame are input into the reinforcement learning model, and the adjusted target stitching parameters output by the reinforcement learning model are obtained. Apply the adjusted target stitching parameters to the next set of video frames for stitching, and calculate the stitching quality index and latency index after stitching. Repeat the above steps until there are no new multi-channel video stream inputs or a stop command is received.

2. The multi-channel video stitching method according to claim 1, characterized in that, The reinforcement learning model includes a policy network and a value network. The step of inputting the stitching quality and latency metrics of the currently stitched video frames into the reinforcement learning model and obtaining the adjusted target stitching parameters output by the reinforcement learning model includes: Obtain the video stitching quality index, latency index, stitching parameters, video frame scene change index, and strategy parameters of the decision period corresponding to the video frames within the preset decision period; For each group of video frames, a state vector of the video frame is constructed based on the stitching quality index, the video frame scene change index, and the latency index. Based on the splicing parameters, a continuous parameter vector corresponding to the decision cycle is constructed using the policy network; Based on the state vector, the continuous parameter vector, and the preset joint reward function, the advantage function corresponding to the policy parameters is determined using a value network; A truncation objective function is constructed based on the advantage function, and the policy parameters are updated according to the truncation objective function; Based on the updated strategy parameters, the target splicing parameters are determined.

3. The multi-channel video stitching method according to claim 2, characterized in that, The truncation objective function is: in, The truncation objective function is... The probability ratio between the old and new strategies. For the aforementioned advantage function, This is a truncation function. The cutoff factor is , For the strategy parameters, For the updated policy network, For the policy network before the update, For the continuous parameter vector, Let be the state vector.

4. The multi-channel video stitching method according to claim 1, characterized in that, The adjusted target stitching parameters are applied to the next set of video frames for stitching, and the stitching quality and latency metrics are calculated, including: Perform feature extraction on the next set of video frames to obtain the key point set of each video frame in the next set of video frames and the descriptor corresponding to the key point set, and determine the feature extraction time. For each group of adjacent video frames in the next group of video frames, feature matching and filtering are performed on the key point set based on the descriptor to obtain the set of interior points between the adjacent video frames, and the interior point rate and feature matching time are determined, wherein the interior point is the key point where the feature matching is successful. Based on the set of interior points, calculate the homography matrix between each group of adjacent video frames; Based on the homography matrix, all adjacent video frames are geometrically aligned to obtain the geometrically aligned overlapping region image, and the homography estimation time and the geometric reprojection error of the overlapping region image are determined. Exposure difference compensation, stitching line selection and smoothing are performed on the overlapping region image, and the processed overlapping region image is fused to obtain the current stitched frame, and the fusion time is determined; Calculate the stitching quality index and latency index of the current stitched frame, wherein the stitching quality index includes at least edge discontinuity, brightness difference, inlier rate and geometric reprojection error, and the latency index includes at least feature extraction time, feature matching time, homography estimation time and fusion time.

5. The multi-channel video stitching method according to claim 4, characterized in that, The step of performing feature extraction on the next set of video frames to obtain a set of key points for each video frame in the next set of video frames and a descriptor corresponding to the set of key points includes: Obtain the overlapping region image corresponding to the previous set of video frames, and use the region where it is located as the candidate overlapping region for the next set of video frames; For each video frame, a scale-invariant feature transformation algorithm is performed within the candidate overlapping region to obtain a set of candidate keypoints and a candidate descriptor corresponding to the set of candidate keypoints. Calculate the contrast and edge response values ​​for each candidate keypoint; Based on preset contrast thresholds and edge response thresholds, the candidate key point set is filtered to remove candidate key points whose contrast is less than the contrast threshold or whose edge response value is less than the edge response threshold, thereby obtaining the key point set and the descriptor corresponding to the key point set.

6. The multi-channel video stitching method according to claim 4, characterized in that, For each group of adjacent video frames in the next group of video frames, feature matching and filtering of the keypoint set based on the descriptor is performed to obtain the set of interior points between the adjacent video frames, including: For each group of adjacent video frames, based on the descriptor, the K-nearest neighbor matching algorithm is used to determine the initial set of matching points between the adjacent video frames; The initial set of matching points is filtered using a random sampling consensus algorithm to remove mismatched points and obtain the set of interior points.

7. The multi-channel video stitching method according to any one of claims 4-6, characterized in that, The method further includes: The ratio of inlier points between adjacent video frames is determined based on the ratio of the number of inlier points to the number of initial matching points. If the proportion of inliers is less than a preset lower limit for the proportion of inliers, a degradation strategy is triggered and feature extraction is performed again on the next set of video frames. The degradation strategy includes at least expanding the candidate overlapping region.

8. The multi-channel video stitching method according to claim 4, characterized in that, The method further includes: After calculating the homography matrix between each group of adjacent video frames based on the inlier set, the homography matrix is ​​subjected to cross-frame smoothing constraints to obtain the transformation matrix. Based on the transformation matrix, geometric alignment is performed on all adjacent video frames to obtain the geometrically aligned overlapping region image, and the homography estimation time and the geometric reprojection error of the overlapping region image are obtained.

9. The multi-channel video stitching method according to claim 8, characterized in that, The homography matrix is ​​subjected to cross-frame smoothing constraints based on the following formula: in, Let be the transformation matrix. The homography matrix has cross-frame smoothing coefficients. Let be the homography matrix. This is the homography matrix corresponding to the previous set of video frames.

10. The multi-channel video stitching method according to claim 1, characterized in that, The reinforcement learning model is trained using a joint reward function. This joint reward function is used to calculate a stitching error penalty term based on the stitching quality indicators, and a timeout penalty term based on the comparison between the end-to-end latency estimate and a preset target latency threshold. The stitching quality indicators include at least edge discontinuity, brightness difference, inlier rate, and geometric reprojection error. The joint reward function is: in, For the joint reward function, The edge discontinuity, The brightness difference is... The geometric reprojection error is... The interior point rate, This is the estimated end-to-end delay value. The preset target delay threshold, These are the weighting parameters for edge discontinuity, brightness difference, geometric reprojection error, inlier rate, and end-to-end delay estimate, respectively.

11. A multi-channel video splicing device, characterized in that, include: The memory is configured to store instructions; The processor is configured to retrieve the instructions from the memory and, when executing the instructions, to implement the multi-channel video stitching method according to any one of claims 1 to 10.

12. A machine-readable storage medium storing instructions thereon, characterized in that, When executed by a processor, this instruction causes the processor to be configured to perform the multi-channel video stitching method according to any one of claims 1 to 10.

Images