Video stitching method and device, computer storage medium and terminal

Abstract

The invention discloses a video stitching method and device, a computer storage medium and a terminal, and the method comprises the steps: splitting a video stitching task into a first preset number of subtasks, and distributing the split subtasks to a second preset number of graphic processing units (GPUs); each GPU carries out video stitching on videos collected by more than two cameras according to the received subtasks; according to a video stitching result of the GPU, outputting a panoramic stitching video; wherein the first preset value is greater than or equal to the second preset value, and the second preset value is greater than or equal to 2. According to the embodiment of the invention, the splicing task of the videos collected by the multiple cameras is split into the multiple sub-tasks, each GPU processes the sub-tasks allocated to the GPU by using the hardware resources of the GPU, and the videos collected by more than two paths of cameras and the multiple GPUs are subjected to cooperative processing of video splicing, so that the video splicing of the high-resolution videos is realized.

Description

Technical Field

[0001] The present invention relates to multimedia technology, and more particularly to a method, device, computer storage medium and terminal for video splicing. Background Art

[0002] Video surveillance is an important means of public area security management. It has been widely used and popularized. Whether from analog cameras of hardware equipment to various types of high-definition network cameras, or various intelligent analysis and linkage on software, video surveillance technology has always received high attention and rapid development. In large public places such as crowded and complex squares, parks, airports and ports and other special areas, it is difficult for monitors to establish a global and complete spatial concept. They can only keep imagining and track the monitored objects by controlling the switching of pictures, the rotation of the pan-tilt or fast ball. To achieve panoramic dynamic control and full coverage of large scenes, it is necessary to abandon the processing method of displaying a video picture in each window in related technologies, and it is necessary to seamlessly stitch the videos collected by multiple organically connected cameras together in real time through video stitching algorithms to form a complete panoramic stitching video.

[0003] The video stitching algorithms in the related technologies include: 1. Using the front-end all-in-one mode, the videos obtained by the panoramic camera or binocular camera are stitched based on the built-in chip. Although the imaging effect of these cameras has a large visual range, the quality of the panoramic stitching video obtained is very poor, especially the image edge distortion is unacceptable, which cannot meet the high-quality video surveillance requirements; 2. Video stitching is based on a single GPU or a single CPU. A single GPU can only be used for image stitching of small-size videos, and a single CPU, including a CPU with multi-core hyper-threading technology, is limited by computing power and cannot be used to achieve video stitching of multiple high-resolution cameras.

[0004] In summary, how to efficiently stitch videos captured by multiple high-resolution cameras to obtain high-quality panoramic stitching videos has become a problem to be solved. Summary of the invention

[0005] The present application provides a video stitching method, including: Splitting the video stitching task into a first preset value of subtasks, and allocating the split subtasks to a second preset value of GPUs; Each GPU performs video stitching on the videos collected by two or more cameras according to the received subtasks; Output panoramic stitching video according to the result of GPU video stitching; Among them, the first preset value is greater than or equal to the second preset value, and the second preset value is greater than or equal to 2.

[0006] On the other hand, an embodiment of the present application further provides a computer storage medium, wherein the computer storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned video stitching method is implemented.

[0007] On the other hand, an embodiment of the present application further provides a terminal, comprising: a memory and a processor, wherein the memory stores a computer program; wherein: The processor is configured to execute the computer program in the memory; When the computer program is executed by the processor, the method for video splicing as described above is implemented.

[0008] In another aspect, the embodiment of the present application further provides a video splicing device, comprising: an allocation unit, a subtask processing unit and an output unit; wherein: The allocating unit is configured to: split the video stitching task into a first preset value of subtasks, and allocate the subtasks obtained by splitting to a second preset value of GPUs; The subtask processing unit is configured as follows: each GPU performs video stitching on videos collected by two or more cameras according to the received subtasks; The output unit is set to: output a panoramic stitching video according to the result of the video stitching of the GPU; Among them, the first preset value is greater than or equal to the second preset value, and the second preset value is greater than or equal to 2.

[0009] The disclosed embodiment divides the task of stitching videos captured by multiple cameras into multiple subtasks. Each GPU uses its own hardware resources to process the subtask assigned to itself. For videos captured by more than two cameras, multiple GPUs perform collaborative processing of video stitching, thereby realizing video stitching of high-resolution videos.

[0010] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by implementing the present application. Other advantages of the present application can be realized and obtained by the schemes described in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings are used to provide an understanding of the technical solution of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the technical solution of the present application and do not constitute a limitation on the technical solution of the present application.

[0012] Figure 1 A flowchart of a method for video splicing according to an embodiment of the present disclosure; Figure 2 It is a structural block diagram of a device for video splicing according to an embodiment of the present disclosure; Figure 3A schematic diagram of an apparatus for an example of application of the present disclosure; Figure 4 The flowchart of video stitching according to an embodiment of the present disclosure is shown in FIG. DETAILED DESCRIPTION

[0013] The present application describes multiple embodiments, but the description is exemplary rather than restrictive, and it is obvious to those skilled in the art that there may be more embodiments and implementations within the scope of the embodiments described in the present application. Although many possible feature combinations are shown in the drawings and discussed in the specific embodiments, many other combinations of the disclosed features are also possible. Unless specifically limited, any feature or element of any embodiment may be used in combination with any other feature or element in any other embodiment, or may replace any other feature or element in any other embodiment.

[0014] The present application includes and contemplates combinations of features and elements known to those of ordinary skill in the art. The embodiments, features and elements disclosed in the present application may also be combined with any conventional features or elements to form a unique invention scheme. Any features or elements of any embodiment may also be combined with features or elements from other invention schemes to form another unique invention scheme. Therefore, it should be understood that any feature shown and / or discussed in the present application may be implemented individually or in any appropriate combination. Therefore, except for the limitations made according to the attached claims and their equivalents, the embodiments are not subject to other restrictions. In addition, various modifications and changes may be made within the scope of protection of the attached claims.

[0015] In addition, when describing representative embodiments, the specification may have presented the method and / or process as a specific sequence of steps. However, to the extent that the method or process does not rely on the specific order of the steps described herein, the method or process should not be limited to the steps of the specific order described. As will be understood by those of ordinary skill in the art, other sequences of steps are also possible. Therefore, the specific sequence of the steps set forth in the specification should not be interpreted as a limitation to the claims. In addition, the claims for the method and / or process should not be limited to the steps of performing them in the order written, and those skilled in the art can easily understand that these sequences can be changed and still remain within the spirit and scope of the embodiments of the present application.

[0016] Figure 1 Flow chart of the video splicing method according to the embodiment of the present disclosure. Figure 1 As shown, including: Step 101: split the video stitching task into a first preset value of subtasks, and distribute the split subtasks to a second preset value of GPUs; Step 102: Each GPU performs video stitching on videos collected by two or more cameras according to the received subtasks; Step 103: output a panoramic stitching video according to the result of the video stitching by the GPU; Among them, the first preset value is greater than or equal to the second preset value, and the second preset value is greater than or equal to 2.

[0017] The disclosed embodiment divides the task of stitching videos captured by multiple cameras into multiple subtasks. Each GPU uses its own hardware resources to process the subtask assigned to itself. For videos captured by more than two cameras, multiple GPUs perform collaborative processing of video stitching, thereby realizing video stitching of high-resolution videos.

[0018] In an exemplary embodiment, the video captured by more than two cameras in the embodiment of the present disclosure is a video with a single-side resolution greater than a preset value.

[0019] Optionally, according to the subtasks assigned to the GPU, the following one or any combination of stream processor resources are allocated to the GPU: bus and bandwidth.

[0020] In an exemplary instance, the embodiment of the present disclosure can use two GPUs with video memory greater than or equal to 8G to perform video stitching processing on a single-sided 8K resolution video captured and stitched by 4 to 6 cameras, and the GPU used can be an RTX2070 graphics card with a GPU frequency of 1.4GHz; for a single-sided 10K resolution video captured and stitched by 6 to 10 cameras, three RTX2070s with video memory greater than 8G or two RTX3060 graphics cards with video memory greater than 12G can be used to perform video stitching processing; for a single-sided 10K resolution video captured by more than 10 cameras, three GPUs with video memory greater than or equal to 12G can be used, and the GPU used can be an RTX4060Ti graphics card.

[0021] The disclosed embodiments can be applied to video stitching of high-resolution videos captured by multiple cameras, and can achieve stable output of videos with a single-sided resolution exceeding 10K in a collaborative working mode.

[0022] In an exemplary embodiment, the embodiment of the present disclosure divides the video stitching task into a first preset value of subtasks, and distributes the divided subtasks to a second preset value of GPUs, including: The video stitching task is divided into subtasks of a first preset number of stages that are executed sequentially.

[0023] In an exemplary embodiment, the subtasks of the first preset number of stages of the embodiment of the present disclosure include: Decoding and frame synchronization, splicing operations, and encoding output.

[0024] In an exemplary embodiment, the division of subtasks in the first preset numerical stage of the embodiment of the present disclosure can be adjusted by technicians based on experience, and the processing with smaller computational complexity can be merged, and the processing with larger computational complexity can be split.

[0025] In an exemplary embodiment, the embodiment of the present disclosure allocates the split subtasks to a second preset number of GPUs, including: Decoding and frame synchronization are processed by one GPU or two or more GPUs are used for parallel computing; Process the splicing operation and encoding output in series through one GPU or two or more GPUs; The number of GPUs used for encoding output is greater than or equal to the number of GPUs used for splicing operations.

[0026] In an exemplary embodiment, the stitching operation in the embodiment of the present disclosure includes: feature point extraction, image matching and fusion.

[0027] In an exemplary embodiment, the encoding output in the embodiment of the present disclosure includes: selecting a model according to a scene, establishing a histogram mapping table between two adjacent images, and performing an overall mapping transformation on the two adjacent images through the histogram mapping table.

[0028] In an exemplary embodiment, when decoding and frame synchronization are processed in parallel by more than two GPUs, a communication connection is established between the two or more GPUs for parallel processing to transmit data for frame synchronization through the established communication connection.

[0029] The embodiment of the present disclosure further provides a computer storage medium, in which a computer program is stored. When the computer program is executed by a processor, the above-mentioned video splicing method is implemented.

[0030] The embodiment of the present disclosure further provides a terminal, comprising: a memory and a processor, wherein a computer program is stored in the memory; The processor is configured to execute the computer program in the memory; When the computer program is executed by a processor, the above-mentioned video stitching method is implemented.

[0031] Figure 2 FIG. 1 is a structural block diagram of a video splicing device according to an embodiment of the present disclosure, such as Figure 2 As shown, it includes: an allocation unit, a subtask processing unit and an output unit; wherein, The allocating unit is configured to: split the video stitching task into a first preset value of subtasks, and allocate the subtasks obtained by splitting to a second preset value of GPUs; The subtask processing unit is configured as follows: each GPU performs video stitching on videos collected by two or more cameras according to the received subtasks; The output unit is set to: output a panoramic stitching video according to the result of the video stitching of the GPU; Among them, the first preset value is greater than or equal to the second preset value, and the second preset value is greater than or equal to 2.

[0032] In an exemplary embodiment, the video captured by more than two cameras in the embodiment of the present disclosure is a video with a single-side resolution greater than a preset value.

[0033] In an exemplary embodiment, the allocation unit of the embodiment of the present disclosure is further configured to allocate corresponding one or any combination of the following stream processor resources for the GPU according to the subtasks allocated to the GPU: Bus and bandwidth.

[0034] In an exemplary embodiment, the allocation unit of the embodiment of the present disclosure is configured to split the video stitching task into a first preset number of subtasks, including: The video stitching task is divided into subtasks of a first preset number of stages that are executed sequentially.

[0035] In an exemplary embodiment, the subtasks of the first preset number of stages in the embodiment of the present disclosure include: Decoding and frame synchronization, splicing operations, and encoding output.

[0036] In an exemplary embodiment, the allocation unit of the embodiment of the present disclosure is configured to allocate the split subtasks to a second preset number of GPUs, including: Decoding and frame synchronization are processed by one GPU or two or more GPUs are used for parallel computing; Process the splicing operation and encoding output in series through one GPU or two or more GPUs; The number of GPUs used for encoding output is greater than or equal to the number of GPUs used for splicing operations.

[0037] The following briefly describes the embodiments of the present disclosure through application examples. The application examples are only used to illustrate the embodiments of the present disclosure and are not used to limit the protection scope of the embodiments of the present disclosure.

[0038] Application Examples The disclosed embodiment can determine the number of GPUs according to the number of cameras and video resolution of the front-end camera, and can also adjust the number of cameras connected to the system according to the number of GPUs; Figure 3 A schematic diagram of an apparatus for an example of application of the present disclosure is shown in FIG. Figure 3As shown, the front-end camera can be composed of multiple high-resolution monocular gun-type cameras, and the pixel of the camera is generally more than 8 million pixels; the video stitching device can be a video analysis server, and the video analysis server has two built-in GPUs, which is the core of realizing video stitching; the video stitching device of the disclosed embodiment can be connected to the monitoring center and the video display terminal through LAN / Internet; the built-in two GPUs form the core of video stitching, and are connected to the front-end camera through a switch for signal acquisition; the video display terminal can be a monitor or a large monitoring screen, which displays the panoramic stitching video of the large scene after video stitching in real time. The disclosed embodiment provides a video stitching device based on the large scene multi-GPU mode, which overcomes the technical problems of video freeze, slow processing of large-resolution video, limited monitoring angle of a single camera, and inability to meet the overall monitoring of large scenes in the related art, and realizes high-resolution panoramic real-time monitoring of large scenes.

[0039] The video analysis server of the disclosed embodiment receives video captured by a front-end camera (video captured by two or more cameras). Assuming that the video analysis server splits the video stitching task into two subtasks, the video analysis server distributes the split subtasks to two GPUs. In addition, the video analysis server can also perform process management and internal video memory management according to the subtasks executed by the GPU.

[0040] Figure 4 Flow chart of video splicing in the embodiment of the present disclosure, as shown in FIG. Figure 4 As shown, after the video analysis server receives the video captured by the front-end camera, it allocates one GPU to perform video decoding and frame synchronization to achieve real-time synchronization of multiple videos; in the embodiment of the present disclosure, the GPU is also used to pre-process the video when performing frame synchronization processing, and the pre-processing includes denoising processing; in the embodiment of the present disclosure, a second GPU is allocated to perform splicing operations and encoding output on the decoded and frame synchronized videos.

[0041] The disclosed embodiment allocates a second GPU to perform stitching operations and encoding outputs on decoded and frame-synchronized videos, including: extracting feature points from each frame of the decoded and frame-synchronized videos, which may be automatically extracted or manually selected, and may be detected by SIFT feature points, or may be screened by the RANSAC algorithm, and image matching and image fusion may be performed after 4 or more feature points are found; the disclosed embodiment selects a model according to the size of the scene, and selects a spherical model for a large scene; matches are performed based on the detected feature point set to obtain a corresponding transformation matrix, and the multi-channel images are converted into a coordinate system in combination with the parameters of the model (the large scene is a spherical model), and before image fusion, a histogram mapping table between the two adjacent images is established based on the relationship between the overlapping areas of the two adjacent images, and the two images are subjected to an overall mapping transformation through the mapping table, so as to ultimately achieve overall brightness and color correction consistency.

[0042] In the embodiment of the present disclosure, it is assumed that the original coordinate pixel point , the pixel coordinates after mapping are , defined by the mapping coordinates, the transformation matrix is ​​calculated according to the two-dimensional affine transformation: ; The second GPU of the disclosed embodiment performs feature extraction, matching and fusion calculation and parallel processing on each frame of the video based on the CUDA architecture. The specific performance is uniform scale transformation with consistent scale transformation coefficients in all directions or non-uniform scale transformation and shear transformation with inconsistent transformation coefficients, etc., to describe translational motion, rotational motion, and small-scale scaling and deformation. By re-encoding and outputting the fused image, real-time video stream is obtained after splicing multiple videos.

[0043] The disclosed embodiments can be applied to ultra-high resolution video splicing in large scenes, and the stability of video output can be increased by reasonably allocating GPU, video memory and bus data transmission volume.

[0044] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all components may be implemented as software executed by a processor, such as a digital signal processor or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or transient medium). As is known to those skilled in the art, the term "computer storage medium" includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

Claims

1. A video splicing method, characterized in that: include: Splitting the video stitching task into a first preset value of subtasks, and allocating the split subtasks to a second preset value of graphics processing units GPU; Each GPU performs video stitching on the videos collected by two or more cameras according to the received subtasks; Output panoramic stitching video according to the result of GPU video stitching; Among them, the first preset value is greater than or equal to the second preset value, and the second preset value is greater than or equal to 2.

2. The method according to claim 1, characterized in that The videos captured by the two or more cameras are videos with a single-side resolution greater than a preset value.

3. The method according to claim 1, characterized in that The method further includes allocating corresponding one or any combination of the following stream processor resources for the GPU according to the subtasks assigned to the GPU: Bus and bandwidth.

4. The method according to any one of claims 1 to 3, characterized in that: The step of splitting the video stitching task into a first preset value of subtasks, and allocating the split subtasks to a second preset value of GPUs includes: The video stitching task is divided into subtasks of the first preset number of stages that are executed sequentially.

5. The method according to claim 4, characterized in that The subtasks of the first preset number of stages include: Decoding and frame synchronization, splicing operations, and encoding output.

6. The method according to claim 5, characterized in that The step of allocating the split subtasks to a second preset number of GPUs includes: Processing the decoding and frame synchronization through one GPU or performing parallel computing on two or more GPUs; Processing the splicing operation and the encoding output in series through one GPU or more than two GPUs; The number of GPUs used for the encoding output is greater than or equal to the number of GPUs used for the splicing operation.

7. A computer storage medium, wherein a computer program is stored in the computer storage medium, and when the computer program is executed by a processor, the video stitching method according to any one of claims 1 to 6 is implemented.

8. A terminal, comprising: A memory and a processor, wherein the memory stores a computer program; wherein, The processor is configured to execute the computer program in the memory; When the computer program is executed by the processor, the video stitching method according to any one of claims 1 to 6 is implemented.

9. A video splicing device, comprising: allocation unit, subtask processing unit and output unit; wherein, The allocating unit is configured to: split the video stitching task into a first preset value of subtasks, and allocate the subtasks obtained by splitting to a second preset value of GPUs; The subtask processing unit is configured as follows: each GPU performs video stitching on videos collected by two or more cameras according to the received subtasks; The output unit is set to: output a panoramic stitching video according to the result of the video stitching of the GPU; Among them, the first preset value is greater than or equal to the second preset value, and the second preset value is greater than or equal to 2.

10. The device according to claim 9, characterized in that The videos captured by the two or more cameras are videos with a single-side resolution greater than a preset value.

Images