Video stitching method and device, electronic equipment and storage medium
By hierarchically processing and encoding the initial video file and splicing according to the hierarchical sequence number, the problem of difficulty in splicing large parallax scenes is solved, efficient video splicing is achieved, and data volume and difficulty are reduced.
Patent Information
- Application Number
- CN202411074993.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-06
- Publication Date
- 2024-12-24
AI Technical Summary
The existing video stitching technology is difficult to quickly and accurately process scene images or videos with large viewing angle gaps, resulting in difficulty in splicing large parallax scenes and large data volume, which increases the difficulty.
By layering the initial video file, multiple hierarchical images are obtained, and the encoding frame rate is determined according to the hierarchical sequence number, encoding process and splicing are performed. After obtaining the intermediate video file, the target video file is decoded, which reduces the splicing of large parallax scenes The amount of data required and reduces difficulty.
On the premise of ensuring image quality, the amount of data required for splicing in large parallax scenes is reduced, and the difficulty of splicing is reduced, and the splicing efficiency is improved.
Smart Images
Figure CN119182957A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of three-dimensional imaging technology, and more specifically, to a video stitching method, device, electronic device and storage medium. Background Art
[0002] With the continuous development of virtual reality technology (VR), how to splice real scenes into the virtual world has become particularly important.
[0003] At present, existing stitching technologies are often used to stitch multiple scenes with small parallax into a virtual world. Therefore, it is very important to quickly and accurately achieve stitching when there is a large difference in perspective between the collected scene images or videos. Summary of the invention
[0004] The present application proposes a video splicing method, device, electronic device and storage medium to improve the above-mentioned defects.
[0005] In terms of the initial aspect, an embodiment of the present application provides a video splicing method, the method comprising: based on pre-acquired initial video files of multiple different perspectives, obtaining multiple hierarchical images corresponding to a single image frame in each of the initial video files; determining the encoding frame rate based on the hierarchical sequence number of the hierarchical image, wherein the encoding frame rate is negatively correlated with the hierarchical sequence number; encoding the multiple hierarchical images in each of the initial video files according to the encoding frame rate corresponding to the hierarchical images with different hierarchical sequence numbers to obtain multiple encoded hierarchical images, wherein each of the encoded hierarchical images has the same hierarchical sequence number as the corresponding hierarchical image; obtaining an intermediate video file after splicing the encoded hierarchical images with the same hierarchical sequence number in different initial video files; and decoding the intermediate video file to obtain a target video file.
[0006] In a second aspect, an embodiment of the present application further provides a video splicing device, the device comprising: a layering module, for acquiring, based on pre-acquired initial video files of multiple different perspectives, multiple hierarchical images corresponding to each image frame in each of the initial video files; a determination module, for determining a coding frame rate based on a hierarchical sequence number of the hierarchical image, wherein the coding frame rate is negatively correlated with the hierarchical sequence number; a coding and decoding module, for encoding multiple hierarchical images in each of the initial video files according to the coding frame rates corresponding to hierarchical images with different hierarchical sequence numbers to obtain corresponding multiple coded hierarchical images, wherein each of the coded hierarchical images has the same hierarchical sequence number as the corresponding hierarchical image; a splicing module, for splicing coded hierarchical images with the same hierarchical sequence number in different initial video files to obtain an intermediate video file; a decoding module, for decoding the intermediate video file to obtain a target video file.
[0007] In a third aspect, an embodiment of the present application also provides an electronic device, comprising: one or more processors; a memory; one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more applications are configured to execute the above method.
[0008] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a program code executable by a processor, and when the program code is executed by the processor, the processor executes the above method.
[0009] Therefore, an embodiment of the present application provides a video splicing method, device, electronic device and storage medium, the method comprising: based on pre-acquired initial video files of multiple different perspectives, obtaining multiple hierarchical images corresponding to a single image frame in each of the initial video files; determining the encoding frame rate based on the hierarchical sequence number of the hierarchical image, wherein the encoding frame rate is negatively correlated with the hierarchical sequence number; encoding the multiple hierarchical images in each of the initial video files according to the encoding frame rate corresponding to the hierarchical images with different hierarchical sequence numbers to obtain multiple encoded hierarchical images, wherein each of the encoded hierarchical images has the same hierarchical sequence number as the corresponding hierarchical image; obtaining an intermediate video file after splicing the encoded hierarchical images with the same hierarchical sequence number in different initial video files; and decoding the intermediate video file to obtain a target video file. The method of the present application obtains hierarchical images with multiple level numbers after layering processing of the initial video file, and encodes the hierarchical images according to the level numbers using the encoding frame rate corresponding to the level numbers to obtain corresponding encoded level images. This reduces the amount of data required for large parallax scene stitching while ensuring image quality, and subsequently obtains the target video file after layering and stitching multiple encoded level images based on the same level number, thereby reducing the difficulty of large parallax scene stitching.
[0010] Other features and advantages of the embodiments of the present application will be described in the subsequent description, and partly become apparent from the description, or can be understood by practicing the embodiments of the present application. The purposes and other advantages of the embodiments of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.
[0012] Figure 1 A method flow chart of a video stitching method proposed in an embodiment of the present application is shown.
[0013] Figure 2 A method flow chart of another video stitching method proposed in an embodiment of the present application is shown.
[0014] Figure 3 A method flow chart of another video stitching method proposed in an embodiment of the present application is shown.
[0015] Figure 4A method flow chart of another video stitching method proposed in an embodiment of the present application is shown.
[0016] Figure 5 A structural block diagram of a video splicing device provided in an embodiment of the present application is shown.
[0017] Figure 6 A structural block diagram of an electronic device provided in an embodiment of the present application is shown.
[0018] Figure 7 A structural block diagram of a computer-readable storage medium provided in an embodiment of the present application is shown. DETAILED DESCRIPTION
[0019] In order to make those skilled in the art better understand the present application scheme, the technical scheme in the present application embodiment will be clearly and completely described below in conjunction with the drawings in the present application embodiment. Obviously, the described embodiment is only a part of the present application embodiment, rather than all the embodiments. The components of the present application embodiment usually described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the present application for protection, but merely represents the selected embodiment of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.
[0020] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.
[0021] Specifically, see Figure 1 , Figure 1 A method flow chart of a video stitching method proposed in an embodiment of the present application is shown, and the method comprises the following steps:
[0022] S110: Based on a plurality of pre-acquired initial video files of different viewing angles, a plurality of hierarchical images corresponding to a single image frame in each of the initial video files are acquired.
[0023] As an implementation method, the initial video file can be obtained based on a video acquisition device such as a camera, etc. Further, multiple initial video files are obtained by different multiple video acquisition devices, so that the viewing angle of each initial video file corresponds to the position of the video acquisition device. The positions of the video acquisition devices are different, and the viewing angles of the initial video files obtained are also different. When the positions of the multiple video acquisition devices are far apart from each other, the corresponding observation viewing angles of the multiple initial video files are also different, that is, the parallax between the multiple initial video files is large. Further, a spherical coordinate system for imaging can be established in advance, and the image coordinates of the video acquisition device can be mapped to the spherical coordinate system according to a specific mapping relationship, so as to ensure that the image frames of different initial video files can be spliced under a unified standard.
[0024] As an implementation method, each initial video file contains multiple image frames. It can be understood that the effect of video playback can be achieved by playing continuous image frames at a certain frame rate. Therefore, the splicing process performed on the video can be refined into the splicing performed on each image frame in the video file, and the image frames can be output as a video after the splicing is completed. Furthermore, based on multiple initial video files, multiple image frames in each initial video file can be obtained, so as to obtain multiple hierarchical images corresponding to a single image frame according to each image frame.
[0025] Exemplarily, hierarchical images can be divided according to the scene distance where the object is located in a single image frame. Taking a certain image frame in an initial video file as an example, layering it and obtaining its corresponding multiple hierarchical images, the method for dividing the target image frame into hierarchical images can be: by analyzing multiple consecutive frames in the initial video file to establish a background model, and then comparing the difference between the target image frame and the background model to regard the area with smaller difference as the background hierarchical image, and the area with larger difference as the foreground hierarchical image; or, the inter-frame difference method can be used, that is, comparing the difference between two or more consecutive frames adjacent to the target image frame in the initial video file to detect moving objects, and regard the static or slowly changing area as the background hierarchical image, and the fast changing area as the foreground hierarchical image. The area with fast changes is regarded as the foreground level image; or the target image frame can be separated into multiple level images according to the movement speed of the object in the target image frame. Specifically, the inventors found that the objects in the distant part of the image are farther away from the observer, so their displacement in the field of view is relatively small. Therefore, compared with the objects in the near part of the image, the objects in the distant part have a slower movement speed. Based on this rule, the object in the target image frame can be divided into different ranges of scene distance according to the different movement speeds of the objects, thereby, according to the different ranges of scene distance, it can be divided into multiple levels and the corresponding level images can be obtained.
[0026] S120: Determine a coding frame rate based on the hierarchical sequence number of the hierarchical image, wherein the coding frame rate is negatively correlated with the hierarchical sequence number.
[0027] Video encoding refers to the process of converting video files into coded data by removing a large amount of redundant information in video files through specific technologies. In existing video encoding technologies, a fixed encoding frame rate is often used to encode videos.
[0028] In this embodiment, the encoding frame rate can be used to characterize the number of frames of the encoded data obtained after the encoding is completed. For example, if the initial video file is 24fps (i.e., contains 24 image frames per second), if the encoding frame rate is 10fps, it means that the encoded data obtained after the encoding is completed contains 10 image frames per second. Further, in the process of encoding the initial video file, since the multiple image frames in the initial video file are layered in advance, and then multiple hierarchical images with different hierarchical numbers corresponding to a single image frame are obtained, it is possible to encode based on the different hierarchical numbers of the hierarchical images, and the size of the encoding frame rate is determined based on the hierarchical number of the hierarchical image. Further, the encoding frame rate is negatively correlated with the hierarchical number, that is, the hierarchical image with a larger hierarchical number has a smaller encoding frame rate during the encoding process. It should be understood that the larger the level number, the larger the corresponding distance range of the level image, and the smaller the level number, the smaller the corresponding distance range of the level image. In general, it is believed that distant scene images contain relatively static backgrounds, while close scene images contain more dynamic and active objects. Therefore, for level images with smaller level numbers, the higher the encoding frame rate used, the more it can ensure that the data obtained after encoding processing can better reflect dynamic changes. For level images with larger level numbers, the lower the encoding frame rate used, which reduces the amount of data required for the encoding and decoding process and subsequent data transmission.
[0029] S130: According to the encoding frame rates corresponding to the hierarchical images with different hierarchical serial numbers, encoding is performed on the multiple hierarchical images in each of the initial video files to obtain multiple encoded hierarchical images, wherein each of the encoded hierarchical images has the same hierarchical serial number as the corresponding hierarchical image.
[0030] According to the above analysis, after determining the coding frame rate of the hierarchical images with different hierarchical numbers, the multiple hierarchical images in each initial video file can be coded based on the coding frame rate to obtain multiple coded hierarchical images. It can be understood that the hierarchical number of the coded hierarchical image is the same as the hierarchical number of the corresponding pre-coded hierarchical image, and the coded hierarchical image also corresponds to a single image frame in the initial video file. As an implementation method, the video coding process can be implemented by multiple compression technologies such as intra-frame image data compression technology, inter-frame image data compression technology, and entropy coding compression technology.
[0031] Exemplarily, the encoding process for an initial video file is taken as an example for explanation. It is assumed that each image frame in the initial video file is only layered into a first-level image and a second-level image, wherein the first-level image corresponds to a smaller range of scene distances, and the second-level image corresponds to a larger range of scene distances. Then, according to the above analysis, it can be known that the encoding frame rate of the first-level image is higher than the encoding frame rate of the second-level image. Here, the initial video file is still taken as 24fps as an example. It is assumed that the encoding frame rate of the first-level image is 15 frames, and the encoding frame rate of the second-level image is 10 frames. After frame rate compression and encoding of the first-level image and the second-level image in multiple image frames at the corresponding encoding frame rate, the corresponding encoded-level image is obtained, and the encoded-level image obtained by encoding the first-level image also has a level sequence number of one, and the encoded-level image obtained by encoding the second-level image also has a level sequence number of two.
[0032] S140: After splicing the coding level images with the same level sequence number in different initial video files, an intermediate video file is obtained.
[0033] Exemplarily, the process of splicing coded level images with the same level number in different initial video files is to splice together the coded level images with level number 1 in each initial video file, splice together the coded level images with level number 2 in each initial video file... and so on.
[0034] As an implementation method, the stitching between coding level images with the same level sequence number can be achieved by the method of feature point stitching. Before the stitching is performed, the feature points suitable for stitching between the coding level images are first found, and then image registration is performed. The matching feature point pairs are used to calculate the spatial position relationship between the coding level images to achieve spatial alignment of coding level images with different level sequence numbers, and then image fusion and optimization processing are performed to obtain multiple panoramic coding level images after stitching, each panoramic coding level image still has a corresponding level sequence number, so that multiple panoramic coding level images are integrated into an intermediate video file. Exemplarily, these feature points can be areas with similar pixels, cross areas with the same features, or corner points with the same features.
[0035] S150: Decode the intermediate video file to obtain a target video file.
[0036] Specifically, video decoding refers to decompressing the encoded data and restoring it to the original video signal. In this embodiment, the decoding process of the intermediate video file only needs to use the decompression algorithm and technology corresponding to the video encoding process.
[0037] It should be understood that after decoding the intermediate video file, a new video stream corresponding to the panoramic coding level images of different level numbers is obtained, and its frame rate is the same as the set coding frame rate. Therefore, if you want to obtain a target video file with a uniform frame rate, you can take certain frame-filling measures for the panoramic coding level images with a lower frame rate, so as to supplement the number of video stream frames restored by the panoramic coding level images with a lower frame rate to the same number of video stream frames restored by the panoramic coding level images with a higher frame rate. Further, the frame rate of the target video file obtained by decoding is consistent with the panoramic coding level images with a higher frame rate. Exemplarily, frame rate compression can be achieved through a variety of methods including frame dropping, frame rate upsampling and downsampling, key frames and difference frames (I / P / B frame technology), etc.; frame rate supplementation can be achieved by frame sampling or frame mixing. For example, if the encoding frame rate of the panoramic coding level image corresponding to the first-level image obtained based on the initial video file is 15 frames, and the encoding frame rate of the panoramic coding level image corresponding to the second-level image is 10 frames, then the frame rate of the video stream obtained after encoding and decoding the panoramic coding level image corresponding to the first-level image is 15fps, and the frame rate of the video stream obtained after encoding and decoding the panoramic coding level image corresponding to the second-level image is 10fps. Therefore, the video stream obtained after encoding and decoding the panoramic coding level image corresponding to the second-level image can be first supplemented to 15fps, and then video fusion is performed to obtain a target video file with a frame rate of 15fps.
[0038] Therefore, by layering the image frames in the initial video file, hierarchical images with different hierarchical numbers are obtained, and then the hierarchical images are encoded using different encoding frame rates according to the hierarchical numbers to obtain the corresponding encoded hierarchical images, and then the target video file is obtained after splicing based on the hierarchical numbers of the encoded hierarchical images, which greatly reduces the amount of data in the processing process. In addition, since a higher encoding frame rate is used for hierarchical images with closer scene distances, good image quality is also guaranteed.
[0039] Therefore, the present embodiment provides a video splicing method, which includes: based on pre-acquired initial video files of multiple different perspectives, obtaining multiple hierarchical images corresponding to a single image frame in each of the initial video files; determining the encoding frame rate based on the hierarchical sequence number of the hierarchical image, wherein the encoding frame rate is negatively correlated with the hierarchical sequence number; encoding the multiple hierarchical images in each of the initial video files according to the encoding frame rate corresponding to the hierarchical images with different hierarchical sequence numbers to obtain multiple encoded hierarchical images, wherein each of the encoded hierarchical images has the same hierarchical sequence number as the corresponding hierarchical image; obtaining an intermediate video file after splicing the encoded hierarchical images with the same hierarchical sequence number in different initial video files; and decoding the intermediate video file to obtain a target video file. This embodiment obtains hierarchical images with multiple level numbers after layering processing of the initial video file, and encodes them according to the level numbers using the encoding frame rate corresponding to the level numbers. This reduces the amount of data required for stitching large parallax scenes while ensuring image quality. In addition, the target video file is obtained after subsequent layered stitching of multiple encoded level images based on the same level number, thereby reducing the difficulty of stitching large parallax scenes.
[0040] See also Figure 2 , Figure 2 A method flow chart of another video stitching method proposed in an embodiment of the present application is shown, comprising the following steps:
[0041] S210: Acquire a plurality of image frames in each of the initial video files, and separate the image frames into a plurality of the hierarchical images according to a movement speed of an object in each of the image frames.
[0042] In this embodiment, multiple image frames are acquired based on each initial video file, and then the image frames are separated into multiple hierarchical images according to the movement speed of the object in each image frame. Specifically, the scene distance range to which the object belongs can be determined according to the movement speed of the object in each image frame, and then the hierarchical sequence number is determined based on the scene distance range to which the object belongs and the hierarchical image is acquired.
[0043] As an implementation method, an optical flow algorithm can be used to calculate the movement speed of pixels in an image frame by analyzing the pixel displacement between adjacent image frames of an image frame that needs to be layered. Furthermore, objects in the background area of the image can be considered to have a movement speed of zero. As the scene distance of the area in the image becomes closer, the object speed will increase accordingly. Therefore, the first correspondence between the movement speed and the scene distance can be set to be a negative correlation, that is, the smaller the object speed, the longer the corresponding scene distance. Depending on the application scenario, the specific parameters in the first correspondence may also be different, and there is no limitation here. Furthermore, after obtaining the movement speed of pixels in each image frame, the pixels corresponding to the object in the image frame can be divided into different scene distance ranges based on the movement speed and the first correspondence between the movement speed and the scene distance, that is, the scene distance range to which the object belongs is determined.
[0044] As an implementation method, a second correspondence between the field of view range to which an object belongs and the hierarchical sequence number can be established in advance. Thus, after the field of view range to which the object belongs is determined, the object is divided into hierarchical images with corresponding hierarchical sequence numbers according to the second correspondence. Thus, after the hierarchical sequence numbers of all objects in the image frame are divided, the corresponding hierarchical images can be obtained, and each hierarchical image has a specific hierarchical sequence number.
[0045] Specifically, the second correspondence between the level sequence number and the range of view distance may be a positive correlation. For example, the second correspondence between the range of view distance and the level sequence number may be: when the range of view distance is 0-1 meter, the corresponding level sequence number is 1; when the range of view distance is 1-5 meters, the corresponding level sequence number is 2; when the range of view distance is 5-15 meters, the corresponding level sequence number is 3; when the range of view distance is 15-20 meters, the corresponding level sequence number is 4. According to the above analysis, each image frame can be divided into level images with a variety of level sequence numbers according to the second correspondence, and multiple image frames in each initial video file can be divided into level images with a variety of level sequence numbers according to the same steps. That is to say, in different initial video files, the level sequences of the level images divided by each initial video file (which can be understood here as a file composed of multiple image frames) can correspond one to one, so that in the subsequent video stitching process, the same level images in different video files are stitched together, which can effectively reduce the difficulty of stitching.
[0046] S220: Determine a coding frame rate based on the hierarchical sequence number of the hierarchical image, wherein the coding frame rate is negatively correlated with the hierarchical sequence number.
[0047] S230: According to the encoding frame rates corresponding to the hierarchical images with different hierarchical serial numbers, encoding is performed on the multiple hierarchical images in each of the initial video files to obtain multiple encoded hierarchical images, wherein each of the encoded hierarchical images has the same hierarchical serial number as the corresponding hierarchical image.
[0048] S240: After splicing the coding level images with the same level sequence number in different initial video files, an intermediate video file is obtained.
[0049] S250: Decode the intermediate video file to obtain a target video file.
[0050] The specific implementation of steps S220 to S250 may refer to the implementation of steps S120 to S150 described above, which will not be repeated here.
[0051] Therefore, the present embodiment provides a video splicing method, which includes: obtaining multiple image frames in each of the initial video files, separating the image frames into multiple hierarchical images according to the movement speed of the objects in each of the image frames, and determining the encoding frame rate based on the hierarchical sequence number of the hierarchical image, wherein the encoding frame rate is negatively correlated with the hierarchical sequence number; encoding the multiple hierarchical images in each of the initial video files according to the encoding frame rates corresponding to the hierarchical images with different hierarchical sequence numbers to obtain multiple encoded hierarchical images, wherein each of the encoded hierarchical images has the same hierarchical sequence number as the corresponding hierarchical image; obtaining an intermediate video file after splicing the encoded hierarchical images with the same hierarchical sequence number in different initial video files; and decoding the intermediate video file to obtain a target video file. In this embodiment, the initial video file is layered according to the movement speed of the object, and multiple level images corresponding to different ranges of scene distances can be obtained. Then, the level images are encoded and spliced according to the different level numbers they have, so that the number of level numbers and the corresponding scene distances can be freely selected according to needs, thereby improving the flexibility of splicing.
[0052] As an implementation, see Figure 3 , Figure 3 A hierarchical image stitching method proposed in an embodiment of the present application is shown, comprising the following steps:
[0053] S310: Based on a plurality of pre-acquired initial video files of different viewing angles, a plurality of hierarchical images corresponding to a single image frame in each of the initial video files are acquired.
[0054] S320: Determine a coding frame rate based on the hierarchical sequence number of the hierarchical image, wherein the coding frame rate is negatively correlated with the hierarchical sequence number.
[0055] S330: According to the encoding frame rates corresponding to the hierarchical images with different hierarchical serial numbers, multiple hierarchical images in each of the initial video files are encoded to obtain multiple encoded hierarchical images, wherein each of the encoded hierarchical images has the same hierarchical serial number as the corresponding hierarchical image.
[0056] The specific implementation of steps S310 to S330 may refer to the implementation of steps S110 to S130 described above, which will not be repeated here.
[0057] S340: Acquire feature points of coding-level images having the same level sequence number in different initial video files.
[0058] Exemplarily, a method of obtaining feature points may be to select regions with similar pixels as feature points. One method is based on pixel similarity. By calculating the difference between pixel values in different regions of an image, regions with similar pixels may be found. These regions may be used as feature points for image stitching. For example, when stitching two images, regions with similar pixels in the two images may be selected as feature points, and then the image stitching may be achieved by aligning these feature points.
[0059] Exemplarily, the method for obtaining feature points can also be based on a certain structure or texture in the image as feature points for stitching. For example, cross areas with the same features can be selected as feature points. These cross areas can be extracted by calculating the gradient or texture information of the image. Then, the image stitching can be achieved by aligning these feature points. Exemplarily, the method for obtaining feature points can also be to select corner points with the same features as feature points. Corner points are points with obvious edges in the image, usually corners or intersections in the image. The advantage of selecting corner points as feature points is that they are unique in the image and can provide better alignment and matching performance. By selecting corner points with the same features as feature points, image stitching can be performed more accurately.
[0060] Exemplarily, feature points can also be obtained by a scale-invariant feature transformation method (SI FT). Specifically, feature points can be detected by using scale space extreme points, and then the gradient information of the scale space extreme points and surrounding pixels is used to calculate the precise position and scale of the key points, and one or more main directions are assigned to each key point to make it rotationally invariant, and finally a 128-dimensional descriptor (vector) is generated to describe the features of the key points. The feature points obtained using the SI FT method have scale invariance, rotation invariance, and robustness to illumination changes, and can effectively reduce image stitching errors.
[0061] S350: splicing the coding level images in different initial video files based on the feature points to obtain the intermediate video file.
[0062] As an implementation method, after obtaining the feature points of the coding level images, these feature points are matched to determine the correspondence between different images, wherein the feature point matching is usually achieved by calculating the similarity between the feature point descriptors, such as using the Euclidean distance or the Hamming distance to measure the similarity between the feature points; then, the transformation relationship between the images is calculated using the matched feature point pairs, and the images are registered. Finally, the images are fused and spliced using image fusion methods such as multi-band fusion and Laplace pyramid fusion. After the coding level images of each level number in different initial video files are spliced, a spliced intermediate video file can be obtained.
[0063] S360: Decode the intermediate video file to obtain a target video file.
[0064] The specific implementation of step S360 may refer to the specific implementation of step S150 in the above embodiment, which will not be described in detail here.
[0065] Therefore, the present embodiment provides a video splicing method, which includes: based on pre-acquired initial video files of multiple different perspectives, obtaining multiple hierarchical images corresponding to a single image frame in each of the initial video files; determining the encoding frame rate based on the hierarchical sequence number of the hierarchical image, wherein the encoding frame rate is negatively correlated with the hierarchical sequence number; encoding the multiple hierarchical images in each of the initial video files according to the encoding frame rate corresponding to the hierarchical images with different hierarchical sequence numbers to obtain multiple encoded hierarchical images, wherein each of the encoded hierarchical images has the same hierarchical sequence number as the corresponding hierarchical image; obtaining feature points of encoded hierarchical images with the same hierarchical sequence number in different initial video files; splicing the encoded hierarchical images in different initial video files based on the feature points to obtain the intermediate video file; and decoding the intermediate video file to obtain the target video file. This embodiment obtains hierarchical images with multiple level numbers after layering the initial video file, encodes them at a specific encoding frame rate according to the level numbers, obtains feature points and performs stitching of multiple level images based on the feature points, thereby reducing the error of large parallax stitching.
[0066] See also Figure 4 , Figure 4 A method flow chart of another video stitching method proposed in an embodiment of the present application is shown, and the method comprises the following steps:
[0067] S410: Based on a pre-established spherical coordinate system, a mapping relationship between the image coordinates of the video acquisition device and the coordinates of the spherical coordinate system is acquired; based on the mapping relationship, a coordinate conversion operation is performed on the acquired video to obtain the initial video file.
[0068] In this embodiment, the video acquisition device may be a fisheye lens or multiple cameras for capturing images with a wide-angle field of view, and the spherical coordinate system is a coordinate system based on a sphere and is used to represent the position of each point in a panoramic image or video.
[0069] As an implementation method, a mapping relationship between the image coordinates of a video acquisition device and the coordinates of a spherical coordinate system can be established by a field calibration method, or a mapping curve between the image coordinates of a video acquisition device and a spherical coordinate system can be obtained by fitting existing test data. Different panoramic camera systems may use different mapping methods and parameters to adapt to different application requirements and camera devices. For example, the video acquisition device can be calibrated first to determine its internal and external parameters, including focal length, distortion, etc., so that the image coordinates of the video acquisition device are converted into three-dimensional coordinates in the camera coordinate system based on these parameters, and then, using spherical projection or other mapping methods, the three-dimensional coordinates in the camera coordinate system are converted into coordinates in the spherical coordinate system, thereby establishing a mapping relationship between the image coordinates of the video acquisition device and the coordinates of the spherical coordinate system. It can be understood that after the mapping relationship between the image coordinates of the video acquisition device and the coordinates of the spherical coordinate system is established, the original video captured by the video acquisition device can be subjected to a coordinate conversion operation based on the mapping relationship, so that it is mapped to a unified spherical coordinate system to obtain an initial video file.
[0070] S420: Based on the pre-acquired initial video files of multiple different viewing angles, a plurality of hierarchical images corresponding to a single image frame in each of the initial video files are acquired.
[0071] S430: Determine a coding frame rate based on the hierarchical sequence number of the hierarchical image, wherein the coding frame rate is negatively correlated with the hierarchical sequence number.
[0072] S440: According to the encoding frame rates corresponding to the hierarchical images with different hierarchical serial numbers, multiple hierarchical images in each of the initial video files are encoded to obtain multiple encoded hierarchical images, wherein each of the encoded hierarchical images has the same hierarchical serial number as the corresponding hierarchical image.
[0073] S450: After splicing the coding level images with the same level sequence number in different initial video files, an intermediate video file is obtained.
[0074] S460: Decode the intermediate video file to obtain a target video file.
[0075] The specific implementation of steps S420 to S460 may refer to the implementation of steps S110 to S150 described above, and will not be repeated here.
[0076] See also Figure 5 , which shows a structural block diagram of a video splicing device 500 provided in an embodiment of the present application, the device may include a layering module 501, a determination module 502, an encoding module 503, a splicing module 504 and a decoding module 505. Specifically, the layering module 501 is used to obtain a plurality of hierarchical images corresponding to each image frame in each of the initial video files based on a plurality of pre-acquired initial video files of different viewing angles; the determination module 502 is used to determine the encoding frame rate based on the hierarchical sequence number of the hierarchical image, wherein the encoding frame rate is negatively correlated with the hierarchical sequence number; the encoding module 503 is used to encode the plurality of hierarchical images in each of the initial video files according to the encoding frame rate corresponding to the hierarchical images with different hierarchical sequence numbers to obtain a plurality of corresponding encoded hierarchical images, wherein each of the encoded hierarchical images has the same hierarchical sequence number as the corresponding hierarchical image; the splicing module 504 is used to splice the encoded hierarchical images with the same hierarchical sequence number in different initial video files to obtain an intermediate video file; the decoding module 505 is used to decode the intermediate video file to obtain a target video file.
[0077] Optionally, the layering module 501 is further used to obtain a plurality of image frames in each of the initial video files, and separate the image frames into a plurality of the layered images according to the movement speed of the object in each of the image frames.
[0078] Optionally, the layering module 501 is also used to determine the field of view range to which the object belongs according to the movement speed of the object in each image frame; determine the layer sequence number based on the field of view range to which the object belongs, and obtain the layer image, wherein the layer sequence number is positively correlated with the field of view range.
[0079] Optionally, the splicing module 504 is further used to obtain feature points of coding level images with the same level sequence number in different initial video files; and splice the coding level images in different initial video files based on the feature points to obtain the intermediate video file.
[0080] Optionally, the stitching module 504 is further configured to obtain the feature points by using a scale-invariant feature conversion method.
[0081] Optionally, the video stitching device 500 further includes a conversion module, which is used to obtain a plurality of captured videos based on a plurality of video capture devices; and perform a coordinate conversion operation on the captured videos to obtain the initial video file.
[0082] Optionally, the conversion module is further used to obtain a mapping relationship between the image coordinates of the video acquisition device and the coordinates of the spherical coordinate system based on a pre-established spherical coordinate system; and perform a coordinate conversion operation on the acquired video based on the mapping relationship to obtain the initial video file.
[0083] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.
[0084] In several embodiments provided in the present application, the coupling between modules may be electrical, mechanical or other forms of coupling.
[0085] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules.
[0086] Please refer to Figure 6 , which shows a block diagram of an electronic device 600 provided in an embodiment of the present application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The electronic device 600 in the present application may include one or more of the following components: a processor 610, a memory 620, and one or more application programs, wherein one or more application programs may be stored in the memory 620 and configured to be executed by one or more processors 610, and one or more programs are configured to execute the method described in the aforementioned method embodiment.
[0087] The processor 610 may include one or more processing cores. The processor 610 uses various interfaces and lines to connect various parts of the entire electronic device 600, and executes various functions of the electronic device 600 and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 620, and calling data stored in the memory 620. Optionally, the processor 610 can be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 610 can integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It is understandable that the above-mentioned modem may not be integrated into the processor 610, but may be implemented by a communication chip alone. The memory 620 may include a random access memory (RAM) or a read-only memory (ROM). The memory 620 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 620 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc. The data storage area may also store data (such as a phone book, audio and video data, chat record data) created by the electronic device 600 during use.
[0088] Please refer to Figure 7 , which shows a structural block diagram of a computer-readable storage medium provided in an embodiment of the present application. The computer-readable medium 700 stores program codes, which can be called by a processor to execute the method described in the above method embodiment.
[0089] The computer readable storage medium 700 can be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM. Optionally, the computer readable storage medium 700 includes a non-transitory computer-readable storage medium. The computer readable storage medium 700 has a storage space for program code 710 that performs any method step in the above method. These program codes can be read from or written into one or more computer program products. The program code 710 can be compressed, for example, in an appropriate form.
[0090] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.
Claims
1. A video splicing method, characterized in that: The method comprises: Based on a plurality of pre-acquired initial video files of different viewing angles, acquiring a plurality of hierarchical images corresponding to a single image frame in each of the initial video files; Determining a coding frame rate based on a hierarchical sequence number of the hierarchical image, wherein the coding frame rate is negatively correlated with the hierarchical sequence number; According to the coding frame rates corresponding to the hierarchical images with different hierarchical sequence numbers, encoding is performed on the multiple hierarchical images in each of the initial video files to obtain multiple coded hierarchical images, wherein each of the coded hierarchical images has the same hierarchical sequence number as the corresponding hierarchical image; After splicing the coding level images with the same level sequence number in different initial video files, an intermediate video file is obtained; The intermediate video file is decoded to obtain a target video file.
2. The method according to claim 1, characterized in that The step of obtaining a plurality of hierarchical images corresponding to a single image frame in each of the initial video files based on the pre-acquired initial video files of different viewing angles includes: A plurality of image frames in each of the initial video files are acquired, and the image frames are separated into a plurality of the hierarchical images according to the movement speed of the object in each of the image frames.
3. The method according to claim 2, characterized in that The step of separating the image frame into a plurality of hierarchical images according to the moving speed of the object in each of the image frames comprises: Determining the scene distance range to which the object belongs according to the moving speed of the object in each of the image frames; A level sequence number is determined based on the viewing distance range to which the object belongs, and the level image corresponding to the level sequence number is obtained, wherein the level sequence number is positively correlated with the viewing distance range.
4. The method according to claim 1, characterized in that: After the coding level images with the same level sequence number in different initial video files are spliced, an intermediate video file is obtained, including: Acquire feature points of coding level images having the same level sequence number in different initial video files; The coding level images in different initial video files are spliced based on the feature points to obtain the intermediate video file.
5. The method according to claim 4, characterized in that The obtaining of feature points of coding level images having the same level sequence number in different initial video files includes: The feature points are obtained by a scale-invariant feature transformation method.
6. The method according to claim 1, characterized in that Before obtaining a plurality of hierarchical images corresponding to each image frame in each of the pre-acquired initial video files based on the pre-acquired initial video files, the method further includes: Based on multiple video acquisition devices, multiple acquisition videos are acquired; A coordinate conversion operation is performed on the captured video to obtain the initial video file.
7. The method according to claim 6, characterized in that The coordinate conversion operation includes: Based on a pre-established spherical coordinate system, obtaining a mapping relationship between the image coordinates of the video acquisition device and the coordinates of the spherical coordinate system; A coordinate conversion operation is performed on the captured video based on the mapping relationship to obtain the initial video file.
8. A video splicing device, characterized in that: The device comprises: A layering module, for obtaining, based on a plurality of pre-acquired initial video files of different viewing angles, a plurality of layered images corresponding to each image frame in each of the initial video files; A determination module, configured to determine a coding frame rate based on a hierarchical sequence number of the hierarchical image, wherein the coding frame rate is negatively correlated with the hierarchical sequence number; An encoding module, configured to encode the multiple hierarchical images in each of the initial video files according to the encoding frame rates corresponding to the hierarchical images with different hierarchical sequence numbers, to obtain a corresponding plurality of encoded hierarchical images, wherein each of the encoded hierarchical images has the same hierarchical sequence number as the corresponding hierarchical image; A splicing module, used for splicing the coded level images with the same level sequence number in different initial video files to obtain an intermediate video file; A decoding module is used to decode the intermediate video file to obtain a target video file.
9. An electronic device, characterized in that: include: one or more processors; Memory; One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more applications are configured to execute the method according to any one of claims 1-7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program code executable by a processor, and when the program code is executed by the processor, the processor executes the method according to any one of claims 1 to 7.