Panoramic video transition rendering methods, apparatus and computer equipment

Abstract

This application relates to a panoramic video transition rendering method, apparatus, computer device, storage medium, and computer program product. The method includes: performing feature matching on the unoccluded image regions of each pair of adjacent frames in at least one image group within an image set, obtaining matching point pairs between the unoccluded image regions of each pair of adjacent frames; optimizing the translation amount between each pair of adjacent frames based on the matching point pairs; and determining the forward direction of the virtual camera at the corresponding moment of each original video frame based on the optimized translation amount between each pair of adjacent frames. This avoids mismatches during feature matching due to occluded regions. Furthermore, histogram statistical analysis of the optimized translation amount between each pair of adjacent frames effectively solves the problem of poor forward direction calculation in scenes with weak textures and near-static scenes.

Description

Technical Field

[0001] This application relates to the field of panoramic video processing technology, and in particular to a panoramic video transition rendering method, apparatus, computer equipment, storage medium, and computer program product. Background Technology

[0002] In related technologies, when rendering panoramic videos, the virtual camera's viewpoint is typically based on either a fixed absolute viewpoint of the panoramic camera or a smoothly rotated viewpoint. However, using a fixed absolute viewpoint may not be the viewpoint the user is consistently interested in; similarly, using a smoothly rotated viewpoint may result in most views not being of interest to the user. In reality, users generally focus more on what's in front of the panoramic camera. The current methods for rendering panoramic videos fail to meet these user needs.

[0003] To address the aforementioned issues, a patent titled "Automatic Viewpoint Adjustment Panoramic Video Rendering Method" discloses an automatic viewpoint adjustment panoramic video rendering method. This method includes: "obtaining the rotation of the panoramic camera relative to the world coordinate system when capturing the current video frame and the multi-channel fisheye images corresponding to the current and previous video frames of the panoramic video; extracting corner points from the multi-channel fisheye images corresponding to the previous video frame of the panoramic video to obtain a sequence of corner points to be tracked; tracking the sequence of corner points to be tracked to obtain matching point pairs to be tracked in the fisheye images corresponding to the current and previous video frames; optimizing the displacement of the panoramic camera relative to the previous video frame based on the matching point pairs to obtain an optimized displacement; using the optimized displacement as the forward direction of the virtual camera, calculating the rotation matrix of the current virtual camera, and performing transition rendering on the current video frame of the panoramic video using the rotation of the panoramic camera relative to the world coordinate system when capturing the current video frame and the rotation matrix of the current virtual camera."

[0004] While it addresses user needs, panoramic cameras are typically used in practice by people holding them with both hands or mounting them on motorcycles or cars. Because panoramic cameras use fisheye lenses and have a 360-degree field of view, it's inevitable that parts of the lens will be obstructed by people's hands or objects supporting the camera on vehicles during shooting. This can lead to mismatches during feature matching, resulting in inaccurate forward direction of the virtual camera. Summary of the Invention

[0005] Therefore, it is necessary to provide a panoramic video transition rendering method, apparatus, computer equipment, storage medium, and computer program product that can improve the accuracy of the forward direction of the virtual camera in order to address the above-mentioned technical problems.

[0006] Firstly, this application provides a panoramic video transition rendering method. The method includes:

[0007] Obtain an image set, where each image set is obtained by extracting frames from the original video, and each original video frame is synthesized from multiple fisheye images.

[0008] Based on at least one image group in the image group set, determine several occlusion region masks, and based on the several occlusion region masks, determine the unoccluded image region of each frame in the image group set;

[0009] Feature matching is performed on the unoccluded image regions of each pair of adjacent frames in at least one image group in the image group set to obtain matching point pairs between the unoccluded image regions of each pair of adjacent frames.

[0010] Based on the matching point pairs between the unoccluded image regions of each of the two adjacent frames, the translation amount between each of the two adjacent frames is optimized. Based on the optimized translation amount between each of the two adjacent frames, the forward direction of the virtual camera at the corresponding moment of each original video frame is determined.

[0011] Calculate the rotation matrix of the virtual camera at each corresponding moment of each original video frame, based on the forward direction of the virtual camera at each corresponding moment of each original video frame.

[0012] Transition rendering is performed on each original video frame based on the rotation matrix of the virtual camera at the corresponding moment of each original video frame and the rotation of the panoramic camera relative to the world coordinate system when capturing each original video frame.

[0013] In one embodiment, determining a plurality of occlusion region masks based on at least one image group in the image group set includes:

[0014] The multi-channel fisheye image corresponding to each frame of at least one image group in the image group set is divided into blocks to obtain the block region set corresponding to at least one image group in the image group set;

[0015] Based on the average gray value of each block region in the block region set, determine the maximum average gray value, and calculate the difference between the average gray value of each block region and the maximum average gray value.

[0016] The block region corresponding to the difference that is greater than the preset threshold among all the differences is taken as the target block region. The target block region is the occluded area in the fisheye image corresponding to the target block region.

[0017] Based on the occlusion area in the fisheye image corresponding to each frame, determine several occlusion area masks.

[0018] In one embodiment, feature matching is performed on the unoccluded image regions of each pair of adjacent frames in at least one image group in the image group set to obtain matching point pairs between the unoccluded image regions of each pair of adjacent frames, including:

[0019] Feature extraction is performed on the unoccluded image regions in each frame of at least one image group in the image set to obtain feature points within the unoccluded image regions in each frame of at least one image group in the image set.

[0020] Feature point matching is performed on the feature points within the unoccluded image regions of each two adjacent frames in at least one image group in the image set to obtain matching point pairs corresponding to each two adjacent frames in at least one image group in the image set.

[0021] In one embodiment, feature extraction is performed on the unoccluded image regions in each frame of at least one image group in the image set, including:

[0022] For any unoccluded image region, take any unoccluded image region as the current image region and extract feature points in the current image region; wherein, the extraction result meets the preset conditions, the preset conditions include that every two adjacent feature points are equally spaced or the ratio between the area of ​​the region enclosed by all extracted feature points and the area of ​​the current image region is greater than a preset threshold.

[0023] In one embodiment, before performing feature point matching on feature points within the unoccluded image regions of each of at least two adjacent frames in the image set, the method further includes:

[0024] Based on a random sampling algorithm, feature points within the unoccluded image regions of each frame in at least one image group in the image set are filtered.

[0025] In one embodiment, determining the forward direction of the virtual camera at the corresponding moment of each original video frame, based on the optimized translation amount between every two adjacent frames, includes:

[0026] Based on the translation amount before optimization and the translation amount after optimization between two adjacent frames, the preset direction vector of each pair of adjacent frames in the camera coordinate system is determined.

[0027] Based on the preset direction vectors of each two adjacent frames in the camera coordinate system, determine the true principal direction corresponding to each sub-direction sequence;

[0028] Based on the real principal direction corresponding to each sub-direction sequence, determine the forward direction of the virtual camera at the corresponding moment of each original video frame.

[0029] In one embodiment, a preset direction vector in the camera coordinate system is determined based on the translation amount before optimization and the translation amount after optimization between two adjacent frames, including:

[0030] The integrated translation amount between each pair of adjacent frames is obtained by weighting the translation amount before optimization between each pair of adjacent frames and the translation amount after optimization between each pair of adjacent frames.

[0031] The integrated translation between each pair of adjacent frames is transformed into the camera coordinate system to obtain the direction vector of each pair of adjacent frames in the camera coordinate system.

[0032] Calculate the angle between the direction vector of each two adjacent frames in the camera coordinate system and each preset direction vector. Based on the angle between each two adjacent frames, determine the preset direction vector into which the direction vector of each two adjacent frames falls in the camera coordinate system.

[0033] In one embodiment, a direction vector sequence is formed by the direction vectors of every two adjacent frames in the camera coordinate system; the true principal direction corresponding to each sub-direction sequence is determined based on the preset direction vectors of every two adjacent frames in the camera coordinate system, including:

[0034] Based on the temporal sequence in the direction vector sequence, the direction vector sequence is segmented to obtain several sub-direction sequences;

[0035] Based on the preset direction vectors into which the direction vectors of two adjacent frames fall in the camera coordinate system, determine the number of times each sub-direction sequence falls into each preset direction vector, and take the preset direction vector into which each sub-direction sequence falls the most times as the main direction corresponding to each sub-direction sequence.

[0036] If the total number of times the principal direction corresponding to each sub-direction sequence falls within a preset number of times, then the principal direction corresponding to each sub-direction sequence is taken as the real principal direction corresponding to each sub-direction sequence.

[0037] In one embodiment, determining the forward direction of the virtual camera at each moment corresponding to each original video frame, based on the real principal direction corresponding to each sub-direction sequence, includes:

[0038] Smoothing and interpolation are performed on the real principal direction corresponding to each sub-direction sequence to obtain the corrected direction vector at the corresponding moment of each original video frame.

[0039] Transform the corrected direction vector of each original video frame at the corresponding time into the world coordinate system to obtain the direction vector of each original video frame at the corresponding time.

[0040] The direction vector at the corresponding moment of each original video frame is used as the forward direction of the virtual camera at the corresponding moment of each original video frame.

[0041] Secondly, this application also provides a panoramic video transition rendering apparatus. The apparatus includes:

[0042] The acquisition module is used to acquire a set of image groups. Each image group in the set is obtained by extracting frames from the original video. Each original video frame is synthesized from multiple fisheye images.

[0043] The first determining module is used to determine several occlusion region masks based on at least one image group in the image group set, and to determine the unoccluded image region of each frame in the image group set based on the several occlusion region masks.

[0044] The second determining module is used to perform feature matching on the unoccluded image regions of each two adjacent frames of at least one image group in the image group set, so as to obtain matching point pairs between the unoccluded image regions of each two adjacent frames.

[0045] The third determining module is used to optimize the translation amount between each pair of adjacent frames based on the matching point pairs between the unoccluded image regions of each pair of adjacent frames, and to determine the forward direction of the virtual camera at the corresponding moment of each original video frame based on the optimized translation amount between each pair of adjacent frames.

[0046] The calculation module is used to calculate the rotation matrix of the virtual camera at each corresponding moment of each original video frame, based on the forward direction of the virtual camera at each corresponding moment of each original video frame.

[0047] The rendering module is used to perform transition rendering on each original video frame based on the rotation matrix of the virtual camera at the corresponding moment of each original video frame and the rotation of the panoramic camera relative to the world coordinate system when capturing each original video frame.

[0048] Thirdly, this application also provides a computer device, the computer device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0049] Obtain an image set, where each image set is obtained by extracting frames from the original video, and each original video frame is synthesized from multiple fisheye images.

[0050] Based on at least one image group in the image group set, determine several occlusion region masks, and based on the several occlusion region masks, determine the unoccluded image region of each frame in the image group set;

[0051] Feature matching is performed on the unoccluded image regions of each pair of adjacent frames in at least one image group in the image group set to obtain matching point pairs between the unoccluded image regions of each pair of adjacent frames.

[0052] Based on the matching point pairs between the unoccluded image regions of each of the two adjacent frames, the translation amount between each of the two adjacent frames is optimized. Based on the optimized translation amount between each of the two adjacent frames, the forward direction of the virtual camera at the corresponding moment of each original video frame is determined.

[0053] Calculate the rotation matrix of the virtual camera at each corresponding moment of each original video frame, based on the forward direction of the virtual camera at each corresponding moment of each original video frame.

[0054] Transition rendering is performed on each original video frame based on the rotation matrix of the virtual camera at the corresponding moment of each original video frame and the rotation of the panoramic camera relative to the world coordinate system when capturing each original video frame.

[0055] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0056] Obtain an image set, where each image set is obtained by extracting frames from the original video, and each original video frame is synthesized from multiple fisheye images.

[0057] Based on at least one image group in the image group set, determine several occlusion region masks, and based on the several occlusion region masks, determine the unoccluded image region of each frame in the image group set;

[0058] Feature matching is performed on the unoccluded image regions of each pair of adjacent frames in at least one image group in the image group set to obtain matching point pairs between the unoccluded image regions of each pair of adjacent frames.

[0059] Based on the matching point pairs between the unoccluded image regions of each of the two adjacent frames, the translation amount between each of the two adjacent frames is optimized. Based on the optimized translation amount between each of the two adjacent frames, the forward direction of the virtual camera at the corresponding moment of each original video frame is determined.

[0060] Calculate the rotation matrix of the virtual camera at each corresponding moment of each original video frame, based on the forward direction of the virtual camera at each corresponding moment of each original video frame.

[0061] Transition rendering is performed on each original video frame based on the rotation matrix of the virtual camera at the corresponding moment of each original video frame and the rotation of the panoramic camera relative to the world coordinate system when capturing each original video frame.

[0062] Fifthly, this application also provides a computer program product, which includes a computer program that, when executed by a processor, performs the following steps:

[0063] Obtain an image set, where each image set is obtained by extracting frames from the original video, and each original video frame is synthesized from multiple fisheye images.

[0064] Based on at least one image group in the image group set, determine several occlusion region masks, and based on the several occlusion region masks, determine the unoccluded image region of each frame in the image group set;

[0065] Feature matching is performed on the unoccluded image regions of each pair of adjacent frames in at least one image group in the image group set to obtain matching point pairs between the unoccluded image regions of each pair of adjacent frames.

[0066] Based on the matching point pairs between the unoccluded image regions of each of the two adjacent frames, the translation amount between each of the two adjacent frames is optimized. Based on the optimized translation amount between each of the two adjacent frames, the forward direction of the virtual camera at the corresponding moment of each original video frame is determined.

[0067] Calculate the rotation matrix of the virtual camera at each corresponding moment of each original video frame, based on the forward direction of the virtual camera at each corresponding moment of each original video frame.

[0068] Transition rendering is performed on each original video frame based on the rotation matrix of the virtual camera at the corresponding moment of each original video frame and the rotation of the panoramic camera relative to the world coordinate system when capturing each original video frame.

[0069] The aforementioned panoramic video transition rendering method, apparatus, computer equipment, storage medium, and computer program products, because they do not need to consider feature points of occluded areas during feature matching, can avoid mismatches caused by the presence of occluded areas, thereby improving the accuracy of the optimized translation between each pair of adjacent frames. That is, it can improve the accuracy of the virtual camera's forward direction at each corresponding moment in the original video. Furthermore, since determining the occluded area mask or performing feature matching does not require processing every single original video frame, but rather extracts frames from the original video to obtain a set of image groups, and only processes at least one image group, the computational load for determining the occluded area mask or performing feature matching can be reduced. In addition, histogram statistical analysis of the optimized translation between each pair of adjacent frames effectively solves the problem of poor forward direction calculation in scenes with weak textures and near-static scenes. Attached Figure Description

[0070] Figure 1 This is an application environment diagram of a panoramic video transition rendering method in one embodiment;

[0071] Figure 2 This is a flowchart illustrating a panoramic video transition rendering method in one embodiment;

[0072] Figure 3 This is a structural block diagram of a panoramic video transition rendering device in one embodiment;

[0073] Figure 4 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0074] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0075] It is understood that the terms “first,” “second,” etc., used in this application may be used to describe various technical terms, but unless otherwise stated, these technical terms are not limited to these terms. These terms are only used to distinguish one technical term from another.

[0076] In related technologies, when rendering panoramic videos, the virtual camera's viewpoint is typically based on either a fixed absolute viewpoint of the panoramic camera or a smoothly rotated viewpoint. However, using a fixed absolute viewpoint may not be the viewpoint the user is consistently interested in; similarly, using a smoothly rotated viewpoint may result in most views not being of interest to the user. In reality, users generally focus more on what's in front of the panoramic camera. The current methods for rendering panoramic videos fail to meet these user needs.

[0077] To address the aforementioned issues, a patent titled "Automatic Viewpoint Adjustment Panoramic Video Rendering Method" discloses an automatic viewpoint adjustment panoramic video rendering method. This method includes: "obtaining the rotation of the panoramic camera relative to the world coordinate system when capturing the current video frame and the multi-channel fisheye images corresponding to the current and previous video frames of the panoramic video; extracting corner points from the multi-channel fisheye images corresponding to the previous video frame of the panoramic video to obtain a sequence of corner points to be tracked; tracking the sequence of corner points to be tracked to obtain matching point pairs to be tracked in the fisheye images corresponding to the current and previous video frames; optimizing the displacement of the panoramic camera relative to the previous video frame based on the matching point pairs to obtain an optimized displacement; using the optimized displacement as the forward direction of the virtual camera, calculating the rotation matrix of the current virtual camera, and performing transition rendering on the current video frame of the panoramic video using the rotation of the panoramic camera relative to the world coordinate system when capturing the current video frame and the rotation matrix of the current virtual camera."

[0078] While addressing user needs, panoramic cameras are typically used in practice by people holding them with both hands or mounting them on motorcycles or cars. Because panoramic cameras use fisheye lenses and have a 360-degree field of view, it's inevitable that parts of the lens will be obstructed by hands or other objects supporting the camera during shooting. This results in occluded areas in each frame of the video, causing mismatches between the current and previous video frames during feature matching. Consequently, the displacement of the current video frame relative to the previous frame is not accurately calculated, leading to inaccurate projections of the virtual camera's forward direction.

[0079] To address the problems existing in the aforementioned related technologies, this application provides a panoramic video transition rendering method, which can be applied to... Figure 1 Application scenarios in [the context]. Figure 1The system includes a panoramic camera 101 and a server 102. The panoramic camera 101 is a panoramic camera equipped with multiple fisheye lenses, the number of which is unlimited. When shooting panoramic video, the panoramic camera 101 can be held by a person or fixed to a vehicle using a bracket. The panoramic video captured by the panoramic camera 101 is then transmitted to the server 102. The server 102 primarily processes the panoramic video captured by the panoramic camera 101 to obtain a rendered panoramic video. Alternatively, in actual implementation, the processing function of the server 102 can be directly integrated into the panoramic camera 101. That is, the panoramic video captured by the panoramic camera 101 is rendered to obtain the rendered panoramic video, and when the rendered panoramic video is needed later, it can simply be output from the panoramic camera 101.

[0080] Furthermore, the processing device used to process panoramic video does not necessarily have to be a server; it can also be a dedicated processing device, such as a personal computer or laptop computer. This application does not specifically limit this. It should be noted that the quantities such as "multiple" mentioned in the embodiments of this application all refer to "at least two," for example, "multiple" means "at least two."

[0081] Based on this, see Figure 2 This paper presents a method for rendering panoramic video transitions. Taking the application of this method on a server, with the server as the execution entity, as an example, the method includes the following steps:

[0082] 202. Obtain the image group set. Each image group in the image group set is obtained by extracting frames from the original video. Each original video frame in the original video is synthesized from multiple fisheye images.

[0083] The original video can be a panoramic video, and the panoramic video can be a panoramic video that has undergone image stabilization. Furthermore, the frame extraction interval for each image group in the image set can be different.

[0084] 204. Based on at least one image group in the image group set, determine several occlusion area masks, and based on the several occlusion area masks, determine the unoccluded image area of ​​each frame in the image group set.

[0085] It is understandable that, since each frame of the original video is synthesized from multiple fisheye images, and the images in the image group are extracted from the original video, the occlusion of the camera lens capturing the multiple fisheye images will be reflected in the images. Specifically, this will result in several occluded areas in the images. It is also understandable that, since the occlusion position of the camera lens capturing the multiple fisheye images is usually fixed, the occluded areas in each image will be in the same position and have the same shape within their respective images. Therefore, it is also understandable that, in this step, based on at least one image group, the determined occlusion area masks can be applied to every image in every image group within the image group set; that is, different images have the same occlusion area masks.

[0086] 206. Perform feature matching on the unoccluded image regions of each pair of adjacent frames in at least one image group in the image group set to obtain matching point pairs between the unoccluded image regions of each pair of adjacent frames.

[0087] The "at least one image group" mentioned in this step can be completely identical, partially identical, or completely different from the "at least one image group" mentioned in step 204 above. This application embodiment does not specifically limit this. It is understood that if there is more scene overlap between adjacent two frames, more matching point pairs will be formed. Therefore, to ensure that more matching point pairs are formed between the unoccluded image areas of each adjacent two frames, the "at least one image group" mentioned in this step can be obtained by extracting frames from the original video based on the smallest possible frame extraction interval, to guarantee more scene overlap between each adjacent two frames. It is also understood that the more matching point pairs there are, the more accurate the translation amount will be.

[0088] It should be noted that, in actual implementation, the frame extraction interval for both the "at least one image group" mentioned in this step and the "at least one image group" mentioned in step 204 above can be determined based on the motion speed of the captured fisheye images. Specifically, if the panoramic camera moves faster, the corresponding scene changes more rapidly, and the frame extraction interval can be appropriately chosen to be smaller; conversely, if the panoramic camera moves slower, the corresponding scene changes more slowly, and the frame extraction interval can be appropriately chosen to be larger.

[0089] For ease of understanding, we will take any two adjacent frames as an example, treating them as the previous and next frames respectively. In practice, matching point pairs between the previous and next frames can be obtained through feature matching or optical flow tracing.

[0090] 208. Based on the matching point pairs between the unoccluded image regions of each of the two adjacent frames, optimize the translation amount between each of the two adjacent frames. Based on the optimized translation amount between each of the two adjacent frames, determine the forward direction of the virtual camera at the corresponding moment of each original video frame.

[0091] It should be noted that in actual implementation, multiple fisheye images are captured by multiple fisheye lenses, with each fisheye lens capturing one fisheye image. These multiple fisheye lenses can be integrated into a single panoramic camera. The "translation amount between each two adjacent frames" mentioned in this step can be used as the translation amount of the next frame in each pair of adjacent frames, representing the movement vector of the panoramic camera when capturing the next frame relative to when capturing the previous frame.

[0092] To facilitate understanding, let's take any image group in the "at least one image group" mentioned in step 206 as an example to illustrate the process of optimizing the translation amount between each pair of adjacent frames in the image group: Initialize the translation amount of the first frame in the image group as (0, 0, 1); Taking any two adjacent frames as the (t-2)th frame, the (t-1)th frame, and the tth frame as examples, we can first obtain the translation amount between the (t-2)th frame and the (t-1)th frame, and then use the translation amount between the (t-2)th frame and the (t-1)th frame as the translation amount between the (t-1)th frame and the tth frame.

[0093] Similarly, taking the (t-1)th frame image and the tth frame image as an example, the translation amount between the (t-1)th frame image and the tth frame image can be optimized through the objective function, which can be expressed by the following formula (1):

[0094]

[0095] Among them, F i (T t The function can be represented by the following formula (2):

[0096]

[0097] In the above formulas (1) and (2), T t T represents the translation amount between the (t-1)th frame and the tth frame. t-1 Let be the translation amount between the (t-2)th frame and the (t-1)th frame, F be the cost function to be optimized, and i be the matching point pair index; ρ(s) be the loss function, N be the number of feature points to be optimized, and a be the smoothing parameter, which can be 0.1; d t,i and d t-1,i They are respectively and The corresponding depth, and Let represent the matching point pairs between the unoccluded image regions of the t-th frame and the (t-1)-th frame in the world coordinate system, respectively. The loss function ρ(s) can be expressed by the following formula (3):

[0098]

[0099] Specifically, F i (t t The optimization iterative process can be as follows: First, the depth of all matching point pairs between the unoccluded image regions of frame t and frame t-1 is solved using the least squares method. Then, the residual is calculated. and F i (T t Jacob functions Where k is the number of iterations, T t k T after k iterations t , and d after k iterations t,i and d t-1,i , and They are respectively and The inverse depth.

[0100] The specific process for solving the depth can be shown in the following formula (4):

[0101]

[0102] It should be noted that the result of formula (4) can be used Indicates. When At that time, when At that time,

[0103] It is understandable that occlusion occurs when a panoramic camera captures multiple fisheye images because the panoramic camera is usually fixed to a movable platform, and the fixing process may obstruct the fisheye lens. The movable platform can be a person or a mobile device; this application does not specifically limit this. The main concept of this application is to abstract the panoramic camera and the movable platform into a movable virtual camera. Transition rendering is performed by calculating the rotation matrix of this virtual camera at each corresponding moment in the original video frame. In this step, the movement direction of the virtual camera at each corresponding moment in the original video frame can be determined first. Since the movable platform usually moves in the direction it faces, this step mainly involves determining the movement direction of the virtual camera at each corresponding moment in the original video frame. In this application embodiment, the translation amount between each two adjacent frames of the image group can be interpolated to obtain the translation amount of the virtual camera at each corresponding moment in the original video frame, which serves as the movement direction of the virtual camera at each corresponding moment in the original video frame.

[0104] 210. Based on the forward direction of the virtual camera at each corresponding moment of the original video frame, calculate the rotation matrix of the virtual camera at each corresponding moment of the original video frame.

[0105] Specifically, for any original video frame, the time corresponding to that original video frame is denoted as t, and the forward direction of the virtual camera at that time is denoted as t. according to Calculate the rotation matrix Rv of the virtual camera at the corresponding moment in the original frame. Rv can be denoted as [e0, e1, e2]. e2 = e0 * e1,

[0106] 212. Based on the rotation matrix of the virtual camera at the corresponding moment of each original video frame and the rotation of the panoramic camera relative to the world coordinate system when capturing each original video frame, perform transition rendering on each original video frame.

[0107] Specifically, the amount of rotation of the panoramic camera relative to the world coordinate system when capturing each raw video frame. It can be calculated based on the gyroscope value of the panoramic camera when each original video frame is captured, specifically using the following formula: in, This represents the rotation of the gyroscope to the world coordinate system at the corresponding moment for each original video frame. This represents the rotation amount from the panoramic camera to the gyroscope at the corresponding moment for each original video frame. and These are the gyroscope values ​​of the panoramic camera at the corresponding moment for each original video frame.

[0108] The method provided in this application avoids mismatches caused by occlusion during feature matching because it does not require consideration of feature points in the occluded region. This improves the accuracy of the optimized translation between adjacent frames, thus enhancing the accuracy of the virtual camera's forward direction at each frame in the original video. Furthermore, since determining the occlusion mask or performing feature matching does not require processing every single original video frame, but rather extracts frames from the original video to obtain a set of image groups, and processes only at least one image group, the computational load for determining the occlusion mask or performing feature matching is reduced.

[0109] In conjunction with the above embodiments, in one embodiment, determining several occlusion region masks based on at least one image group in the image group set includes: dividing the multi-channel fisheye images corresponding to each frame of at least one image group in the image group set into blocks to obtain a block region set corresponding to at least one image group in the image group set; determining the maximum grayscale average value based on the grayscale average value of each block region in the block region set, and calculating the difference between the grayscale average value of each block region and the maximum grayscale average value; taking the block region corresponding to the difference greater than a preset threshold among all differences as the target block region, and the target block region is the occlusion region in the fisheye image corresponding to the target block region; and determining several occlusion region masks based on the occlusion region in the fisheye image corresponding to each frame image.

[0110] Specifically, for any image group in the image group set, taking an image group containing two frames A and B, where both image A and image B are synthesized from two fisheye images a1 and a2, each fisheye image is divided into 4 blocks. The average grayscale values ​​of each block after dividing the a1 fisheye image of image A into 4 blocks are 113, 123, 178 and 146, respectively. The average grayscale values ​​of each block after dividing the a2 fisheye image of image A into 4 blocks are 120, 172, 166 and 159, respectively. The average grayscale values ​​of each block after dividing the a1 fisheye image of image B into 4 blocks are 115, 125, 174 and 145, respectively. The average grayscale values ​​of each block after dividing the a2 fisheye image of image B into 4 blocks are 124, 166, 160 and 148, respectively. And the preset threshold is 40.

[0111] Therefore, this image group can correspond to a total of 16 block regions. Combining the grayscale average value of each block region listed above, the maximum grayscale average value can be determined to be 178. Next, the differences between the grayscale average value of each block region and the maximum grayscale average value can be calculated as 65, 55, 0, 32, 58, 6, 12, 19, 63, 53, 4, 33, 54, 12, 18, and 30, respectively. Since the preset threshold is 40, the block regions with grayscale average values ​​of 113, 123, 120, 115, 125, and 124 can be used as target block regions. Among them, the block region with a grayscale average value of 113 and the block region with a grayscale average value of 123 are set as the occluded regions in the a1 channel fisheye image of image A, and the block region with a grayscale average value of 120 is set as the occluded region in the a2 channel fisheye image of image A. The block regions with an average grayscale value of 115 and 125 are set as the occluded regions in the a1 channel fisheye image of image B, and the block region with an average grayscale value of 124 is set as the occluded region in the a2 channel fisheye image of image B.

[0112] Therefore, by integrating the occlusion regions in the a1-way fisheye images of image A, the a2-way fisheye images of image A, the a1-way fisheye images of image B, and the a2-way fisheye images of image B, the location of the occlusion region mask in the image can be determined. As mentioned in the above embodiments, based on at least one image group, the determined occlusion region masks can be applied to each image in each image group within the image group set; that is, different images have the same number of occlusion region masks. Therefore, the location of the occlusion region mask in the image determined in this step is the location of the occlusion region mask for each image in the image group set.

[0113] The method provided in this application embodiment can accurately determine the location of the occluded area in the fisheye image based on the gray value difference, since the occluded area usually has a high gray value. In this way, the location of the occluded area mask in the image can be determined more accurately.

[0114] In conjunction with the above embodiments, in one embodiment, feature matching is performed on the unoccluded image regions of each pair of adjacent frames in at least one image group in the image group set to obtain matching point pairs between the unoccluded image regions of each pair of adjacent frames. This includes: extracting features from the unoccluded image regions of each frame in at least one image group in the image group set to obtain feature points within the unoccluded image regions of each frame in at least one image group in the image group set; and performing feature point matching on the feature points within the unoccluded image regions of each pair of adjacent frames in at least one image group in the image group set to obtain matching point pairs corresponding to each pair of adjacent frames in at least one image group in the image group set.

[0115] The feature extraction and matching methods can be one or more of the following: SIFT (Scale-invariant feature transform), SURF (Speeded Up Robust Features), ORB (Oriented FAST and Rotated BRIEF), Harris corner detection, and GoodFeatureToTrack. This application does not specifically limit the methods used.

[0116] The method provided in this application embodiment, since the image group is obtained by extracting frames from the original video, can determine the translation amount between each two adjacent original video frames based on the matching point pairs between the unoccluded image regions of each two adjacent frames of at least one image group, thereby reducing the overall computational load.

[0117] In conjunction with the above embodiments, in one embodiment, feature extraction is performed on the unoccluded image region in each frame of at least one image group in the image group set, including: for any unoccluded image region, taking any unoccluded image region as the current image region, and extracting feature points in the current image region; wherein, the extraction result meets preset conditions, the preset conditions including that every two adjacent feature points are equally spaced or the ratio between the area of ​​the region enclosed by all extracted feature points and the area of ​​the current image region is greater than a preset threshold.

[0118] The purpose of setting preset conditions is primarily to prevent the extracted feature points from concentrating in one or a few localized areas when extracting feature points from the current image region. Therefore, by setting the preset condition that every two adjacent feature points are equally spaced, it ensures that the extracted feature points cover the entire current image region. Setting the preset condition that every two adjacent feature points are equally spaced also ensures that the extracted feature points are evenly distributed, facilitating subsequent feature point allocation. Furthermore, setting the preset condition that the ratio between the area enclosed by all extracted feature points and the area of ​​the current image region is greater than a preset threshold is also to prevent the extracted feature points from concentrating in one or a few localized areas.

[0119] The method provided in this application embodiment can avoid the extracted feature points being limited to one or a few local areas in the image, thereby improving the coverage of feature point pairs and thus improving the accuracy of the optimized translation amount between each two adjacent frames in the subsequent image group.

[0120] In conjunction with the above embodiments, in one embodiment, before performing feature point matching on feature points within the unoccluded image regions of each of at least one image group in the image group set, the method further includes: filtering feature points within the unoccluded image regions of each frame of at least one image group in the image group set based on a random sampling algorithm.

[0121] The method provided in this application uses a random sampling algorithm to filter the extracted feature points, which can eliminate erroneous feature points caused by factors such as illumination, imaging angle, geometric deformation and changes in ground features, thereby improving the accuracy of feature point extraction and, in turn, improving the accuracy of the optimized translation amount between each adjacent two frames of images.

[0122] In conjunction with the above embodiments, in one embodiment, determining the forward direction of the virtual camera at the corresponding moment of each original video frame based on the optimized translation amount between each pair of adjacent frames includes: determining a preset direction vector in the camera coordinate system for each pair of adjacent frames based on the unoptimized translation amount and the optimized translation amount between each pair of adjacent frames; determining the real principal direction corresponding to each sub-direction sequence based on the preset direction vector in the camera coordinate system for each pair of adjacent frames; and determining the forward direction of the virtual camera at the corresponding moment of each original video frame based on the real principal direction corresponding to each sub-direction sequence.

[0123] The forward direction of the virtual camera at each original video frame can be determined by histogram statistical analysis of the optimized translation amount between each two adjacent frames.

[0124] The method provided in this application addresses the issue that weak textures in some images within an image group can easily lead to mismatches during feature matching, resulting in numerous mismatches in the formed matching point pairs. Consequently, the optimized translation amount obtained based on these mismatched feature point pairs is also inaccurate. Furthermore, since near-static scenes lack a definite direction of travel, errors in calculating the direction of travel can easily occur. By determining the direction of travel of the virtual camera at each corresponding moment in the original video based on the true principal direction corresponding to each sub-direction sequence, the errors caused by weak textures in some images of the original video and near-static scenes in certain time periods of the original video can be eliminated, thereby improving the accuracy of the direction of travel of the virtual camera at each corresponding moment in the original video frame.

[0125] In conjunction with the above embodiments, in one embodiment, determining a preset direction vector in the camera coordinate system for each pair of adjacent frames based on the translation amount before optimization and the translation amount after optimization between each pair of adjacent frames includes: weighting the translation amount before optimization and the translation amount after optimization between each pair of adjacent frames to obtain an integrated translation amount between each pair of adjacent frames; converting the integrated translation amount between each pair of adjacent frames to the camera coordinate system to obtain a direction vector in the camera coordinate system for each pair of adjacent frames; calculating the angle between the direction vector in the camera coordinate system for each pair of adjacent frames and each preset direction vector; and determining the preset direction vector into which the direction vector in the camera coordinate system for each pair of adjacent frames falls based on the angle between each pair of adjacent frames.

[0126] Specifically, for any image group in at least one image group, taking any two adjacent frames in that image group as an example, the direction vectors of the two adjacent frames in the camera coordinate system can be expressed by the following formula (5):

[0127] Tc = Rc_w1 * Tw; (5)

[0128] Where Tw represents the integrated translation amount between the two adjacent frames, and Rc_w1 represents the rotation amount of the panoramic camera relative to the world coordinate system when capturing the latter frame of the two adjacent frames.

[0129] For example, consider an image group consisting of three frames, T1, T2, and T3, denoted as C, D, E, and F, respectively. If the angles between the direction vectors of adjacent frames T1 and T2 in the camera coordinate system and the preset direction vectors C, D, E, and F are 30 degrees, 25 degrees, 20 degrees, and 5 degrees, respectively, and the angles between the direction vectors of adjacent frames T2 and T3 in the camera coordinate system and the preset direction vectors C, D, E, and F are 30 degrees, 25 degrees, 5 degrees, and 3 degrees, respectively, then the condition for determining which preset direction vector the direction vectors of adjacent frames fall into is that the angle between the two adjacent frames is less than a preset threshold of 7 degrees. Therefore, the preset direction vectors that the direction vectors of T1 and T2 fall into in the camera coordinate system are both F and F.

[0130] In conjunction with the above embodiments, in one embodiment, a direction vector sequence is formed by the direction vectors of each two adjacent frames in the camera coordinate system; the true principal direction corresponding to each sub-direction sequence is determined according to the preset direction vectors of each two adjacent frames in the camera coordinate system, including: segmenting the direction vector sequence based on the temporal sequence to obtain several sub-direction sequences; determining the number of times each sub-direction sequence falls on each preset direction vector according to the preset direction vectors into which the direction vectors of each two adjacent frames in the camera coordinate system fall, and taking the preset direction vector into which each sub-direction sequence falls the most times as the principal direction corresponding to each sub-direction sequence; if the total number of times the principal direction corresponding to each sub-direction sequence falls within a preset number range, then the principal direction corresponding to each sub-direction sequence is taken as the true principal direction corresponding to each sub-direction sequence.

[0131] Specifically, taking an image group consisting of 10 frames T1, T2, T3, T4, T5, T6, T7, T8, T9, and T10 as an example, and denoting the direction vectors used to calculate the included angle as C, D, E, and F respectively, the direction vector sequence is segmented to obtain three sub-direction sequences Y1, Y2, and Y3. Sub-direction sequence Y1 is composed of the direction vectors of three pairs of adjacent frames (T1 and T2, T2 and T3, and T3 and T4) in the camera coordinate system. Sub-direction sequence Y2 is composed of the direction vectors of three pairs of adjacent frames (T4 and T5, T5 and T6, and T6 and T7) in the camera coordinate system. Sub-direction sequence Y3 is composed of the direction vectors of three pairs of adjacent frames (T7 and T8, T8 and T9, and T9 and T10) in the camera coordinate system.

[0132] After counting the total number of times each sub-direction sequence Y1 falls into the four preset direction vectors C, D, E, and F, the number of times they fall into these vectors are 0, 0, 1, and 3, respectively; the number of times the sub-direction sequence Y2 falls into these vectors are 2, 1, 0, and 0, respectively; and the number of times the sub-direction sequence Y3 falls into these vectors are 2, 3, 0, and 0, respectively. Therefore, it can be determined that the principal direction corresponding to sub-direction sequence Y1 is the preset direction vector F, the principal direction corresponding to sub-direction sequence Y2 is the preset direction vector C, and the principal direction corresponding to sub-direction sequence Y3 is the preset direction vector D. In summary, the principal direction corresponding to sub-direction sequence Y1 can be taken as the true principal direction corresponding to sub-direction sequence Y1, that is, the preset direction vector F can be taken as the true principal direction corresponding to sub-direction sequence Y1. Similarly, the principal direction corresponding to sub-direction sequence Y3 can be taken as the true principal direction corresponding to sub-direction sequence Y3, that is, the preset direction vector D can be taken as the true principal direction corresponding to sub-direction sequence Y3.

[0133] In conjunction with the above embodiments, in one embodiment, determining the forward direction of the virtual camera at each original video frame time according to the real principal direction corresponding to each sub-direction sequence includes: smoothing and interpolating the real principal direction corresponding to each sub-direction sequence to obtain the corrected direction vector at each original video frame time; transforming the corrected direction vector at each original video frame time to the world coordinate system to obtain the direction vector at each original video frame time; and using the direction vector at each original video frame time as the forward direction of the virtual camera at each original video frame time.

[0134] The smoothing process can use an nth-order weighted smoothing algorithm or a smoothing function; this application does not specifically limit the method used. Interpolation can use quaternion transformation spherical linear interpolation or three-dimensional rotational spherical linear interpolation; this application does not specifically limit the method used.

[0135] The method provided in this application embodiment does not require processing every two adjacent frames of the original video. Instead, it performs smoothing and interpolation processing on the real principal direction corresponding to each sub-direction sequence to obtain the direction vector at the corresponding moment of each original video frame, thereby reducing the amount of computation.

[0136] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0137] Based on the same inventive concept, this application also provides a panoramic video transition rendering apparatus for implementing the panoramic video transition rendering method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more panoramic video transition rendering apparatus embodiments provided below can be found in the limitations of the panoramic video transition rendering method described above, and will not be repeated here.

[0138] In one embodiment, such as Figure 3 As shown, a panoramic video transition rendering device is provided, including: an acquisition module 301, a first determination module 302, a second determination module 303, a third determination module 304, a calculation module 305, and a rendering module 306, wherein:

[0139] The acquisition module 301 is used to acquire a set of image groups. Each image group in the set of image groups is obtained by extracting frames from the original video. Each original video frame in the original video is synthesized from multiple fisheye images.

[0140] The first determining module 302 is used to determine a plurality of occlusion region masks based on at least one image group in the image group set, and to determine the unoccluded image region of each frame in the image group set based on the plurality of occlusion region masks.

[0141] The second determining module 303 is used to perform feature matching on the unoccluded image regions of each two adjacent frames of at least one image group in the image group set, so as to obtain matching point pairs between the unoccluded image regions of each two adjacent frames.

[0142] The third determining module 304 is used to optimize the translation amount between each pair of adjacent frames based on the matching point pairs between the unoccluded image regions of each pair of adjacent frames, and to determine the forward direction of the virtual camera at the corresponding moment of each original video frame based on the optimized translation amount between each pair of adjacent frames.

[0143] The calculation module 305 is used to calculate the rotation matrix of the virtual camera at each time corresponding to each original video frame, based on the forward direction of the virtual camera at each time corresponding to each original video frame.

[0144] The rendering module 306 is used to perform transition rendering on each original video frame based on the rotation matrix of the virtual camera at the corresponding moment of each original video frame and the rotation amount of the panoramic camera relative to the world coordinate system when capturing each original video frame.

[0145] The apparatus provided in this application does not need to consider feature points in occluded areas during feature matching, thus avoiding mismatches caused by occlusion and improving the accuracy of the optimized translation between adjacent frames. In other words, it improves the accuracy of the virtual camera's forward direction at each frame in the original video. Furthermore, since determining the occlusion mask or performing feature matching does not require processing every single original video frame, but rather extracts frames from the original video to obtain a set of image groups, and processes only at least one image group, the computational load for determining the occlusion mask or performing feature matching is reduced.

[0146] In one embodiment, the first determining module 302 includes:

[0147] The segmentation unit is used to segment the multi-channel fisheye image corresponding to each frame of at least one image group in the image group set to obtain a set of segmented regions corresponding to at least one image group in the image group set.

[0148] The first determining unit is used to determine the maximum average gray value based on the average gray value of each block region in the block region set, and to calculate the difference between the average gray value of each block region and the maximum average gray value.

[0149] As a unit, it is used to take the block region corresponding to the difference greater than the preset threshold among all the differences as the target block region. The target block region is the occluded area in the fisheye image corresponding to the target block region.

[0150] The second determining unit is used to determine several occlusion area masks based on the occlusion area in the fisheye image corresponding to each frame image.

[0151] In one embodiment, the second determining module 303 includes:

[0152] The feature extraction unit is used to extract features from the unoccluded image regions in each frame of at least one image group in the image set, and obtain feature points within the unoccluded image regions in each frame of at least one image group in the image set.

[0153] The matching unit is used to perform feature point matching on feature points in the unoccluded image regions of each two adjacent frames of at least one image group in the image group set, so as to obtain matching point pairs corresponding to each two adjacent frames of at least one image group in the image group set.

[0154] In one embodiment, the feature extraction unit is further configured to, for any unoccluded image region, take any unoccluded image region as the current image region and extract feature points in the current image region; wherein the extraction result satisfies preset conditions, the preset conditions including that every two adjacent feature points are equally spaced or the ratio between the area of ​​the region enclosed by all extracted feature points and the area of ​​the current image region is greater than a preset threshold.

[0155] In one embodiment, the second determining module 303 further includes:

[0156] The filtering unit is used to filter feature points in the unoccluded image regions of each frame of at least one image group in the image group set based on a random sampling algorithm.

[0157] In one embodiment, the third determining module 304 includes:

[0158] The third determining unit is used to determine the preset direction vector of each pair of adjacent frames in the camera coordinate system based on the translation amount before optimization and the translation amount after optimization between each pair of adjacent frames.

[0159] The fourth determining unit is used to determine the true principal direction corresponding to each sub-direction sequence based on the preset direction vector of each two adjacent frames in the camera coordinate system.

[0160] The fifth determining unit is used to determine the forward direction of the virtual camera at the corresponding moment of each original video frame based on the real main direction corresponding to each sub-direction sequence.

[0161] In one embodiment, the third determining unit includes:

[0162] The weighted subunit is used to weight the translation amount before optimization between each pair of adjacent frames and the translation amount after optimization between each pair of adjacent frames to obtain the integrated translation amount between each pair of adjacent frames.

[0163] The first transformation subunit is used to transform the integrated translation amount between each two adjacent frames into the camera coordinate system, so as to obtain the direction vector of each two adjacent frames in the camera coordinate system.

[0164] The calculation subunit is used to calculate the angle between the direction vector of each two adjacent frames in the camera coordinate system and each preset direction vector. Based on the angle between each two adjacent frames, the preset direction vector into which the direction vector of each two adjacent frames in the camera coordinate system falls is determined.

[0165] In one embodiment, a direction vector sequence is formed by the direction vectors of every two adjacent frames in the camera coordinate system; the fourth determining unit includes:

[0166] The segmented sub-unit is used to segment the direction vector sequence based on the time sequence in the direction vector sequence to obtain several sub-direction sequences;

[0167] A sub-unit is defined to determine the number of times each sub-direction sequence falls on each preset direction vector based on the preset direction vectors into which the direction vectors of each two adjacent frames fall in the camera coordinate system. The preset direction vector into which each sub-direction sequence falls the most times is taken as the principal direction corresponding to each sub-direction sequence. If the total number of times the principal direction corresponding to each sub-direction sequence falls within a preset number, then the principal direction corresponding to each sub-direction sequence is taken as the true principal direction corresponding to each sub-direction sequence.

[0168] In one embodiment, the fifth determining unit includes:

[0169] The smoothing subunit is used to smooth and interpolate the real principal direction corresponding to each sub-direction sequence to obtain the corrected direction vector at the corresponding moment of each original video frame.

[0170] The second transformation subunit is used to transform the corrected direction vector of each original video frame to the world coordinate system to obtain the direction vector of each original video frame at the corresponding time.

[0171] As a sub-unit, it is used to take the direction vector of each original video frame at the corresponding moment as the forward direction of the virtual camera at the corresponding moment of each original video frame.

[0172] Each module in the aforementioned panoramic video transition rendering device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0173] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data such as the original video and the rendered original video. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements a panoramic video transition rendering method.

[0174] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0175] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0176] Obtain an image set, where each image set is obtained by extracting frames from the original video, and each original video frame is synthesized from multiple fisheye images.

[0177] Based on at least one image group in the image group set, determine several occlusion region masks, and based on the several occlusion region masks, determine the unoccluded image region of each frame in the image group set;

[0178] Feature matching is performed on the unoccluded image regions of each pair of adjacent frames in at least one image group in the image group set to obtain matching point pairs between the unoccluded image regions of each pair of adjacent frames.

[0179] Based on the matching point pairs between the unoccluded image regions of each of the two adjacent frames, the translation amount between each of the two adjacent frames is optimized. Based on the optimized translation amount between each of the two adjacent frames, the forward direction of the virtual camera at the corresponding moment of each original video frame is determined.

[0180] Calculate the rotation matrix of the virtual camera at each corresponding moment of each original video frame, based on the forward direction of the virtual camera at each corresponding moment of each original video frame.

[0181] Transition rendering is performed on each original video frame based on the rotation matrix of the virtual camera at the corresponding moment of each original video frame and the rotation of the panoramic camera relative to the world coordinate system when capturing each original video frame.

[0182] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0183] The multi-channel fisheye image corresponding to each frame of at least one image group in the image group set is divided into blocks to obtain the block region set corresponding to at least one image group in the image group set;

[0184] Based on the average gray value of each block region in the block region set, determine the maximum average gray value, and calculate the difference between the average gray value of each block region and the maximum average gray value.

[0185] The block region corresponding to the difference that is greater than the preset threshold among all the differences is taken as the target block region. The target block region is the occluded area in the fisheye image corresponding to the target block region.

[0186] Based on the occlusion area in the fisheye image corresponding to each frame, determine several occlusion area masks.

[0187] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0188] Feature extraction is performed on the unoccluded image regions in each frame of at least one image group in the image set to obtain feature points within the unoccluded image regions in each frame of at least one image group in the image set.

[0189] Feature point matching is performed on the feature points within the unoccluded image regions of each two adjacent frames in at least one image group in the image set to obtain matching point pairs corresponding to each two adjacent frames in at least one image group in the image set.

[0190] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0191] For any unoccluded image region, take any unoccluded image region as the current image region and extract feature points in the current image region; wherein, the extraction result meets the preset conditions, the preset conditions include that every two adjacent feature points are equally spaced or the ratio between the area of ​​the region enclosed by all extracted feature points and the area of ​​the current image region is greater than a preset threshold.

[0192] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0193] Based on a random sampling algorithm, feature points within the unoccluded image regions of each frame in at least one image group in the image set are filtered.

[0194] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0195] Based on the translation amount before optimization and the translation amount after optimization between two adjacent frames, the preset direction vector of each pair of adjacent frames in the camera coordinate system is determined.

[0196] Based on the preset direction vectors of each two adjacent frames in the camera coordinate system, determine the true principal direction corresponding to each sub-direction sequence;

[0197] Based on the real principal direction corresponding to each sub-direction sequence, determine the forward direction of the virtual camera at the corresponding moment of each original video frame.

[0198] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0199] The integrated translation amount between each pair of adjacent frames is obtained by weighting the translation amount before optimization between each pair of adjacent frames and the translation amount after optimization between each pair of adjacent frames.

[0200] The integrated translation between each pair of adjacent frames is transformed into the camera coordinate system to obtain the direction vector of each pair of adjacent frames in the camera coordinate system.

[0201] Calculate the angle between the direction vector of each two adjacent frames in the camera coordinate system and each preset direction vector. Based on the angle between each two adjacent frames, determine the preset direction vector into which the direction vector of each two adjacent frames falls in the camera coordinate system.

[0202] In one embodiment, a direction vector sequence is formed by the direction vectors of every two adjacent frames in the camera coordinate system; the processor also implements the following steps when executing the computer program:

[0203] Based on the temporal sequence in the direction vector sequence, the direction vector sequence is segmented to obtain several sub-direction sequences;

[0204] Based on the preset direction vectors into which the direction vectors of two adjacent frames fall in the camera coordinate system, determine the number of times each sub-direction sequence falls into each preset direction vector, and take the preset direction vector into which each sub-direction sequence falls the most times as the main direction corresponding to each sub-direction sequence.

[0205] If the total number of times the principal direction corresponding to each sub-direction sequence falls within a preset number of times, then the principal direction corresponding to each sub-direction sequence is taken as the real principal direction corresponding to each sub-direction sequence.

[0206] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0207] Smoothing and interpolation are performed on the real principal direction corresponding to each sub-direction sequence to obtain the corrected direction vector at the corresponding moment of each original video frame.

[0208] Transform the corrected direction vector of each original video frame at the corresponding time into the world coordinate system to obtain the direction vector of each original video frame at the corresponding time.

[0209] The direction vector at the corresponding moment of each original video frame is used as the forward direction of the virtual camera at the corresponding moment of each original video frame.

[0210] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0211] Obtain an image set, where each image set is obtained by extracting frames from the original video, and each original video frame is synthesized from multiple fisheye images.

[0212] Based on at least one image group in the image group set, determine several occlusion region masks, and based on the several occlusion region masks, determine the unoccluded image region of each frame in the image group set;

[0213] Feature matching is performed on the unoccluded image regions of each pair of adjacent frames in at least one image group in the image group set to obtain matching point pairs between the unoccluded image regions of each pair of adjacent frames.

[0214] Based on the matching point pairs between the unoccluded image regions of each of the two adjacent frames, the translation amount between each of the two adjacent frames is optimized. Based on the optimized translation amount between each of the two adjacent frames, the forward direction of the virtual camera at the corresponding moment of each original video frame is determined.

[0215] Calculate the rotation matrix of the virtual camera at each corresponding moment of each original video frame, based on the forward direction of the virtual camera at each corresponding moment of each original video frame.

[0216] Transition rendering is performed on each original video frame based on the rotation matrix of the virtual camera at the corresponding moment of each original video frame and the rotation of the panoramic camera relative to the world coordinate system when capturing each original video frame.

[0217] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0218] The multi-channel fisheye image corresponding to each frame of at least one image group in the image group set is divided into blocks to obtain the block region set corresponding to at least one image group in the image group set;

[0219] Based on the average gray value of each block region in the block region set, determine the maximum average gray value, and calculate the difference between the average gray value of each block region and the maximum average gray value.

[0220] The block region corresponding to the difference that is greater than the preset threshold among all the differences is taken as the target block region. The target block region is the occluded area in the fisheye image corresponding to the target block region.

[0221] Based on the occlusion area in the fisheye image corresponding to each frame, determine several occlusion area masks.

[0222] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0223] Feature extraction is performed on the unoccluded image regions in each frame of at least one image group in the image set to obtain feature points within the unoccluded image regions in each frame of at least one image group in the image set.

[0224] Feature point matching is performed on the feature points within the unoccluded image regions of each two adjacent frames in at least one image group in the image set to obtain matching point pairs corresponding to each two adjacent frames in at least one image group in the image set.

[0225] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0226] For any unoccluded image region, take any unoccluded image region as the current image region and extract feature points in the current image region; wherein, the extraction result meets the preset conditions, the preset conditions include that every two adjacent feature points are equally spaced or the ratio between the area of ​​the region enclosed by all extracted feature points and the area of ​​the current image region is greater than a preset threshold.

[0227] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0228] Based on a random sampling algorithm, feature points within the unoccluded image regions of each frame in at least one image group in the image set are filtered.

[0229] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0230] Based on the translation amount before optimization and the translation amount after optimization between two adjacent frames, the preset direction vector of each pair of adjacent frames in the camera coordinate system is determined.

[0231] Based on the preset direction vectors of each two adjacent frames in the camera coordinate system, determine the true principal direction corresponding to each sub-direction sequence;

[0232] Based on the real principal direction corresponding to each sub-direction sequence, determine the forward direction of the virtual camera at the corresponding moment of each original video frame.

[0233] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0234] The integrated translation amount between each pair of adjacent frames is obtained by weighting the translation amount before optimization between each pair of adjacent frames and the translation amount after optimization between each pair of adjacent frames.

[0235] The integrated translation between each pair of adjacent frames is transformed into the camera coordinate system to obtain the direction vector of each pair of adjacent frames in the camera coordinate system.

[0236] Calculate the angle between the direction vector of each two adjacent frames in the camera coordinate system and each preset direction vector. Based on the angle between each two adjacent frames, determine the preset direction vector into which the direction vector of each two adjacent frames falls in the camera coordinate system.

[0237] In one embodiment, a sequence of direction vectors is formed by the direction vectors of every two adjacent frames in the camera coordinate system; when the computer program is executed by the processor, it also performs the following steps:

[0238] Based on the temporal sequence in the direction vector sequence, the direction vector sequence is segmented to obtain several sub-direction sequences;

[0239] Based on the preset direction vectors into which the direction vectors of two adjacent frames fall in the camera coordinate system, determine the number of times each sub-direction sequence falls into each preset direction vector, and take the preset direction vector into which each sub-direction sequence falls the most times as the main direction corresponding to each sub-direction sequence.

[0240] If the total number of times the principal direction corresponding to each sub-direction sequence falls within a preset number of times, then the principal direction corresponding to each sub-direction sequence is taken as the real principal direction corresponding to each sub-direction sequence.

[0241] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0242] Smoothing and interpolation are performed on the real principal direction corresponding to each sub-direction sequence to obtain the corrected direction vector at the corresponding moment of each original video frame.

[0243] Transform the corrected direction vector of each original video frame at the corresponding time into the world coordinate system to obtain the direction vector of each original video frame at the corresponding time.

[0244] The direction vector at the corresponding moment of each original video frame is used as the forward direction of the virtual camera at the corresponding moment of each original video frame.

[0245] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps:

[0246] Obtain an image set, where each image set is obtained by extracting frames from the original video, and each original video frame is synthesized from multiple fisheye images.

[0247] Based on at least one image group in the image group set, determine several occlusion region masks, and based on the several occlusion region masks, determine the unoccluded image region of each frame in the image group set;

[0248] Feature matching is performed on the unoccluded image regions of each pair of adjacent frames in at least one image group in the image group set to obtain matching point pairs between the unoccluded image regions of each pair of adjacent frames.

[0249] Based on the matching point pairs between the unoccluded image regions of each of the two adjacent frames, the translation amount between each of the two adjacent frames is optimized. Based on the optimized translation amount between each of the two adjacent frames, the forward direction of the virtual camera at the corresponding moment of each original video frame is determined.

[0250] Calculate the rotation matrix of the virtual camera at each corresponding moment of each original video frame, based on the forward direction of the virtual camera at each corresponding moment of each original video frame.

[0251] Transition rendering is performed on each original video frame based on the rotation matrix of the virtual camera at the corresponding moment of each original video frame and the rotation of the panoramic camera relative to the world coordinate system when capturing each original video frame.

[0252] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0253] The multi-channel fisheye image corresponding to each frame of at least one image group in the image group set is divided into blocks to obtain the block region set corresponding to at least one image group in the image group set;

[0254] Based on the average gray value of each block region in the block region set, determine the maximum average gray value, and calculate the difference between the average gray value of each block region and the maximum average gray value.

[0255] The block region corresponding to the difference that is greater than the preset threshold among all the differences is taken as the target block region. The target block region is the occluded area in the fisheye image corresponding to the target block region.

[0256] Based on the occlusion area in the fisheye image corresponding to each frame, determine several occlusion area masks.

[0257] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0258] Feature extraction is performed on the unoccluded image regions in each frame of at least one image group in the image set to obtain feature points within the unoccluded image regions in each frame of at least one image group in the image set.

[0259] Feature point matching is performed on the feature points within the unoccluded image regions of each two adjacent frames in at least one image group in the image set to obtain matching point pairs corresponding to each two adjacent frames in at least one image group in the image set.

[0260] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0261] For any unoccluded image region, take any unoccluded image region as the current image region and extract feature points in the current image region; wherein, the extraction result meets the preset conditions, the preset conditions include that every two adjacent feature points are equally spaced or the ratio between the area of ​​the region enclosed by all extracted feature points and the area of ​​the current image region is greater than a preset threshold.

[0262] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0263] Based on a random sampling algorithm, feature points within the unoccluded image regions of each frame in at least one image group in the image set are filtered.

[0264] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0265] Based on the translation amount before optimization and the translation amount after optimization between two adjacent frames, the preset direction vector of each pair of adjacent frames in the camera coordinate system is determined.

[0266] Based on the preset direction vectors of each two adjacent frames in the camera coordinate system, determine the true principal direction corresponding to each sub-direction sequence;

[0267] Based on the real principal direction corresponding to each sub-direction sequence, determine the forward direction of the virtual camera at the corresponding moment of each original video frame.

[0268] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0269] The integrated translation amount between each pair of adjacent frames is obtained by weighting the translation amount before optimization between each pair of adjacent frames and the translation amount after optimization between each pair of adjacent frames.

[0270] The integrated translation between each pair of adjacent frames is transformed into the camera coordinate system to obtain the direction vector of each pair of adjacent frames in the camera coordinate system.

[0271] Calculate the angle between the direction vector of each two adjacent frames in the camera coordinate system and each preset direction vector. Based on the angle between each two adjacent frames, determine the preset direction vector into which the direction vector of each two adjacent frames falls in the camera coordinate system.

[0272] In one embodiment, a sequence of direction vectors is formed by the direction vectors of every two adjacent frames in the camera coordinate system; when the computer program is executed by the processor, it also performs the following steps:

[0273] Based on the temporal sequence in the direction vector sequence, the direction vector sequence is segmented to obtain several sub-direction sequences;

[0274] Based on the preset direction vectors into which the direction vectors of two adjacent frames fall in the camera coordinate system, determine the number of times each sub-direction sequence falls into each preset direction vector, and take the preset direction vector into which each sub-direction sequence falls the most times as the main direction corresponding to each sub-direction sequence.

[0275] If the total number of times the principal direction corresponding to each sub-direction sequence falls within a preset number of times, then the principal direction corresponding to each sub-direction sequence is taken as the real principal direction corresponding to each sub-direction sequence.

[0276] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0277] Smoothing and interpolation are performed on the real principal direction corresponding to each sub-direction sequence to obtain the corrected direction vector at the corresponding moment of each original video frame.

[0278] Transform the corrected direction vector of each original video frame at the corresponding time into the world coordinate system to obtain the direction vector of each original video frame at the corresponding time.

[0279] The direction vector at the corresponding moment of each original video frame is used as the forward direction of the virtual camera at the corresponding moment of each original video frame.

[0280] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0281] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0282] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A panoramic video transition rendering method, characterized in that, The method includes: Obtain an image set, wherein each image set is obtained by extracting frames from the original video, and each original video frame is synthesized from multiple fisheye images; Based on at least one image group in the image group set, determine several occlusion area masks, and based on the several occlusion area masks, determine the unoccluded image area of ​​each frame in the image group set; Feature matching is performed on the unoccluded image regions of each pair of adjacent frames in at least one image group in the image group set to obtain matching point pairs between the unoccluded image regions of each pair of adjacent frames. Based on the matching point pairs between the unoccluded image regions of each of the two adjacent frames, the translation amount between each of the two adjacent frames is optimized. Based on the optimized translation amount between each of the two adjacent frames, the forward direction of the virtual camera at the corresponding moment of each original video frame is determined. Specifically, the translation amount between each of the two adjacent frames is interpolated to obtain the translation amount of the virtual camera at the corresponding moment of each original video frame, which is used as the forward direction of the virtual camera at the corresponding moment of each original video frame. Calculate the rotation matrix of the virtual camera at each corresponding moment of each original video frame based on the forward direction of the virtual camera at each corresponding moment of each original video frame. Based on the rotation matrix of the virtual camera at each corresponding moment of each original video frame and the rotation of the panoramic camera relative to the world coordinate system when capturing each original video frame, transition rendering is performed on each original video frame.

2. The method according to claim 1, characterized in that, The step of determining a plurality of occlusion region masks based on at least one image group in the image group set includes: The multi-channel fisheye image corresponding to each frame of at least one image group in the image group set is divided into blocks to obtain a block region set corresponding to at least one image group in the image group set; Based on the average gray value of each block region in the block region set, determine the maximum average gray value, and calculate the difference between the average gray value of each block region and the maximum average gray value. The segmented regions corresponding to all differences that are greater than a preset threshold are taken as target segmented regions, and the target segmented regions are the occluded regions in the fisheye image corresponding to the target segmented regions. The mask for the plurality of occlusion regions is determined based on the occlusion region in the fisheye image corresponding to each frame.

3. The method according to claim 1, characterized in that, The step of performing feature matching on the unoccluded image regions of each pair of adjacent frames in at least one image group in the image group set to obtain matching point pairs between the unoccluded image regions of each pair of adjacent frames includes: Feature extraction is performed on the unoccluded image regions in each frame of at least one image group in the image set to obtain feature points within the unoccluded image regions in each frame of at least one image group in the image set; Feature point matching is performed on feature points within the unoccluded image regions of at least one image group in the image set to obtain matching point pairs corresponding to each pair of adjacent frames in at least one image group in the image set.

4. The method according to claim 3, characterized in that, The step of extracting features from the unoccluded image regions in each frame of at least one image group in the image set includes: For any unoccluded image region, the unoccluded image region is taken as the current image region, and feature points are extracted in the current image region; wherein, the extraction result meets the preset conditions, the preset conditions include that every two adjacent feature points are equally spaced or the ratio between the area of ​​the region enclosed by all extracted feature points and the area of ​​the current image region is greater than a preset threshold.

5. The method according to claim 3, characterized in that, Before performing feature point matching on feature points within the unoccluded image regions of each pair of adjacent frames in the image set, the method further includes: Based on a random sampling algorithm, feature points within the unoccluded image region of each frame image in at least one image group in the image group set are filtered.

6. The method according to claim 1, characterized in that, Determining the forward direction of the virtual camera at the corresponding moment of each original video frame based on the optimized translation amount between each two adjacent image frames includes: Based on the translation amount before optimization and the translation amount after optimization between each pair of adjacent frames, a preset direction vector in the camera coordinate system is determined for each pair of adjacent frames. Based on the preset direction vector of each two adjacent frames in the camera coordinate system, determine the real main direction corresponding to each sub-direction sequence; Based on the real principal direction corresponding to each sub-direction sequence, the forward direction of the virtual camera at the corresponding moment of each original video frame is determined.

7. The method according to claim 6, characterized in that, The step of determining the preset direction vector of each pair of adjacent frames in the camera coordinate system based on the translation amount before optimization and the translation amount after optimization between each pair of adjacent frames includes: The integrated translation amount between each pair of adjacent frames is obtained by weighting the translation amount before optimization between each pair of adjacent frames and the translation amount after optimization between each pair of adjacent frames. The integrated translation amount between each pair of adjacent frames is transformed into the camera coordinate system to obtain the direction vector of each pair of adjacent frames in the camera coordinate system. Calculate the angle between the direction vector of each two adjacent frames in the camera coordinate system and each preset direction vector. Based on the angle between each two adjacent frames, determine the preset direction vector into which the direction vector of each two adjacent frames in the camera coordinate system falls.

8. The method according to claim 7, characterized in that, A direction vector sequence is formed by the direction vectors of every two adjacent frames in the camera coordinate system; the step of determining the true principal direction corresponding to each sub-direction sequence based on the preset direction vectors of every two adjacent frames in the camera coordinate system includes: Based on the timing sequence in the direction vector sequence, the direction vector sequence is segmented to obtain several sub-direction sequences; Based on the preset direction vector into which the direction vectors of each two adjacent frames fall in the camera coordinate system, determine the number of times each sub-direction sequence falls in each preset direction vector, and take the preset direction vector into which each sub-direction sequence falls the most times as the main direction corresponding to each sub-direction sequence. If the total number of times the principal direction corresponding to each sub-direction sequence falls within a preset number of times, then the principal direction corresponding to each sub-direction sequence is taken as the real principal direction corresponding to each sub-direction sequence.

9. The method according to claim 6, characterized in that, Determining the forward direction of the virtual camera at the corresponding moment of each original video frame based on the real principal direction corresponding to each sub-direction sequence includes: Smoothing and interpolation are performed on the real principal direction corresponding to each sub-direction sequence to obtain the corrected direction vector at the corresponding moment of each original video frame. Transform the corrected direction vector of each original video frame at the corresponding time into the world coordinate system to obtain the direction vector of each original video frame at the corresponding time. The direction vector at the corresponding moment of each original video frame is used as the forward direction of the virtual camera at the corresponding moment of each original video frame.

10. A panoramic video transition rendering device, characterized in that, The device includes: The acquisition module is used to acquire a set of image groups, wherein each image group in the set is obtained by extracting frames from the original video, and each original video frame in the original video is synthesized from multiple fisheye images. The first determining module is used to determine a plurality of occlusion region masks based on at least one image group in the image group set, and to determine the unoccluded image region of each frame in the image group set based on the plurality of occlusion region masks. The second determining module is used to perform feature matching on the unoccluded image regions of each two adjacent frames of at least one image group in the image group set, to obtain matching point pairs between the unoccluded image regions of each two adjacent frames. The third determining module is used to optimize the translation amount between each pair of adjacent frames based on the matching point pairs between the unoccluded image regions of each pair of adjacent frames, and to determine the forward direction of the virtual camera at the corresponding moment of each original video frame based on the optimized translation amount between each pair of adjacent frames; wherein, the translation amount of the virtual camera at the corresponding moment of each original video frame is obtained by interpolation processing through the translation amount between each pair of adjacent frames, and is used as the forward direction of the virtual camera at the corresponding moment of each original video frame; The calculation module is used to calculate the rotation matrix of the virtual camera at each corresponding moment of each original video frame, based on the forward direction of the virtual camera at each corresponding moment of each original video frame. The rendering module is used to perform transition rendering on each original video frame based on the rotation matrix of the virtual camera at the corresponding moment of each original video frame and the rotation amount of the panoramic camera relative to the world coordinate system when capturing each original video frame.

11. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 9.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 9.

Images