Panoramic video three-dimensional point cloud sparse reconstruction method and device integrating key frame screening

Abstract

The invention provides a panoramic video three-dimensional point cloud sparse reconstruction method and device integrated with key frame screening, and the method comprises the steps: obtaining a panoramic video of a target scene, and carrying out the video quality evaluation of the panoramic video, and obtaining a video evaluation result; determining a target resolution of downsampling according to the video evaluation result; carrying out downsampling processing on the panoramic video based on the target resolution to obtain a downsampled second video; for the second video, dynamically adjusting a frame extraction interval based on a preset key frame selection mechanism, and extracting a frame index of the key frame from the second video according to the frame extraction interval to obtain a plurality of frame indexes; and obtaining a plurality of corresponding key frames in the panoramic video based on the plurality of frame indexes, and obtaining the three-dimensional scene sparse point cloud of the target scene based on the plurality of key frames, thereby solving the technical problem of low processing efficiency in the prior art.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and in particular to a method and device for sparse reconstruction of three-dimensional point clouds of panoramic videos with integrated key frame screening. Background Art

[0002] 3D reconstruction has been widely used in virtual reality, augmented reality, and scene restoration. However, existing technologies still face challenges in sparse reconstruction of 3D scenes using panoramic videos.

[0003] Since panoramic cameras can capture 25 to 30 frames per second, and a video takes at least 1 to 2 minutes to shoot, a panoramic video contains a lot of frame data. Reconstructing a three-dimensional point cloud of a scene based on a panoramic video requires a high level of computation and results in low reconstruction efficiency. Furthermore, traditional methods extract frames from the video at preset intervals. This fixed frame extraction interval may ignore the effects of dynamic changes in the scene and may miss key details. In static or slowly changing scenes, this unnecessarily increases the amount of data and computational burden. Furthermore, existing methods only perform point cloud reconstruction and are unable to trace the video capture trajectory or view the images corresponding to each capture position based on the trajectory. Summary of the Invention

[0004] The present application aims to solve one of the technical problems in the related art at least to a certain extent.

[0005] To this end, the first purpose of this application is to propose a sparse reconstruction method for panoramic video three-dimensional point cloud with integrated key frame screening to improve reconstruction efficiency.

[0006] The second purpose of this application is to propose a sparse reconstruction device for panoramic video three-dimensional point cloud with integrated key frame screening.

[0007] The third objective of this application is to provide an electronic device.

[0008] The fourth object of this application is to provide a computer-readable storage medium.

[0009] A fifth object of this application is to provide a computer program product.

[0010] To achieve the above objectives, the first embodiment of the present application proposes a method for sparse reconstruction of 3D point clouds from panoramic videos with integrated keyframe screening, comprising:

[0011] Acquire a panoramic video of the target scene, and perform video quality assessment on the panoramic video to obtain a video assessment result;

[0012] Determining a downsampling target resolution based on the video evaluation result; and downsampling the panoramic video based on the target resolution to obtain a downsampled second video;

[0013] For the second video, dynamically adjusting a frame extraction interval based on a preset key frame selection mechanism, and extracting frame indexes of key frames from the second video according to the frame extraction interval to obtain multiple frame indexes;

[0014] A plurality of corresponding key frames in the panoramic video are acquired based on the plurality of frame indexes, and a three-dimensional scene sparse point cloud of the target scene is obtained based on the plurality of key frames.

[0015] To achieve the above objectives, the second embodiment of the present application proposes a sparse reconstruction device for panoramic video 3D point cloud with integrated key frame screening, comprising:

[0016] A video evaluation module is used to obtain a panoramic video of the target scene and perform video quality evaluation on the panoramic video to obtain a video evaluation result;

[0017] a video processing module, configured to determine a target resolution for downsampling based on the video evaluation result; and perform downsampling processing on the panoramic video based on the target resolution to obtain a downsampled second video;

[0018] a video frame extraction module, configured to dynamically adjust a frame extraction interval for the second video based on a preset key frame selection mechanism, and extract frame indexes of key frames from the second video according to the frame extraction interval to obtain a plurality of frame indexes;

[0019] A point cloud reconstruction module is used to obtain a plurality of corresponding key frames in the panoramic video based on the plurality of frame indexes, and obtain a three-dimensional scene sparse point cloud of the target scene based on the plurality of key frames.

[0020] To achieve the above-mentioned purpose, the third aspect embodiment of the present application proposes an electronic device, comprising: a processor, and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the method described in the first aspect.

[0021] To achieve the above-mentioned purpose, the fourth embodiment of the present application proposes a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the method described in the first aspect.

[0022] To achieve the above-mentioned purpose, the fifth embodiment of the present application proposes a computer program product, including a computer program, which implements the method described in the first aspect when executed by a processor.

[0023] The present application provides a method, device, electronic device and storage medium for sparse reconstruction of three-dimensional point clouds from panoramic videos with integrated key frame screening. A key frame selection mechanism based on downsampling is introduced in the preprocessing of panoramic videos to achieve dynamic adjustment of the frame extraction interval, which can more effectively capture important information. It reduces the processing of redundant frames, improves computational efficiency, and ensures that important details are not lost, thereby improving the efficiency of point cloud reconstruction and significantly optimizing the subsequent three-dimensional sparse reconstruction process. At the same time, by selectively selecting key frames, the data used in the subsequent three-dimensional point cloud reconstruction process is made more accurate, thereby improving the quality and robustness of the three-dimensional point cloud. The present invention not only improves the efficiency of data processing, but also enhances the adaptability of the three-dimensional point cloud model to dynamic scenes and its ability to capture detailed information, which can significantly improve the accuracy and integrity of the reconstruction results.

[0024] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0026] Figure 1 A flowchart of a method for sparse reconstruction of 3D point clouds from panoramic videos with integrated keyframe screening provided in an embodiment of the present application;

[0027] Figure 2 This is an example diagram of the three-dimensional point cloud and camera trajectory provided in the embodiments of the present application.

[0028] Figure 3 A block diagram of a sparse reconstruction device for panoramic video 3D point cloud with integrated key frame screening provided in an embodiment of the present application;

[0029] Figure 4 A block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] The following describes in detail embodiments of the present application, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0031] The following describes the method, device and equipment for sparse reconstruction of panoramic video three-dimensional point cloud with integrated key frame screening according to the embodiments of the present application with reference to the accompanying drawings.

[0032] Figure 1 A flowchart of a method for sparse reconstruction of three-dimensional point clouds from panoramic videos with integrated key frame screening provided in an embodiment of the present application.

[0033] It should be noted that the executor of the method for sparse reconstruction of panoramic video three-dimensional point cloud with integrated key frame filtering in the embodiment of the present application is the sparse reconstruction device for panoramic video three-dimensional point cloud with integrated key frame filtering in the embodiment of the present application. The sparse reconstruction device for panoramic video three-dimensional point cloud with integrated key frame filtering can be configured in an electronic device so that the electronic device can perform the sparse reconstruction function of panoramic video three-dimensional point cloud with integrated key frame filtering.

[0034] like Figure 1 As shown in FIG, the sparse reconstruction method of panoramic video 3D point cloud with integrated key frame screening includes the following steps:

[0035] Step S101: Obtain a panoramic video of a target scene, and perform video quality evaluation on the panoramic video to obtain a video evaluation result.

[0036] As an implementation method, a video quality assessment is performed on a panoramic video to obtain a video assessment result; this includes: obtaining the brightness, contrast, noise level, color balance and edge clarity of the panoramic video; and performing weighted summation on the brightness, contrast, noise level, color balance and edge clarity to obtain a video assessment result.

[0037] As an implementation method, the video quality of the panoramic video of the target scene is evaluated by the following video quality evaluation function, which is expressed as follows:

[0038] Q=w1×B+w2×C+w3×N1+w4×CB+w5×E

[0039] Among them, Q is the comprehensive quality score, ranging from 0 to 1, where 0 represents the lowest quality and 1 represents the highest quality; B is brightness, I i is the brightness value of the pixel in the LAB color space, and N is the total number of pixels;

[0040] C is the contrast, L i is the value of the brightness channel in the LAB color space, is the average value of the brightness channel;

[0041] N1 is the noise level, is the Laplace transform of the grayscale image, Var represents the variance;

[0042] CB is color balance, R, G, and B are the average values of the red, green, and blue channels, respectively;

[0043] E is edge clarity, δ j Indicates the edge strength in the edge detection result, M is the total number of edge pixels;

[0044] w1, w2, w3, w4, w5 are the weights of each indicator, satisfying: w1+w2+w3+w4+w5=1.

[0045] Step S102 : determining a downsampling target resolution based on the video evaluation result; and performing downsampling processing on the panoramic video based on the target resolution to obtain a downsampled second video.

[0046] As an implementation method, a method for determining a target resolution for downsampling based on a video evaluation result includes: obtaining a maximum allowable resolution and a minimum allowable resolution, and calculating a difference between the maximum allowable resolution and the minimum allowable resolution;

[0047] Calculate the product of the video evaluation result and the difference; sum the product and the minimum allowed resolution as the target resolution for downsampling.

[0048] In this step, in order to determine the target resolution of downsampling, a linear mapping function is defined as follows:

[0049] R(Q)=R max -Q×(R max -R min )

[0050] Among them, R(Q) is the target resolution calculated according to the quality score Q, R min and R max They are the minimum and maximum allowed resolutions, respectively, and are used to indicate the upper and lower limits of resolution adjustment.

[0051] This step uses downsampling, that is, reducing the resolution of the original panoramic video to reduce the computational burden, so as to be used for the rapid evaluation of inter-frame changes in the subsequent steps.

[0052] Step S103 : for the second video, dynamically adjust the frame extraction interval based on a preset key frame selection mechanism, and extract frame indexes of key frames from the second video according to the frame extraction interval to obtain multiple frame indexes.

[0053] In this step, the frame interval is determined by downsampling the video, thereby obtaining the frame indexes of multiple key frames so that point cloud reconstruction can be performed based on the key frames.

[0054] As an implementation method, for the second video, the frame extraction interval is dynamically adjusted based on a preset key frame selection mechanism, and the frame index of the key frame is extracted from the second video according to the frame extraction interval to obtain multiple frame indexes; including:

[0055] The following steps are executed in a loop until the second video frame extraction is completed;

[0056] Based on the frame index of the current frame image, obtaining the next frame image of the current frame image from the second video;

[0057] Obtaining an inter-frame change parameter between a current frame image and a next frame image and a visual perspective influence parameter of the next frame image;

[0058] Determine the current frame extraction interval based on the inter-frame change parameter and the visual perspective influence parameter;

[0059] Comparing the determined current frame extraction interval with the frame extraction interval lower limit and the frame extraction interval upper limit, and determining a final current frame extraction interval according to the comparison results;

[0060] Obtain the frame index of the target frame image that is at the current frame extraction interval from the current frame image; wherein, the frame index of the target frame image is the frame index of the current frame image in the next cycle, and the current frame image of the first cycle is determined from the second video according to a preset first frame extraction rule, and the preset first frame extraction rule is to select the first frame of the video.

[0061] In some embodiments, the inter-frame change parameter is the square of the gradient of the frame content change, and the visual perspective impact parameter is the average visual perspective impact.

[0062] In some embodiments, determining a current frame extraction interval based on an inter-frame variation parameter and a visual perspective impact parameter includes:

[0063] The current frame extraction interval is determined based on the inter-frame change parameter and the visual perspective influence parameter using the following formula, which is expressed as follows:

[0064]

[0065] Among them, T is the frame extraction interval, T base As the basic frame interval, it serves as a reference standard; is the square of the gradient of the frame content change, which quantifies the change amplitude of the image; θ avg is the average visual perspective effect, which is used to evaluate the complexity of the scene from the current perspective; α and β are adjustment factors used to balance the nonlinear effect of scene changes on the frame interval; κ is the coefficient used to adjust the impact of perspective changes, and ε is a small constant to avoid division by zero errors.

[0066] Among them, the visual perspective influence parameter is obtained by the following formula:

[0067]

[0068] Where num_lines is the number of significant line segments detected from the Hough transform, texture_variation is a measure of the image texture variation, which can be calculated using the Local Binary Pattern (LBP) method. γ1 and γ2 are weight coefficients used to balance the contribution of each part to the perspective effect. max_lines and max_texture are the maximum possible values of each feature. max_lines is the maximum number of detectable line segments in the image, which is pre-set according to the image resolution. max_texture is the maximum value of the texture variation calculated by the LBP method and is used for normalization.

[0069] The squared gradient of the frame content change in the embodiment of the present application is an indicator for quantifying the magnitude of image change, and is particularly used to measure the difference between two frames. When obtaining the squared gradient of the frame content change, the Sobel operator can be used to calculate the gradient of each frame, and the sum of the squared gradients of the current frame image and the next frame image can be compared to obtain the difference between the sums of the squared gradients of adjacent frames, i.e., the squared gradient of the frame content change. The larger the value, the greater the change between frames and the smaller the frame extraction interval; conversely, the smaller the data, the larger the frame extraction interval. The squared gradient of the frame content change is used to measure the magnitude of the content change between two frames and can be used to reflect the movement of the camera. If the camera movement causes the scene to change, this value will increase.

[0070] The processing of the Sobel operator includes:

[0071] 1. Image preprocessing: Convert the image into grayscale to simplify calculation.

[0072] 2. Convolution operation: Use Gx and Gy convolution kernels to convolve with the image respectively to obtain the horizontal and vertical gradient components.

[0073] 3. Calculate the gradient magnitude and direction: Calculate the gradient magnitude and direction based on Gx and Gy.

[0074] 4. Threshold processing: Set a threshold based on the gradient magnitude, and mark pixels with a gradient magnitude greater than the threshold as edges.

[0075] Exemplarily, in the first loop, the first frame of the panoramic video is used as the current frame image, the second frame is obtained from the second video, the inter-frame change parameters between the second frame and the first frame and the visual perspective influence parameters of the second frame are calculated, and the current frame extraction frequency is obtained according to the above formula to determine which subsequent frame to extract; and so on, to achieve dynamic frame extraction of the panoramic video.

[0076] In order to ensure that the frame extraction interval is not too small or too large, this solution also introduces upper and lower limit constraints on the frame extraction interval T, setting the lower limit value of the frame extraction interval T min and the upper limit of the frame extraction interval T max , that is, Tmin ≤T≤T max When the current frame interval is less than T min When the final current frame interval is determined to be T min ; When the current frame interval is greater than T max When the final current frame interval is determined to be T max .

[0077] For example, the basic frame interval T base Set to 1 / 5 of the video frame rate. In order to ensure that rapidly changing scenes can be captured, the lower limit of the frame interval T min Set to half of the basic frame extraction interval; in order to reduce redundant frames, the upper limit of the frame extraction interval T max Set to twice the basic frame interval or greater.

[0078] In some embodiments, before determining the final frame extraction interval, the method further includes:

[0079] The square of the gradient of the acquired frame content change comparing with a first preset change threshold and a second preset change threshold respectively;

[0080] If the square of the gradient of the frame content change is less than or equal to a first preset change threshold, adjusting the lower limit value and the upper limit value of the frame extraction interval based on the first prediction adjustment rule;

[0081] If the square of the gradient of the frame content change is greater than or equal to a second preset change threshold, the lower limit value and the upper limit value of the frame extraction interval are adjusted based on a second prediction adjustment rule.

[0082] Therefore, this scheme introduces an inter-frame change threshold judgment before calculating the frame extraction interval. If the square of the gradient of the frame content change is less than or equal to the first preset change threshold, the current scene is considered to be static, and the constraint value of the frame extraction interval can be appropriately increased to reduce the extraction of redundant frames; if the square of the gradient of the frame content change is greater than or equal to the second preset change threshold, the current scene is considered to be dynamic, and the constraint value of the frame extraction interval can be appropriately reduced to ensure that the key frame is captured.

[0083] As an example, the first prediction adjustment rule is: when the current scene is considered to be static, the lower limit value and the upper limit value of the frame extraction interval are adjusted by the following formula:

[0084] T min =max(T min ×k min ,1)

[0085] T max =min(T max ×k max ,Tmax _ limit )

[0086] Among them, k min and k max It is a coefficient greater than 1 and is used to adjust the lower limit and upper limit of the frame extraction interval.

[0087] As an example, the second prediction adjustment rule is: when the current scene is considered to be dynamic, the lower limit value and the upper limit value of the frame extraction interval are adjusted by the following formula:

[0088] T min =max(T min / k min ,1)

[0089] T max =max(T max / k max ,T min )

[0090] Also, to ensure T min and T max Within a reasonable range, set the following boundary conditions:

[0091] T min ≥T min _li m i t

[0092] T max ≥T min

[0093] T max ≤T max_limit

[0094] Among them, T min _ limit is the lower limit constraint value of the frame extraction interval, T max _ limit is the upper limit constraint value of the frame extraction interval. In the dynamic adjustment mechanism of the frame extraction interval, T min _ limit and T max _ limit It is a hard constraint on the frame extraction interval to ensure that the frame extraction interval T is within a reasonable range. min _ limit and T max _ limit The value range of T is coordinated with the parameters in the frame extraction formula. min _ limit It is used to ensure that the frame extraction interval is not too small, usually set to 1 frame; T max _ limitUsed to ensure that the frame extraction interval is not too large, usually set to 2 to 3 times the basic frame extraction interval.

[0095] As an example, k min and k max The value of is usually set between 1.2 and 2, and the specific value can be adjusted according to the dynamic characteristics of the video and computing resources. min and k max It will more actively increase the frame extraction interval and reduce redundant frames; smaller k min and k max The frame rate will be adjusted more conservatively.

[0096] The embodiment of the present application uses frame indexing, that is, using integers to represent the positions of key frames in a video sequence, to implement the results obtained from low-resolution analysis through frame indexing, and is used to mark the positions of key frames in the original high-resolution video.

[0097] Step S104: Acquire multiple key frames corresponding to the panoramic video based on the multiple frame indexes, and obtain a three-dimensional scene sparse point cloud of the target scene based on the multiple key frames.

[0098] In some embodiments, a plurality of key frames corresponding to the panoramic video are acquired based on the plurality of frame indexes, and a 3D scene sparse point cloud of the target scene is obtained based on the plurality of key frames; including:

[0099] Based on multiple frame indexes, multiple key frames corresponding to the panoramic video are obtained, and the SFM method is used to obtain the three-dimensional scene sparse point cloud of the target scene based on the multiple key frames.

[0100] The point cloud reconstruction effect of the dynamic frame extraction based on downsampling of the present invention is compared with the point cloud reconstruction effect of the traditional fixed frame extraction, and the effect comparison table 1 is obtained.

[0101] Table 1: Effect comparison table

[0102]

[0103] In some embodiments, a plurality of key frames corresponding to the panoramic video are acquired based on the plurality of frame indexes, and a 3D scene sparse point cloud of the target scene is obtained based on the plurality of key frames; including:

[0104] Obtain each frame index from the multiple frame indexes one by one in sequence as the current frame index, and loop through the following steps until the number of loops is equal to the number of the multiple frame indexes, thereby obtaining a sparse point cloud of the three-dimensional scene of the target scene;

[0105] Get the current key frame image corresponding to the current frame index from the panoramic video;

[0106] Based on the current key frame image, obtain the first point cloud;

[0107] Based on the first point cloud, first camera position information corresponding to the current key frame image is obtained; and the first camera position information is added to the camera trajectory information obtained in the previous cycle to obtain the camera trajectory information of the current cycle;

[0108] Based on the feature matching fusion algorithm, the first point cloud is fused with the fused point cloud obtained in the previous cycle to obtain the fused point cloud of the current cycle;

[0109] Among them, the fused point cloud obtained in the previous cycle of the first cycle is empty, and the camera trajectory information obtained in the previous cycle of the first cycle is empty.

[0110] Therefore, according to the frame extraction results, key frames are obtained one by one, point cloud is generated for each key frame, and the camera position is obtained based on the generated point cloud. Finally, the fused point cloud corresponding to multiple key frames is the three-dimensional scene sparse point cloud of the target scene, and the multiple camera position information constitutes the camera trajectory information.

[0111] It should be noted that obtaining point clouds from panoramic images is a state-of-the-art technique and will not be elaborated on here. For example, the SFM method, including feature point extraction, feature point matching, eigenmatrix, triangulation, and bundle adjustment algorithms, can be used. Based on the point cloud data and matched feature point pairs, the perspective N-point method (PNP) can be used to determine the camera pose. The PNP problem is relatively simple to describe and define; its purpose is to solve the problem of 3D-2D point pair motion. Simply put, given the coordinates of n 3D points (relative to a specified coordinate system A) and their 2D projections, how can the camera pose (i.e., the camera's orientation in coordinate system A) be estimated? For example, if the relative coordinates of at least four points in an image are known in 3D space, the camera pose relative to these points, or the pose of these 3D points in the camera's coordinate system, can be estimated. The pose or posture includes both position and orientation, representing a 6-degree-of-freedom state. The camera pose can be used to determine the angular transformation between two adjacent camera poses, and the camera's direction of movement, i.e., the camera heading, can be derived from this angular transformation.

[0112] In some embodiments, the camera trajectory information includes camera position information and camera orientation, and the fused point cloud of the current cycle is obtained. Thereafter, it also includes: adjusting the camera orientation in the camera trajectory information of the current cycle according to the fused point cloud of the current cycle.

[0113] It can be understood that the fused point cloud can obtain the angle of the camera's movement in the camera trajectory. For example, if a person holding a camera is moving to the right front, the camera is moving to the right. If there is only the camera position, it can be said that the camera is facing forward in both positions, but based on the fused point cloud, it can be determined that the camera is moving to the right. For example, after fusing the point cloud obtained by the first keyframe and the second keyframe, the relative movement direction of the camera corresponding to the two keyframes can be obtained; then, after fusing the point cloud of the third keyframe, the relative movement direction of the three camera positions between the first keyframe and the third keyframe can be obtained, so that the relative movement direction between the previous two keyframes can be adjusted, for example, the relative movement angle of the second keyframe relative to the first keyframe can be adjusted from 60 degrees to 45 degrees.

[0114] In some embodiments, the 3D scene sparse point cloud and camera trajectory of the target scene are displayed in a visual manner, such as Figure 2 As shown, Figure 2 The small rectangle in the middle is the camera position. Clicking each camera position in the camera track will pop up the corresponding panoramic image (i.e. key frame), that is, viewing the image through the panoramic sensor according to the frame index.

[0115] By implementing this embodiment, a keyframe selection mechanism is introduced during panoramic video preprocessing, dynamically adjusting the frame extraction interval based on inter-frame variation parameters and visual perspective influence parameters. This allows for more effective capture of important information, reduces redundant frame processing, improves computational efficiency, and ensures that important details are not lost, thereby enhancing the efficiency of point cloud reconstruction and significantly optimizing the subsequent 3D sparse reconstruction process. The present invention takes perspective changes into account, effectively reducing information loss caused by varying viewing angles. Furthermore, through targeted keyframe selection, the data used in the subsequent 3D point cloud reconstruction process is more accurate, improving the quality and robustness of the 3D point cloud. This not only improves data processing efficiency but also enhances the 3D point cloud model's adaptability to dynamic scenes and its ability to capture detailed information, significantly improving the accuracy and completeness of the reconstruction results. The present invention also simultaneously captures the camera trajectory while reconstructing the 3D point cloud model, visually displaying panoramic images of each position along the motion trajectory. Based on the camera trajectory, the camera's movement trajectory can be restored, facilitating understanding of the target scene. For example, in a fire scene, 3D scene information and the firefighter's movement trajectory can be obtained from a panoramic video of the fire captured by a firefighter, providing a basis for determining firefighting strategies.

[0116] In order to implement the above embodiments, the present application also proposes a panoramic video three-dimensional point cloud sparse reconstruction device integrated with key frame screening. Figure 3 This is a schematic diagram of the structure of a sparse reconstruction device for panoramic video 3D point cloud with integrated key frame screening provided in an embodiment of the present application. Figure 3As shown, the panoramic video 3D point cloud sparse reconstruction device with integrated key frame screening may include: a video evaluation module 301 , a video processing module 302 , a video frame extraction module 303 and a point cloud reconstruction module 304 .

[0117] The video evaluation module 301 is used to obtain a panoramic video of a target scene and perform video quality evaluation on the panoramic video to obtain a video evaluation result;

[0118] The video processing module 302 is configured to determine a target resolution for downsampling based on the video evaluation result; and downsample the panoramic video based on the target resolution to obtain a downsampled second video;

[0119] The video frame extraction module 303 is configured to dynamically adjust the frame extraction interval for the second video based on a preset key frame selection mechanism, and extract the frame index of the key frame from the second video according to the frame extraction interval to obtain multiple frame indexes;

[0120] The point cloud reconstruction module 304 is configured to obtain a plurality of corresponding key frames in the panoramic video based on the plurality of frame indexes, and obtain a 3D scene sparse point cloud of the target scene based on the plurality of key frames.

[0121] Furthermore, in a possible implementation of the embodiment of the present application, the video frame extraction module 303 is specifically configured to:

[0122] The following steps are executed in a loop until the second video frame extraction is completed;

[0123] Based on the frame index of the current frame image, obtaining the next frame image of the current frame image from the second video;

[0124] Obtaining an inter-frame change parameter between a current frame image and a next frame image and a visual perspective influence parameter of the next frame image;

[0125] Determine the current frame extraction interval based on the inter-frame change parameter and the visual perspective influence parameter;

[0126] Comparing the determined current frame extraction interval with the frame extraction interval lower limit and the frame extraction interval upper limit, and determining a final current frame extraction interval according to the comparison results;

[0127] Get the frame index of the target frame image that is at the current frame extraction interval from the current frame image; wherein, the frame index of the target frame image is the frame index of the current frame image in the next loop, that is, the target frame image obtained in this loop is the current frame image of the next loop; the current frame image of the first loop is determined from the second video according to the preset first frame extraction rule, and the preset first frame extraction rule is to select the first frame of the video.

[0128] Furthermore, in a possible implementation of the embodiment of the present application, the inter-frame change parameter is the square of the gradient of the frame content change, and the visual perspective influence parameter is the average visual perspective influence; when the video frame extraction module 303 determines the current frame extraction interval based on the inter-frame change parameter and the visual perspective influence parameter, it is configured to:

[0129] The current frame extraction interval is determined based on the inter-frame variation parameter and the visual perspective influence parameter through the first formula. The first formula is expressed as follows:

[0130]

[0131] Among them, T is the frame extraction interval, T base is the basic frame interval, is the square of the gradient of the frame content change, θ avg is the average visual perspective effect, α and β are adjustment factors, κ is the coefficient used to adjust the impact of perspective change, and ε is a small constant to avoid division by zero errors.

[0132] Furthermore, in a possible implementation of the embodiment of the present application, when obtaining the visual perspective influence parameter of the next frame image, the video frame extraction module 303 is configured to:

[0133] The visual perspective influence parameter of the next frame image is obtained by the second formula, which is expressed as follows:

[0134]

[0135] Where num_lines is the number of significant line segments detected from the Hough transform, texture_variation is a measure of the image texture variation, which can be calculated using the Local Binary Pattern (LBP) method. γ1 and γ2 are weight coefficients used to balance the contribution of each part to the perspective effect. max_lines and max_texture are the maximum possible values of each feature. max_lines is the maximum number of detectable line segments in the image, which is pre-set according to the image resolution. max_texture is the maximum value of the texture variation calculated by the LBP method and is used for normalization.

[0136] Furthermore, in a possible implementation of the embodiment of the present application, when the video evaluation module 301 performs video quality evaluation on the panoramic video and obtains the video evaluation result, it is configured to:

[0137] Obtain the brightness, contrast, noise level, color balance, and edge clarity of panoramic videos;

[0138] The video evaluation result is obtained by weighted summing of brightness, contrast, noise level, color balance and edge clarity.

[0139] Furthermore, in a possible implementation of the embodiment of the present application, when determining the target resolution for downsampling based on the video evaluation result, the video processing module 302 is configured to:

[0140] Obtaining the highest allowed resolution and the lowest allowed resolution, and calculating the difference between the highest allowed resolution and the lowest allowed resolution;

[0141] Calculate the product of the video evaluation result and the difference;

[0142] The sum of the product and the minimum allowed resolution is used as the downsampling target resolution.

[0143] Furthermore, in a possible implementation of the embodiment of the present application, a plurality of key frames corresponding to the panoramic video are obtained based on a plurality of frame indexes, and a 3D scene sparse point cloud of the target scene is obtained based on the plurality of key frames; including:

[0144] Obtain each frame index from the multiple frame indexes one by one in sequence as the current frame index, and loop through the following steps until the number of loops is equal to the number of the multiple frame indexes, thereby obtaining a sparse point cloud of the three-dimensional scene of the target scene;

[0145] Get the current key frame image corresponding to the current frame index from the panoramic video;

[0146] Based on the current key frame image, obtain the first point cloud;

[0147] Based on the first point cloud, first camera position information corresponding to the current key frame image is obtained; and the first camera position information is added to the camera trajectory information obtained in the previous cycle to obtain the camera trajectory information of the current cycle;

[0148] Based on the feature matching fusion algorithm, the first point cloud is fused with the fused point cloud obtained in the previous cycle to obtain the fused point cloud of the current cycle;

[0149] Among them, the fused point cloud obtained in the previous cycle of the first cycle is empty, and the camera trajectory information obtained in the previous cycle of the first cycle is empty.

[0150] Furthermore, in a possible implementation of the embodiment of the present application, the camera trajectory information includes camera position information and camera orientation. After obtaining the fused point cloud of the current cycle, the point cloud reconstruction module 304 is further configured to:

[0151] Adjust the camera orientation in the camera trajectory information of the current loop according to the fused point cloud of the current loop.

[0152] It should be noted that the above explanation of the embodiment of the panoramic video 3D point cloud sparse reconstruction method with integrated key frame screening is also applicable to the panoramic video 3D point cloud sparse reconstruction device with integrated key frame screening of this embodiment, and will not be repeated here.

[0153] In order to implement the above embodiment, the present application also proposes an electronic device. Figure 4 , Figure 4 Schematic diagram of the structure of the electronic device provided in the embodiment of the present application. Figure 4 As shown, the electronic device 400 includes: a processor 401, and a memory 402 communicatively connected to the processor 401; the memory 402 stores computer-executable instructions; the processor 401 executes the computer-executable instructions stored in the memory to implement the method provided in the aforementioned embodiment.

[0154] In order to implement the above embodiments, the present application also proposes a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the methods provided by the above embodiments.

[0155] In order to implement the above embodiments, the present application also proposes a computer program product, including a computer program, which implements the methods provided by the above embodiments when executed by a processor.

[0156] In the descriptions of the foregoing embodiments, the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are mutually inconsistent.

[0157] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of such features. Throughout the description of this application, "plurality" means at least two, for example, two, three, etc., unless otherwise specifically defined.

[0158] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present application belong.

[0159] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and a portable compact disc read-only memory (CDROM). Furthermore, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing it in another suitable manner if necessary, and then storing it in a computer memory.

[0160] It should be understood that various parts of the present application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used to implement: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0161] Those skilled in the art will understand that all or part of the steps in the method of the above embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0162] In addition, the functional units in the various embodiments of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into a module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0163] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A sparse reconstruction method for 3D point cloud from panoramic video with integrated keyframe screening, characterized in that: The following steps are involved: Acquire a panoramic video of the target scene, and perform video quality assessment on the panoramic video to obtain a video assessment result; Determining a target resolution for downsampling based on the video evaluation result; and performing downsampling processing on the panoramic video based on the target resolution to obtain a downsampled second video; For the second video, dynamically adjusting a frame extraction interval based on a preset key frame selection mechanism, and extracting frame indexes of key frames from the second video according to the frame extraction interval to obtain multiple frame indexes; A plurality of corresponding key frames in the panoramic video are acquired based on the plurality of frame indexes, and a three-dimensional scene sparse point cloud of the target scene is obtained based on the plurality of key frames.

2. The method according to claim 1, characterized in that dynamically adjusting the frame extraction interval for the second video based on a preset key frame selection mechanism, and extracting frame indexes of key frames from the second video according to the frame extraction interval to obtain multiple frame indexes; include: Looping through the following steps until the second video frame extraction is complete; Based on the frame index of the current frame image, obtaining the next frame image of the current frame image from the second video; Acquire an inter-frame variation parameter between the current frame image and the next frame image and a visual perspective influence parameter of the next frame image; Determining a current frame extraction interval based on the inter-frame variation parameter and the visual perspective influence parameter; Comparing the determined current frame extraction interval with the frame extraction interval lower limit and the frame extraction interval upper limit, and determining a final current frame extraction interval according to the comparison results; Obtain the frame index of the target frame image that is at the current frame extraction interval from the current frame image; wherein the frame index of the target frame image is the frame index of the current frame image in the next cycle, and the current frame image of the first cycle is determined from the second video according to a preset first frame extraction rule.

3. The method according to claim 2, characterized in that The inter-frame variation parameter is a square of the gradient of the frame content variation, and the visual perspective influence parameter is an average visual perspective influence; and determining a current frame extraction interval based on the inter-frame variation parameter and the visual perspective influence parameter includes: The current frame extraction interval is determined based on the inter-frame variation parameter and the visual perspective influence parameter using a first formula, wherein the first formula is expressed as follows: Among them, T is the frame extraction interval, T base is the basic frame extraction interval, is the square of the gradient of the frame content change, θ avg is the average visual perspective effect, α and β are adjustment factors, κ is the coefficient used to adjust the impact of perspective change, and ε is a small constant to avoid division by zero errors.

4. The method according to claim 3, characterized in that Get the visual perspective impact parameters of the next frame of image, including: The visual perspective influence parameter of the next frame image is obtained by the second formula, which is expressed as follows: Where num_lines is the number of significant line segments detected from the Hough transform, texture_variation is a measure of the image texture variation, which can be calculated using the Local Binary Pattern (LBP) method. γ1 and γ2 are weight coefficients used to balance the contribution of each part to the perspective effect. max_lines and max_texture are the maximum possible values of each feature. max_lines is the maximum number of detectable line segments in the image, which is pre-set according to the image resolution. max_texture is the maximum value of the texture variation calculated by the LBP method and is used for normalization.

5. The method according to claim 1, wherein The performing video quality assessment on the panoramic video to obtain a video assessment result includes: Obtaining brightness, contrast, noise level, color balance, and edge clarity of the panoramic video; A weighted sum is performed on the brightness, contrast, noise level, color balance, and edge clarity to obtain a video evaluation result.

6. The method according to claim 1, wherein Determining a target resolution for downsampling based on the video evaluation result includes: Obtaining a maximum allowable resolution and a minimum allowable resolution, and calculating a difference between the maximum allowable resolution and the minimum allowable resolution; Calculating the product of the video evaluation result and the difference; The sum of the product and the minimum allowed resolution is used as the target resolution of downsampling.

7. The method according to claim 1, characterized in that The method of acquiring a plurality of corresponding key frames in the panoramic video based on the plurality of frame indexes, and obtaining a three-dimensional scene sparse point cloud of the target scene based on the plurality of key frames comprises: Sequentially acquiring each frame index from the plurality of frame indexes as a current frame index, and looping through the following steps until the number of loops is equal to the number of the plurality of frame indexes, thereby obtaining a three-dimensional scene sparse point cloud of the target scene; Obtaining a current key frame image corresponding to the current frame index from the panoramic video; Obtaining a first point cloud based on the current key frame image; Based on the first point cloud, first camera position information corresponding to the current key frame image is obtained; and the first camera position information is added to the camera trajectory information obtained in the previous cycle to obtain the camera trajectory information of the current cycle; Based on the feature matching fusion algorithm, the first point cloud is fused with the fused point cloud obtained in the previous cycle to obtain the fused point cloud of the current cycle; wherein, the fused point cloud obtained in the cycle before the first cycle is empty, and the camera trajectory information obtained in the cycle before the first cycle is empty.

8. The method according to claim 7, characterized in that The camera trajectory information includes camera position information and camera orientation, and the fused point cloud obtained in the current cycle is then further comprised of: Adjust the camera orientation in the camera trajectory information of the current cycle according to the fused point cloud of the current cycle.

9. A sparse reconstruction device for 3D point cloud of panoramic video with integrated key frame screening, characterized in that: include: A video evaluation module is used to obtain a panoramic video of the target scene and perform video quality evaluation on the panoramic video to obtain a video evaluation result; A video processing module, configured to determine a target resolution for downsampling based on the video evaluation result; and performing downsampling processing on the panoramic video based on the target resolution to obtain a downsampled second video; a video frame extraction module, configured to dynamically adjust a frame extraction interval for the second video based on a preset key frame selection mechanism, and extract frame indexes of key frames from the second video according to the frame extraction interval to obtain a plurality of frame indexes; A point cloud reconstruction module is used to obtain a plurality of corresponding key frames in the panoramic video based on the plurality of frame indexes, and obtain a three-dimensional scene sparse point cloud of the target scene based on the plurality of key frames.

10. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 8 when executed by a processor.

Images