Video generation method and device

By acquiring a set of sample images of the target scene, a neural radiation field model is generated, and incremental learning is performed by supplementing sample images. This solves the problem of insufficient viewpoint information in existing technologies and enables the efficient generation of high-quality panoramic videos.

CN117544829BActive Publication Date: 2024-11-08ALIPAY (HANGZHOU) INFORMATION TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202311338422.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-10-16
Publication Date
2024-11-08
Estimated Expiration
2043-10-16

AI Technical Summary

Technical Problem

Existing VR panoramic video generation technology requires professional photography teams to collect high-quality scene image data, and the generated panoramic video does not have enough perspective information and lacks consistency across multiple perspectives.

Method used

By acquiring a set of sample images of the target scene, a neural radiation field model is generated, and incremental learning is performed by supplementing sample images to optimize the viewpoint information and generate a high-quality panoramic video.

Benefits of technology

It enables the generation of high-quality panoramic videos with rich perspective information under low image data acquisition and training costs, thereby improving generation efficiency and quality.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117544829B_ABST
    Figure CN117544829B_ABST
Patent Text Reader

Abstract

The embodiments of this specification disclose a video generation method and device, wherein the video generation method collects supplementary sample images to perform incremental learning training on the neural radiation field model of the target scene, and continuously supplements the perspective information to learn and optimize the representation ability of the target scene.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of this specification belong to the field of computer technology, and particularly to a video generation method and device. Background Art

[0002] VR (Virtual Reality) panoramic immersive display solutions use advanced virtual reality technology to allow users to experience virtual scenes in person. Whether visiting places of interest, exploring natural scenery, participating in exhibitions, or experiencing products, users can get a real feeling in the virtual world, as if they were there in person. This immersive experience adds more fun and attraction to the display activities.

[0003] Panoramic videos of VR panoramic immersive display solutions usually require professional photography teams to shoot and collect high-quality scene image data in one go, and the collection process is complicated. In addition, additional training optimization and rendering trajectory selection are required. Large-scale AI painting tools can generate multi-frame panoramic videos with changing perspectives based on a single image and the description in this article. Since the training data does not contain multi-perspective image information and corresponding semantic annotations, the perspective information contained in the generated panoramic video is not rich enough, and there is multi-perspective consistency. Summary of the invention

[0004] The embodiments of this specification provide a video generation method, device, storage medium and electronic device, and the technical solutions thereof are as follows:

[0005] In a first aspect, an embodiment of the present specification provides a video generation method, including:

[0006] Acquire a sample image set of a target scene, wherein the sample image set includes a plurality of original sample images of the target scene;

[0007] Based on the sample image set, generating a browsing trajectory of the target scene and a neural radiation field model of the target scene;

[0008] Based on the browsing trajectory and the neural radiation field model, generating a rendered image of the target scene corresponding to the browsing trajectory;

[0009] Performing quality assessment on the rendered images to obtain quality assessment results of each rendered image; if the quality assessment result of the rendered image is lower than expected, obtaining a supplementary sample image corresponding to the rendered image; adding the supplementary sample image to the sample image set to obtain an updated sample image set; regenerating the neural radiation field model of the target scene based on the updated sample image set to obtain an updated neural radiation field model; regenerating the rendered image of the target scene corresponding to the browsing trajectory based on the browsing trajectory and the updated neural radiation field model; repeating this step until the quality assessment results of the rendered image of the target scene corresponding to the browsing trajectory are not lower than expected;

[0010] Based on the rendered image of the target scene corresponding to the browsing trajectory, a panoramic video of the target scene is generated.

[0011] In a second aspect, an embodiment of the present specification provides a video generation device, including:

[0012] An image acquisition module, which acquires a sample image set of a target scene, wherein the sample image set includes a plurality of original sample images of the target scene;

[0013] An image processing module, which generates a browsing trajectory of the target scene and a neural radiation field model of the target scene based on the sample image set;

[0014] A rendering module, which generates a rendered image of the target scene corresponding to the browsing trajectory based on the browsing trajectory and the neural radiation field model;

[0015] The quality assessment and incremental acquisition module performs quality assessment on the rendered images to obtain quality assessment results of each rendered image; if the quality assessment result of the rendered image is lower than expected, obtain a supplementary sample image corresponding to the rendered image; add the supplementary sample image to the sample image set to obtain an updated sample image set; regenerate the neural radiation field model of the target scene based on the updated sample image set to obtain an updated neural radiation field model; regenerate the rendered image of the target scene corresponding to the browsing trajectory based on the browsing trajectory and the updated neural radiation field model; repeat this step until the quality assessment results of the rendered image of the target scene corresponding to the browsing trajectory are not lower than expected;

[0016] The video generation module generates a panoramic video of the target scene based on a rendered image of the target scene corresponding to the browsing track.

[0017] In a third aspect, an embodiment of this specification provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method described in the first aspect above are implemented.

[0018] In a fourth aspect, an embodiment of the present specification provides an electronic device, including:

[0019] one or more processors, and

[0020] A memory associated with the one or more processors, the memory being used to store program instructions, wherein when the program instructions are read and executed by the one or more processors, the steps of the method described in the first aspect are executed.

[0021] The beneficial effects brought about by the technical solutions provided by one or more embodiments of this specification include at least:

[0022] By collecting supplementary sample images, the neural radiation field model of the target scene is incrementally trained, and the perspective information is continuously supplemented to learn and optimize the representation ability of the target scene. Based on lower image data collection and training costs, higher-quality panoramic videos are generated. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of this specification, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0024] Figure 1 A flowchart of a video generation method provided in an embodiment of this specification.

[0025] Figure 2 for Figure 1 Schematic diagram of the process of step 102 in FIG.

[0026] Figure 3 A schematic diagram of an exemplary application scenario of a video generation method provided in an embodiment of this specification.

[0027] Figure 4 A schematic diagram of the structure of a video generating device provided in an embodiment of this specification.

[0028] Figure 5 A schematic diagram of the architecture of an exemplary electronic device. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in the embodiments of this specification will be clearly and completely described below in combination with the embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0030] The terms "including" and "having" and any variations thereof in this specification and claims and the above-mentioned drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices. Depending on the context, the word "if" as applied herein can be interpreted as "at..." or "when..." or "in response to determining" or "in response to detecting".

[0031] The synthesis of traditional panoramic videos usually requires a one-time acquisition of high-quality scene image data and additional training optimization and rendering trajectory selection. The panoramic videos generated by large-scale AI painting tools based on single images and the description in this article contain insufficient perspective information, insufficient consistency between multiple perspectives, and other technical problems. The embodiment of the present application manual proposes a video generation method that collects supplementary sample images to perform incremental learning training on a neural radiation field model of a target scene. By collecting supplementary sample images to supplement the perspective information, the method learns and optimizes the representation capability of the target scene, thereby achieving higher quality panoramic videos at lower image data acquisition and training costs.

[0032] The technical solutions provided by the embodiments of this specification are described in detail below in conjunction with the accompanying drawings.

[0033] Figure 1 This is a flow chart of a video generation method provided in an embodiment of this specification. Figure 1 , the method may include the following steps:

[0034] 102. Acquire a sample image set of a target scene, where the sample image set includes a plurality of original sample images of the target scene.

[0035] The target scene can be any object that needs to be displayed, such as scenes such as streets, shopping malls, highways, shops, squares, etc., or objects such as shoes, tables, and cars. In the immersive panoramic display application scenario, a three-dimensional model of the target scene is required. The model content seen from different perspectives can be determined based on the three-dimensional model of the target scene, and then images from different perspectives can be rendered based on the model content, thereby generating a panoramic video of the target scene. Before the target scene is three-dimensionally reconstructed, a sample image set of the target scene is obtained, and the sample image set includes multiple original sample images of the target scene, so as to facilitate three-dimensional reconstruction of the target scene based on the original sample images. In some embodiments, the target scene in the real world can be photographed from different shooting angles to obtain multiple original sample images of the target scene or obtain a video of the target scene, and extract multiple original sample images of the target scene from the video. In other embodiments, the target scene can be photographed in a circle around the inside or outside of the target scene to improve the accuracy of the three-dimensional model of the reconstructed target scene, thereby improving the image rendering quality based on the three-dimensional model. Different original sample images can correspond to different camera poses, and the camera pose includes the position and posture of the shooting device when shooting the image. The shooting device may be any device capable of acquiring the image data of the target scene, such as but not limited to a camera, a mobile phone with a shooting function, a tablet computer, a wearable device, etc. For example, a mobile phone device with a camera function with a high popularity rate may be used to shoot a video of the target scene within 1-2 minutes, and extract multiple original sample images of the target scene from the video, and the original sample images may be image data in RGB format.

[0036] 104. Based on the sample image set, generate a browsing trajectory of the target scene and a neural radiation field model of the target scene.

[0037] Based on the sample image set, the browsing trajectory and neural radiation field model of the target scene can be generated. The representation model of the target scene can be a fully connected neural network, which is also called a multi-layer perceptron (Multi-LayerPerceptron, MLP). In one embodiment of the present specification, the target scene can be characterized as a neural radiation field (Neural Radiance Field, NERF). The neural radiation field model can predict the volume density and color information of the input spatial point based on the spatial coordinates and viewing angle information of the input spatial point. Among them, the neural radiation field model can be expressed as:

[0038] σ,c=F(d,X)

[0039] Wherein, X = (x, y, z), X is the spatial coordinates (x, y, z) of the spatial point; Recorded as the viewing angle information of the spatial point θ is the azimuth angle, is the elevation angle; c = (r, g, b), c is recorded as the color information of the spatial point (r, g, b), r refers to red (Red, R), g refers to green (Green, G), and b refers to red (Blue, B); σ is recorded as the volume density of the spatial point. After obtaining the sample image set of the target scene, the camera pose corresponding to each original sample image is calculated respectively, and the multiple sight lines emitted by the camera when shooting each original sample image and the viewing angle information of each sight line are determined according to the camera pose corresponding to each original sample image and the camera internal parameters and other data. Spatial point sampling is performed on each sight line to obtain multiple spatial points. It should be understood that the viewing angle information of the spatial points sampled from the same sight line is the viewing angle information of the sight line. After obtaining multiple spatial points, the spatial coordinates of the multiple spatial points and their viewing angle information are used for three-dimensional reconstruction based on a neural network. This process can be a process that is executed in batches multiple times, and finally a neural radiation field model of the target scene can be obtained.

[0040] The browsing trajectory of a target scene usually refers to the path that a camera or observer moves in the target scene. This path records the position and posture changes of the camera or observer when observing the target scene. The browsing trajectory can be a straight line, a curve, or a complex irregular path. The browsing path can be expressed as a time series composed of multiple discrete browsing points, or as a function of browsing points related to time. Figure 2 As shown, in some embodiments, step 102 may include:

[0041] 2022. Feature point matching and pose estimation are performed on the sample image set. Through feature point matching, points with similar features can be found between two or more images, and a corresponding relationship between them can be established, so as to find the same object or scene between multiple original sample images. Pose estimation can calculate the camera pose of each original sample image by matching known feature points. In some embodiments, a SLAM (simultaneous localization and mapping) algorithm can be used to more accurately calculate the camera pose corresponding to each sample image. Specifically, when calculating the camera pose, the SLAM algorithm first extracts the feature points in each sample image, then establishes a matching relationship between the feature points of two adjacent sample images, calculates the relative camera pose between the two adjacent sample images based on the matching relationship between the feature points of the two adjacent sample images, and calculates the camera pose corresponding to each sample image based on the relative camera pose of the pairwise sample image quality inspection. In some embodiments, a neural network model can be used to process each sample image in the sample image set to extract feature points in each sample image. Feature points can be corner points (such as Harris corner points, Shi-Tomasi corner points), edges, half points, SIFT feature points and other points with significant properties. For two or more sample images in the sample image set, the feature points extracted by the neural network are matched to determine the corresponding relationship between the feature points. The matching algorithm can be distance-based matching (such as nearest neighbor matching, nearest neighbor distance ratio matching), set relationship-based matching (such as robust matching based on RANSAC), etc. Through the corresponding relationship of feature point matching, the open source computer vision software package Colmap is used to perform pose estimation, thereby obtaining a data pair of the original sample image and its pose. In some embodiments, the global pose optimization algorithm can also be used to optimize the pose estimation results to improve the accuracy and stability of camera pose estimation and three-dimensional reconstruction of the target scene.

[0042] 2024. Based on the results of the feature point matching and the pose estimation, a coarse reconstruction representation of the target scene is generated, and based on the coarse reconstruction representation of the target scene, a browsing trajectory of the target scene is generated. In some embodiments, the target scene can be coarsely reconstructed as a geometric model based on sparse three-dimensional feature points. For example, based on the results of the feature point matching and the pose estimation, sparse three-dimensional point cloud data of the target scene can be extracted, and based on the sparse three-dimensional point cloud data, a coarse reconstruction model of the target scene is generated; based on the coarse reconstruction model of the target scene, a browsing trajectory of the target scene is generated. Sparse three-dimensional point cloud data is a portion of discrete, representative three-dimensional points extracted from the target scene, which can provide sufficient information to describe the geometry and structure of the scene. By extracting sparse key points for triangulation calculation, a smaller point cloud data set can be obtained, thereby reducing the amount of calculation and storage requirements.

[0043] The browsing trajectory of the target scene can be obtained by processing based on the acquisition trajectory corresponding to the sample image set. Based on the pose estimation of the sample image set, the camera pose of each sample image in the sample image set is obtained, and each camera pose corresponds to a collection point of a sample image. The time series of the camera pose of each sample image in the sample image set constitutes the acquisition trajectory of the sample image set. For the original sample images collected by video shooting, the shooting trajectory during video shooting is usually in line with the user's observation habits, and the acquisition trajectory of the sample image set is the shooting trajectory when the user shoots the video. In some embodiments, the acquisition trajectory is interpolated to obtain a browsing trajectory that conforms to the user's observation habits. Among them, the interpolation processing method can be linear interpolation, sample interpolation, Bezier curve interpolation, etc. In other embodiments, the center and direction of the acquisition trajectory can be calculated to generate a circular trajectory as the browsing trajectory. Among them, the center point of the acquisition trajectory is used to determine the center of the generated browsing trajectory, and the direction of the acquisition trajectory is used to determine the starting angle of the generated browsing trajectory.

[0044] 2026. Generate a neural radiation field model of the target scene based on the results of the feature point matching and the pose estimation. In the embodiment of this specification, based on the feature point matching and pose estimation, a data pair of a sample image and a camera pose can be obtained, and all the data pairs of the sample images in the sample image set and their camera poses are used for training the neural radiation field of the target scene, so that a neural radiation field model of the target scene can be obtained.

[0045] 106. Based on the browsing trajectory and the neural radiation field model, generate a rendered image of the target scene corresponding to the browsing trajectory.

[0046] In some embodiments, the browsing point includes a browsing position and a browsing perspective. The browsing position is used to record the position of a camera or an observer when observing a target scene, which can be represented as a spatial point coordinate in the target scene. The browsing perspective is used to record the posture change of the camera or the observer when observing the target scene, which can be represented as the perspective information of the spatial point. The spatial point coordinates represented by the browsing position and the perspective information represented by the browsing perspective are input into the implicit three-dimensional representation model of the target scene for prediction, and the volume density and color information corresponding to the spatial coordinates can be obtained, so as to render a rendered image of the target scene corresponding to the browsing point.

[0047] 108. Perform quality assessment on the rendered images to obtain quality assessment results of each rendered image; if the quality assessment result of the rendered image is lower than expected, obtain a supplementary sample image corresponding to the rendered image; add the supplementary sample image to the sample image set to obtain an updated sample image set; regenerate an implicit three-dimensional representation model of the target scene based on the updated sample image set to obtain an updated implicit three-dimensional representation model; based on the browsing trajectory and the updated implicit three-dimensional representation model, regenerate the rendered image of the target scene corresponding to the browsing trajectory; repeat this step until the quality assessment results of the rendered image of the target scene corresponding to the browsing trajectory are not lower than expected.

[0048] In the embodiment of this specification, there is no restriction on the quality and quantity requirements of the original sample images of the target scene obtained in step 104, so as to improve the efficiency of obtaining the sample image set of the target scene and reduce the sample acquisition cost. Therefore, the number and viewing angle of the original sample images collected in the sample image set are limited, and the neural radiation field model obtained by training based on the original sample images is usually a low-quality neural radiation field model with incomplete viewing angles. The quality of the rendered images of the neural radiation field model under different viewing angles is uneven. Therefore, the rendered images of some browsing points in the browsing trajectory may be low.

[0049] In some embodiments, the indicators used to evaluate the quality of the rendered image may include peak signal-to-noise ratio (PSNR), structural similarity (SSIM), image sharpness, etc. For example, the indicators used for quality evaluation may be weighted and summed, and the quality evaluation result of the rendered image may be obtained after normalization. The quality evaluation result is compared with the preset quality evaluation index value: if the quality evaluation result is greater than or equal to the preset quality evaluation index value, it means that the quality evaluation result of the rendered image is not lower than expected; if the quality evaluation result is lower than the preset quality evaluation index value, it means that the quality evaluation result of the rendered image is lower than expected. If the quality evaluation result of the rendered image is lower than expected, it means that incremental learning of the neural radiation field model of the target scene generated in step 102 is required.

[0050] In some embodiments, if the quality evaluation result of the rendered image is lower than expected, obtaining a supplementary sample image corresponding to the rendered image includes: obtaining a browsing point corresponding to the rendered image whose quality evaluation result is lower than expected; based on the obtained browsing point, initiating a supplementary acquisition of sample images of the target scene corresponding to the browsing point, so as to obtain a supplementary sample image of the rendered image. For example, the browsing position of the browsing point can be marked in the real-time rendering platform of the target scene, or a rendered image with a poor quality evaluation result can be output, thereby indicating the viewing angle to be adopted when the image is supplemented, so that the camera position and camera posture of the supplementary sample image acquired are the same or similar to the browsing position and browsing viewing angle of the rendered image.

[0051] It should be noted that step 106 and step 108 can be performed alternately or sequentially. That is, each time a rendered image is generated, a quality assessment can be performed on the generated rendered image, and whether incremental learning is required can be determined based on the result of the quality assessment. It is also possible that after all the rendered views of the browsing trajectory are generated in step 106, the quality of each generated rendered view can be assessed in step 108, and whether incremental learning is required can be determined based on the result of the quality assessment. The latter is adopted in the embodiment of this specification, and in step 108, sample images are supplemented for each browsing point corresponding to all the rendered images whose quality assessment results do not meet expectations. The embodiment of this specification does not limit the number of supplementary sample images for each browsing point, and one or more images can be collected. Specifically, a guiding trajectory for auxiliary supplementary collection can also be generated based on multiple browsing points to indicate the collection trajectory for video shooting, and the video shot according to the collection trajectory can include supplementary sample images of browsing points corresponding to each rendered image whose quality assessment results do not meet expectations.

[0052] In this embodiment, the supplementary sample image is added to the sample image set to obtain an updated sample image set. The updated image sample set includes sample images corresponding to the original viewing angle with poor rendering quality. The updated sample image set is subjected to feature point matching and pose estimation by means similar to step 2022 to obtain feature points and corresponding camera poses of each sample image. Based on the results of feature matching and pose estimation, the neural radiation field model of the target scene is trained again using the updated sample image set. Among them, the retraining of the neural radiation field model can be based on the neural radiation field model obtained by previous training to increase the training of training data, or it can be to train a new neural radiation field model from the beginning. Because the supplementary sample images are collected for the browsing points of the rendered images with rendering quality lower than expected, the incremental training of the neural radiation field model based on the updated sample image set training can make up for the representation defects more targeted and efficiently, and the representation effect of the updated neural radiation field model at the corresponding browsing point is improved. In this way, by repeating this step, the representation defects of the neural radiation field model of the target scene at different viewing angles can be continuously repaired. Finally, the quality evaluation results of all rendered images of the target scene corresponding to the browsing trajectory can meet expectations. Finally, a high-quality neural radiation field model of the target scene can be obtained.

[0053] 110. Generate a panoramic video of the target scene based on a rendered image of the target scene corresponding to the browsing trajectory.

[0054] In some embodiments, an image stitching algorithm is used to stitch all rendered images corresponding to the browsing trajectory into a panoramic image of the target scene. The image stitching algorithm can automatically stack and fuse images according to the overlapping areas and camera parameters in the image. For example, algorithms such as feature matching, image registration, and image fusion. In some embodiments, a video processing library is used to convert the panoramic image of the target scene into a panoramic video. In other embodiments, the generated panoramic video can also be edited twice, such as stylization processing, resolution tuning processing, etc., to obtain a high-quality panoramic video of the target scene.

[0055] The video service method provided in the embodiments of this specification can collect additional sample images and add them to the sample image set based on the quality evaluation results of the rendered images to perform incremental learning on the neural radiation field of the target scene, thereby improving the generation efficiency and quality of the panoramic video.

[0056] For ease of understanding, the following introduces a concentrated scenario implementation example to explain in detail the video generation method provided in the embodiment of the present application.

[0057] Scenario Example 1:

[0058] In e-commerce scenarios, the video generation method provided in the embodiments of the present application can be used to create an immersive panoramic video of a product or store. Figure 3 In actual applications, merchants can use shooting devices 302 such as mobile phones, tablet computers, wearable smart devices, smart home devices, cameras, etc. to shoot a video in a way that surrounds the target scene 301 (which can be a commodity or a store object, etc.). Merchants can initiate a picture or video upload operation on the human-computer interaction interface (such as a Web interface) provided by the client 303 to upload pictures or videos including multiple target scenes to the server 304 that executes the video generation method. Among them, the client 303 can be independent of the shooting device 302, for example, a mobile phone, tablet computer, computer, etc. with a video service client app or applet installed. The client 303 can also be integrated with the shooting device 302 in the same device, such as a mobile phone, tablet computer, etc. with a shooting function and a client app or applet installed. The server 304 is a distributed server cluster composed of a single server or multiple servers. Further optionally, the server 304 can be a cloud server. The server 304 runs the video generation method of this embodiment based on multiple pictures or videos of the target scene, generates a panoramic video of the target scene and sends it to the client for display by the client.

[0059] Scenario Example 2:

[0060] In the field of AI (Artificial Intelligent) shopping malls, in order to provide consumers with a 3D scene-based immersive experience in online shopping, a three-dimensional model can be created for the store. To this end, the designer can use a mobile phone to shoot a video around the store (target scene) in the real scene, and upload the video to a device that executes the video generation method. The video generation device generates a panoramic video of the store based on multiple store-related pictures in the video to complete the AI ​​scene creation task of the store.

[0061] In addition, Figure 1 Corresponding to the video generating method shown, the embodiment of this specification also provides a video generating device. Figure 4 4 is a schematic diagram of the structure of a video generating device 400 provided in an embodiment of this specification. It includes:

[0062] The image acquisition module 401 acquires a sample image set of a target scene, wherein the sample image set includes a plurality of original sample images of the target scene. Before performing three-dimensional reconstruction on the target scene, a sample image set of the target scene is acquired, wherein the sample image set includes a plurality of original sample images of the target scene, so as to facilitate three-dimensional reconstruction of the target scene based on the original sample images.

[0063] The image processing module 402 generates a browsing trajectory of the target scene and a neural radiation field model of the target scene based on the sample image set. The neural radiation field model can predict the volume density and color information of the input spatial point based on the spatial coordinates and viewing angle information of the spatial point. The browsing trajectory of the target scene usually refers to the path that the camera or observer moves in the target scene. The path records the position and posture changes of the camera or observer when observing the target scene. The browsing trajectory can be a straight line, a curve or a complex irregular path. The browsing path can be represented as a time series composed of multiple discrete browsing points, or it can be represented as a function of browsing points related to time.

[0064] In some embodiments, the image processing module 402 includes:

[0065] The feature matching and pose estimation unit 4021 performs feature point matching and pose estimation on the sample image set. Through feature point matching, points with similar features can be found between two or more images, and a correspondence between them can be established, so as to find the same object or scene between multiple original sample images. Pose estimation can calculate the camera pose of each original sample image through known feature point matching, so as to obtain a data pair of the original sample image and its pose.

[0066] The trajectory generation unit 4022 generates a rough reconstruction representation of the target scene based on the result of the feature point matching and the pose estimation, and generates a browsing trajectory of the target scene based on the rough reconstruction representation of the target scene. In one embodiment of the present specification, the trajectory generation unit includes:

[0067] The 3D point cloud processing subunit extracts the sparse 3D point cloud data of the target scene based on the results of the feature point matching and the pose estimation, and generates a rough reconstruction model of the target scene based on the sparse 3D point cloud data. Sparse 3D point cloud data is a part of discrete and representative 3D points extracted from the target scene, which can provide sufficient information to describe the geometric shape and structure of the scene. By extracting sparse key points for triangulation calculation, a smaller point cloud data set can be obtained, thereby reducing the amount of calculation and storage requirements.

[0068] The trajectory generation subunit generates a browsing trajectory of the target scene based on the coarse reconstruction model of the target scene.

[0069] The browsing trajectory of the target scene can be obtained by processing based on the acquisition trajectory corresponding to the sample image set. Based on the pose estimation of the sample image set, the camera pose of each sample image in the sample image set is obtained, and each camera pose corresponds to a collection point of a sample image. The time series of the camera pose of each sample image in the sample image set constitutes the acquisition trajectory of the sample image set. For the original sample images collected by video shooting, the shooting trajectory during video shooting is usually in line with the user's observation habits, and the acquisition trajectory of the sample image set is the shooting trajectory when the user shoots the video. In some embodiments, the trajectory generation subunit performs interpolation processing on the acquisition trajectory to obtain a browsing trajectory that conforms to the user's observation habits. Among them, the interpolation processing method can be linear interpolation, sample interpolation, Bezier curve interpolation, etc. In other embodiments, the trajectory generation subunit can calculate the center and direction of the acquisition trajectory and generate a circular trajectory as the browsing trajectory. Among them, the center point of the acquisition trajectory is used to determine the center of the generated browsing trajectory, and the direction of the acquisition trajectory is used to determine the starting angle of the generated browsing trajectory.

[0070] The model training unit 4023 generates a neural radiation field model of the target scene based on the results of the feature point matching and the pose estimation. Based on the feature point matching and pose estimation, a data pair of a sample image and a camera pose can be obtained, and all the data pairs of the sample images in the sample image set and their camera poses are used for training the neural radiation field of the target scene, so that a neural radiation field model of the target scene can be obtained.

[0071] The rendering module 403 generates a rendered image of the target scene corresponding to the browsing trajectory based on the browsing trajectory and the neural radiation field model. In some embodiments, the browsing point includes a browsing position and a browsing perspective. The browsing position is used to record the position of the camera or observer when observing the target scene, which can be expressed as a spatial point coordinate in the target scene. The browsing perspective is used to record the posture change of the camera or observer when observing the target scene, which can be expressed as the perspective information of the spatial point. The spatial point coordinates represented by the browsing position and the perspective information represented by the browsing perspective are input into the implicit three-dimensional representation model of the target scene for prediction, and the volume density and color information corresponding to the spatial coordinates can be obtained, thereby rendering a rendered image of the target scene corresponding to the browsing point.

[0072] In the embodiments of this specification, the quality and quantity requirements of the original sample images of the target scene acquired by the image acquisition module 401 are not restricted, so as to improve the efficiency of acquiring the sample image set of the target scene and reduce the sample acquisition cost. Therefore, the number and viewing angle of the original sample images acquired in the sample image set are limited, and the neural radiation field model trained based on the original sample images is usually a low-quality neural radiation field model with incomplete viewing angles, and the quality of the rendered images of the neural radiation field model under different viewing angles is uneven. Therefore, the rendered images of some browsing points in the browsing trajectory may be low.

[0073] The quality assessment and supplementary acquisition module 404 performs quality assessment on the rendered images to obtain quality assessment results of each rendered image; if the quality assessment result of the rendered image is lower than expected, obtain a supplementary sample image corresponding to the rendered image; add the supplementary sample image to the sample image set to obtain an updated sample image set; regenerate the neural radiation field model of the target scene based on the updated sample image set to obtain an updated neural radiation field model; regenerate the rendered image of the target scene corresponding to the browsing trajectory based on the browsing trajectory and the updated neural radiation field model; repeat this step until the quality assessment results of the rendered images of the target scene corresponding to the browsing trajectory are not lower than expected.

[0074] For the rendered images whose quality assessment does not meet expectations, the quality assessment and supplementary acquisition module 404 adds the supplementary sample images to the sample image set to obtain an updated sample image set. The updated image sample set includes sample images corresponding to the original viewing angles with poor rendering quality. The feature matching and pose estimation unit 4021 performs feature point matching and pose estimation on the updated sample image set to obtain feature points and corresponding camera poses of each sample image. Based on the results of feature matching and pose estimation, the model training unit 4023 uses the updated sample image set to train the neural radiation field model of the target scene again. Because the supplementary sample images are collected for the browsing points of the rendered images whose rendering quality is lower than expected, the incremental training of the neural radiation field model based on the updated sample image set training can make up for the representation defects more targetedly and efficiently, and the representation effect of the updated neural radiation field model at the corresponding browsing points is improved. In this way, after a cycle, the representation defects of the neural radiation field model of the target scene at different viewing angles can be continuously repaired. Finally, the quality assessment results of all the rendered images of the target scene corresponding to the browsing trajectory generated by the rendering module 403 can meet expectations. Ultimately, a high-quality neural radiation field model of the target scene can be obtained.

[0075] The video generation module 405 generates a panoramic video of the target scene based on the rendered image of the target scene corresponding to the browsing trajectory.

[0076] The video generation device of the embodiment of the present specification can collect additional sample images and add them to the sample image set according to the quality evaluation results of the rendered image to perform incremental learning on the neural radiation field of the target scene, thereby improving the generation efficiency and quality of the panoramic video.

[0077] In some embodiments, the video generating device further includes: a stylized processing module for performing stylized processing on the panoramic video.

[0078] In some embodiments, the video generating device further includes: a tuning module for performing super-resolution tuning processing on the panoramic video.

[0079] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0080] In addition, the embodiments of the present application also provide another computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of any of the methods in the aforementioned embodiments are implemented. If the components of the above-mentioned device are implemented in the form of software functional units and sold or used as independent products, they can be stored in the computer-readable storage medium.

[0081] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of this specification is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (Digital Subscriber Line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server, data center, etc. that contains one or more available media integrated. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a digital versatile disc (DVD)), or a semiconductor medium (eg, a solid state disk (SSD)).

[0082] The present application also provides an electronic device, including:

[0083] one or more processors, and

[0084] A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of the method described in any one of the aforementioned method embodiments.

[0085] The present application also provides a computer program product, including a computer program, which implements the steps of any one of the methods in the aforementioned method embodiments when executed by a processor.

[0086] in, Figure 5 The architecture of the electronic device 500 is shown as an example, which may include: a processor 510, a disk drive 520, an input / output interface 530, a network interface 540, and a memory 550. The processor 510, the disk drive 520, the input / output interface 530, the network interface 540, and the memory 550 may be communicatively connected via a communication bus 560.

[0087] The processor 510 may be implemented by a general-purpose CPU, a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, etc., and is used to execute relevant programs to implement the technical solutions provided in this application.

[0088] The memory 550 can be implemented in the form of ROM (Read Only Memory), RAM (Read Access Memory), static memory, dynamic storage device, etc. The memory 550 can store an operating system 551 for controlling the operation of the electronic device 500 and a basic input and output system (BIOS) 552 for controlling the low-level operation of the electronic device 500. In addition, a web browser 553, a data storage management system 554, etc. can also be stored. In short, when the technical solution provided by the present application is implemented by software or firmware, the relevant program code is stored in the memory 550 and is called and executed by the processor 510.

[0089] The input / output interface 530 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, a prompt light, etc.

[0090] The network interface 540 is used to connect to a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0091] Bus 560 comprises a pathway that transmits information between the various components of the device (eg, processor 510, disk drive 520, input / output interface 530, network interface 540, and memory 550).

[0092] It should be noted that, although the above device only shows a processor 510, a disk drive 520, an input / output interface 530, a network interface 540, a memory 550, a bus 560, etc., in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may only include components necessary for implementing the method of the present application, and does not necessarily include all the components shown in the figure.

[0093] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk and other media that can store program codes. In the absence of conflict, the technical features in this embodiment and the implementation scheme can be combined arbitrarily.

[0094] The embodiments described above are merely preferred embodiments of this specification and are not intended to limit the scope of this specification. Without departing from the design spirit of this specification, various modifications and improvements made to the technical solutions of this specification by ordinary technicians in this field should fall within the scope of protection determined by the claims of this specification.

Claims

1. Video generation service method, including: Acquire a sample image set of a target scene, wherein the sample image set includes a plurality of original sample images of the target scene; Based on the sample image set, generating a browsing trajectory of the target scene and a neural radiation field model of the target scene; Based on the browsing trajectory and the neural radiation field model, generating a rendered image of the target scene corresponding to the browsing trajectory; Performing quality assessment on the rendered images to obtain quality assessment results of each rendered image; if the quality assessment result of the rendered image is lower than expected, obtaining a supplementary sample image corresponding to the rendered image; adding the supplementary sample image to the sample image set to obtain an updated sample image set; regenerating the neural radiation field model of the target scene based on the updated sample image set to obtain an updated neural radiation field model; regenerating the rendered image of the target scene corresponding to the browsing trajectory based on the browsing trajectory and the updated neural radiation field model; Repeat this step until the quality assessment results of the rendered images of the target scene corresponding to the browsing trajectory are not lower than the expectation; Based on the rendered image of the target scene corresponding to the browsing trajectory, a panoramic video of the target scene is generated.

2. The method according to claim 1, wherein generating the browsing trajectory of the target scene and the neural radiation field model of the target scene based on the sample image set comprises: Performing feature point matching and pose estimation on the sample image set; Based on the results of the feature point matching and the pose estimation, a coarse reconstruction representation of the target scene is generated, and based on the coarse reconstruction representation of the target scene, a browsing trajectory of the target scene is generated; Based on the results of the feature point matching and the pose estimation, a neural radiation field model of the target scene is generated.

3. The method according to claim 2, wherein generating a coarse reconstruction representation of the target scene based on the result of the feature point matching and the pose estimation, and generating a browsing trajectory of the target scene based on the coarse reconstruction representation of the target scene comprises: Extracting sparse three-dimensional point cloud data of the target scene based on the results of the feature point matching and the pose estimation, and generating a rough reconstruction model of the target scene based on the sparse three-dimensional point cloud data; Based on the coarse reconstruction model of the target scene, a browsing trajectory of the target scene is generated.

4. The method according to claim 2, wherein generating a browsing trajectory of the target scene based on the coarsely reconstructed representation of the target scene comprises: generating a collection trajectory of the sample image set based on the coarsely reconstructed representation of the target scene; The collection trajectory includes a plurality of collection points; Interpolation processing is performed on the acquisition trajectory to obtain the browsing trajectory.

5. The method according to claim 2, wherein generating a browsing trajectory of the target scene based on the coarsely reconstructed representation of the target scene comprises: generating a collection trajectory of the sample image set based on the coarsely reconstructed representation of the target scene; Based on the center and orientation of the acquisition trajectory, a circular trajectory is generated as the browsing trajectory.

6. The method according to claim 1, wherein the browsing track comprises a plurality of browsing points, wherein the browsing points comprise browsing positions and browsing viewing angles corresponding to the browsing positions; The step of generating a rendered image of the target scene corresponding to the browsing trajectory based on the browsing trajectory and the neural radiation field model comprises: The browsing position and browsing perspective of the browsing point are input into the neural radiation field model of the target scene to obtain the rendering color and rendering density data of the target scene corresponding to the browsing point, and a rendering image of the target scene corresponding to the browsing point is generated based on the rendering color and rendering density data.

7. The method according to claim 6, if the quality evaluation result of the rendered image is lower than expected, obtaining a supplementary sample image corresponding to the rendered image, comprising: Obtain browsing points corresponding to rendered images whose quality evaluation results are lower than expected; Based on the acquired browsing point, a supplementary acquisition of a sample image of the target scene corresponding to the browsing point is initiated to acquire a supplementary sample image of the rendered image.

8. The method according to claim 1, after generating a panoramic video of the target scene based on the rendered image of the target scene corresponding to the browsing trajectory, further comprises: The panoramic video is stylized.

9. The method according to claim 1, after generating a panoramic video of the target scene based on the rendered image of the target scene corresponding to the browsing trajectory, further comprises: Perform super-resolution tuning processing on the panoramic video.

10. A video generating device, comprising: An image acquisition module, which acquires a sample image set of a target scene, wherein the sample image set includes a plurality of original sample images of the target scene; An image processing module, which generates a browsing trajectory of the target scene and a neural radiation field model of the target scene based on the sample image set; A rendering module, which generates a rendered image of the target scene corresponding to the browsing trajectory based on the browsing trajectory and the neural radiation field model; A quality assessment and supplementary acquisition module performs quality assessment on the rendered images to obtain quality assessment results of each rendered image; if the quality assessment result of the rendered image is lower than expected, obtains a supplementary sample image corresponding to the rendered image; adds the supplementary sample image to the sample image set to obtain an updated sample image set; regenerates the neural radiation field model of the target scene based on the updated sample image set to obtain an updated neural radiation field model; regenerates the rendered image of the target scene corresponding to the browsing trajectory based on the browsing trajectory and the updated neural radiation field model; Repeat this step until the quality assessment results of the rendered images of the target scene corresponding to the browsing trajectory are not lower than the expectation; The video generation module generates a panoramic video of the target scene based on a rendered image of the target scene corresponding to the browsing track.

11. The device according to claim 10, wherein the image processing module comprises: A feature matching and posture estimation unit, performing feature point matching and posture estimation on the sample image set; A trajectory generating unit, which generates a coarse reconstruction representation of the target scene based on the results of the feature point matching and the pose estimation, and generates a browsing trajectory of the target scene based on the coarse reconstruction representation of the target scene; A model training unit generates a neural radiation field model of the target scene based on the results of the feature point matching and the pose estimation.

12. The device according to claim 11, wherein the trajectory generating unit comprises: A three-dimensional point cloud processing subunit, which extracts sparse three-dimensional point cloud data of the target scene based on the results of the feature point matching and the pose estimation, and generates a rough reconstruction model of the target scene based on the sparse three-dimensional point cloud data; The trajectory generation subunit generates a browsing trajectory of the target scene based on the coarse reconstruction model of the target scene.

13. The device according to claim 10, further comprising: The stylization processing module performs stylization processing on the panoramic video.

14. The apparatus according to claim 10, further comprising: The tuning module performs super-resolution tuning processing on the panoramic video.

15. A computer storable medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method according to any one of claims 1 to 9.

16. Electronic equipment, including: one or more processors, and A memory associated with the one or more processors, the memory being used to store program instructions, wherein when the program instructions are read and executed by the one or more processors, the steps of the method according to any one of claims 1 to 9 are executed.