Video generation methods, apparatuses, electronic devices, storage media, and software products

Abstract

This disclosure presents a video generation method, apparatus, electronic device, computer-readable storage medium, and computer program product. By generating a rough 3D world of a target scene formed by a set of 3D Gaussian spheres, and by automatically finding multiple offset viewpoints at the same coordinate position as new viewpoints, and stitching the image data of these new viewpoints before image restoration, a multi-view joint restoration mechanism is achieved. This ensures the geometric and semantic consistency of image data between different viewpoints, resulting in consistency of scene structure, texture, and semantic information under different viewing angles. This constructs a continuous, coherent, and high-quality 3D world, solving the problem of inconsistent multi-viewpoints in traditional methods, improving the image quality of new viewpoints under large viewpoint changes, eliminating image quality problems such as floating objects, artifacts, and structural distortions, making the generated images more stable and realistic, improving spatial coherence, and enhancing the user's immersion and exploration experience.

Description

Technical Field

[0001] This disclosure relates to the field of video generation technology, and in particular to a video generation method, apparatus, electronic device, computer-readable storage medium, and computer program product. Background Technology

[0002] Existing 3D (Three-Dimensional) world generation methods primarily start with a single image or text description, generating images or panoramas from multiple perspectives, and then converting them into a 3D spatial representation. Representative methods include Text2Room (a text-driven 3D space construction tool) and LucidDreamer (an innovative 3D scene generation technology capable of generating high-quality 3D scenes based on text or image cues). They generate multi-view images from input images and text, and then construct an initial 3D scene based on these. Other 3D world generation methods use pre-trained text-to-panoramic diffusion models to generate scene images, and then construct the 3D world. However, while these methods can generate a degree of interactive 3D space, because training relies on only a few perspectives for supervision, the generated 3D world often fails to maintain high-quality image performance when the user moves their viewpoint significantly or explores further. Problems such as floating objects, artifacts, and structural distortions easily appear, limiting the immersive experience. Summary of the Invention

[0003] To address the technical problems in related technologies, embodiments of this disclosure provide a video generation method, apparatus, electronic device, computer-readable storage medium, and computer program product.

[0004] According to a first aspect of the present disclosure, a video generation method is provided, the method comprising:

[0005] Generate the first three-dimensional world of the target scene. The first three-dimensional world is a set of three-dimensional Gaussian spheres. Each three-dimensional Gaussian sphere includes position coordinate parameters, covariance parameters, and spherical harmonic function coefficients.

[0006] Based on the first three-dimensional world, image data of a target viewpoint group is determined, wherein the target viewpoint group includes multiple offset viewpoints at the same coordinate position;

[0007] The image data of the target viewpoint group are stitched together to obtain stitched panoramic image data.

[0008] The stitched panoramic image data is subjected to image restoration processing to obtain restored panoramic image data;

[0009] The first 3D world is optimized based on the repaired panoramic image data to obtain a second 3D world, and a target 3D video is generated based on the second 3D world.

[0010] As an optional embodiment, the first three-dimensional world of the generated target scene is a set of three-dimensional Gaussian spheres. Each three-dimensional Gaussian sphere includes position coordinate parameters, covariance parameters, and spherical harmonic function coefficients, including:

[0011] In response to any received scene image and scene description prompt text, extract the image semantic feature distribution of the scene image and the text description feature distribution of the scene description prompt text;

[0012] For any pixel in the scene image, determine the three-dimensional coordinate value of the pixel based on the image semantic feature distribution;

[0013] Create a first three-dimensional Gaussian sphere for any pixel based on its three-dimensional coordinates, so as to obtain a set of first three-dimensional Gaussian spheres for all pixels in the scene image;

[0014] By adjusting and optimizing the Gaussian sphere parameters in the first three-dimensional Gaussian sphere set through the text description feature distribution, a second Gaussian sphere set is obtained, and the second Gaussian sphere set constitutes the first three-dimensional world.

[0015] As an optional embodiment, the step of determining image data of a target viewpoint group based on the first three-dimensional world, wherein the target viewpoint group includes multiple angle-shifted viewpoints at the same coordinate position, includes:

[0016] Receive any coordinate value input from the user to find the coordinate value from a set of known viewpoints, wherein the set of known viewpoints is the observation pose data used in the process of establishing the first three-dimensional world of the target scene;

[0017] In response to the fact that the coordinate value is not found from the known set of viewpoints, the coordinate value is determined to be a first viewpoint;

[0018] Based on the first viewpoint, at least one adjacent offset viewpoint is determined according to a preset offset rule. The at least one adjacent offset viewpoint is a second viewpoint, and the first viewpoint and the second viewpoint are the target viewpoint group.

[0019] Image rendering is performed on the first viewpoint and the second viewpoint to obtain multiple rendered images, and these multiple rendered images are determined as the image data of the target viewpoint group.

[0020] As an optional embodiment, the step of stitching together the image data of the target viewpoint group to obtain stitched panoramic image data includes:

[0021] Feature extraction is performed on each image data of the target viewpoint group to obtain the image features of the image data of each viewpoint, and image feature pairs between each viewpoint are constructed.

[0022] Based on the image features, the image data of the target viewpoint group is aligned using a preset image alignment algorithm to obtain aligned image data.

[0023] The aligned image data is stitched together using a preset image stitching algorithm to obtain the stitched panoramic image data.

[0024] As an optional embodiment, the step of stitching the aligned image data using a preset image stitching algorithm to obtain the stitched panoramic image data includes:

[0025] The frequency domain decomposition of the aligned image data is performed using the preset image stitching algorithm to obtain the frequency domain signal features of the image data from each viewpoint in the aligned image data.

[0026] According to a preset weighting rule, the frequency domain signal features of the image data from each perspective in the aligned image data are weighted and fused to obtain the stitched panoramic image data.

[0027] As an optional embodiment, the image restoration processing of the stitched panoramic image data to obtain restored panoramic image data includes:

[0028] The stitched panoramic image data is input into a video restoration model to determine the area to be restored in the stitched panoramic image data, and the area to be restored is then restored to obtain restored panoramic image data.

[0029] As an optional embodiment, the step of inputting the stitched panoramic image data into a video restoration model to determine the area to be restored in the stitched panoramic image data through the video restoration model, and performing restoration processing on the area to be restored to obtain restored panoramic image data, includes:

[0030] The stitched panoramic image data is input into a video restoration model, and the video restoration model is used to perform mask positioning on the stitched panoramic image data to determine the area to be restored in the stitched panoramic image data.

[0031] For the area to be repaired in the stitched panoramic image data, spatial repair processing is performed using the video repair model to obtain the first repaired image;

[0032] The first restored image is subjected to temporal restoration processing using the video restoration model to obtain a second restored image, which is then identified as the restored panoramic image data.

[0033] As an optional embodiment, the method further includes:

[0034] The training dataset is input into a preset video restoration network. The training dataset includes multiple first video frame images with mask markings and corresponding second video frame images. The first video frame images are video images with defect features in the mask marking areas, and the second video frame images are video images without defect features. The preset video restoration network is a video processing network based on a diffusion model.

[0035] The first video frame image with a mask identifier is processed using the preset video restoration network to obtain a predicted video image;

[0036] Calculate the preset distance between the predicted video image and the second video frame image to obtain the prediction loss value;

[0037] The video restoration model is obtained by iteratively optimizing the network parameters of the preset video restoration network based on the predicted loss value.

[0038] As an optional embodiment, the method further includes:

[0039] Initialize a 3D world reconstruction model and obtain a set of rendered video frame images based on the 3D world reconstruction model. The set of rendered video frame images is a combination of video frame image sequences obtained from 3D world rendering of multiple scenes.

[0040] In response to iteratively optimizing the 3D world reconstruction model according to a preset number of iterations, the third video frame image and the fourth video frame image rendered based on the 3D world reconstruction model during the iterative optimization process are collected according to the preset number of iterations and the preset sampling step size. The third video frame image is the video frame image output before reaching the preset number of iterations, and the fourth video frame image is the video frame image output when the preset number of iterations is reached.

[0041] By comparing the third video frame image and the fourth video frame image, the region with defective features in the third video frame image is determined, and a masking mark is made on the region with defective features by a preset masking mechanism to obtain a third video frame image with masking mark.

[0042] The third video frame image with the mask identifier and the fourth video frame image are identified as the second data pair, and the second data pair corresponding to at least one frame image in the set of rendered video frame images constitutes the training dataset.

[0043] According to a second aspect of the present disclosure, a video generation apparatus is provided, the apparatus comprising:

[0044] The 3D world generation module is used to generate the first 3D world of the target scene. The first 3D world is a set of 3D Gaussian spheres. Each 3D Gaussian sphere includes position coordinate parameters, covariance parameters, and spherical harmonic function coefficients.

[0045] A multi-view image determination module is used to determine image data of a target view group based on the first three-dimensional world, wherein the target view group includes multiple offset viewpoints at the same coordinate position;

[0046] The image stitching module is used to stitch together the image data of the target viewpoint group to obtain stitched panoramic image data.

[0047] The image restoration module is used to perform image restoration processing on the stitched panoramic image data to obtain restored panoramic image data.

[0048] The reverse optimization module is used to optimize the first 3D world based on the repaired panoramic image data to obtain a second 3D world, and then generate a target 3D video based on the second 3D world.

[0049] As an optional embodiment, the three-dimensional world generation module includes:

[0050] The first feature extraction unit is used to extract the image semantic feature distribution of the scene image and the text description feature distribution of the scene description prompt text in response to any received scene image and scene description prompt text;

[0051] A three-dimensional coordinate determination unit is used to determine the three-dimensional coordinate value of any pixel in the scene image based on the distribution of the image semantic features.

[0052] The Gaussian sphere creation unit is used to create the first three-dimensional Gaussian sphere for any pixel based on its three-dimensional coordinates, so as to obtain the first three-dimensional Gaussian sphere set for all pixels of the scene image.

[0053] The Gaussian sphere adjustment unit is used to adjust and optimize the Gaussian sphere parameters in the first three-dimensional Gaussian sphere set through the text description feature distribution to obtain a second Gaussian sphere set, and the second Gaussian sphere set constitutes the first three-dimensional world.

[0054] As an optional embodiment, the multi-view image determination module includes:

[0055] A coordinate value receiving unit is used to receive any coordinate value input by the user in order to find the coordinate value from a set of known viewpoints, wherein the set of known viewpoints is the observation pose data used in the process of establishing the first three-dimensional world of the target scene;

[0056] The first-viewpoint determination unit is used to determine the coordinate value as a first-viewpoint in response to the fact that the coordinate value is not found from the known viewpoint set.

[0057] A viewpoint group determination unit is used to determine at least one adjacent offset viewpoint of the first viewpoint according to a preset offset rule based on the first viewpoint, wherein the at least one adjacent offset viewpoint is a second viewpoint, and the first viewpoint and the second viewpoint are the target viewpoint group.

[0058] The graphics rendering unit is used to render images of the first viewpoint and the second viewpoint to obtain multiple rendered images, and to determine the multiple rendered images as the image data of the target viewpoint group.

[0059] As an optional embodiment, the image stitching module includes:

[0060] The second feature extraction unit is used to extract features from each image data of the target viewpoint group, obtain the image features of the image data of each viewpoint, and construct image feature pairs between each viewpoint.

[0061] An image alignment unit is used to align the image data of the target viewpoint group based on the image features using a preset image alignment algorithm to obtain aligned image data.

[0062] The image stitching unit is used to stitch the aligned image data using a preset image stitching algorithm to obtain the stitched panoramic image data.

[0063] As an optional embodiment, the image stitching unit is further configured to:

[0064] The frequency domain decomposition of the aligned image data is performed using the preset image stitching algorithm to obtain the frequency domain signal features of the image data from each viewpoint in the aligned image data.

[0065] According to a preset weighting rule, the frequency domain signal features of the image data from each perspective in the aligned image data are weighted and fused to obtain the stitched panoramic image data.

[0066] As an optional embodiment, the image restoration module includes:

[0067] The image restoration unit is used to input the stitched panoramic image data into the video restoration model, so as to determine the area to be restored in the stitched panoramic image data through the video restoration model, and to perform restoration processing on the area to be restored to obtain restored panoramic image data.

[0068] As an optional embodiment, the image restoration unit is further configured to:

[0069] The stitched panoramic image data is input into a video restoration model, and the video restoration model is used to perform mask positioning on the stitched panoramic image data to determine the area to be restored in the stitched panoramic image data.

[0070] For the area to be repaired in the stitched panoramic image data, spatial repair processing is performed using the video repair model to obtain the first repaired image;

[0071] The first restored image is subjected to temporal restoration processing using the video restoration model to obtain a second restored image, which is then identified as the restored panoramic image data.

[0072] As an optional embodiment, the device further includes:

[0073] The data input module is used to input the training dataset into a preset video restoration network. The training dataset includes multiple first video frame images with mask markings and corresponding second video frame images. The first video frame images are video images with defect features in the mask-marked areas, and the second video frame images are video images without defect features. The preset video restoration network is a video processing network based on a diffusion model.

[0074] The image prediction module is used to process the first video frame image with a mask identifier using the preset video restoration network to obtain a predicted video image;

[0075] The loss calculation module is used to calculate the preset distance between the predicted video image and the second video frame image to obtain the predicted loss value;

[0076] The parameter optimization module is used to iteratively optimize the network parameters of the preset video restoration network based on the predicted loss value, so as to obtain the video restoration model.

[0077] As an optional embodiment, the device further includes:

[0078] The image set acquisition module is used to initialize the three-dimensional world reconstruction model and acquire a set of rendered video frame images based on the three-dimensional world reconstruction model. The set of rendered video frame images is a combination of video frame image sequences obtained from three-dimensional world rendering based on multiple scenes.

[0079] The video frame image acquisition module is used to respond to iteratively optimizing the three-dimensional world reconstruction model according to a preset number of iterations, and to acquire the third video frame image and the fourth video frame image rendered based on the three-dimensional world reconstruction model during the iterative optimization process according to the preset number of iterations and the preset sampling step size. The third video frame image is the video frame image output before reaching the preset number of iterations, and the fourth video frame image is the video frame image output when the preset number of iterations is reached.

[0080] The first data pair forming module is used to determine the region with defective features in the third video frame image by comparing the third video frame image and the fourth video frame image, and to mark the region with defective features by a preset occlusion masking mechanism to obtain a third video frame image with masking identification.

[0081] The data pair set determination module is used to determine the third video frame image with the mask identifier and the fourth video frame image as a second data pair, and the second data pair corresponding to at least one frame image in the set of rendered video frame images constitutes the training dataset.

[0082] According to a third aspect of the present disclosure, an electronic device is provided, comprising:

[0083] Memory, used to store computer program products;

[0084] A processor is configured to execute a computer program product stored in the memory, and when the computer program product is executed, to implement the method described in the first aspect above.

[0085] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided that stores computer program instructions thereon, which, when executed by a processor, implement the method described in the first aspect.

[0086] According to a fifth aspect of the present disclosure, a computer program product is provided, including computer program instructions that, when executed by a processor, implement the method described in the first aspect above.

[0087] In this embodiment, a rough 3D world (first 3D world) of the target scene formed by a set of 3D Gaussian spheres is generated. Then, by automatically finding multiple offset viewpoints with the same coordinate position as new viewpoints, and stitching the image data of the new viewpoints before image restoration, a multi-view joint restoration mechanism is realized. This ensures the geometric and semantic consistency of image data between different viewpoints, and makes the scene structure, texture, and semantic information consistent under different viewing angles. This constructs a continuous, coherent, and high-quality 3D world, solving the problem of inconsistency among multiple viewpoints in traditional methods. It improves the image quality of new viewpoints under large viewpoint changes, eliminates image quality problems such as floating objects, artifacts, and structural distortions in the image, and makes the generated image more stable and realistic, improving spatial coherence and enhancing the user's immersion and exploration experience. Furthermore, obtaining information from the new viewpoints can improve the modeling ability of unseen areas of the 3D world. In addition, the rough 3D world is updated and optimized using the restored image data, forming a closed-loop modeling from the rough 3D world to the high-quality 3D world (second 3D world), which improves the integrity of the modeling.

[0088] The technical solutions of this disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0089] The accompanying drawings, which form part of this specification, illustrate embodiments of this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0090] This disclosure will become clearer with reference to the accompanying drawings and the following detailed description, wherein:

[0091] Figure 1 This is one of the flowcharts for a video generation method according to an embodiment of the method disclosed herein.

[0092] Figure 2 This is a second flowchart of a video generation method according to an embodiment of the present disclosure.

[0093] Figure 3 This is a third flowchart of a video generation method according to an embodiment of the present disclosure.

[0094] Figure 4 This is a flowchart of a video generation method according to an embodiment of the present disclosure.

[0095] Figure 5 This is the fifth flowchart of a video generation method according to an embodiment of the present disclosure.

[0096] Figure 6 This is a schematic diagram of image comparison for a multi-view joint repair method according to an embodiment of the present disclosure.

[0097] Figure 7This is a schematic diagram of image data for a multi-scene degradation-original video pair, representing one embodiment of the method disclosed herein.

[0098] Figure 8 This is a schematic diagram illustrating the process of generating image data for multi-scene degradation-original video pairs according to an embodiment of the method disclosed herein.

[0099] Figure 9 This is a structural block diagram of a video generation apparatus according to an embodiment of the present disclosure.

[0100] Figure 10 This is a structural block diagram of an electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0101] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this disclosure.

[0102] In the related technologies, there are two core drawbacks in generating 3D worlds: First, the image quality of new perspectives is unstable, especially when making large-scale viewpoint movements or exploring forward, which can easily lead to floating objects, artifacts and structural distortions; Second, there is a lack of consistency across perspectives, with inconsistencies in geometric structure, semantic information and visual appearance between different perspectives, resulting in poor overall spatial coherence and seriously affecting the immersive exploration experience.

[0103] To address the technical problems in related technologies, this disclosure provides a video generation method, apparatus, electronic device, computer-readable storage medium, and computer program product, which will be described in detail below with reference to the accompanying drawings.

[0104] Figure 1 This is one of the flowcharts for a video generation method according to an embodiment of the method disclosed herein. Figure 1 As shown, a video generation method may include the following steps:

[0105] Step 101: Generate the first three-dimensional world of the target scene, wherein the first three-dimensional world is a set of three-dimensional Gaussian spheres, and any three-dimensional Gaussian sphere includes position coordinate parameters, covariance parameters, and spherical harmonic function coefficients.

[0106] The target scene is a specific location or space. In this embodiment, the target scene can be a three-dimensional space and its content uniquely determined by a specific image (scene image) and a specific text description (scene description prompt text) provided by the user. It does not refer to a general type of location (such as "library"), but specifically to the specific space and content described by the user. For example, suppose the user inputs an image, which is a low-angle view of the Trinity College Library at Cambridge University, and inputs a specific text description, "The oak bookshelves are covered in dust, and sunlight shines in from the high window on the right." Then the target scene is the Trinity College Library at Cambridge University, with a dust effect added to the oak bookshelves and light from the right.

[0107] The first three-dimensional world is a three-dimensional Gaussian distribution of multi-view video frame images of the target scene acquired by the acquisition device under multiple different acquisition poses. It consists of a dynamically optimized set of 3D Gaussian spheres (using a Gaussian distribution to describe points in three-dimensional space), representing the spatial structure and rules of the target scene. In this embodiment, the first three-dimensional world is a coarse three-dimensional world, and the image quality of the video rendered from it is low, prone to artifacts, floating objects, and other problems. Therefore, it is also a three-dimensional world that needs to be dynamically optimized.

[0108] In this embodiment of the disclosure, this step can be implemented through the following steps: In response to any received scene image and scene description prompt text, extract the image semantic feature distribution of the scene image and the text description feature distribution of the scene description prompt text; then, for any pixel in the scene image, determine the three-dimensional coordinate value of the pixel based on the image semantic feature distribution; then, create a first three-dimensional Gaussian sphere for the pixel based on the three-dimensional coordinate value of the pixel to obtain a set of first three-dimensional Gaussian spheres for all pixels of the scene image; finally, adjust and optimize the Gaussian sphere parameters in the set of first three-dimensional Gaussian spheres through the text description feature distribution to obtain a second set of Gaussian spheres, which constitute the first three-dimensional world. It is understood that this embodiment of the disclosure uses a scene image (providing visual content) and a related scene description prompt text (providing supplementary information and constraints) to quickly generate a preliminary 3D (three-dimensional) scene model (i.e., the first three-dimensional world) containing basic geometry, object layout, and general style. This model is "rough," meaning it lacks detail, may be inaccurate, and contains errors (e.g., floating objects, unreasonable structures), but it provides a crucial starting point for subsequent iterative optimization.

[0109] Specifically, it can receive user input such as a single scene image and scene description prompt text. The scene image depicts the target scene or object and serves as the primary source of visual information. The scene description prompt text is text provided to the user related to the content of the scene image. It can include information not fully shown in the scene image (e.g., the scene image only shows the front of the room, and the scene description prompt text describes "there is a window and a desk at the back of the room"), abstract concepts or styles (e.g., "a sci-fi style control room", "a sunny country cottage", "a cartoon cat sitting on a sofa"), explanations or emphasis on the content of the image (e.g., "the robot in the center of the image is the main character, and it is holding a glowing energy core"), and so on. Then, feature extraction (i.e., extraction of image semantic feature distribution and text description feature distribution) and understanding are performed on individual scene images and scene description prompts. Specifically, pre-trained deep learning models (which can be Transformer-based models, such as CLIP (Contrastive Language-Image Pre-training) or similar multimodal models) can be used to analyze scene images and scene description prompts separately. For example: ① Identify the main objects in the image and their approximate positions and boundaries (e.g., identify "sofa", "cat", "table", "window"); ② Understand the overall layout and perspective of the scene (e.g., determine that it is an indoor scene with a foreground sofa and a background window); ③ Extract basic visual style features (such as color tone, material texture, and lighting atmosphere); ④ Parse the semantic content of the text description (mentioned objects, attributes, relationships, actions, and scene types); ⑤ Extract keywords and key concepts (such as "science fiction", "control console", "energy core", "sunny"); ⑥ Understand the supplementary and corrective information of the text to the image (e.g., "there is a desk behind the room"). Furthermore, multimodal feature fusion processing is performed on the extracted image semantic feature distribution and text description feature distribution to allow the scene description prompt text to guide and correct ambiguities or omissions when inferring 3D structures from a single scene image. For example, the scene image may only show the front of the sofa, and the scene description prompt text describes "the sofa is L-shaped". After fusion, an L-shaped sofa is determined to be generated. Moreover, if the scene description prompt text describes "science fiction style", the futuristic elements in the image will be enhanced during fusion.Secondly, the fused features are used to predict the 3D structure, including but not limited to depth estimation, normal estimation, and 3D object placement and pose estimation, to determine the three-dimensional coordinates of each pixel in the 3D structure. Based on these three-dimensional coordinates, a three-dimensional Gaussian sphere is created. The Gaussian sphere parameters in the first set of three-dimensional Gaussian spheres can be further adjusted and optimized according to the feature distribution described in the text to adjust the position, size, etc. of the Gaussian spheres, thereby adjusting the layout of the 3D structure. Finally, a 3D spatial structure represented by a set of three-dimensional Gaussian spheres, i.e., the first three-dimensional world, is roughly determined.

[0110] Step 102: Based on the first three-dimensional world, determine the image data of the target view group, wherein the target view group includes multiple offset viewpoints at the same coordinate position.

[0111] In this embodiment of the disclosure, in order to improve the ability to model unseen areas, information can be obtained from new perspectives, wherein the new perspective is the perspective corresponding to the observation pose data that was not used in the process of building the first three-dimensional world of the target scene, corresponding to the "target perspective group".

[0112] A target viewpoint group can include multiple offset viewpoints at the same coordinate position. Specifically, it can receive any 3D coordinate value input by the user, and determine whether the 3D coordinate value exists by querying the first 3D world observation pose data. If not, the viewpoint corresponding to the 3D coordinate value is determined as a new viewpoint, denoted as θ. Then, multiple offset viewpoints are obtained around this new viewpoint θ, such as θ+α, θ-α, etc. Thus, three adjacent viewpoints, θ+α, θ, and θ-α, are determined as a target viewpoint group. Image data of this target viewpoint group is then obtained through image rendering, for example, using 3D Gaussian splashing. It is understood that the number of offset viewpoints included in the target viewpoint group is not limited to three; the required number of adjacent offset viewpoints can be obtained according to the actual image data requirements, and this is not limited here.

[0113] It is understood that angular offset or offset angle refers to image data acquired by rotating the acquisition device around the optical center by a small angle (e.g., <10°) at the same observation position (same coordinate position). As an optional embodiment, the offset amount or offset step size can be controlled within the range where continuous viewing angles can overlap to ensure the continuity of image data.

[0114] By acquiring image data from the target viewpoint group (new viewpoint), it can not only be used for display, but also provide incremental optimization signals for the first three-dimensional world, so as to improve the overall (first three-dimensional world) reconstruction accuracy through subsequent repair of the new viewpoint image data.

[0115] Step 103: The image data of the target viewpoint group is stitched together to obtain stitched panoramic image data.

[0116] In this step, the image data of the target viewpoint group is stitched together to form a panoramic image. This involves stitching together image data from adjacent offset viewpoints to obtain image data with a larger field of view. In some embodiments, the image data from multiple adjacent offset viewpoints in the target viewpoint group can first undergo coordinate transformation to make these image data coplanar. For example, the transformation matrix can be calculated using feature matching to make the images coplanar, including but not limited to homography transformation. For instance, both tilted viewpoints θ+α and θ-α can be projected onto the imaging plane of viewpoint θ. Then, the image data is fused (e.g., multi-band blending can be used) to complete the stitching process and obtain stitched panoramic image data. In this way, multiple offset views at the same coordinate position are stitched together to form a panoramic image, providing spatial consistency assurance for subsequent joint restoration.

[0117] Step 104: Perform image restoration processing on the stitched panoramic image data to obtain restored panoramic image data.

[0118] In this step, stitched panoramic image data is used as the data foundation, enabling the processing of the global structure of the image within a unified coordinate system and satisfying the consistency constraints between viewpoints. As an optional embodiment, image inpainting processing can be performed on the stitched panoramic image data based on a diffusion model to repair defects in the panoramic image and ensure temporal continuity.

[0119] In this embodiment of the disclosure, the repair of stitched panoramic image data may include at least the processes of defect detection, diffusion model repair, and temporal consistency optimization, so as to achieve spatial repair and / or temporal repair of the stitched panoramic image data. Specifically, the spatial structure of the stitched panoramic image data can be repaired by prior knowledge, such as structure reconstruction and material replacement. For video frame images, temporal consistency optimization can also be performed, and finally the repaired panoramic image data is output.

[0120] In some embodiments, defect detection can employ an occlusion masking mechanism. For example, a three-channel mask (structural distortion, floating objects, and texture loss) can be used. Different masks can be used for different defects in the stitched panoramic image data to enhance the repair capability for complex defect areas during the repair process.

[0121] In some embodiments, if the stitched panoramic image data is video frame image data, the diffusion-based inpainting takes the input video as a conditional input, starts with pure Gaussian noise, and gradually generates a complete video frame sequence with spatial rationality (spatial inpainting) and temporal consistency (temporal inpainting) through a multi-step denoising process, while forcibly keeping the unoccluded area unchanged during the denoising process, and finally outputs the inpainted video (inpainted panoramic image data).

[0122] In this embodiment, this step can specifically be implemented by inputting the stitched panoramic image data into a video restoration model to determine the region to be restored in the stitched panoramic image data, and then performing restoration processing on the region to be restored to obtain restored panoramic image data. Further, the stitched panoramic image data is input into the video restoration model, and the video restoration model performs masking localization on the stitched panoramic image data to determine the region to be restored in the stitched panoramic image data. For the region to be restored in the stitched panoramic image data, spatial restoration processing is performed using the video restoration model to obtain a first restored image. Then, temporal restoration processing is performed on the first restored image using the video restoration model to obtain a second restored image, and the second restored image is determined as the restored panoramic image data. As an example, the video restoration model can be a pre-trained restoration model based on a diffusion model.

[0123] like Figure 6 The diagram shown is a schematic representation of image comparison for multi-view joint restoration according to an embodiment of the method disclosed herein. It includes three columns of comparison images: A, B, and C. The first row of each column contains the original degraded video frame image data, which is the image data to be restored. The second row contains single-view restored video frame images, and the third row contains multi-view coordinated restored video frames (i.e., restoration of stitched panoramic image data). The single-view restored video frame images in the second row, as marked in the rectangular box, are prone to problems such as floating objects, artifacts, and structural distortions. The multi-view coordinated restored video frames in the third row (i.e., restoration of stitched panoramic image data) solve the image problems in the second row, resulting in better image quality and clearer images after restoration.

[0124] Step 105: Optimize the first 3D world based on the repaired panoramic image data to obtain the second 3D world, and generate the target 3D video based on the second 3D world.

[0125] In this embodiment, the repaired stitched panoramic image data (i.e., the repaired panoramic image data) is used as supervisory information to back-optimize the 3D representation (e.g., Gaussian splatting, i.e., the first 3D world). In this way, passive supervision is transformed into active exploration (the repaired panoramic image data from a new perspective), and the ability to model unobserved areas is improved through an iterative mechanism.

[0126] As an optional embodiment, the Gaussian parameters of the first 3D world can be optimized in reverse based on repairing the rendering differences between the panoramic image data and the first 3D world. Optimization of the Gaussian parameters includes, but is not limited to, adding Gaussian spheres (e.g., to fill in missing areas), deleting Gaussian spheres (e.g., to remove floating objects), and moving Gaussian spheres (e.g., to repair structural distortion). For example, the first 3D world corresponds to an ancient architectural scene, which includes brackets (assuming 12840 Gaussian spheres), glazed tiles (assuming 23500 Gaussian spheres), and damaged pillars (assuming 8620 Gaussian spheres). After rendering and stitching the target viewpoint group, an image data with cracks between tiles (Gaussian sphere position deviation) is obtained. This image data with cracks between tiles is repaired to generate a crack-free reference image. The gradient of the cracked area in the reference image is calculated, and the Gaussian spheres in the first 3D world are updated based on this gradient. Specifically, the cracks are closed by moving the Gaussian sphere positions at the tile edges. Finally, the second 3D world obtained after updating the Gaussian spheres is output. After rendering based on this second 3D world, various video frame images are obtained to constitute the target 3D video. In this way, reverse optimization can generate 3D scenes with high visual quality and explorability. Through a closed-loop mechanism of "rendering (image data of the target view group and the first 3D world) - repair (image repair processing of the stitched panoramic image data) - reconstruction (optimization of the first 3D world)," the rendering quality and spatial consistency of the 3D world under new perspectives can be continuously improved.

[0127] To enable those skilled in the art to clearly understand the technical solutions of the embodiments of this disclosure, this disclosure provides, as follows: Figure 5 The full process example diagram shown is as follows: Figure 5 As shown, a rough 3D world (the first 3D world) is initialized. Based on the rough 3D world, new perspective information is searched to realize multi-view new trajectory rendering video, obtaining image data of the target perspective group. This data, along with random Gaussian noise images, is input into a world repairer (a video repairer based on a diffusion model) to perform view repair with multi-view consistency. This involves stitching together the image data from multiple perspectives, then repairing the stitched image data to obtain repaired panoramic image data. The video decoder then decodes the repaired panoramic image data into multiple image data corresponding to each perspective. These image data are then input into, for example, a 3D Gaussian sphere optimizer (3D world optimization) to perform optimization processing on the rough 3D world. The optimized rough 3D world image includes the original trajectory view (the original rough 3D world) and the new trajectory view (i.e., the image data of the target perspective group).

[0128] This disclosure also provides another optional embodiment. Figure 2 This is a second flowchart illustrating a video generation method according to an embodiment of the present disclosure. Figure 2As shown, step 102 in a video generation method may further include:

[0129] Step 1021: Receive any coordinate value input from the user to find the coordinate value from the known viewpoint set, where the known viewpoint set is the observation pose data used in the process of establishing the first three-dimensional world of the target scene.

[0130] Step 1022: In response to the fact that the coordinate value is not found in the known set of viewpoints, the coordinate value is determined to be the first viewpoint.

[0131] In this embodiment of the disclosure, the observation pose data used in establishing the first three-dimensional world of the target scene can be recorded and stored as a set of known viewpoints. The coordinate values ​​input by the user can be represented using pose data. The pose data includes the position and orientation information of an object. The position can be represented by the object's position coordinates (x, y, z) to represent the object's three-dimensional spatial position relative to a reference point. The orientation of the object can be described using three angles (yaw, pitch, and roll) to represent the object's rotation around the X, Y, and Z axes of the reference coordinate system, respectively. It is understood that the position and orientation information of the object can also be represented in other ways (e.g., rotation matrix representation, pose vector, etc.), and this embodiment of the disclosure does not limit this.

[0132] For example, suppose the first 3D world is a pipe model constructed based on 50 panoramic viewpoints, and its corresponding set of known viewpoints stores 50 poses (covering 60% of the pipe's outer wall). The user input coordinate value is, for example, P = [5.7m, -2.1m, 0.3m, 30°, 0°, 0°] (pointing to the back of the valve). As one embodiment, directly search for P in the set of 50 known viewpoints. If it exists, P cannot be used as a new viewpoint (i.e., the first viewpoint); otherwise, P is used as the new viewpoint (i.e., the first viewpoint). As another embodiment, search for the known viewpoint closest to the input coordinate value P in the set of known viewpoints, and then calculate the positional distance and angle deviation between the known viewpoint and P. If the positional distance is greater than or equal to a preset distance threshold and the angle deviation is greater than or equal to a preset angle threshold, then P can be determined as the new viewpoint (i.e., the first viewpoint); otherwise, P is not a new viewpoint (i.e., the first viewpoint).

[0133] To further illustrate, suppose the most recently known viewpoint is found to be [5.6m, -2.0m, 0.4m, 28°, 0°, 0°]. The distance between the user-input coordinate value P and the most recently known viewpoint is √[(0.1)²+(0.1)²+(0.1)²]≈ 0.17m>0.05m (greater than the preset distance threshold), and the angle deviation is |30°-28°|=2° (equal to the preset angle threshold). Therefore, coordinate value P is determined to be the new viewpoint (i.e., the first-person viewpoint).

[0134] Step 1023: Based on the first viewpoint, determine at least one adjacent offset viewpoint of the first viewpoint according to a preset offset rule. The at least one adjacent offset viewpoint is the second viewpoint, and the first viewpoint and the second viewpoint are the target viewpoint group.

[0135] In this embodiment of the disclosure, the preset offset rule may include an angle step size and an offset level. The offset viewpoint can be determined by the first viewpoint, the offset level, and the angle step size, with the latter suitable for obtaining more offset viewpoints. For example, assuming the first viewpoint is θ, the angle step size is β, and the offset level is n (a natural number), then assuming the offset level n is 1, the offset viewpoint α = θ ± nβ, and the target viewpoint group is composed of θ and the offset viewpoint α = θ ± nβ.

[0136] As an optional embodiment, the coordinate values ​​corresponding to the target viewpoint group are recorded and stored in the known viewpoint combination.

[0137] Step 1024: Render images from the first and second perspectives to obtain multiple rendered images, and determine these multiple rendered images as the image data of the target perspective group.

[0138] In this embodiment of the disclosure, for each offset viewpoint, intrinsic parameters (e.g., focal length, principal point) and extrinsic parameters (i.e., pose) of the acquisition device can be set. The intrinsic parameters of the acquisition device remain unchanged, while the extrinsic parameters are adjusted according to the offset angle. Then, for the pose of the acquisition device at each offset viewpoint, a differentiable renderer (e.g., a 3D Gaussian Splatting renderer) is used to generate a rendered image. Exemplarily, the rendering process may include: projecting the 3D Gaussian sphere onto a 2D image plane based on the 3D Gaussian sphere corresponding to each offset viewpoint; sorting the 3D Gaussian sphere according to its depth to the acquisition device; calculating the color of each pixel in a back-to-front order (or using alpha blending); performing rendering for each offset viewpoint pose to obtain a set of rendered images (typically 3 or 5 images, depending on the offset level; the more offset levels and the more viewpoints, the more rendered images). This set of rendered images is then determined as the image data for the target viewpoint group.

[0139] In this disclosure, another embodiment is also provided. Figure 3 This is a flowchart of a video generation method according to an embodiment of the present disclosure, as shown in Figure 3. Figure 3 As shown, step 103 may include the following steps:

[0140] Step 1031: Extract features from each image data of the target viewpoint group to obtain the image features of each viewpoint and construct image feature pairs between each viewpoint.

[0141] For image data from adjacent viewpoints (e.g., θ+α, θ, θ-α), feature detection and extraction can focus on stable local structures across viewpoints. For example, image features include, but are not limited to, corner features, speckle features, edge features, texture features, and material features. Based on the extracted characteristic features of each viewpoint, image feature pairs are established between each viewpoint, and the association between image features (or matching points) across viewpoints is established.

[0142] For example, consider the rendered image data of a wind turbine blade from three perspectives: θ+α, θ, and θ-α. Feature detection and extraction are performed as follows: Left perspective (θ-α): detect the corner point of the leading edge protective strip (SIFT); Main perspective (θ): capture the edge of the bolt hole (Canny+ORB); Right perspective (θ+α): identify the trailing edge wear texture (SURF). For instance, image feature pairs are established between each pair of perspectives. For example, the left perspective (θ-α) searches for image features of the main perspective (θ) using a search tree or similar method, and performs bidirectional feature matching to form an image feature pair between the left perspective (θ-α) and the main perspective (θ), thereby establishing the association and constraint of image features between the left perspective (θ-α) and the main perspective (θ).

[0143] Step 1032: Based on image features, the image data of the target viewpoint group is aligned using a preset image alignment algorithm to obtain aligned image data.

[0144] As an optional embodiment, the preset image alignment algorithm can be a coordinate transformation algorithm, optionally a homography transformation algorithm. Exemplarily, coordinate transformation (e.g., homography transformation) can be used to project three viewpoint image data onto the same imaging plane (not simply coplanar), establishing a projection mapping relationship between viewpoints and achieving coordinate unification of different viewpoints, while being constrained by image feature pairs between viewpoints. For example, an initial homography matrix is ​​established through homography transformation to project the three viewpoint image data onto the same imaging plane. Then, the initial homography matrix is ​​fine-tuned based on the mutual image feature pairs (i.e., the mutual image feature pairs between viewpoints locate stable local structures across viewpoints, providing anchor points for subsequent transformations), resulting in the final homography matrix. This achieves the alignment of image viewpoints between target viewpoints on the same imaging coordinate system plane, outputting aligned image data.

[0145] Step 1033: Use a preset image stitching algorithm to stitch the aligned image data to obtain stitched panoramic image data.

[0146] The stitching process for aligned image data involves fusing image data from various viewpoints to achieve seamless stitching. As an optional implementation, a preset image stitching algorithm can be used to perform frequency domain decomposition on the aligned image data, obtaining the frequency domain signal features of the image data from each viewpoint within the aligned image data. Then, weighted fusion calculations are performed on the frequency domain signal features of the image data from each viewpoint according to preset weighting rules to obtain the stitched panoramic image data. The preset image stitching algorithm can be a multi-band fusion weighting algorithm (e.g., Laplacian pyramid fusion), and the preset weighting rule can be that the weights gradually change as the image approaches the stitching boundary (e.g., weights decrease). This effectively handles edge and detail issues in image stitching, maintaining image clarity and coherence. Through multi-band fusion, weights can be flexibly adjusted across different frequency bands to achieve a more natural image transition effect. This provides a consistent data foundation for image restoration, improving the quality of image restoration.

[0147] As one implementation method, this disclosure also provides other embodiments, Figure 4 This is flowchart four of a video generation method according to an embodiment of the method disclosed herein. Figure 4 As shown, the video generation method disclosed herein may further include the following steps:

[0148] Step 401: Initialize the 3D world reconstruction model and obtain a set of rendered video frame images based on the 3D world reconstruction model. The set of rendered video frame images is a combination of video frame image sequences obtained from 3D world rendering of multiple scenes.

[0149] Step 402: In response to iteratively optimizing the 3D world reconstruction model according to a preset number of iterations, the third video frame image and the fourth video frame image rendered based on the 3D world reconstruction model during the iterative optimization process are acquired according to the preset number of iterations and the preset sampling step size. The third video frame image is the video frame image output before reaching the preset number of iterations, and the fourth video frame image is the video frame image output when the preset number of iterations is reached.

[0150] Step 403: By comparing the third video frame image and the fourth video frame image, the region with defective features in the third video frame image is determined, and a masking mark is made on the region with defective features by a preset occlusion masking mechanism to obtain the third video frame image with masking mark.

[0151] Step 404: The third and fourth video frame images with mask identifiers are identified as the second data pair, and the second data pair corresponding to at least one frame image in the set of rendered video frame images is used to form the training dataset.

[0152] Steps 401 to 404 are used to construct degradation-original video pair datasets for various styles and scenes. Through this embodiment, training samples can be automatically extracted from synthetic and real scenes without manual annotation, covering different types of degradation. This results in a training dataset that enables the video restoration model to have good generalization ability and cross-domain adaptability, thereby improving the overall practicality and stability of the system. For example, 3DGS can be used to reconstruct the scene. This reconstruction process requires continuous iteration to achieve good results. Assuming x iterations (i.e., the preset iteration count is x), during the iteration process, image data rendered from each iteration of the reconstructed scene can be collected. Because the number of iterations is insufficient, these image data may be blurry or of poor quality; therefore, they are defined as degradation video frame images (third video frame images). Image data obtained after reaching the preset iteration count x is of better quality and is determined as the restored video frame image (fourth video frame image).

[0153] like Figure 8 As shown, in the 3D world reconstruction model, the model, physical textures, materials, and lighting conditions in the scene can be identified as digital assets in the 3D world reconstruction process. Based on these digital assets, the 3D world reconstruction of the target scene is performed. Through iterative optimization of the script (including preset iteration counts and iteration steps), after each iteration, the rendering platform (e.g., Gaussian splashing technology) renders the image, obtaining multiple third video frame images. Finally, the third video frame images and the fourth video frame images after iteration constitute multiple second data pairs (as shown in the multi-track rendering frames 1, 2, and 3 in the figure), which then form the training dataset. See also... Figure 7 As shown in the figure, the second data pair is an example. A and A1, B and B1 are two training sample data pairs. Since A and B have not yet completed the iteration, their image quality is relatively poor, so they correspond to the degraded video frame (the third video frame image) respectively. A1 and B1 correspond to the repaired video frame (the fourth video frame image) respectively.

[0154] Step 405: Input the training dataset into the preset video restoration network. The training dataset includes multiple first video frame images with mask markings and their corresponding second video frame images. The first video frame images are video images with defect features in the mask marking regions, and the second video frame images are video images without defect features. The preset video restoration network is a video processing network based on a diffusion model.

[0155] Step 406: The first video frame image with the mask identifier is processed using a preset video restoration network to obtain the predicted video image.

[0156] Step 407: Calculate the preset distance between the predicted video image and the second video frame image to obtain the prediction loss value.

[0157] Step 408: Iteratively optimize the network parameters of the preset video restoration network based on the predicted loss value to obtain the video restoration model.

[0158] In this embodiment, the video restoration model is a diffusion-based restoration module used to eliminate floating objects, artifacts, and structural distortions in 3D world rendering. During the training phase, it learns from a self-constructed multi-scene training dataset (including degradation-original video pairs) and introduces an occlusion masking mechanism to enhance its ability to restore complex areas.

[0159] By generating a rough 3D world (the first 3D world) of the target scene formed by a set of 3D Gaussian spheres, and then automatically finding multiple offset viewpoints at the same coordinate position as new viewpoints, and stitching the image data of the new viewpoints before image restoration, a multi-view joint restoration mechanism is realized. This ensures the geometric and semantic consistency of image data between different viewpoints, and makes the scene structure, texture and semantic information consistent under different viewing angles. This constructs a continuous, coherent and high-quality 3D world, solving the problem of inconsistency in multiple viewpoints in traditional methods, improving the image quality of new viewpoints under large viewpoint changes, eliminating image quality problems such as floating objects, artifacts and structural distortions in the image, making the generated image more stable and realistic, improving spatial coherence, and enhancing the user's immersion and exploration experience. Furthermore, obtaining information from the new viewpoints can improve the modeling ability of unseen areas of the 3D world. In addition, the rough 3D world is updated and optimized using the restored image data, forming a closed-loop modeling from the rough 3D world to the high-quality 3D world (the second 3D world), which improves the integrity of the modeling.

[0160] Correspondingly, this disclosure also provides device embodiments corresponding to the foregoing method embodiments. The device embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. Figure 9 This is a structural block diagram of a video generation apparatus according to one embodiment of the present disclosure, such as... Figure 9 As shown, a video generation apparatus may include:

[0161] The 3D world generation module 901 is used to generate the first 3D world of the target scene. The first 3D world is a set of 3D Gaussian spheres. Each 3D Gaussian sphere includes position coordinate parameters, covariance parameters, and spherical harmonic function coefficients.

[0162] The multi-view image determination module 902 is used to determine image data of a target view group based on the first three-dimensional world, wherein the target view group includes multiple offset viewpoints at the same coordinate position.

[0163] The image stitching module 903 is used to stitch the image data of the target viewpoint group to obtain stitched panoramic image data.

[0164] Image restoration module 904 is used to perform image restoration processing on the stitched panoramic image data to obtain restored panoramic image data;

[0165] The reverse optimization module 905 is used to optimize the first three-dimensional world based on the repaired panoramic image data to obtain a second three-dimensional world, and to generate a target three-dimensional video based on the second three-dimensional world.

[0166] In this embodiment, a rough 3D world (first 3D world) of the target scene formed by a set of 3D Gaussian spheres is generated. Then, by automatically finding multiple offset viewpoints with the same coordinate position as new viewpoints, and stitching the image data of the new viewpoints before image restoration, a multi-view joint restoration mechanism is realized. This ensures the geometric and semantic consistency of image data between different viewpoints, and makes the scene structure, texture, and semantic information consistent under different viewing angles. This constructs a continuous, coherent, and high-quality 3D world, solving the problem of inconsistency among multiple viewpoints in traditional methods. It improves the image quality of new viewpoints under large viewpoint changes, eliminates image quality problems such as floating objects, artifacts, and structural distortions in the image, and makes the generated image more stable and realistic, improving spatial coherence and enhancing the user's immersion and exploration experience. Furthermore, obtaining information from the new viewpoints can improve the modeling ability of unseen areas of the 3D world. In addition, the rough 3D world is updated and optimized using the restored image data, forming a closed-loop modeling from the rough 3D world to the high-quality 3D world (second 3D world), which improves the integrity of the modeling.

[0167] As an optional embodiment, the three-dimensional world generation module includes:

[0168] The first feature extraction unit is used to extract the image semantic feature distribution of the scene image and the text description feature distribution of the scene description prompt text in response to any received scene image and scene description prompt text;

[0169] A three-dimensional coordinate determination unit is used to determine the three-dimensional coordinate value of any pixel in the scene image based on the distribution of the image semantic features.

[0170] The Gaussian sphere creation unit is used to create the first three-dimensional Gaussian sphere for any pixel based on its three-dimensional coordinates, so as to obtain the first three-dimensional Gaussian sphere set for all pixels of the scene image.

[0171] The Gaussian sphere adjustment unit is used to adjust and optimize the Gaussian sphere parameters in the first three-dimensional Gaussian sphere set through the text description feature distribution to obtain a second Gaussian sphere set, and the second Gaussian sphere set constitutes the first three-dimensional world.

[0172] As an optional embodiment, the multi-view image determination module includes:

[0173] A coordinate value receiving unit is used to receive any coordinate value input by the user in order to find the coordinate value from a set of known viewpoints, wherein the set of known viewpoints is the observation pose data used in the process of establishing the first three-dimensional world of the target scene;

[0174] The first-viewpoint determination unit is used to determine the coordinate value as a first-viewpoint in response to the fact that the coordinate value is not found from the known viewpoint set.

[0175] A viewpoint group determination unit is used to determine at least one adjacent offset viewpoint of the first viewpoint according to a preset offset rule based on the first viewpoint, wherein the at least one adjacent offset viewpoint is a second viewpoint, and the first viewpoint and the second viewpoint are the target viewpoint group.

[0176] The graphics rendering unit is used to render images of the first viewpoint and the second viewpoint to obtain multiple rendered images, and to determine the multiple rendered images as the image data of the target viewpoint group.

[0177] As an optional embodiment, the image stitching module includes:

[0178] The second feature extraction unit is used to extract features from each image data of the target viewpoint group, obtain the image features of the image data of each viewpoint, and construct image feature pairs between each viewpoint.

[0179] An image alignment unit is used to align the image data of the target viewpoint group based on the image features using a preset image alignment algorithm to obtain aligned image data.

[0180] The image stitching unit is used to stitch the aligned image data using a preset image stitching algorithm to obtain the stitched panoramic image data.

[0181] As an optional embodiment, the image stitching unit is further configured to:

[0182] The frequency domain decomposition of the aligned image data is performed using the preset image stitching algorithm to obtain the frequency domain signal features of the image data from each viewpoint in the aligned image data.

[0183] According to a preset weighting rule, the frequency domain signal features of the image data from each perspective in the aligned image data are weighted and fused to obtain the stitched panoramic image data.

[0184] As an optional embodiment, the image restoration module includes:

[0185] The image restoration unit is used to input the stitched panoramic image data into the video restoration model, so as to determine the area to be restored in the stitched panoramic image data through the video restoration model, and to perform restoration processing on the area to be restored to obtain restored panoramic image data.

[0186] As an optional embodiment, the image restoration unit is further configured to:

[0187] The stitched panoramic image data is input into a video restoration model, and the video restoration model is used to perform mask positioning on the stitched panoramic image data to determine the area to be restored in the stitched panoramic image data.

[0188] For the area to be repaired in the stitched panoramic image data, spatial repair processing is performed using the video repair model to obtain the first repaired image;

[0189] The first restored image is subjected to temporal restoration processing using the video restoration model to obtain a second restored image, which is then identified as the restored panoramic image data.

[0190] As an optional embodiment, the device further includes:

[0191] The data input module is used to input the training dataset into a preset video restoration network. The training dataset includes multiple first video frame images with mask markings and corresponding second video frame images. The first video frame images are video images with defect features in the mask-marked areas, and the second video frame images are video images without defect features. The preset video restoration network is a video processing network based on a diffusion model.

[0192] The image prediction module is used to process the first video frame image with a mask identifier using the preset video restoration network to obtain a predicted video image;

[0193] The loss calculation module is used to calculate the preset distance between the predicted video image and the second video frame image to obtain the predicted loss value;

[0194] The parameter optimization module is used to iteratively optimize the network parameters of the preset video restoration network based on the predicted loss value, so as to obtain the video restoration model.

[0195] As an optional embodiment, the device further includes:

[0196] The image set acquisition module is used to initialize the three-dimensional world reconstruction model and acquire a set of rendered video frame images based on the three-dimensional world reconstruction model. The set of rendered video frame images is a combination of video frame image sequences obtained from three-dimensional world rendering based on multiple scenes.

[0197] The video frame image acquisition module is used to respond to iteratively optimizing the three-dimensional world reconstruction model according to a preset number of iterations, and to acquire the third video frame image and the fourth video frame image rendered based on the three-dimensional world reconstruction model during the iterative optimization process according to the preset number of iterations and the preset sampling step size. The third video frame image is the video frame image output before reaching the preset number of iterations, and the fourth video frame image is the video frame image output when the preset number of iterations is reached.

[0198] The first data pair forming module is used to determine the region with defective features in the third video frame image by comparing the third video frame image and the fourth video frame image, and to mark the region with defective features by a preset occlusion masking mechanism to obtain a third video frame image with masking identification.

[0199] The data pair set determination module is used to determine the third video frame image with the mask identifier and the fourth video frame image as a second data pair, and the second data pair corresponding to at least one frame image in the set of rendered video frame images constitutes the training dataset.

[0200] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this disclosure. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0201] Below, for reference Figure 10This describes an electronic device according to embodiments of the present disclosure. The electronic device may be either or both of a first device and a second device, or a standalone device independent of them, which may communicate with the first device and the second device to receive acquired input signals from them.

[0202] Figure 10 A block diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0203] like Figure 10 As shown, the electronic device includes one or more processors and memory.

[0204] A processor can be a central processing unit (CPU) or other form of processing unit with data processing and / or instruction execution capabilities, and can control other components in an electronic device to perform desired functions.

[0205] The memory can store one or more computer program products, and the memory can include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program products can be stored on the computer-readable storage medium, and the processor can run the computer program products to implement the video generation methods of the various embodiments of this disclosure described above and / or other desired functions.

[0206] In one example, the electronic device may also include input devices and output devices, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0207] In addition, the input device may also include, for example, a keyboard, a mouse, etc.

[0208] This output device can output various information to the outside, including determined distance information, direction information, etc. The output device may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0209] Of course, for the sake of simplicity, Figure 10 Only some of the components of the electronic device relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device may include any other suitable components depending on the specific application.

[0210] In addition to the methods and apparatus described above, embodiments of this disclosure may also be computer program products comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps in the video generation methods according to various embodiments of this disclosure as described in the foregoing portion of this specification.

[0211] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0212] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps in the video generation methods according to various embodiments of this disclosure as described in the foregoing portion of this specification.

[0213] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0214] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0215] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0216] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0217] The methods and apparatus of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.

[0218] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.

[0219] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0220] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A video generation method, characterized in that, The method includes: Generate the first three-dimensional world of the target scene. The first three-dimensional world is a set of three-dimensional Gaussian spheres. Each three-dimensional Gaussian sphere includes position coordinate parameters, covariance parameters, and spherical harmonic function coefficients. Based on the first three-dimensional world, image data of a target view group is determined. The target view group includes multiple offset viewpoints at the same coordinate position, and the target view group is the viewpoint corresponding to the observation pose data that was not used in the process of establishing the first three-dimensional world of the target scene. The image data of the target viewpoint group are stitched together to obtain stitched panoramic image data. The stitched panoramic image data is subjected to image restoration processing to obtain restored panoramic image data; The first 3D world is optimized based on the repaired panoramic image data to obtain the second 3D world, and the target 3D video is generated based on the second 3D world; The first three-dimensional world of the generated target scene includes: In response to any received scene image and scene description prompt text, extract the image semantic feature distribution of the scene image and the text description feature distribution of the scene description prompt text; For any pixel in the scene image, determine the three-dimensional coordinate value of the pixel based on the image semantic feature distribution; Create a first three-dimensional Gaussian sphere for any pixel based on its three-dimensional coordinates, so as to obtain a set of first three-dimensional Gaussian spheres for all pixels in the scene image; By adjusting and optimizing the Gaussian sphere parameters in the first three-dimensional Gaussian sphere set through the text description feature distribution, a second Gaussian sphere set is obtained, and the second Gaussian sphere set constitutes the first three-dimensional world.

2. The method according to claim 1, characterized in that, The image data for determining a target viewpoint group based on the first three-dimensional world, wherein the target viewpoint group includes multiple offset viewpoints at the same coordinate position, including: Receive any coordinate value input from the user to find the coordinate value from a set of known viewpoints, wherein the set of known viewpoints is the observation pose data used in the process of establishing the first three-dimensional world of the target scene; In response to the fact that the coordinate value is not found from the known set of viewpoints, the coordinate value is determined to be a first viewpoint; Based on the first viewpoint, at least one adjacent offset viewpoint is determined according to a preset offset rule. The at least one adjacent offset viewpoint is a second viewpoint, and the first viewpoint and the second viewpoint are the target viewpoint group. Image rendering is performed on the first viewpoint and the second viewpoint to obtain multiple rendered images, and these multiple rendered images are determined as the image data of the target viewpoint group.

3. The method according to claim 1, characterized in that, The step of stitching together the image data of the target viewpoint group to obtain stitched panoramic image data includes: Feature extraction is performed on each image data of the target viewpoint group to obtain the image features of the image data of each viewpoint, and image feature pairs between each viewpoint are constructed. Based on the image features, the image data of the target viewpoint group is aligned using a preset image alignment algorithm to obtain aligned image data. The aligned image data is stitched together using a preset image stitching algorithm to obtain the stitched panoramic image data.

4. The method according to claim 3, characterized in that, The step of stitching the aligned image data using a preset image stitching algorithm to obtain the stitched panoramic image data includes: The frequency domain decomposition of the aligned image data is performed using the preset image stitching algorithm to obtain the frequency domain signal features of the image data from each viewpoint in the aligned image data. According to a preset weighting rule, the frequency domain signal features of the image data from each perspective in the aligned image data are weighted and fused to obtain the stitched panoramic image data.

5. The method according to claim 1, characterized in that, The step of performing image restoration processing on the stitched panoramic image data to obtain restored panoramic image data includes: The stitched panoramic image data is input into a video restoration model to determine the area to be restored in the stitched panoramic image data, and the area to be restored is then restored to obtain restored panoramic image data.

6. The method according to claim 5, characterized in that, The step of inputting the stitched panoramic image data into a video restoration model to determine the area to be restored in the stitched panoramic image data, and performing restoration processing on the area to be restored to obtain restored panoramic image data includes: The stitched panoramic image data is input into a video restoration model, and the video restoration model is used to perform mask positioning on the stitched panoramic image data to determine the area to be restored in the stitched panoramic image data. For the area to be repaired in the stitched panoramic image data, spatial repair processing is performed using the video repair model to obtain the first repaired image; The first restored image is subjected to temporal restoration processing using the video restoration model to obtain a second restored image, which is then identified as the restored panoramic image data.

7. The method according to claim 5, characterized in that, The method further includes: The training dataset is input into a preset video restoration network. The training dataset includes multiple first video frame images with mask markings and corresponding second video frame images. The first video frame images are video images with defect features in the mask marking areas, and the second video frame images are video images without defect features. The preset video restoration network is a video processing network based on a diffusion model. The first video frame image with a mask identifier is processed using the preset video restoration network to obtain a predicted video image; Calculate the preset distance between the predicted video image and the second video frame image to obtain the prediction loss value; The video restoration model is obtained by iteratively optimizing the network parameters of the preset video restoration network based on the predicted loss value.

8. The method according to claim 7, characterized in that, The method further includes: Initialize a 3D world reconstruction model and obtain a set of rendered video frame images based on the 3D world reconstruction model. The set of rendered video frame images is a combination of video frame image sequences obtained from 3D world rendering of multiple scenes. In response to iteratively optimizing the 3D world reconstruction model according to a preset number of iterations, the third video frame image and the fourth video frame image rendered based on the 3D world reconstruction model during the iterative optimization process are collected according to the preset number of iterations and the preset sampling step size. The third video frame image is the video frame image output before reaching the preset number of iterations, and the fourth video frame image is the video frame image output when the preset number of iterations is reached. By comparing the third video frame image and the fourth video frame image, the region with defective features in the third video frame image is determined, and a masking mark is made on the region with defective features by a preset masking mechanism to obtain a third video frame image with masking mark. The third video frame image with the mask identifier and the fourth video frame image are identified as the second data pair, and the second data pair corresponding to at least one frame image in the set of rendered video frame images constitutes the training dataset.

9. A video generation apparatus, characterized in that, The device includes: The 3D world generation module is used to generate the first 3D world of the target scene. The first 3D world is a set of 3D Gaussian spheres. Each 3D Gaussian sphere includes position coordinate parameters, covariance parameters, and spherical harmonic function coefficients. A multi-view image determination module is used to determine image data of a target view group based on the first three-dimensional world. The target view group includes multiple angle offset views at the same coordinate position, and the target view group is the view corresponding to the observation pose data that was not used in the process of establishing the first three-dimensional world of the target scene. The image stitching module is used to stitch together the image data of the target viewpoint group to obtain stitched panoramic image data. The image restoration module is used to perform image restoration processing on the stitched panoramic image data to obtain restored panoramic image data. The reverse optimization module is used to optimize the first three-dimensional world based on the repaired panoramic image data to obtain the second three-dimensional world, and to generate the target three-dimensional video based on the second three-dimensional world; The three-dimensional world generation module is specifically used for: In response to any received scene image and scene description prompt text, extract the image semantic feature distribution of the scene image and the text description feature distribution of the scene description prompt text; For any pixel in the scene image, determine the three-dimensional coordinate value of the pixel based on the image semantic feature distribution; Create a first three-dimensional Gaussian sphere for any pixel based on its three-dimensional coordinates, so as to obtain a set of first three-dimensional Gaussian spheres for all pixels in the scene image; By adjusting and optimizing the Gaussian sphere parameters in the first three-dimensional Gaussian sphere set through the text description feature distribution, a second Gaussian sphere set is obtained, and the second Gaussian sphere set constitutes the first three-dimensional world.

10. An electronic device, characterized in that, include: Memory, used to store computer program products; A processor for executing a computer program product stored in the memory, wherein when the computer program product is executed, it implements the method described in any one of claims 1-8.

11. A computer-readable storage medium storing computer program instructions thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1-8.

12. A computer program product comprising computer program instructions, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1-8.

Images