Model training method, electronic equipment and computer readable storage medium

Abstract

The invention discloses a model training method, electronic equipment and a computer readable storage medium, and relates to the technical field of large model and image generation. The method comprises the steps that training data are acquired, the training data comprise a reference image set, a first camera pose set and a second camera pose set, and the first camera pose set is different from the second camera pose set; obtaining prior information based on the reference image set, the first camera pose set and the second camera pose set; the initial multi-view-angle generation model is trained through the reference image set, the prior information, the first camera pose set, the second camera pose set and preset mask information, a target multi-view-angle generation model is obtained, and the target multi-view-angle generation model is used for conducting multi-view-angle generation on an input original-view-angle image to obtain an expanded-view-angle image. The technical problem that a three-dimensional multi-view-angle generation algorithm in related technologies cannot reach a good generation level in the aspect of scene-level generation is solved.

Description

Technical Field

[0001] The present application relates to the field of large models and image generation technology, and more specifically, to a model training method, an electronic device, and a computer-readable storage medium. Background Art

[0002] With the rise of artificial intelligence generated content (AIGC) technology, three-dimensional (3D) multi-view generation algorithms have experienced significant progress. However, although 3D multi-view generation algorithms have achieved remarkable results in object-level generation, they are still limited in scene-level generation due to task complexity, the quality of available multi-view data, and the limitations of open source models, and cannot achieve a good generation level.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present application provide a model training method, an electronic device and a computer-readable storage medium to at least solve the technical problem that the three-dimensional multi-view generation algorithm in the related technology cannot achieve a good generation level at the scene level.

[0005] According to one aspect of an embodiment of the present application, a model training method is provided, comprising: obtaining training data, wherein the training data comprises: a reference image set, a first camera pose set, and a second camera pose set, the first camera pose set being different from the second camera pose set; obtaining prior information based on the reference image set, the first camera pose set, and the second camera pose set; training an initial multi-view generation model using the reference image set, the prior information, the first camera pose set, the second camera pose set, and preset mask information to obtain a target multi-view generation model, wherein the target multi-view generation model is used to perform multi-view generation on an input original view image to obtain an extended view image.

[0006] According to another aspect of an embodiment of the present application, an image generation method is also provided, including: obtaining an original perspective image; performing multi-perspective generation on the original perspective image using a target multi-perspective generation model to obtain an extended perspective image; wherein the target multi-perspective generation model is obtained using any one of the above-mentioned model training methods.

[0007] According to another aspect of an embodiment of the present application, an image generation method is also provided, including: obtaining an image generation request through a first application programming interface, wherein the request data carried in the image generation request includes: an original perspective image; returning an image generation response through a second application programming interface, wherein the response data carried in the image generation response includes: an extended perspective image, wherein the extended perspective image is obtained by performing multi-perspective generation of the original perspective image using a target multi-perspective generation model, and the target multi-perspective generation model is obtained using any one of the above-mentioned model training methods.

[0008] According to another aspect of an embodiment of the present application, an image generation method is also provided, including: obtaining a currently input image generation dialogue request, wherein the request data carried in the image generation dialogue request includes: an original perspective image; in response to the image generation dialogue request, returning an image generation dialogue reply, wherein the information carried in the image generation dialogue reply includes: an extended perspective image, wherein the extended perspective image is obtained by performing multi-perspective generation of the original perspective image using a target multi-perspective generation model, wherein the target multi-perspective generation model is obtained using any one of the above-mentioned model training methods; and displaying the extended perspective image in a graphical user interface.

[0009] According to another aspect of an embodiment of the present application, an electronic device is also provided, including: a memory storing an executable program; and a processor for running the program, wherein when the program is running, any one of the above-mentioned model training methods or image generation methods is executed.

[0010] According to another aspect of an embodiment of the present application, a computer-readable storage medium is also provided, which includes a stored executable program, wherein when the executable program is running, the device where the computer-readable storage medium is located is controlled to execute any one of the above-mentioned model training methods or image generation methods.

[0011] According to another aspect of an embodiment of the present application, a computer program product is also provided, including a computer program, which implements any one of the above-mentioned model training methods or image generation methods when executed by a processor.

[0012] In an embodiment of the present application, by obtaining training data, that is, obtaining a reference image set, a first camera pose set, and a second camera pose set, and then obtaining prior information based on the reference image set, the first camera pose set, and the second camera pose set, finally using the reference image set, the prior information, the first camera pose set, the second camera pose set, and the preset mask information to train the initial multi-view generation model, a target multi-view generation model is obtained, and the target multi-view generation model is used to perform multi-view generation on the input original view image to obtain an extended view image, thereby achieving the purpose of training a high-quality multi-view generation model, maximizing the use of 3D prior information, view transformation data, and mask information, thereby significantly improving the quality of image generation, the generalization ability of view transformation, and the processing of complex scenes, and achieving the technical effect of generating high-quality scene-level multi-view images, thereby solving the technical problem that the three-dimensional multi-view generation algorithm in the related technology cannot achieve a good generation level at the scene level.

[0013] It is easy to notice that the above general description and the following detailed description are only for the purpose of exemplifying and explaining the present application, and do not constitute a limitation of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0015] Figure 1 is a schematic diagram of an application scenario of a model training method according to an embodiment of the present application;

[0016] Figure 2 is a flow chart of a model training method according to an embodiment of the present application;

[0017] Figure 3 is a diagram of a metric depth alignment process according to an embodiment of the present application;

[0018] Figure 4 is a model reasoning flow chart according to an embodiment of the present application;

[0019] Figure 5 is a flow chart of an image generating method according to an embodiment of the present application;

[0020] Figure 6 is a flow chart of an image generating method according to an embodiment of the present application;

[0021] Figure 7 is a flow chart of an image generating method according to an embodiment of the present application;

[0022] Figure 8is a structural schematic diagram of a model training device according to an embodiment of the present application;

[0023] Fig. 9 is a structural schematic diagram of an image generating device according to an embodiment of the present application;

[0024] Fig.10 is a structural schematic diagram of another image generating device according to an embodiment of the present application;

[0025] Fig.11 is a structural schematic diagram of another image generating device according to an embodiment of the present application;

[0026] Fig.12 It is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0028] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0029] The technical solution provided in this application is mainly implemented by large-scale model technology. The large model here refers to a deep learning model with large-scale model parameters, which can usually contain hundreds of millions, tens of billions, hundreds of billions, trillions or even more than 10 trillion model parameters. The large model can also be called a cornerstone model / foundation model (Foundation Model). The large model is pre-trained through large-scale unlabeled corpus to produce a pre-trained model with more than 100 million parameters. This model can adapt to a wide range of downstream tasks, and the model has good generalization ability, such as large-scale language model (Large Language Model, LLM), multi-modal pre-training model (multi-modal pre-training model), etc.

[0030] It should be noted that when the large model is actually used, the pre-trained model can be fine-tuned through a small number of samples so that the large model can be applied to different tasks. For example, the large model can be widely used in natural language processing (NLP), computer vision, speech processing and other fields, and can be specifically applied to computer vision tasks such as visual question answering (VQA), image caption (IC), image generation, etc. It can also be widely used in text-based sentiment classification, text summary generation, machine translation and other natural language processing tasks. Therefore, the main application scenarios of the large model include but are not limited to digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, etc. In the embodiment of the present application, multi-view image generation is performed by a trained multi-view generation model in an image generation scenario as an example for explanation.

[0031] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following explanations:

[0032] Fourier Embedding: Also known as Fourier Feature Mapping, it is a technique used in neural networks and deep learning models to enhance the model's representation capabilities, especially when processing data with periodic or frequency characteristics. The core idea is to use the properties of Fourier transform to map the input data to a higher-dimensional space, which can effectively transform the linear model into a nonlinear model, thereby enhancing the model's expressive power.

[0033] Random Sample Consensus (RANSAC) correction: is a commonly used algorithm in computer vision and image processing, which is used to estimate model parameters from noisy data, especially when the data contains outliers, it can effectively fit the model. In scenarios such as 3D reconstruction, multi-view stereo matching, and camera calibration, RANSAC is often used to extract the most suitable parameter set for the model from a set of point clouds, depth maps, feature points, or camera poses.

[0034] Structure from Motion (SFM): is a 3D reconstruction technique widely used in the field of computer vision. It mainly estimates the structure of the 3D scene and the motion parameters of the camera by analyzing the motion changes of objects in a series of image sequences. That is, by analyzing a series of 2D images, the 3D scene is reconstructed.

[0035] Colmap: A classic SFM algorithm.

[0036] Dust3R: A well-known SFM algorithm.

[0037] AI-generated content (AIGC): various types of content generated using AI technology, such as text, images, audio, and video.

[0038] 3D generation algorithms: Generate various 3D representations through generative models, including but not limited to generating multi-view consistent images, meshes, and point clouds of the target based on the camera pose.

[0039] 3D reconstruction algorithm: 3D reconstruction is performed based on the known camera pose and the corresponding multi-view images. Related algorithms include but are not limited to the Neural Radiance Fields (NeRF) and 3D Generative Synthesis (3DGS) algorithms.

[0040] Metric depth: General depth (monocular depth) is scale-agnostic and only provides relative depth relationships, while metric depth provides absolute depth.

[0041] Multi-view stereo (MVS): It is used to obtain the corresponding metric depth map by matching the features of multiple viewpoints based on the known camera pose.

[0042] Plücker ray: A dense representation of camera pose, which is the cross product of the dense vector from the optical center to the image plane and the vector from the origin to the optical center.

[0043] Exponential Moving Average (EMA): During the model training process, after each optimization step, the weighted sum of the current optimized model and all previous model weights is taken, which is equivalent to model integration to improve stability.

[0044] Models trained based on traditional 3D multi-view generation algorithms are limited in scene-level generation due to task complexity, available multi-view data quality, and open source model limitations, and cannot achieve a good generation level. For example, the ZeroNVS model relies on sample-by-sample optimization (Sequential Deep Synthesis loss, SDSloss), and the generation effect is poor. Another example is the MotionCtrl model, which is based on a video model. Due to the limited training data set, the change in viewpoint is greatly restricted, and the generation effect is also poor.

[0045] The model obtained by training the related technology based on the traditional 3D multi-view generation algorithm has the following defects.

[0046] Defect 1: The model trained based on the traditional 3D multi-view generation algorithm is limited by the difficulty of the generation task, the quality of multi-view data and the capabilities of the open source model, and cannot achieve a good generation level.

[0047] With respect to the above-mentioned defects, no effective solution has been proposed before the present application.

[0048] According to an embodiment of the present application, a model training method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0049] Considering the huge amount of model parameters of the large model and the limited computing resources of the mobile terminal, the above model training method provided in the embodiment of the present application can be applied to Figure 1 The application scenarios shown are not limited to these. Figure 1In the application scenario shown, the large model is deployed in the server 10, and the server 10 can be connected to one or more client devices 20 through a local area network connection, a wide area network connection, an Internet connection, or other types of data networks. The client device 20 here may include but is not limited to: a smart phone, a tablet computer, a laptop computer, a PDA, a personal computer, a smart home device, a vehicle-mounted device, etc. The client device 20 can interact with the user through a graphical user interface to implement the call of the large model, thereby implementing the method provided in the embodiment of the present application.

[0050] In an embodiment of the present application, a system composed of a client device and a server can perform the following steps: the client device executes steps of acquiring training data, such as acquiring training data such as a reference image set, a first camera pose set, and a second camera pose set, and sending the training data to the server; the server executes steps of acquiring prior information based on the acquired reference image set, the first camera pose set, and the second camera pose set, and training an initial multi-view generation model using the reference image set, the prior information, the first camera pose set, the second camera pose set, and the preset mask information to obtain a target multi-view generation model for performing multi-view generation on an input original view image to obtain an extended view image, and returning the target multi-view generation model to the client device. It should be noted that, when the operating resources of the client device can meet the deployment and operating conditions of the large model, the embodiment of the present application can be performed in the client device.

[0051] It should be noted that, with the rapid development of high-performance computing units, in other application scenarios, the above method provided in the embodiment of the present application can also be applied to the model all-in-one machine. In an optional embodiment, a plurality of models are built into the model all-in-one machine, and the user can choose to adjust a model as needed to obtain the user's own model, so that the high-performance computing unit built into the model all-in-one machine can directly call the adjusted model to execute the above method provided in the embodiment of the present application. In another optional embodiment, a trained model is built into the large model all-in-one machine, so that the high-performance computing unit built into the model all-in-one machine can directly call the model to execute the above method provided in the embodiment of the present application.

[0052] Furthermore, when users need to train their own models, they can also upload their own data sets through the client, which are sent by the client to the server, so that the server can adjust the pre-trained model with the data set to obtain the user's own model and then deploy it to the production environment. In order to facilitate users' needs for model adjustment, the server can provide complete adjustment tools, development frameworks and processes, and can support multiple adjustment strategies, so that the adjusted model can better adapt to applications in different fields and achieve a high degree of customization.

[0053] Under the above operating environment, this application provides Figure 2 The model training method shown. Figure 2 is a flow chart of a model training method according to an embodiment of the present application. Figure 2 As shown, the method may include the following steps:

[0054] Step S21, acquiring training data, wherein the training data includes: a reference image set, a first camera pose set, and a second camera pose set, wherein the first camera pose set is different from the second camera pose set;

[0055] Step S22, acquiring prior information based on the reference image set, the first camera pose set, and the second camera pose set;

[0056] Step S23, using the reference image set, the prior information, the first camera pose set, the second camera pose set and the preset mask information to train the initial multi-view generation model to obtain a target multi-view generation model, wherein the target multi-view generation model is used to perform multi-view generation on the input original perspective image to obtain an extended perspective image.

[0057] In an embodiment of the present application, a reference image set may be understood as a multi-view image data set, i.e., images of the same scene taken from different viewpoints. The reference image set includes multi-view images of multiple scenes, thereby providing comprehensive information about the scene appearance and geometric structure for subsequent model training.

[0058] It should be noted that, in addition to the current mainstream multi-view image datasets, the reference image set of this application further collects and acquires a large amount of available high-quality multi-view data, such as aerial photography datasets, indoor scene datasets, object datasets, natural and scenic scene datasets, 360° street panoramas, etc., which are not limited here. The actual total data volume of the reference image set of this application can reach more than 4 times the amount of data used for traditional model training, or even more.

[0059] The first camera pose set and the second camera pose set are sets of camera position coordinates and camera rotation angles of two different perspectives, which are used to accurately describe the pose of the camera in three-dimensional space. Among them, the first camera pose set is a camera pose set of a reference view, that is, a camera pose set of a known perspective, which can be understood as the camera pose of the camera corresponding to the image that has been taken or acquired when shooting.

[0060] The second camera pose set can be understood as the camera pose set of the target view, which can be understood as the camera pose that the user wants or the task requires to observe the scene from a specific position and direction. The target view can be a new view that has never been taken, or a specific, user-defined view, which is not limited here.

[0061] In an embodiment of the present application, during the model training process, a reference image set, a first camera pose set, and a second camera pose set are obtained, that is, a large number of available multi-view image data sets are obtained, as well as a camera pose set under a reference perspective and a camera pose set under a target perspective. It can be seen that the reference image set provides multi-view images of different scenes, different objects, and different conditions, so that the model can learn a wide range of changes and patterns during training, so that it can better generalize and adapt when encountering new scenes. The first camera pose set and the second camera pose set help the model understand the spatial relationship and perspective changes between images. By associating images of different perspectives with their corresponding camera poses, the model can learn how perspective changes affect image content, thereby maintaining consistency and accuracy in the three-dimensional space when generating new perspective images.

[0062] After obtaining the reference image set, the first camera pose set, and the second camera pose set, the embodiment of the present application introduces 3D priors for the obtained data, that is, based on the reference image set, the first camera pose set, and the second camera pose set, prior information is obtained. The prior information is the 3D prior, which helps the model or algorithm to make more accurate predictions or generate additional knowledge. The prior information is used to guide and constrain tasks such as 3D reconstruction, multi-view image generation, and scene understanding to improve the accuracy and efficiency of the algorithm.

[0063] It can be seen that the present application can generate images containing 3D spatial position information based on the reference image set, the first camera pose set and the second camera pose set, which provides key 3D prior knowledge for the subsequent multi-view image data and 3D reconstruction, thereby being able to guide and enhance the effect of scene-level multi-view image generation based on 3D prior. By introducing 3D prior, combined with camera pose and image content, it can help the model to more accurately understand and generate new perspective images that conform to the 3D spatial structure.

[0064] After obtaining the prior information, the embodiment of the present application uses the reference image set, the prior information, the first camera pose set, the second camera pose set, and the preset mask information to train the initial multi-view generation model to obtain the target multi-view generation model. The preset mask information can be understood as a mask, such as a binary mask, which is used to distinguish the reference features under the reference perspective and the target features under the target perspective, thereby helping the algorithm to accurately identify and distinguish the two.

[0065] The initial multi-view generation model can be understood as a multi-view generation model before training, and the target multi-view generation model can be understood as a multi-view generation model obtained after model training of the initial multi-view generation model. Exemplarily, the initial multi-view generation model and the target multi-view generation model can be large models or other deep learning models, which are not limited here.

[0066] The initial multi-view generation model is trained using a reference image set, prior information, a first camera pose set, a second camera pose set, and preset mask information. This can be understood as training the initial multi-view generation model through a training data set containing visual information, 3D priors, camera poses under multiple perspectives, and preset masks, so that the initial multi-view generation model can learn how to transform from one perspective to another, thereby generating new visual content, and finally obtaining the target multi-view generation model.

[0067] The target multi-view generation model is used to perform multi-view generation on the input original view image to obtain an extended view image, that is, the target multi-view generation model can perform multi-view generation (multi-viewsynthesis, MV synthesis) on the input original view image, thereby obtaining a series of images under new view angles, namely, extended view images. It can be understood that the target multi-view generation model trained in this application can receive an original view image as input, that is, receive an existing scene image taken from a specific view angle, and then the model will use the input original view image, combined with its built-in generation mechanism and 3D prior knowledge, to synthesize or generate a series of images under new view angles, namely, extended view images (novelviews), thereby expanding the observation angle and visual information of the same scene.

[0068] In an embodiment of the present application, by obtaining training data, that is, obtaining a reference image set, a first camera pose set, and a second camera pose set, then obtaining prior information based on the reference image set, the first camera pose set, and the second camera pose set, finally using the reference image set, the prior information, the first camera pose set, the second camera pose set, and the preset mask information to train the initial multi-view generation model, a target multi-view generation model is obtained, and the target multi-view generation model is used to generate multi-views for the input original view image to obtain an extended view image. Thus, by obtaining reference image sets from multiple sources, the diversity and quantity of training data are increased, which helps the model learn a wider and deeper scene representation, thereby improving the diversity and adaptability of its generated images. In addition, the first camera pose set and the second camera pose set are used for model training, so that the model can learn and adapt to a wide range of perspective changes. At the same time, using prior information for model training, the model can better understand the three-dimensional structure and spatial layout of the scene, which helps the model to accurately handle occlusion, perspective changes, and object position relationships when generating new perspective images, thereby generating new perspective images that are more visually coherent and realistic. In addition, by using prior information and preset masks, the generation results of the model are no longer solely dependent on the image input of a single perspective, thereby enhancing the robustness of the model to the input data, so that it can still stably generate high-quality new perspective images when faced with reference images of different qualities.

[0069] The above-mentioned model training method provided in the embodiment of the present application can be applied to, but is not limited to, application scenarios involving multi-perspective generation model training in the fields of e-commerce services, educational services, legal services, medical services, conference services, social network services, financial product services, logistics services and navigation services, for example: multi-perspective generation model training for e-commerce services, multi-perspective generation model training for educational services, multi-perspective generation model training for legal services, etc., which are not limited here.

[0070] By adopting the embodiment of the present application, by obtaining training data, that is, obtaining a reference image set, a first camera pose set and a second camera pose set, then based on the reference image set, the first camera pose set and the second camera pose set, obtaining prior information, and finally using the reference image set, the prior information, the first camera pose set, the second camera pose set and the preset mask information to train the initial multi-view generation model, a target multi-view generation model is obtained, and the target multi-view generation model is used to perform multi-view generation on the input original view image to obtain an extended view image, thereby achieving the purpose of training a high-quality multi-view generation model, maximizing the use of 3D prior information, view transformation data and mask information, thereby significantly improving the quality of image generation, the generalization ability of view transformation and the processing of complex scenes, and achieving the technical effect of generating high-quality scene-level multi-view images, thereby solving the technical problem that the three-dimensional multi-view generation algorithm in the related technology cannot achieve a good generation level at the scene level.

[0071] In an optional embodiment, in step S22, obtaining prior information based on the reference image set, the first camera pose set, and the second camera pose set includes the following method steps:

[0072] Step S221, using the monocular depth map of the reference image set to obtain a metric depth map of the reference image set, and using the color pixel values ​​of the reference image set to obtain a unified spatial coordinate corresponding to the color pixel values;

[0073] Step S222, performing back projection based on the color pixel values, the unified space coordinates, the metric depth map, the relative viewing angle between the first camera pose set and the second camera pose set, the intrinsic parameter matrix corresponding to the first camera pose set, and the intrinsic parameter matrix corresponding to the second camera pose set to obtain prior information, wherein the prior information includes: the projected color pixel values ​​and the projected unified space coordinates.

[0074] In an embodiment of the present application, when obtaining prior information based on a reference image set, a first camera pose set, and a second camera pose set, a metric depth map of the reference image set can be obtained using a monocular depth map of the reference image set. Among them, the monocular depth map of the reference image set is used to analyze a single red, green, and blue (RGB) image to estimate the scene depth of the image, that is, to obtain a metric depth map of the reference image set. The metric depth map of the reference image set is used to provide an absolute value of the scene depth, which is helpful for accurate 3D reconstruction and helps the model correct depth estimation, thereby generating more accurate multi-view images.

[0075] At the same time, the color pixel values ​​of the reference image set can be used to obtain the unified spatial coordinates corresponding to the color pixel values, where the color pixel values ​​(RGB Pixels) of the reference image set can be understood as the RGB pixels of the reference image set. The unified spatial coordinates (Canonical Coordinate Maps, CCM) corresponding to the color pixel values ​​can be understood as the process of mapping all pixel points in the reference image set to a unified three-dimensional spatial coordinate system. Therefore, even if the pixel points are located at different image positions under different perspectives, the position of the pixel points in the unified spatial coordinate system can also reflect their relative position relationship in the real scene, thereby helping the model understand the position of the pixel points in three-dimensional space, and correctly place and render colors when generating new perspective images.

[0076] After that, back-projection is performed based on the color pixel values, unified spatial coordinates, metric depth map, relative viewing angle between the first camera pose set and the second camera pose set, the intrinsic parameter matrix corresponding to the first camera pose set, and the intrinsic parameter matrix corresponding to the second camera pose set to obtain prior information. Among them, the prior information includes the projected color pixel values ​​(warped RGB pixels) and the projected unified spatial coordinates (warped canonical coordinate maps). The projected color pixel values ​​can be recorded as The unified space coordinates after projection can be recorded as

[0077] Exemplarily, the above back-projection process can be expressed by formula (1).

[0078]

[0079] Among them, the color pixel value can be recorded as I i , the unified space coordinate can be recorded as C i , C i It is obtained by back-projecting the two-dimensional coordinates into a standardized three-dimensional unified space and then encoding it through Fourier embedding. The metric depth map can be recorded as The known perspective represented by the first camera pose set can be recorded as perspective i, and the target perspective represented by the second camera pose set can be recorded as perspective j. The relative perspective between the first camera pose set and the second camera pose set is the relative viewpoint of the camera from perspective i to perspective j, which can be recorded as P i→j , the intrinsic parameter matrix corresponding to the first camera pose set can be recorded as K i , the internal parameter matrix corresponding to the second camera pose set can be recorded as K j .

[0080] Warp(·) is a geometric transformation function that can transform an image from one coordinate system to another. In this application, warp(·) can be used to measure the depth map. Back-project the pixels of view i to view j. Thus, the relationship between the two-dimensional coordinates can be expressed as formula (2).

[0081]

[0082] Among them, x j Represents x i The two-dimensional position in view j. It should be noted that It must be a metric depth, not a relative depth aligned with the SFM's extrinsic matrix translation.

[0083] Through the above back-projection process, after each pixel point is mapped to the three-dimensional space, these points are re-projected back to the new two-dimensional image plane according to the camera posture of the target perspective, so as to obtain the color value of each pixel point at the new perspective, that is, the projected color pixel value Should It helps the model understand the color changes of the same 3D position under different viewing angles, and obtain the spatial coordinates of each pixel under the new viewing angle, that is, the unified spatial coordinates after projection. Should Helps to handle complex scene elements such as occlusion and perspective changes.

[0084] Therefore, obtaining this prior information through back-projection can help the model consider not only the color information but also the position information of the pixels in 3D space when generating new perspective images, as well as the depth and spatial layout changes caused by perspective changes. Therefore, the multi-perspective generation method based on 3D prior proposed in this application helps to generate more coherent images that are more in line with the physical rules of the real world, especially in tasks such as dynamic perspective transformation, complex scene understanding, and 3D content editing.

[0085] In an optional embodiment, in step S221, obtaining a metric depth map of a reference image set using a monocular depth map of a reference image set includes the following method steps:

[0086] Step S2211, obtaining a depth map to be aligned;

[0087] Step S2212: performing image correction on the monocular depth based on the depth map to be aligned to obtain a metric depth map.

[0088] This application takes into account that monocular depth often has better generalization and provides very dense depth information, which can provide more valuable 3D priors. At the same time, considering that metric depth is often very sparse but more accurate, combining the two can complement each other and build a better quality 3D prior.

[0089] Therefore, in an embodiment of the present application, when using the monocular depth map of the reference image set to obtain the metric depth map of the reference image set, the depth map to be aligned can be obtained, such as colmap SFM points, MVS depth, Dust3R depth / point cloud, etc., which is not limited here.

[0090] Then, the monocular depth is corrected based on the depth map to be aligned to obtain a metric depth map, which can be understood as performing RANSAC correction on the monocular depth based on the depth map to be aligned to obtain a metric depth map.

[0091] For example, in order to obtain the metric depth during the training phase This application uses RANSAC correction to align the monocular depth D with the sparse SFM points. Figure 3 As shown, Figure 3 It is a diagram of the metric depth alignment process according to an embodiment of the present application. First, the monocular depth of the input image and the depth map to be aligned (such as MVS depth (mask filter) / SFM point / Dust3R depth) are obtained, and then the monocular depth is RANSAC corrected based on the depth map to be aligned to obtain the scaling coefficient (Rescale, r) and the offset coefficient (Shift, s), and finally the r coefficient and the s coefficient are used to align the monocular depth to the metric depth through a simple linear change. The above process can be expressed by formula (3).

[0092]

[0093] In order to prevent the model from degenerating into an overly simple solution (ie, r=0), the present application requires that the minimum value of r be ≥ 0.1 times the ratio of the Dense Structure from Motion (DSFM) standard deviation to the D standard deviation.

[0094] As a result, more accurate and reliable metric depth maps can be extracted and constructed from monocular depth maps, which not only enhances the quality of depth information, but also provides more effective training data for multi-view generation models based on 3D priors, thereby achieving better performance in image generation, 3D reconstruction and other related tasks.

[0095] In an optional embodiment, in step S221, using the color pixel values ​​of the reference image set to obtain the unified spatial coordinates corresponding to the color pixel values ​​includes the following method steps:

[0096] Step S2113, projecting the color pixel value to the camera coordinate system using the intrinsic parameter matrix corresponding to the metric depth map and the first camera pose set to obtain a first projection result;

[0097] Step S2114, projecting the first projection result to the world coordinate system according to the extrinsic parameter matrix corresponding to the first camera pose set to obtain a second projection result;

[0098] Step S2115, normalizing the second projection result to obtain unified space coordinates.

[0099] In the embodiment of the present application, when the color pixel values ​​of the reference image set are used to obtain the unified spatial coordinates corresponding to the color pixel values, the color pixel values ​​can be projected to the camera coordinate system using the intrinsic parameter matrix corresponding to the metric depth map and the first camera pose set to obtain a first projection result, and then the first projection result is projected to the world coordinate system according to the extrinsic parameter matrix corresponding to the first camera pose set to obtain a second projection result, and finally the second projection result is normalized to obtain the unified spatial coordinates. It can be understood that RGB pixels are projected to the camera coordinate system through the intrinsic parameter matrix and the metric depth map corresponding to the first camera pose set, and then projected to the world coordinate system through the camera2world extrinsic parameter matrix, and finally normalized to obtain the unified spatial coordinates.

[0100] Among them, the extrinsic parameter matrix corresponding to the first camera pose set, namely the camera2world extrinsic parameter matrix, is a mathematical matrix that describes the position and posture (direction) of the camera relative to the world coordinate system.

[0101] It can be seen that this application uses the metric depth map and the internal parameters of the camera (i.e., the intrinsic parameter matrix) to calculate the three-dimensional position of each color pixel in the camera coordinate system, that is, to obtain the first projection result. Then it is necessary to convert these points from the camera coordinate system to the world coordinate system to build a three-dimensional model of the entire scene. Therefore, by combining the first projection result with the camera's external parameter matrix, the three-dimensional point coordinates in the world coordinate system can be obtained, that is, the second projection result is obtained. Finally, the three-dimensional point coordinates in these world coordinate systems are normalized to obtain unified spatial coordinates to ensure that all point clouds or three-dimensional models are in a unified, standardized spatial coordinate system, which is helpful for subsequent three-dimensional model construction, multi-view image generation and other tasks to ensure the consistency and accuracy of all data.

[0102] In an optional embodiment, in step S21, obtaining the first camera pose set and the second camera pose set includes the following method steps:

[0103] The first camera pose set and the second camera pose set are obtained by using a preset ray method and a reference image set.

[0104] In an embodiment of the present application, when obtaining the first camera pose set and the second camera pose set, the first camera pose set and the second camera pose set can be obtained by using a preset ray method and a reference image set. The preset ray method can be understood as presetting a series of light directions starting from the optical center of the camera in three-dimensional space. These light directions can be based on a specific geometric model, such as Plücker rays, or rays starting from pixel coordinates in a more general sense. The preset ray can be used to assist in estimating the pose of the camera, especially in technologies such as MVS or SFM.

[0105] Exemplarily, taking the preset ray mode as the Plücker ray mode as an example, when obtaining the first camera pose set and the second camera pose set, dense camera pose information is provided by Plücker ray, combined with the depth information in the reference image set, and a matching algorithm is used to determine which rays correspond to the same scene point, and then the camera pose is estimated, thereby obtaining the first camera pose set and the second camera pose set.

[0106] In an optional embodiment, in step S23, the initial multi-view generation model is trained using the reference image set, the prior information, the first camera pose set, the second camera pose set, and the preset mask information to obtain the target multi-view generation model, including the following method steps:

[0107] Step S231, training the initial multi-view generation model using the reference image set, the prior information, the first camera pose set, the second camera pose set, and the preset mask information to obtain an intermediate multi-view generation model;

[0108] Step S232, using the verification set to verify the intermediate multi-view generation model to obtain a verification result;

[0109] Step S233, in response to the verification result not meeting the preset condition, adjusting the model parameters of the intermediate multi-view generation model until the verification result meets the preset condition, thereby obtaining the target multi-view generation model.

[0110] In an embodiment of the present application, when the initial multi-view generation model is trained using a reference image set, prior information, a first camera pose set, a second camera pose set, and preset mask information to obtain a target multi-view generation model, the initial multi-view generation model can be trained using a reference image set, prior information, a first camera pose set, a second camera pose set, and preset mask information to obtain an intermediate multi-view generation model. It can be understood that the model learns how to generate images of new perspectives based on existing perspectives (i.e., reference image sets and camera poses), and uses prior information and preset masks to guide this process, thereby obtaining a preliminary trained intermediate multi-view generation model.

[0111] Then, the validation set is used to validate the intermediate multi-view generation model to obtain the validation results, wherein the validation set may consist of a set of images that have not been used in the training phase, and is used to objectively evaluate the performance of the model. The intermediate multi-view generation model is used to predict the images in the validation set, and then the prediction results are compared with the true results to obtain the validation results, which are used to evaluate whether the effect of the intermediate multi-view generation model meets expectations.

[0112] If the verification result does not meet the preset conditions, for example, the performance indicators of the intermediate multi-view generation model do not reach the predefined threshold or standard, the model parameters of the intermediate multi-view generation model are adjusted until the verification result meets the preset conditions, for example, the performance indicators of the intermediate multi-view generation model reach the predefined threshold or standard, or the verification result of the verification set no longer improves, indicating that the preset conditions are met, thereby obtaining the target multi-view generation model.

[0113] In an optional embodiment, the preset mask information includes: a first mask set and a second mask set. In step S231, the initial multi-view generation model is trained using the reference image set, the prior information, the first camera pose set, the second camera pose set, and the preset mask information to obtain an intermediate multi-view generation model, including the following method steps:

[0114] Step S2311, determining a reference input feature based on the color pixel value, the first mask set, the first camera pose set and the unified space coordinates, and determining a target input feature based on the projected color pixel value, the second mask set, the second camera pose set and the projected unified space coordinates, wherein the binary value of the first mask set is opposite to the binary value of the second mask set;

[0115] Step S2312, using the reference input features and the target input features, the initial multi-view generation model is trained to obtain an intermediate multi-view generation model.

[0116] In the embodiment of the present application, the preset mask information includes: a first mask set and a second mask set. Taking the preset mask information as a binary mask as an example, the first mask set and the second mask set can be understood as mask sets with opposite binary values, that is, the first mask set and the second mask set can cover the entire image together, but under different masks, the areas or information of interest are different. For example, the first mask set may focus on foreground objects, while the second mask set focuses on background or occluded areas, which can help the model process different parts of the image more carefully.

[0117] When the initial multi-view generation model is trained using the reference image set, the prior information, the first camera pose set, the second camera pose set, and the preset mask information to obtain the intermediate multi-view generation model, the reference input features (Observed input features) can be determined based on the color pixel values, the first mask set, the first camera pose set, and the unified space coordinates, and the target input features (Target input features) can be determined based on the projected color pixel values, the second mask set, the second camera pose set, and the unified space coordinates after projection. Then, the initial multi-view generation model is trained using the reference input features and the target input features to obtain the intermediate multi-view generation model.

[0118] As a result, the intermediate multi-view generation model can learn how to generate images with consistent perspectives and accurate content under different camera positions. And by using the preset mask information, it can help the intermediate multi-view generation model distinguish between foreground and background, occluded areas and non-occluded areas, thus helping to handle complex scenes such as multi-object occlusion and dynamic environmental changes.

[0119] Figure 4 is a model reasoning flow chart according to an embodiment of the present application, such as Figure 4 As shown in the figure, the metric depth of the objects in the scene is estimated by extracting the monocular depth of the reference image, and then the metric depth is aligned. After that, the depth information can be projected to other perspectives by combining the camera pose of the reference perspective and the camera pose of the target perspective, so as to achieve the alignment of the depth information between different perspectives and obtain 3D prior data. In addition, multi-perspective generation can be performed based on the camera pose of the reference perspective and the camera pose of the target perspective, and then 3D reconstruction (3D Generative Synthesis, 3DGS) is performed based on the multi-perspective images generated from the multi-perspective generation.

[0120] The multi-view generation in the multi-view generation model trained in this application is completed by a latent diffusion model (LDM), namely a multi-view latent diffusion model (MV-LDM). For example, given a set of reference images I r , C r Represents the unified spatial coordinates of the reference image, which is also the camera pose of the reference view and the camera pose of the target view (P r , P t ) provides the Plücker ray embedding, and the 3D prior mentioned above, LDMθ learns to synthesize the image from the target perspective, see formula (4).

[0121]

[0122] Among them, θ represents the model parameters of LDM, z r and z t Represent the potential features of the reference view and the target view respectively. M is a binary mask used to distinguish the reference features from the target features. represents the unified space coordinates after projection, Represents the color pixel value after projection. This application introduces Plücker ray embedding to confirm the dense camera representation, and all the above features are spliced ​​to form the input of the model. Studies have shown that this direct introduction of 3D priors in this application significantly improves the model effect.

[0123] Furthermore, the multi-view generation model determines the reference input features based on the color pixel values ​​(RGB pixels), the preset rays (Plückerray), the unified space coordinates (CCM) and the clear latent representation (Clear latent), and determines the target input features based on the projected color pixel values ​​(Warped RGB pixels), the preset rays (Plücker ray), the projected unified space coordinates (Warped CCM) and the latent representation of noise (Noised latent). In addition, when determining the reference input features and the target input features, a first mask set and a second mask set with opposite binary values ​​are introduced, see the above description, which will not be repeated here.

[0124] Since the data used in this application includes the corrected depth as a 3D prior, the camera pose and depth are used to obtain the corresponding projected images (warped RGB pixels) and the projected unified spatial coordinate positions (warped canonical coordinate maps). In addition, preset rays (such as Plücker rays) are used as additional inputs to enhance the ability to generate models. Among them, the projected images are used to provide a 3D prior for display, CCM is used to provide spatial position information to supplement the specific order of pixels, and Plücker rays are used to provide dense camera pose information.

[0125] The multi-view generation model trained in the present application generates a multi-view image, i.e., a new multi-view image, based on a variational autoencoder (VAE) decoder for the input image when performing multi-view generation.

[0126] In an optional embodiment, the initial multi-view generation model includes: multiple self-attention modules, wherein the multiple self-attention modules are used to rescale the key features corresponding to the reference image set according to a preset scale coefficient to calculate a key-value pair attention map.

[0127] After research, it was found that when the number of M is too large, the performance of the trained multi-view generation model will drop significantly, and the model will have an overfitting problem. The root cause of this overfitting problem is attention dilution, that is, attention is allocated to too many positions, resulting in weakened guidance for the reference image, resulting in poor quality of the generated multi-view image.

[0128] To this end, the initial multi-view generation model in the embodiment of the present application includes multiple self-attention modules, and the present application proposes key-rescale to enhance the reference guidance in the self-attention module.

[0129] Furthermore, key scale adjustment can be understood as multiplying a constant γ by the key features of the reference view in all self-attention modules, that is, multiple self-attention modules are used to rescale the key features corresponding to the reference image set according to a preset scale factor to calculate the key-value pair attention map, where the key-value pair attention map is used to reflect the correlation between the query (usually the image features from the target generation perspective) and the key (the image features from the reference image set). By adjusting the scale of the key features (i.e., rescaling), the attention calculation is optimized, so that the model can more intelligently capture and utilize the intrinsic correlation of multi-view images during the generation process, eliminating the problem of attention dilution and improving the quality and perspective coherence of the generated images.

[0130] In an optional embodiment, the reference image set includes: multiple domain image sets for joint training, wherein the multiple domain image sets are respectively assigned labels of different categories.

[0131] The reference image set in the embodiment of the present application includes a plurality of domain image sets for joint training. Since the data of a single domain may not be sufficient to train a model with generalization ability, the present application uses image data sets from a plurality of different domains (such as indoor, outdoor, street, natural scenery, etc.) for joint training, thereby helping to enhance the generalization ability of the model, so that it can generate high-quality images when facing different scenes and perspectives.

[0132] Among them, multiple domain image sets are each assigned different category labels. It can be understood that the embodiment of the present application has also made some improvements at the model level. In order to better process images from different fields, the present application adds category labels for different data sets, that is, a specific label is assigned to each field or data set.

[0133] For example, labels can be domain information (such as "indoor", "outdoor"), scene types (such as "mountain", "city"), object categories (such as "furniture", "vehicle"), etc. During the training process, the model will learn to associate specific labels with corresponding features and generation logic. When the model generates images from a new perspective, it can adjust the generation strategy based on the input category label to generate images that are more in line with the characteristics of the category, thereby improving the generalization of the algorithm.

[0134] In addition, the present application also designs a detailed dropout strategy to prevent the model from being overly dependent on 3D priors. For example, during the training process, randomly discarding some input features can encourage the model to learn more comprehensive and robust representations, and avoid performance degradation when the model processes test data that does not provide prior information.

[0135] In addition, this application also makes a lot of basic engineering model optimizations, including but not limited to exponential moving average (EMA) and multi-scale training. Among them, the use of EMA during model training can improve the stability of the model, reduce oscillations during training, and help obtain smoother and more stable model weight updates. Multi-scale training aims to enable the model to handle inputs of different scales and enhance its performance on images of different resolutions.

[0136] It can be seen that, with the goal of achieving scene-level multi-perspective generation effect, this application designs a scene-level multi-perspective generation model based on 3D prior assistance from the algorithm level and the data level, thereby greatly improving the effect of scene generation.

[0137] It is easy to understand that the beneficial effects of the model training method provided in this application include the following points.

[0138] Beneficial effect (1): A large amount of multi-view image data has been collected. Furthermore, traditional training data is limited to four datasets: Co3dv2, MVImageNet, Real10k, and DL3DV. This application collects and constructs a high-quality multi-view image dataset that is far larger than ever before, including aerial photography, indoor scenes, objects, and natural and scenic spots scene data. The total data volume reaches 1.2 million scenes, which significantly improves the diversity and richness of training data, thereby improving the generalization ability of the model.

[0139] Beneficial effect (2): For all multi-view image data, monocular and metric depths are extracted to construct corresponding 3D priors. Furthermore, traditional model training does not introduce 3D priors. This application combines monocular depth and metric depth to provide high-quality 3D priors to drive scene image generation under large angle changes, providing the model with richer three-dimensional spatial clues. The generalization and density of monocular depth combined with the accuracy of metric depth optimize the 3D prior, thereby achieving better results in multi-view image generation under large angle changes.

[0140] Beneficial effect (3): In the multi-view generation model, the present application improves the generation capability and accuracy of the model by using projected images, unified spatial coordinate positions (CCM) and Plücker rays as additional inputs. In addition, by introducing category labels, detailed dropout strategies and EMA techniques, the model is further optimized, and the stability and generalization of the algorithm are enhanced.

[0141] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0142] In addition, it should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

[0143] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD), including several instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0144] According to the embodiment of the present application, there is also provided Figure 5 An image generation method shown in FIG. Figure 5 is a flow chart of an image generation method according to an embodiment of the present application, such as Figure 5 As shown, the method includes:

[0145] Step S51, obtaining the original viewing angle image;

[0146] Step S52, using a target multi-view generation model to perform multi-view generation on the original view image to obtain an extended view image; wherein the target multi-view generation model is obtained by using any one of the above-mentioned model training methods.

[0147] In the embodiment of the present application, the original perspective image can be understood as a scene image taken from a specific angle and position, which is the basis for the subsequent generation of multi-perspective images. The original perspective image can be one or more images under the original perspective, which can be taken from a scene in the real world or a rendered image of a virtual scene, which is not limited here.

[0148] The target multi-view generation model is used to generate multi-views from the original view images to obtain extended view images. That is, after obtaining the original view images, these images are processed by the previously trained multi-view generation model (target multi-view generation model) to generate scene images seen from different viewpoints, i.e. extended view images. The target multi-view generation model is obtained by the model training method mentioned above, see the above description, and will not be repeated here.

[0149] In an embodiment of the present application, by acquiring the original perspective image, a target multi-perspective generation model obtained by any one of the above-mentioned model training methods is used to perform multi-perspective generation on the original perspective image to obtain an extended perspective image, thereby being able to generate perspective images of multiple angles and positions, which greatly enriches the understanding and observation angles of the scene, and the generated extended perspective image has perspective diversity and adaptability, and has high image quality, and is a new perspective image that is more visually coherent and realistic.

[0150] The above-mentioned image generation method provided in the embodiment of the present application can be applied to, but is not limited to, application scenarios involving image generation in the fields of e-commerce services, educational services, legal services, medical services, conference services, social network services, financial product services, logistics services and navigation services, for example: image generation of e-commerce services, image generation of educational services, image generation of legal services, etc., which are not limited here.

[0151] By adopting the embodiment of the present application, the original perspective image is obtained, and the target multi-perspective generation model obtained by any one of the above-mentioned model training methods is used to perform multi-perspective generation on the original perspective image to obtain an extended perspective image, thereby achieving the purpose of training a high-quality multi-perspective generation model and generating a multi-perspective image with perspective diversity, adaptability and high quality, thereby achieving the technical effect of generating high-quality scene-level multi-perspective images, and further solving the technical problem that the three-dimensional multi-perspective generation algorithm in the related technology cannot achieve a good generation level at the scene level.

[0152] In an optional embodiment, the image generation method further includes:

[0153] A three-dimensional reconstruction operation or a video generation operation is performed based on the original view image and the extended view image.

[0154] In an embodiment of the present application, a three-dimensional reconstruction operation can be performed based on the original perspective image and the extended perspective image. It can be understood that the original perspective image and the extended perspective image together constitute a multi-perspective image set. Since the extended perspective image is synthesized from the original perspective image through a multi-perspective generation model, the two maintain the continuity and consistency of the scene. Therefore, they are used together as input for three-dimensional reconstruction, which can improve the accuracy and coverage of the three-dimensional reconstruction and help to more comprehensively understand the three-dimensional structure of the scene.

[0155] In addition, video generation operations can also be performed based on the original view images and the extended view images. Video generation usually involves creating a continuous sequence of images to simulate the dynamic changes of the scene or to show the scene observed from different viewpoints. The original view images and the extended view images generated by the model can be used as the basis for constructing video frames. By arranging these images in a time series or generating a continuous image stream through perspective transformation, a video can be obtained, which shows the process of observing the scene from different perspectives. In video generation, the use of multi-view images can not only provide changes in perspective, but also show the changes in the appearance of the scene at different time points by combining the dynamic generation capabilities of the model.

[0156] Therefore, the three-dimensional reconstruction or video generation operation based on the original perspective image and the extended perspective image can be understood as the use of a multi-perspective image set (including the original perspective images taken actually and the extended perspective images generated by the model) to restore the three-dimensional structure of the scene or generate a video with continuous perspective changes, which can significantly enhance the final three-dimensional model accuracy and video quality, while providing a wider range of application possibilities.

[0157] It should be noted that the preferred implementation of this embodiment can refer to the relevant description in the embodiment, which will not be repeated here.

[0158] According to the embodiment of the present application, there is also provided Figure 6A method for generating an image is shown. Figure 6 is a flow chart of an image generation method according to an embodiment of the present application, such as Figure 6 As shown, the method includes:

[0159] Step S61, obtaining an image generation request through a first application programming interface, wherein the request data carried in the image generation request includes: an original viewing angle image;

[0160] Step S62, returning an image generation response through a second application programming interface, wherein the response data carried in the image generation response includes: an extended perspective image, the extended perspective image is obtained by performing multi-perspective generation on the original perspective image using a target multi-perspective generation model, and the target multi-perspective generation model is obtained using any one of the above-mentioned model training methods.

[0161] The first application programming interface and the second application programming interface may be the same application programming interface or different application programming interfaces. In an optional embodiment, the interface parameters in the first application programming interface and the second application programming interface may include but are not limited to: interface global identifier, interface signature key, interface timestamp, interface request identifier, system call credential identifier, etc. The first application programming interface may use GET or POST as the interface request method to obtain the file processing request. The second application programming interface may use JSON format to feedback the file processing response.

[0162] The request data carried in the image generation request includes: the original view image. The image generation request can be understood as a request for multi-view generation of the original view image. The original view image can be understood as a scene image taken from a specific angle and position, which is the basis for subsequent multi-view image generation. The original view image can be one or more images under the original view, which can be taken from a real-world scene or a rendered image of a virtual scene, which is not limited here.

[0163] The image generation response can be understood as a reply to the image generation request. The response data carried in the image generation response includes: an extended view image, which is obtained by using the target multi-view generation model to generate multi-view images from the original view image. That is, after obtaining the original view image, the previously trained multi-view generation model (target multi-view generation model) is used to process these images to generate scene images seen from different viewpoints, i.e., extended view images. Among them, the target multi-view generation model is obtained by the model training method mentioned above, see the above description, and will not be repeated here.

[0164] In the embodiment of the present application, an image generation request is obtained through a first application programming interface, wherein the request data carried in the image generation request includes: the original view image, and then an image generation response is returned through a second application programming interface, wherein the response data carried in the image generation response includes: an extended view image, and the extended view image is obtained by performing multi-view generation of the original view image using a target multi-view generation model, and the target multi-view generation model is obtained using any of the above-mentioned model training methods. In this way, multiple view images of different angles and positions can be generated, which greatly enriches the understanding and observation angles of the scene, and the generated extended view image has view diversity and adaptability, and has high image quality, and is a new view image that is more visually coherent and realistic.

[0165] The above-mentioned image generation method provided in the embodiment of the present application can be applied to, but is not limited to, application scenarios involving image generation in the fields of e-commerce services, educational services, legal services, medical services, conference services, social network services, financial product services, logistics services and navigation services, for example: image generation of e-commerce services, image generation of educational services, image generation of legal services, etc., which are not limited here.

[0166] According to an embodiment of the present application, an image generation request is obtained through a first application programming interface, wherein the request data carried in the image generation request includes: an original perspective image; and then an image generation response is returned through a second application programming interface, wherein the response data carried in the image generation response includes: an extended perspective image, wherein the extended perspective image is obtained by performing multi-perspective generation of the original perspective image using a target multi-perspective generation model, and the target multi-perspective generation model is obtained using any one of the above-mentioned model training methods, thereby achieving the purpose of training a high-quality multi-perspective generation model to generate multi-perspective images with perspective diversity, adaptability and high quality, thereby achieving the technical effect of generating high-quality scene-level multi-perspective images, and further solving the technical problem that the three-dimensional multi-perspective generation algorithm in the related technology cannot achieve a good generation level at the scene level.

[0167] It should be noted that the preferred implementation of this embodiment can refer to the relevant description in the embodiment, which will not be repeated here.

[0168] According to the embodiment of the present application, there is also provided Figure 7 A method for generating an image is shown. Figure 7 is a flow chart of an image generation method according to an embodiment of the present application, such as Figure 7 As shown, the method includes:

[0169] Step S71, obtaining the currently input image generation dialogue request, wherein the request data carried in the image generation dialogue request includes: the original viewing angle image;

[0170] Step S72, in response to the image generation dialogue request, returning an image generation dialogue reply, wherein the information carried in the image generation dialogue reply includes: an extended view image, the extended view image is obtained by performing multi-view generation on the original view image using a target multi-view generation model, and the target multi-view generation model is obtained using any one of the above-mentioned model training methods;

[0171] Step S73: display the extended viewing angle image in the graphical user interface.

[0172] The image generation dialogue request can be understood as a dialogue request initiated by a user when communicating with artificial intelligence. The request data carried in the image generation dialogue request includes: original view images, which can be understood as scene images taken from a specific angle and position, and are the basis for subsequent multi-view image generation. The original view image can be one or more images at the original view, which can be taken from a real-world scene or a rendered image of a virtual scene, and there is no restriction here.

[0173] The image generation dialogue reply can be understood as a reply to the image generation dialogue request. The information carried in the image generation dialogue reply includes: extended view images, that is, after obtaining the original view images, these images are processed by the previously trained multi-view generation model (target multi-view generation model) to generate scene images seen from different viewpoints, that is, extended view images. Among them, the target multi-view generation model is obtained by the model training method mentioned above, see the previous description, and will not be repeated here.

[0174] In the embodiment of the present application, by obtaining the currently input image generation dialogue request, wherein the request data carried in the image generation dialogue request includes: the original view image, then responding to the image generation dialogue request, returning the image generation dialogue reply, wherein the information carried in the image generation dialogue reply includes: the extended view image, the extended view image is obtained by using the target multi-view generation model to generate the original view image from multiple viewpoints, the target multi-view generation model is obtained by using any of the above-mentioned model training methods, and finally the extended view image is displayed in the graphical user interface to provide feedback to the user. In this way, multiple view images of different angles and positions can be generated, which greatly enriches the understanding and observation angle of the scene, and the generated extended view image has view diversity and adaptability, and has high image quality, and is a new view image that is more visually coherent and realistic.

[0175] The above-mentioned image generation method provided in the embodiment of the present application can be applied to, but is not limited to, application scenarios involving image generation in the fields of e-commerce services, educational services, legal services, medical services, conference services, social network services, financial product services, logistics services and navigation services, for example: image generation of e-commerce services, image generation of educational services, image generation of legal services, etc., which are not limited here.

[0176] According to an embodiment of the present application, a currently input image generation dialogue request is obtained, wherein the request data carried in the image generation dialogue request includes: an original view image, and then in response to the image generation dialogue request, an image generation dialogue reply is returned, wherein the information carried in the image generation dialogue reply includes: an extended view image, wherein the extended view image is obtained by performing multi-view generation on the original view image using a target multi-view generation model, wherein the target multi-view generation model is obtained using any one of the above-mentioned model training methods, and finally the extended view image is displayed in a graphical user interface to provide feedback to the user, thereby achieving the purpose of training a high-quality multi-view generation model and generating a multi-view image with view diversity, adaptability and high quality, thereby achieving the technical effect of generating high-quality scene-level multi-view images, and further solving the technical problem that the three-dimensional multi-view generation algorithm in the related art cannot achieve a good generation level in scene-level generation.

[0177] It should be noted that the preferred implementation of this embodiment can refer to the relevant description in the embodiment, which will not be repeated here.

[0178] According to an embodiment of the present application, a device embodiment for implementing the above-mentioned model training method is also provided. Figure 8 is a structural diagram of a model training device according to an embodiment of the present application, such as Figure 8 As shown, the device comprises:

[0179] A first acquisition module 801 is used to acquire training data, wherein the training data includes: a reference image set, a first camera pose set, and a second camera pose set, wherein the first camera pose set is different from the second camera pose set;

[0180] A second acquisition module 802, configured to acquire prior information based on the reference image set, the first camera pose set, and the second camera pose set;

[0181] The training module 803 is used to train the initial multi-view generation model using the reference image set, the prior information, the first camera pose set, the second camera pose set and the preset mask information to obtain the target multi-view generation model, wherein the target multi-view generation model is used to perform multi-view generation on the input original perspective image to obtain the extended perspective image.

[0182] Optionally, the second acquisition module 802 is also used to: use the monocular depth map of the reference image set to obtain the metric depth map of the reference image set, and use the color pixel values ​​of the reference image set to obtain the unified spatial coordinates corresponding to the color pixel values; perform back projection based on the color pixel values, unified spatial coordinates, the metric depth map, the relative viewing angle between the first camera pose set and the second camera pose set, the intrinsic parameter matrix corresponding to the first camera pose set, and the intrinsic parameter matrix corresponding to the second camera pose set to obtain prior information, wherein the prior information includes: the projected color pixel values ​​and the projected unified spatial coordinates.

[0183] Optionally, the second acquisition module 802 is further used to: acquire a depth map to be aligned; and perform image correction on a monocular depth based on the depth map to be aligned to obtain a metric depth map.

[0184] Optionally, the second acquisition module 802 is also used to: project the color pixel value to the camera coordinate system using the intrinsic parameter matrix corresponding to the metric depth map and the first camera pose set to obtain a first projection result; project the first projection result to the world coordinate system according to the extrinsic parameter matrix corresponding to the first camera pose set to obtain a second projection result; and normalize the second projection result to obtain a unified spatial coordinate.

[0185] Optionally, the first acquisition module 801 is further used to: acquire the first camera pose set and the second camera pose set by using a preset ray method and a reference image set.

[0186] Optionally, the above-mentioned training module 803 is also used to: train the initial multi-view generation model using a reference image set, prior information, a first camera pose set, a second camera pose set and preset mask information to obtain an intermediate multi-view generation model; verify the intermediate multi-view generation model using a verification set to obtain a verification result; in response to the verification result not meeting the preset conditions, adjust the model parameters of the intermediate multi-view generation model until the verification result meets the preset conditions to obtain the target multi-view generation model.

[0187] Optionally, the preset mask information includes: a first mask set and a second mask set, and the above-mentioned training module 803 is also used to: determine reference input features based on color pixel values, the first mask set, the first camera pose set and the unified space coordinates, and determine target input features based on the projected color pixel values, the second mask set, the second camera pose set and the unified space coordinates after projection, wherein the binary values ​​of the first mask set are opposite to the binary values ​​of the second mask set; use the reference input features and the target input features to train the initial multi-view generation model to obtain an intermediate multi-view generation model.

[0188] Optionally, the initial multi-view generation model includes: multiple self-attention modules, wherein the multiple self-attention modules are used to rescale the key features corresponding to the reference image set according to a preset scale factor to calculate a key-value pair attention map.

[0189] Optionally, the reference image set includes: multiple domain image sets for joint training, wherein the multiple domain image sets are respectively assigned labels of different categories.

[0190] By adopting the embodiment of the present application, by obtaining training data, that is, obtaining a reference image set, a first camera pose set and a second camera pose set, then based on the reference image set, the first camera pose set and the second camera pose set, obtaining prior information, and finally using the reference image set, the prior information, the first camera pose set, the second camera pose set and the preset mask information to train the initial multi-view generation model, a target multi-view generation model is obtained, and the target multi-view generation model is used to perform multi-view generation on the input original view image to obtain an extended view image, thereby achieving the purpose of training a high-quality multi-view generation model, maximizing the use of 3D prior information, view transformation data and mask information, thereby significantly improving the quality of image generation, the generalization ability of view transformation and the processing of complex scenes, and achieving the technical effect of generating high-quality scene-level multi-view images, thereby solving the technical problem that the three-dimensional multi-view generation algorithm in the related technology cannot achieve a good generation level at the scene level.

[0191] It should be noted that the first acquisition module 801, the second acquisition module 802 and the training module 803 correspond to steps S21 to S23 in the embodiment, and the three modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above embodiment. It should be noted that the above modules or units can be hardware components or software components stored in a memory and processed by one or more processors, and the above modules can also run in the server 10 provided in the embodiment.

[0192] According to an embodiment of the present application, another device embodiment for implementing the above-mentioned image generating method is also provided. Fig. 9 is a schematic diagram of the structure of another image generating device according to an embodiment of the present application. Fig. 9 As shown, the device comprises:

[0193] The third acquisition module 901 is used to acquire the original viewing angle image;

[0194] The first processing module 902 is used to use a target multi-view generation model to generate a multi-view image for the original view image to obtain an extended view image; wherein the target multi-view generation model is obtained by using any one of the above-mentioned model training methods.

[0195] Optionally, the device further includes: a second processing module, configured to perform a three-dimensional reconstruction operation or a video generation operation based on the original viewing angle image and the extended viewing angle image.

[0196] By adopting the embodiment of the present application, the original perspective image is obtained, and the target multi-perspective generation model obtained by any one of the above-mentioned model training methods is used to perform multi-perspective generation on the original perspective image to obtain an extended perspective image, thereby achieving the purpose of training a high-quality multi-perspective generation model and generating a multi-perspective image with perspective diversity, adaptability and high quality, thereby achieving the technical effect of generating high-quality scene-level multi-perspective images, and further solving the technical problem that the three-dimensional multi-perspective generation algorithm in the related technology cannot achieve a good generation level at the scene level.

[0197] It should be noted that the third acquisition module 901, the first processing module 902 and the second processing module correspond to step S51 and step S52 in the embodiment, and the two modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above embodiment. It should be noted that the above modules or units can be hardware components or software components stored in a memory and processed by one or more processors, and the above modules can also run in the server 10 provided in the embodiment.

[0198] According to an embodiment of the present application, another device embodiment for implementing the above-mentioned image generating method is also provided. Fig.10 is a structural schematic diagram of another image generating device according to an embodiment of the present application, such as Fig.10 As shown, the device comprises:

[0199] The fourth acquisition module 1001 is used to acquire an image generation request through the first application programming interface, wherein the request data carried in the image generation request includes: an original view image;

[0200] The first return module 1002 is used to return an image generation response through a second application programming interface, wherein the response data carried in the image generation response includes: an extended perspective image, which is obtained by performing multi-perspective generation on the original perspective image using a target multi-perspective generation model, and the target multi-perspective generation model is obtained using any one of the above-mentioned model training methods.

[0201] According to an embodiment of the present application, an image generation request is obtained through a first application programming interface, wherein the request data carried in the image generation request includes: an original perspective image; and then an image generation response is returned through a second application programming interface, wherein the response data carried in the image generation response includes: an extended perspective image, wherein the extended perspective image is obtained by performing multi-perspective generation of the original perspective image using a target multi-perspective generation model, and the target multi-perspective generation model is obtained using any one of the above-mentioned model training methods, thereby achieving the purpose of training a high-quality multi-perspective generation model to generate multi-perspective images with perspective diversity, adaptability and high quality, thereby achieving the technical effect of generating high-quality scene-level multi-perspective images, and further solving the technical problem that the three-dimensional multi-perspective generation algorithm in the related technology cannot achieve a good generation level at the scene level.

[0202] It should be noted that the fourth acquisition module 1001 and the first return module 1002 correspond to step S61 and step S62 in the embodiment, and the two modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above embodiment. It should be noted that the above modules or units can be hardware components or software components stored in a memory and processed by one or more processors, and the above modules can also run in the server 10 provided in the embodiment.

[0203] According to an embodiment of the present application, another device embodiment for implementing the above-mentioned image generating method is also provided. Fig.11 is a structural schematic diagram of another image generating device according to an embodiment of the present application, such as Fig.11 As shown, the device comprises:

[0204] The fifth acquisition module 1101 is used to acquire the currently input image generation dialogue request, wherein the request data carried in the image generation dialogue request includes: the original view image;

[0205] The second returning module 1102 is used to respond to the image generation dialogue request and return an image generation dialogue reply, wherein the information carried in the image generation dialogue reply includes: an extended view image, which is obtained by performing multi-view generation on the original view image using a target multi-view generation model, and the target multi-view generation model is obtained using any one of the above-mentioned model training methods;

[0206] The display module 1103 is used to display the extended viewing angle image in the graphical user interface.

[0207] According to an embodiment of the present application, a currently input image generation dialogue request is obtained, wherein the request data carried in the image generation dialogue request includes: an original view image, and then in response to the image generation dialogue request, an image generation dialogue reply is returned, wherein the information carried in the image generation dialogue reply includes: an extended view image, wherein the extended view image is obtained by performing multi-view generation on the original view image using a target multi-view generation model, wherein the target multi-view generation model is obtained using any one of the above-mentioned model training methods, and finally the extended view image is displayed in a graphical user interface to provide feedback to the user, thereby achieving the purpose of training a high-quality multi-view generation model and generating a multi-view image with view diversity, adaptability and high quality, thereby achieving the technical effect of generating high-quality scene-level multi-view images, and further solving the technical problem that the three-dimensional multi-view generation algorithm in the related art cannot achieve a good generation level in scene-level generation.

[0208] It should be noted that the fifth acquisition module 1101, the second return module 1102 and the display module 1103 correspond to steps S71 to S73 in the embodiment, and the three modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above embodiment. It should be noted that the above modules or units can be hardware components or software components stored in a memory and processed by one or more processors, and the above modules can also run in the server 10 provided in the embodiment.

[0209] It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the scheme provided in the embodiments, as well as the application scenario and implementation process, but is not limited to the scheme provided in the embodiments.

[0210] The embodiment of the present application may provide an electronic device, which may be any electronic device in a group of electronic devices. Optionally, in this embodiment, the electronic device may also be replaced by a terminal device such as a mobile terminal. Optionally, in this embodiment, the electronic device may be located in at least one network device among a plurality of network devices in a computer network.

[0211] In this embodiment, the above-mentioned electronic device can execute the program code of the following steps in the model training method: obtaining training data, wherein the training data includes: a reference image set, a first camera pose set and a second camera pose set, the first camera pose set is different from the second camera pose set; obtaining prior information based on the reference image set, the first camera pose set and the second camera pose set; using the reference image set, the prior information, the first camera pose set, the second camera pose set and the preset mask information to train the initial multi-view generation model to obtain a target multi-view generation model, wherein the target multi-view generation model is used to perform multi-view generation on the input original view image to obtain an extended view image.

[0212] Optionally, Fig.12 is a structural block diagram of an electronic device according to an embodiment of the present application. Fig.12 As shown, taking electronic device A as an example, the electronic device A may include: one or more (only one is shown in the figure) processors 1202, a memory 1204, a storage controller, and a peripheral interface, wherein the peripheral interface is connected to a radio frequency module, an audio module, and a display.

[0213] Among them, the memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the model training method and device in the embodiment of the present application, and the processor executes various functional applications and data processing by running the stored software programs and modules, that is, realizing the above-mentioned model training method. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories can be connected to the electronic device A via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0214] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: obtain training data, wherein the training data includes: a reference image set, a first camera pose set and a second camera pose set, the first camera pose set is different from the second camera pose set; based on the reference image set, the first camera pose set and the second camera pose set, obtain prior information; use the reference image set, the prior information, the first camera pose set, the second camera pose set and the preset mask information to train the initial multi-view generation model to obtain a target multi-view generation model, wherein the target multi-view generation model is used to perform multi-view generation on the input original view image to obtain an extended view image.

[0215] By adopting the embodiment of the present application, by obtaining training data, that is, obtaining a reference image set, a first camera pose set and a second camera pose set, then based on the reference image set, the first camera pose set and the second camera pose set, obtaining prior information, and finally using the reference image set, the prior information, the first camera pose set, the second camera pose set and the preset mask information to train the initial multi-view generation model, a target multi-view generation model is obtained, and the target multi-view generation model is used to perform multi-view generation on the input original view image to obtain an extended view image, thereby achieving the purpose of training a high-quality multi-view generation model, maximizing the use of 3D prior information, view transformation data and mask information, thereby significantly improving the quality of image generation, the generalization ability of view transformation and the processing of complex scenes, and achieving the technical effect of generating high-quality scene-level multi-view images, thereby solving the technical problem that the three-dimensional multi-view generation algorithm in the related technology cannot achieve a good generation level at the scene level.

[0216] It can be understood by those skilled in the art that Fig.12 The structure shown is for illustration only, and the electronic device A may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (Mobile Internet Devices, MID), a PAD, or other terminal devices. Fig.12 It does not limit the structure of the above electronic device. For example, the electronic device A may also include Fig.12 More or fewer components (such as network interfaces, display devices, etc.) shown in, or having Fig.12 Different configurations shown.

[0217] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0218] The embodiment of the present application also provides a computer-readable storage medium. Optionally, in this embodiment, the computer-readable storage medium can be used to store the program code executed by the model training method provided in the first embodiment.

[0219] Optionally, in this embodiment, the computer-readable storage medium may be located in any one of the computer terminals in a computer terminal group in a computer network, or in any one of the mobile terminals in a mobile terminal group.

[0220] Optionally, in this embodiment, a computer-readable storage medium is configured to store program code for performing the following steps: obtaining training data, wherein the training data includes: a reference image set, a first camera pose set, and a second camera pose set, the first camera pose set being different from the second camera pose set; obtaining prior information based on the reference image set, the first camera pose set, and the second camera pose set; training an initial multi-view generation model using the reference image set, the prior information, the first camera pose set, the second camera pose set, and preset mask information to obtain a target multi-view generation model, wherein the target multi-view generation model is used to perform multi-view generation on an input original view image to obtain an extended view image.

[0221] An embodiment of the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements any one of the above-mentioned model training methods.

[0222] Optionally, in this embodiment, the computer program product is used to perform the following steps when the processor executes the program code: obtain training data, wherein the training data includes: a reference image set, a first camera pose set and a second camera pose set, the first camera pose set is different from the second camera pose set; based on the reference image set, the first camera pose set and the second camera pose set, obtain prior information; use the reference image set, the prior information, the first camera pose set, the second camera pose set and preset mask information to train an initial multi-view generation model to obtain a target multi-view generation model, wherein the target multi-view generation model is used to perform multi-view generation on the input original view image to obtain an extended view image.

[0223] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0224] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0225] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0226] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0227] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0228] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0229] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A model training method, characterized in that: include: Acquire training data, wherein the training data includes: a reference image set, a first camera pose set, and a second camera pose set, wherein the first camera pose set is different from the second camera pose set; Acquire prior information based on the reference image set, the first camera pose set, and the second camera pose set; The reference image set, the prior information, the first camera pose set, the second camera pose set and the preset mask information are used to train the initial multi-view generation model to obtain a target multi-view generation model, wherein the target multi-view generation model is used to perform multi-view generation on the input original perspective image to obtain an extended perspective image.

2. The model training method according to claim 1, characterized in that: Based on the reference image set, the first camera pose set, and the second camera pose set, acquiring the prior information includes: Obtaining a metric depth map of the reference image set using the monocular depth map of the reference image set, and obtaining a unified spatial coordinate corresponding to the color pixel value using the color pixel value of the reference image set; Back-projection is performed based on the color pixel values, the unified space coordinates, the metric depth map, the relative viewing angle between the first camera pose set and the second camera pose set, the intrinsic parameter matrix corresponding to the first camera pose set, and the intrinsic parameter matrix corresponding to the second camera pose set to obtain the prior information, wherein the prior information includes: the projected color pixel values ​​and the projected unified space coordinates.

3. The model training method according to claim 2, characterized in that: Acquiring the metric depth map of the reference image set using the monocular depth map of the reference image set includes: Get the depth map to be aligned; Image correction is performed on the monocular depth based on the depth map to be aligned to obtain the metric depth map.

4. The model training method according to claim 2, characterized in that: Using the color pixel value of the reference image set to obtain the unified space coordinates corresponding to the color pixel value includes: Projecting the color pixel value to a camera coordinate system using the metric depth map and an intrinsic parameter matrix corresponding to the first camera pose set to obtain a first projection result; Projecting the first projection result to a world coordinate system according to an extrinsic parameter matrix corresponding to the first camera pose set to obtain a second projection result; The second projection result is normalized to obtain the unified space coordinates.

5. The model training method according to claim 1, characterized in that: Acquiring the first camera pose set and the second camera pose set includes: The first camera pose set and the second camera pose set are acquired by using a preset ray method and the reference image set.

6. The model training method according to claim 2, characterized in that: The initial multi-view generation model is trained using the reference image set, the prior information, the first camera pose set, the second camera pose set, and the preset mask information to obtain the target multi-view generation model, including: Training the initial multi-view generation model using the reference image set, the prior information, the first camera pose set, the second camera pose set, and the preset mask information to obtain an intermediate multi-view generation model; Using a verification set to verify the intermediate multi-view generation model to obtain a verification result; In response to the verification result not satisfying a preset condition, the model parameters of the intermediate multi-view generation model are adjusted until the verification result satisfies the preset condition, thereby obtaining the target multi-view generation model.

7. The model training method according to claim 6, characterized in that: The preset mask information includes: a first mask set and a second mask set. The initial multi-view generation model is trained using the reference image set, the prior information, the first camera pose set, the second camera pose set, and the preset mask information to obtain the intermediate multi-view generation model, including: determining a reference input feature based on the color pixel value, the first mask set, the first camera pose set, and the unified space coordinates, and determining a target input feature based on the projected color pixel value, the second mask set, the second camera pose set, and the projected unified space coordinates, wherein the binary value of the first mask set is opposite to the binary value of the second mask set; The initial multi-view generation model is trained using the reference input features and the target input features to obtain the intermediate multi-view generation model.

8. The model training method according to claim 1, characterized in that: The initial multi-view generation model includes: multiple self-attention modules, wherein the multiple self-attention modules are used to rescale the key features corresponding to the reference image set according to a preset scale coefficient to calculate a key-value pair attention map.

9. The model training method according to claim 1, characterized in that: The reference image set includes: a plurality of domain image sets for joint training, wherein the plurality of domain image sets are respectively assigned labels of different categories.

10. An image generation method, characterized in that: include: Get the original view image; Using the target multi-perspective generation model to perform multi-perspective generation on the original perspective image to obtain an extended perspective image; Wherein, the target multi-view generation model is obtained by using the model training method described in any one of claims 1 to 9.

11. The image generation method according to claim 10, characterized in that: The image generation method further comprises: A three-dimensional reconstruction operation or a video generation operation is performed based on the original viewing angle image and the extended viewing angle image.

12. An image generation method, characterized in that: include: Acquiring an image generation request through a first application programming interface, wherein the request data carried in the image generation request includes: an original view image; An image generation response is returned through a second application programming interface, wherein the response data carried in the image generation response includes: an extended perspective image, wherein the extended perspective image is obtained by performing multi-perspective generation on the original perspective image using a target multi-perspective generation model, and the target multi-perspective generation model is obtained using the model training method described in any one of claims 1 to 9.

13. An image generation method, characterized in that: include: Acquire a currently input image generation dialogue request, wherein the request data carried in the image generation dialogue request includes: an original viewing angle image; In response to the image generation dialogue request, an image generation dialogue reply is returned, wherein the information carried in the image generation dialogue reply includes: an extended view image, wherein the extended view image is obtained by performing multi-view generation on the original view image using a target multi-view generation model, wherein the target multi-view generation model is obtained using the model training method according to any one of claims 1 to 9; The extended viewing angle image is displayed in a graphical user interface.

14. An electronic device, characterized in that: include: A memory storing an executable program; A processor, used to run the program, wherein the program, when running, executes the model training method described in any one of claims 1 to 9 or the image generation method described in any one of claims 10 to 13.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored executable program, wherein when the executable program is running, the device where the computer-readable storage medium is located is controlled to execute the model training method described in any one of claims 1 to 9 or the image generation method described in any one of claims 10 to 13.

16. A computer program product, characterized in that It comprises a computer program which, when executed by a processor, implements the model training method described in any one of claims 1 to 9 or the image generation method described in any one of claims 10 to 13.

Images