Multi-view image generation method and multi-view image generation model training method

Abstract

The invention provides a multi-view image generation method and a multi-view image generation model training method. The multi-view-angle image generation method comprises the steps that a target single-view-angle image of a target object is obtained, the target single-view-angle image is input into a pre-trained multi-view-angle image generation model, m-view-angle images of the target object are output, the image of one view angle comprises n-domain images, m and n are integers larger than 1, and m is an integer larger than 1. The images of different domains of the target object under the same view angle have consistency in physical attributes. A single view angle can be expanded to multiple view angles, and richer visual information is provided; moreover, the physical attributes of the domain images in the multi-view image are consistent, so that vivid illumination and material performance can be ensured, and the details of the generated image are richer and more accurate.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method for generating a multi-view image and a method for training a multi-view image generation model. Background Art

[0002] In certain tasks in the field of computer technology, it is often necessary to obtain images of an object from different perspectives.

[0003] In the related art, images under multiple perspectives can be generated by generating models. Among them, the diffusion model is a generation model, and images under multiple perspectives can be generated by the diffusion model. For example, the multiple perspectives can be 2 perspectives, and the diffusion model can generate 2 images, one perspective corresponding to one image.

[0004] However, the above-mentioned method for generating multi-view images has disadvantages such as low richness of generated images and large differences between images.

[0005] It is worth noting that the contents of the above-mentioned related technologies are merely information known to the inventor personally, which does not mean that the above-mentioned information has entered the public domain before the filing date of this specification, nor does it mean that it can become the prior art of this specification. Summary of the invention

[0006] The present specification provides a method for generating a multi-view image and a method for training a multi-view image generation model to avoid at least one of the above-mentioned technical problems.

[0007] In a first aspect, this specification provides a method for generating a multi-view image, comprising:

[0008] Obtain a target single-view image of a target object;

[0009] Input the target single-view image into a pre-trained multi-view image generation model, and output images of m views of the target object, where one view includes images of n domains;

[0010] Wherein, both m and n are integers greater than 1; and the images of each domain of the target object are consistent in physical properties.

[0011] In a second aspect, this specification provides a training method for a multi-view image generation model, comprising:

[0012] Obtain a sample single-view image of the sample object;

[0013] Inputting the sample single-view image into a basic network model, so as to predict, based on the basic network model and in accordance with physical constraints that follow physical laws, images of the sample object at m viewpoints and images of n domains corresponding to each of the images at the m viewpoints of the sample object, wherein both m and n are integers greater than 1, and the physical constraints are used to constrain the images of the sample object at m viewpoints and images of n domains corresponding to each of the images at the m viewpoints of the sample object to have consistency in physical properties;

[0014] Taking minimizing the difference between the images of the sample object from m perspectives and the preset true values ​​corresponding to each domain image as the training goal, the parameters of the basic network model are updated to obtain a multi-perspective image generation model.

[0015] In a third aspect, this specification provides a system for generating a multi-view image, comprising:

[0016] at least one storage medium storing at least one instruction set for generating multi-view images;

[0017] At least one processor is communicatively connected to the at least one storage medium, wherein when the at least one processor is running, it reads the at least one instruction set and executes the method as described in the first aspect according to the instructions of the at least one instruction set.

[0018] In a fourth aspect, this specification provides a training system for a multi-view image generation model, comprising:

[0019] at least one storage medium storing at least one instruction set for training a multi-view image generation model;

[0020] At least one processor is communicatively connected to the at least one storage medium, wherein when the at least one processor is running, it reads the at least one instruction set and executes the method as described in the second aspect according to the instructions of the at least one instruction set.

[0021] In a fifth aspect, the present specification provides a computer-readable non-temporary storage medium, wherein the computer-readable non-temporary storage medium stores at least one instruction set, and the at least one instruction set is executed by at least one processor to implement the method described in the first aspect or the second aspect.

[0022] It can be seen from the above technical solutions that the multi-view image generation method and the multi-view image generation model training method provided in this specification, in which the multi-view image corresponding to the target single-view image is obtained by predicting the multi-view image generation model, so as to expand from a single view to a multi-view and provide richer visual information; moreover, by having the images in each domain in the multi-view image have consistency in physical properties, realistic lighting and material performance can be ensured, thereby making the details of the generated image richer and more accurate.

[0023] Other functions of the multi-view image generation method and the multi-view image generation model training method provided in this specification will be partially listed in the following description. The creative aspects of the multi-view image generation method and the multi-view image generation model training method provided in this specification can be fully explained by practicing or using the methods, devices and combinations described in the following detailed examples. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of this specification, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0025] Figure 1 A schematic diagram of an application scenario of the method for generating a multi-view image provided in an embodiment of this specification;

[0026] Figure 2 A schematic diagram of the structure of a system for generating multi-view images provided in an embodiment of this specification;

[0027] Figure 3 A schematic diagram of a flow chart of a method for generating a multi-view image according to an embodiment of this specification;

[0028] Figure 4 A schematic diagram of the principle of a method for generating a multi-view image provided in an embodiment of this specification;

[0029] Figure 5 A schematic diagram of a flow chart of a method for training a multi-view image generation model provided in an embodiment of the present disclosure;

[0030] Figure 6 A schematic diagram of the principles of a method for training a multi-view image generation model provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0031] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with this specification. Instead, they are merely examples of devices and methods consistent with some aspects of this specification as detailed in the appended claims.

[0032] It should be understood that the terms "including" and "having" and any variations thereof in the embodiments of this specification are intended to cover but not exclude inclusion. For example, a product or device comprising a series of components is not necessarily limited to those components explicitly listed, but may include other components not explicitly listed or inherent to these products or devices.

[0033] In the embodiments of this specification, the term "and / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0034] The term "plurality" in the embodiments of the present specification refers to two or more than two, and other quantifiers are similar.

[0035] The terms "first", "second", "third", etc. in this specification are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise indicated. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, for example, they can be implemented in an order other than those given in the diagrams or descriptions of the embodiments of this specification.

[0036] The term "unit / module" as used in this specification refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0037] To facilitate the reader's understanding of this specification, at least some of the terms in this specification are explained as follows:

[0038] A generative model is a machine learning model that generates new data samples whose distribution is similar to the training data. For example, by learning the characteristics of the training data, the generative model can create new samples such as images, audio, or text.

[0039] The Diffusion Model is a generative model that is often used for image generation. It generates data based on the process of gradually adding and denoising Gaussian noise. It generates images by modeling the gradual change process of data distribution.

[0040] Physically Based Rendering (PBR) is a rendering technology that aims to simulate the real interaction between light and materials to achieve more realistic visual effects, including lighting, shadows, and material reflections.

[0041] Albedo is the ability of an object's surface to reflect light. It usually refers to the intensity of visible light reflected by an object under white light conditions, which affects the color and brightness of the object.

[0042] Roughness is a parameter that describes the microscopic unevenness of an object's surface, which affects the degree of scattering of light on the surface, thereby affecting the glossiness and effect of reflection.

[0043] Metallic describes the degree to which a material's surface displays metallic properties, which usually affects the reflection and color of light. Metal surfaces have higher reflections and specific color mapping.

[0044] To facilitate readers' understanding of this specification, the application scenarios of this specification are now introduced.

[0045] The technical solution provided in this specification is applicable to scenarios where multi-view images need to be generated based on single-view images.

[0046] For example, a user obtains a single-view image of a certain object (which may be a person or an object). Through the technical solution provided in this specification, the user can obtain images of the object from multiple viewpoints.

[0047] Figure 1 FIG. 1 is a schematic diagram of an application scenario of a method for generating a multi-view image according to an embodiment of the present specification (hereinafter referred to as a generating method), wherein the generating method of the present specification can be applied to: Figure 1 The application scenario 100 shown. Figure 1 As shown, the application scenario 100 may include a target user 101 , a client 102 , a server 103 , and a network 104 .

[0048] The target user 101 may be a user who triggers the generation of a multi-view image of the target object. For example, the target user 101 may perform a target operation on the client 102 to trigger the generation of a multi-view image. In some embodiments, the target user 101 may upload a target single-view image of the target object on the client 102 to trigger the generation of a multi-view image of the target object.

[0049] The client 102 may be an electronic device that provides interactive functions to the target user 101. For example, the client 102 may provide an interactive interface to the target user 101, and the target user 101 may perform interactive operations in the interactive page. In some embodiments, the client 102 executes the generation method described in this specification in response to detecting the target single-view image initiated by the target user 101. At this time, the client 102 may store data or instructions for executing the generation method described in this specification, and may execute or be used to execute the data or instructions. In some embodiments, the client 102 may include a hardware device with a data information processing function and the necessary program required to drive the hardware device to work, so as to execute the generation method described in this specification.

[0050] In some embodiments, the client 102 may include a mobile device, a tablet computer, a laptop computer, a built-in device of a motor vehicle, or the like, or any combination thereof. In some embodiments, the mobile device may include a smart home device, a smart mobile device, a virtual reality device, an augmented reality device, or the like, or any combination thereof. In some embodiments, the smart home device may include a smart TV, a desktop computer, or the like, or any combination thereof. In some embodiments, the smart mobile device may include a smart phone, a personal digital assistant, a gaming device, a navigation device, or the like, or any combination thereof. In some embodiments, the built-in device in a motor vehicle may include an onboard computer, an onboard TV, or the like. In some embodiments, the client 102 may include a capture device for capturing a target single-view image of a target object.

[0051] In some embodiments, the client 102 may be installed with one or more applications (Application, APP). APP can provide the target user 101 with the ability and interface to interact with the outside world through the network 104. APP includes but is not limited to: web browser APP programs, search APP programs, chat APP programs, shopping APP programs, video APP programs, financial management APP programs, instant messaging tools, email clients, social platform software, etc. In some embodiments, a target APP can be installed on the client 102. The target APP can collect a single-view image of the target for the client 102.

[0052] like Figure 1 As shown, the client 102 can be connected to the server 103 for communication. The server 103 can be connected to one client 102 for communication, or can be connected to multiple clients 102 for communication. In some embodiments, the client 102 can interact with the server 103 through the network 104 to receive or send messages, etc. For example, the client 102 can interact with the server 103 through the network 104 to send a target single-view image to the server 103.

[0053] The server 103 may be a server that provides various services. For example, the server 103 may be a cloud server or a local server. The server 103 may be connected to a client 102 for communication and receive data sent by the client 102, or may be connected to multiple clients 102 for communication and receive data sent by each client 102.

[0054] In some embodiments, the generation method described in this specification can be executed on the server 103. In this case, the server 103 can store data or instructions for executing the generation method described in this specification, and can execute or be used to execute the data or instructions. The server 103 can include a hardware device with a data information processing function and a necessary program required to drive the hardware device to work.

[0055] The network 104 is a medium for providing a communication connection between the client 102 and the server 103. The network 104 can facilitate the exchange of information or data. Figure 1 As shown, the client 102 and the server 103 can be connected to the network 104 respectively, and transmit information or data to each other through the network 104.

[0056] In some embodiments, the network 104 may be any type of wired or wireless network, or a combination thereof. For example, the network 104 may include a cable network, a wired network, a fiber optic network, a telecommunications network, an intranet, the Internet, a local area network (LAN), a wide area network (WAN), a wireless local area network (WLAN), a metropolitan area network (MAN), a public switched telephone network (PSTN), a Bluetooth network TM, a short-range wireless network (ZigBee TM), a near field communication (NFC) network, or a similar network.

[0057] In some embodiments, the network 104 may include one or more network access points. For example, the network 104 may include a wired or wireless network access point, such as a base station or an Internet exchange point, through which one or more components of the client 102 and the server 103 can connect to the network 104 to exchange data or information.

[0058] It is worth mentioning that Figure 1The number of clients 102, servers 103, and networks 104 in the example are merely illustrative. Depending on the implementation requirements, there may be any number of clients 102, servers 103, and networks 104. The generation method provided in this specification may be executed entirely on the client 102, entirely on the server 103, or partially on the client 102 and partially on the server 103.

[0059] That is to say, Figure 1 and for Figure 1 The above description is only used to illustrate the application scenarios to which the generation method of this specification may be applicable, and should not be understood as a limitation on the application scenarios.

[0060] Figure 2 The hardware structure diagram of a multi-view image generation system (hereinafter referred to as generation system) 200 provided according to an embodiment of the present specification is shown. The generation system 200 can execute the generation method described in the present specification. The generation method is introduced in other parts of the present specification. When the generation method is executed on the client 102, the generation system 200 can be the client 102. When the generation method is executed on the server 103, the generation system 200 can be the server 103. When the generation method is partially executed on the client 102 and partially executed on the server 103, the generation system 200 can be a system including the client 102 and the server 103.

[0061] like Figure 2 As shown, the generation system 200 may include at least one storage medium 203 and at least one processor 202. In some embodiments, the generation system 200 may further include a communication port 204 and an internal communication bus 201. The generation system 200 may further include an I / O component 205.

[0062] The internal communication bus 201 can connect different system components. For example, the internal communication bus 201 can connect the storage medium 203, the processor 202, the communication port 204 and the I / O component 205.

[0063] I / O component 205 supports input / output between generation system 200 and other components.

[0064] The communication port 204 is used to generate data communication between the system 200 and the outside world. For example, the communication port 204 can be used to generate data communication between the system 200 and the network 104. The communication port 204 can be a wired communication port or a wireless communication port.

[0065] The storage medium 203 may include a data storage device. The data storage device may be a non-temporary storage medium or a temporary storage medium. For example, the data storage device may include one or more of a disk 2031, a read-only storage medium (ROM) 2032, or a random access storage medium (RAM) 2033. The storage medium 203 also includes at least one instruction set stored in the data storage device. The instruction set includes computer program code, and the computer program code may include a program, a routine, an object, a component, a data structure, a process, a module, etc. for executing the generation method provided in this specification.

[0066] At least one processor 202 may be connected in communication with at least one storage medium 203. At least one processor 202 is used to execute the above-mentioned at least one instruction set. When the generation system 200 is running, at least one processor 202 reads the at least one instruction set and executes the generation method provided in this specification according to the instructions of the at least one instruction set. The processor 202 may execute all the steps included in the generation method. The processor 202 may be in the form of one or more processors. In some embodiments, the processor 202 may include one or more hardware processors, such as a microcontroller, a microprocessor, a reduced instruction set computer (RISC), an application-specific integrated circuit (ASIC), an application-specific instruction set processor (ASIP), a central processing unit (CPU), a graphics processing unit (GPU), a physical processing unit (PPU), a microcontroller unit, a digital signal processor (DSP), a field programmable gate array (FPGA), an advanced RISC machine (ARM), a programmable logic device (PLD), any circuit or processor capable of performing one or more functions, etc., or any combination thereof.

[0067] For illustration purposes only, only one processor 202 is shown in the generation system 200 in the accompanying drawings. However, it should be noted that the generation system 200 in this specification may also include multiple processors, and therefore, the operations and / or method steps disclosed in this specification may be performed by one processor or by multiple processors in combination. For example, if the processor 202 of the generation system 200 is described in this specification as performing step A and step B, it should be understood that step A and step B may also be performed jointly or separately by two different processors 202 (e.g., the first processor performs step A, the second processor performs step B, or the first and second processors perform steps A and B together).

[0068] See also Figure 3 , Figure 3 This is a flow chart of the generation method provided in the embodiment of this specification. Figure 3 As shown, the method includes the following S301 and S302:

[0069] S301: Obtain a target single-view image of a target object.

[0070] It can be understood that this embodiment is an introduction to the technical solution of generating a multi-view image based on a multi-view image generation model from the inference stage, and the following also includes an introduction to the technical solution of obtaining a multi-view image generation model through training from the training stage. Therefore, in order to distinguish different stages, the "target" in this embodiment and the "sample" in the following are used to distinguish the same type of technical features, which cannot be understood as a limitation of the technical features.

[0071] For example, the "target" in this embodiment and the "sample" below are intended to facilitate the distinction between the same type of technical features "object" and "single-view image" in the reasoning stage and the training stage, and should not be understood as a limitation on "object" and "single-view image".

[0072] The target object can be a person or an object. Figure 4 As shown, the target object may be a helmet. Accordingly, the target single-view image is an image of the helmet taken from a certain perspective. It should be understood that Figure 4 The target single-view image shown in the figure is just an example of a certain view angle, and cannot be understood as a limitation of the view angle. That is, the input target single-view image can be an image of any view angle.

[0073] The following example can be used to obtain a single-view image of the target:

[0074] In one example, the generation system may be connected to an image acquisition device and receive a target single-view image sent by the image acquisition device.

[0075] In another example, the generation system may provide a tool for loading images, and the user may transfer the target single-view image to the generation system through the tool for loading images.

[0076] Among them, the tool for loading images can be an interface for connecting to an external device, such as an interface for connecting to other storage devices, through which the target monoscopic image transmitted by the external device is obtained; the tool for loading images can also be a display device, such as the generation system can output an interface of the image loading function on the display device, and the user can import the target monoscopic image into the generation system through this interface.

[0077] S302: Input the target single-view image into a pre-trained multi-view image generation model, and output images of the target object from m perspectives, where one perspective image includes images of n domains.

[0078] Wherein, m and n are both integers greater than 1; the images of each domain of the target object are consistent in physical properties.

[0079] Exemplarily, the multi-view image generation model can be pre-trained by a training system. The training system can be the same system as the generation system or a different system, which is not limited in this embodiment. The multi-view generation model is trained to predict a multi-view image based on a single-view image.

[0080] Accordingly, in this embodiment, the multi-view image generation model performs prediction based on the target single-view image to obtain images of m views. Moreover, for each of the m views, the image of the view includes images of multiple domains.

[0081] This embodiment has no other restrictions on m and n. For example, m can be equal to n; m can also be greater than n; m can also be less than n. For example, taking m and n as equal and 6 as an example, the generation system can generate 36 domain images based on the multi-view image generation model. Similarly, this embodiment does not limit the specific perspectives in the multi-view.

[0082] It is worth noting that, in this embodiment, the images in each domain generated by the generation system are consistent in physical properties, that is, the images under different viewing angles can correctly reflect the real physical properties of the target object.

[0083] Exemplarily, the physical attributes may include illumination, geometry, and material attributes. That is, each domain image can truly and consistently reflect the illumination in the imaging environment of the target object, the geometric features of the target object, and the physical material features of the target object.

[0084] This means that domain images under different viewing angles can correctly reflect the material and lighting characteristics of the target object, etc. That is, each domain image can realistically and consistently reflect the lighting effects, geometric characteristics, and material performance.

[0085] Based on the above analysis of S301 and S302, it can be known that in this embodiment, a multi-perspective image corresponding to the target single-perspective image is obtained by predicting based on a multi-perspective image generation model, so as to expand from a single perspective to a multi-perspective and provide richer visual information; by ensuring that the images in each domain in the multi-perspective image have consistency in physical properties, realistic lighting and material representation can be ensured, thereby making the details of the generated image richer and more accurate.

[0086] In some embodiments, the n domains include multiple ones of: color, normal, depth, albedo, roughness, and metallic.

[0087] Exemplarily, combined with the above analysis, n>1, therefore, the domain image of the target object at one viewing angle can be two of the color image, normal image, depth image, albedo image, roughness image, and metalness image, or it can be three or more, up to all six.

[0088] In some embodiments, n is an integer greater than 2, and the n domains are used to characterize geometric properties and / or physical material properties of the target object.

[0089] Exemplarily, for each of the m viewing angles, the generation system generates at least three domain images under the viewing angle. The three domain images can represent the geometric properties of the target object; can also represent the physical material properties of the target object; or can represent both the geometric properties of the target object and the physical material properties of the target object.

[0090] For example, n can be 6. Figure 4 As shown, the images of the six domains of a viewing angle may include a color image, a normal image, a depth image, an albedo image, a roughness image, and a metalness image at the viewing angle.

[0091] In some embodiments, S302 may include the following steps 1 to 3:

[0092] Step 1: Input the target single-view image into the multi-view image generation model to extract key features representing the target object from the target single-view image based on the multi-view image generation model.

[0093] The key features characterizing the target object may be understood as image features in the target single-view image that are used to characterize the characteristics of the target object.

[0094] Exemplary, combined Figure 4 It can be seen that the multi-view image generation model includes a residual block (ResBlock). The residual block can extract key features that characterize the target object from the target single-view image.

[0095] Step 2: Based on the extracted key features characterizing the target object, determine the features used to characterize the interdependencies between the target object in different viewpoints and different domains of the m viewpoints.

[0096] In some embodiments, step 2 may include the following steps 21 and 22:

[0097] Step 21: Based on the extracted key features characterizing the target object, determine features characterizing the dependency relationship between different viewpoints of the target object in the m viewpoints.

[0098] Continue reading Figure 4, the multi-view image generation model can also include multi-view self-attention. The multi-view self-attention mechanism is used to capture the dependencies between different perspectives. That is, through the self-attention mechanism, the multi-view image generation model can determine the global dependencies between images of different perspectives, ensuring that the generated images are consistent under different perspectives.

[0099] Step 22: According to the determined features characterizing the dependency relationship of the target object between different viewpoints of the m viewpoints, determine features characterizing the mutual dependency relationship of the target object under different viewpoints of the m viewpoints and between different domains.

[0100] In some embodiments, step 22 may include the following steps 221 and 222:

[0101] Step 221: According to the determined features characterizing the dependencies between different viewpoints of the target object in the m viewpoints, features characterizing the global dependencies between different domains of the target object in the same viewpoint of the m viewpoints are determined.

[0102] Continue reading Figure 4 , the multi-view image generation model can also include cross-domain self-attention. Cross-domain self-attention is used to process the dependencies between different domains. That is, through cross-domain self-attention, the multi-view image generation model can learn the global dependencies between images in different domains and ensure that the generated images are consistent between different domains.

[0103] In some embodiments, the cross-domain self-attention includes a trained domain switching parameter. Through the trained switching parameter, the cross-domain self-attention determines which different domains need to extract global dependencies.

[0104] For example, if the switching parameter of two domains is 1, it means that the cross-domain self-attention of the two domains needs to be calculated to extract the dependency relationship; conversely, if the switching parameter of a domain is 0, it means that the attention of the domain and other domains is calculated.

[0105] In some embodiments, among the n domains, a main domain may be included. The main domain is used to characterize the need to calculate cross-domain attention between the domain and other domains separately. That is, the generation system can guide the generation of other domains based on the main domain as a reference. This not only improves the quality of the generated images, but also ensures their physical rationality. In this way, the multi-view image generation model is able to generate realistic multi-view images while maintaining geometric and material details.

[0106] Step 222: According to the determined features of the global dependency relationship between different domains of the target object at the same viewpoint of the m viewpoints, features for characterizing the mutual dependency relationship between different domains of the target object at different viewpoints of the m viewpoints are determined.

[0107] Continue reading Figure 4 , the multi-view image generation model can also include cross attention. Cross attention processes the interdependence between images from different perspectives and different domains. That is, through cross attention, the multi-view image generation model can determine the interdependence between images from different perspectives and different domains, ensuring that the generated images are consistent between different perspectives and different domains.

[0108] Step 3: Generate images of the target object from m perspectives according to the determined features of the target object at different perspectives of the m perspectives and the interdependence between different domains.

[0109] Continue reading Figure 4 , the multi-view image generation model can also include UNet. The input of UNet is the output of cross attention; the output of UNet is the image of the target object from m perspectives. For example, the output of UNet includes Figure 4 Images of 6 domains under one viewing angle are shown.

[0110] Based on the above analysis of steps 1 to 3, it can be seen that in this embodiment, by analyzing the interdependence between different perspectives, different domains under the same perspective, and different domains under different perspectives, it is ensured that the generated multi-view images are consistent in physical properties. For example, the normal image should maintain the same geometric structure under different perspectives, the color image should maintain the same material color under different perspectives, and the depth image should maintain a consistent spatial position relationship under different perspectives, etc., which are not listed here one by one. In addition, by analyzing the complex interactions between images in different domains (such as color, normal, depth, etc.), the multi-view image generation model can capture these dependencies and ensure that the generated images are coordinated and consistent between various domains. For example, changes in the normal image will affect the lighting effect, and changes in the color image will affect the material performance. The multi-view image generation model can ensure that these changes are coherent under different perspectives. Moreover, by analyzing the interdependence between different perspectives, different domains under the same perspective, and different domains under different perspectives, the multi-view image generation model can better understand the three-dimensional structure and physical properties of the target object, thereby generating more realistic multi-view images.

[0111] In some embodiments, the multi-view image generation model is trained based on physical constraints, and the physical constraints are used to constrain the generation of images of each domain of the target object to follow physical laws so as to truly and consistently represent the target object.

[0112] Physical constraints can be understood as introducing a series of physical laws and rules during the training process of the multi-view image generation model to guide and restrict the multi-view images generated by the multi-view image generation model to conform to the physical properties of the real world. For example, physical constraints may include geometric consistency, lighting consistency, material consistency, motion consistency, etc.

[0113] Geometric consistency can be understood as the fact that depth images from different perspectives should maintain a consistent spatial structure. For example, the shape, size, and position of an object should remain unchanged from different perspectives.

[0114] Lighting consistency can be understood as the lighting effects at different viewing angles should follow the same lighting model. For example, the position, intensity, and direction of the light source should be consistent in all viewing angles, and the shadow projection and reflection effects should also conform to physical laws.

[0115] Material consistency can be understood as the material properties (such as color, roughness, transparency, etc.) under different viewing angles should remain consistent to ensure that the appearance of the object under different viewing angles is realistic and credible.

[0116] Motion consistency can be understood as, if it involves dynamic scenes, the motion trajectory and speed of the target object should conform to the laws of physical dynamics to ensure the realism of the motion.

[0117] In this embodiment, by introducing physical constraints, the images generated by the multi-view image generation model strictly follow the laws of physics in terms of geometry, lighting, material, etc., making the generated images more realistic and credible. Users can observe objects from different perspectives without feeling any obvious unnaturalness or disobedience.

[0118] In order to further understand the implementation principle of the multi-view image generation model to generate multi-view images based on single-view images, the training process of the multi-view image generation model is now elaborated in detail from the training stage.

[0119] See also Figure 5 , Figure 5 The flowchart of the training method of the multi-view image generation model provided by the embodiment of the present disclosure (hereinafter referred to as the training method) is shown. Figure 5 As shown, the method includes the following S501-S503:

[0120] S501: Obtain a sample single-view image of a sample object.

[0121] It is worth noting that, in order to avoid cumbersome description, the technical features of this embodiment that are the same or similar to those in the above examples will not be described in detail in this embodiment.

[0122] For example, regarding the principle of S501, reference may be made to the description of S301 above.

[0123] S502: Input the sample single-view image into the basic network model to predict the images of the sample object from m perspectives and the images of n domains corresponding to each image in the images of the sample object from m perspectives based on the basic network model according to the physical constraints that follow the laws of physics, wherein m and n are both integers greater than 1, and the physical constraints are used to constrain the images of the sample object from m perspectives and the images of n domains corresponding to each image in the images of the sample object from m perspectives to have consistency in physical properties.

[0124] Exemplarily, a single-view image is input into a basic network model so that the basic network model predicts images of m viewpoints of the sample object and images of n domains corresponding to each viewpoint (such as color images, normal images, depth images, albedo images, roughness images, metalness images, etc.) according to pre-set physical constraints.

[0125] By introducing physical constraints, we ensure that the generated multi-view images and images in each domain have consistency in physical properties. For example, the normal image should follow the geometric structure of the object surface, the depth image should reflect the spatial position relationship of the object, the color image should maintain the color characteristics of the material, and so on.

[0126] S503: Taking minimizing the difference between the images of the sample object from m perspectives and the preset true values ​​corresponding to the images in each domain as the training goal, the parameters of the basic network model are updated to obtain a multi-perspective image generation model.

[0127] By minimizing the difference between the predicted results (including images of m perspectives of sample objects and domain images generated based on the basic network model) and the preset true values ​​(which can be marked in advance), the generated images and domain images are ensured to be as close to the actual situation as possible.

[0128] In some embodiments, S502 may include the following steps 1 to 3:

[0129] Step 1: Based on the basic network model, extract the features of the sample objects under different perspectives of m viewpoints and the interdependence between different domains.

[0130] In some embodiments, step 1 may include: extracting features of global dependencies between different domains of sample objects at the same perspective of m perspectives based on the basic network model, and determining features of mutual dependencies between sample objects at different perspectives of m perspectives and between different domains based on the extracted features of global dependencies between different domains of sample objects at the same perspective of m perspectives.

[0131] Combined with the above examples (such as Figure 4 corresponding examples) and Figure 6 It can be seen that the basic network model can include: residual block, multi-view self-attention, cross-domain self-attention, and cross-attention. The principles of each component in the basic network model can be found in the above examples and will not be repeated here.

[0132] Correspondingly, the training system inputs the sample single-view image into the basic network model, and the basic network model determines the characteristics of the interdependence between the sample objects in m different perspectives and different domains through sequentially connected residual blocks, multi-view self-attention, cross-domain self-attention, and cross-attention.

[0133] It is worth noting that, in some embodiments, the parameters of the basic network model include a domain switching parameter; the domain switching parameter is used to determine the domain from which the basic network model needs to extract features of the global dependency relationship.

[0134] Exemplarily, the cross-domain self-attention includes a switching parameter. Through the switching parameter, the cross-domain self-attention determines which different domains need to extract global dependencies.

[0135] For example, if the switching parameter of two domains is 1, it means that the cross-domain self-attention of the two domains needs to be calculated to extract the dependency relationship; conversely, if the switching parameter of a domain is 0, it means that the attention of the domain and other domains is calculated.

[0136] In some embodiments, the n domains may include a main domain, which is used to indicate that cross-domain attention between the domain and other domains needs to be calculated separately.

[0137] For example, the main domain can be a color domain. So as to reflect the information of the appearance of the object relatively and most completely based on the color domain. That is to say, through the technical solution provided by this embodiment, it is possible to use the color image as a reference to guide the generation of other domains while ensuring the geometric consistency between different domains. This not only improves the quality of the generated images, but also ensures their physical rationality. In this way, the basic network model can learn to generate realistic multi-view images while maintaining geometric and material details.

[0138] It is worth noting that when the cross-domain self-attention includes domain switching parameters and the main domain, the cross-domain attention can be first calculated through the main domain, and then weighted fused with the switching parameters to obtain the features finally captured by the cross-domain self-attention.

[0139] Step 2: Based on the extracted features of the sample object at different viewing angles of the m viewing angles and the interdependence between different domains, an image of the sample object in n domains at each of the m viewing angles is generated.

[0140] Step 3: Perform physical based rendering (PBR) on the images of the n domains at each of the m viewing angles according to the physical constraints to obtain images of the sample object at m viewing angles.

[0141] Exemplary, combined Figure 6 It can be seen that the basic network model can also include UNet and PBR lighting equation (PBRCondition).

[0142] Among them, the input of UNet is the output of cross attention, and the output of UNet is the image of n domains of the sample object in each of the m perspectives. For example, the output of UNet is 36 domain images.

[0143] The input of the PBR lighting equation is the output of UNet. For example, the input of the PBR lighting equation is 36 domain images. Accordingly, the PBR lighting equation performs inverse rendering on the 36 domain images based on physical constraints, and restores the images under 6 perspectives corresponding to the sample single-view image (that is, 6 images are obtained).

[0144] For example, the PBR lighting equation can be expressed by Formula 1:

[0145]

[0146] Where Ω represents the hemispherical domain for integration, ω O is the viewing direction, n is the normal, x is the surface point with the normal, L i is the incident light, ω i is the incident direction, d is the differential, f r is the bidirectional reflectance distribution function.

[0147] Based on the above analysis of steps 1 to 3, it can be seen that in this embodiment, the PBR lighting equation is used as an additional constraint to ensure that the generated image complies with the physical lighting law and other physical laws, such as making the six images truly and consistently represent the sample object.

[0148] In some embodiments, S503 may include steps 1 to 3:

[0149] Step 1: Calculate the image loss between the images of the sample object at m perspectives and the corresponding preset true values.

[0150] Combined with the above example, if m is 6, the image loss is the sum of the losses corresponding to each of the 6 images.

[0151] Step 2: Calculate the domain loss between each domain image of the sample object and the corresponding preset true value.

[0152] Combined with the above example, if m and n are both 6, there are 36 domain images in total, and the domain loss is the sum of the losses corresponding to each of the 36 domain images.

[0153] Step 3: Taking minimizing image loss and domain loss as the training objective, the parameters of the basic network model are updated to obtain a multi-view image generation model, where the weight of image loss is smaller than the weight of domain loss.

[0154] Exemplarily, the image and the domain have weights respectively, and the weight of the domain is greater than the weight of the image. Accordingly, the parameters of the basic network model are updated with minimizing the weighted loss of the image and the domain as the training goal, such as including updating the domain switching parameters in the above example, to obtain a multi-view image generation model.

[0155] It is worth noting that the above examples are only used to illustrate the possible implementation methods of the generation method and training method of this specification, and cannot be understood as limiting the implementation methods of the generation method and training method of this specification. Exemplarily, some of the technical features in the above can be combined to obtain a new embodiment; new technical features can be added on the basis of the above examples to obtain a new embodiment; some technical features can be reduced on the basis of the above examples to obtain a new embodiment; some technical features in the above examples can be replaced with other technical features; some technical features and the order of the above examples can be adjusted to obtain a new embodiment, etc., which will not be listed here one by one.

[0156] This specification also provides a computer-readable non-temporary storage medium, which stores at least one instruction set. When the at least one instruction set is executed by a processor, the steps of the generation method and training method described in this specification are implemented.

[0157] In some possible implementations, various aspects of this specification may also be implemented in the form of a program product, which includes a program code. Taking the generation method as an example (the training method is similar, so it is not repeated), when the program product is run on the generation system 200, the program code is used to make the generation system 200 perform the steps of the generation method described in this specification. The program product for implementing the above method can use a portable compact disk read-only memory (CD-ROM) to include program code, and can be run on the generation system 200. However, the program product of this specification is not limited to this. In this specification, the readable storage medium can be any tangible medium containing or storing a program, which can be used by the instruction execution system or used in combination with it. The program product can use any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples of readable storage media include: an electrical connection with one or more conductors, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. The computer-readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above. Program code for performing the operations of the present specification may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the generation system 200, partially on the generation system 200, as a stand-alone software package, partially on the generation system 200 and partially on a remote generation system, or entirely on a remote generation system 200.

[0158] The above is a description of a specific embodiment of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require a specific order or a continuous order to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0159] In summary, after reading this detailed disclosure, those skilled in the art will appreciate that the foregoing detailed disclosure may be presented only by way of example and may not be limiting. Although not explicitly stated herein, those skilled in the art will appreciate that this specification requires various reasonable changes, improvements and modifications to the embodiments. These changes, improvements and modifications are intended to be proposed by this specification and are within the spirit and scope of the exemplary embodiments of this specification.

[0160] In addition, certain terms in this specification have been used to describe embodiments of this specification. For example, "one embodiment", "an embodiment" and / or "some embodiments" mean that a particular feature, structure or characteristic described in conjunction with the embodiment may be included in at least one embodiment of this specification. Therefore, it can be emphasized and should be understood that two or more references to "an embodiment" or "one embodiment" or "an alternative embodiment" in various parts of this specification do not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics may be appropriately combined in one or more embodiments of this specification.

[0161] It should be understood that in the foregoing description of the embodiments of this specification, in order to help understand a feature and for the purpose of simplifying this specification, this specification combines various features in a single embodiment, figure or its description. However, this does not mean that the combination of these features is necessary. When reading this specification, it is entirely possible for a person skilled in the art to mark out some of the devices as separate embodiments. In other words, the embodiments in this specification can also be understood as the integration of multiple secondary embodiments. And the content of each secondary embodiment is also valid when it is less than all the features of a single aforementioned disclosed embodiment.

[0162] Each patent, patent application, patent application publication, and other material, such as articles, books, specifications, publications, documents, literature, etc., cited herein (excluding any historical review documents related thereto) is hereby incorporated by reference for all purposes relevant to this document, such as in the specification and claims herein. However, if there is any inconsistency or conflict between the descriptions, definitions, and / or terminology of the aforementioned materials and the descriptions, definitions, and / or terminology used herein, the descriptions, definitions, and / or terminology used herein shall prevail.

[0163] Finally, it should be understood that the embodiments of the application disclosed herein are explanations of the principles of the embodiments of this specification. Other modified embodiments are also within the scope of this specification. Therefore, the embodiments disclosed in this specification are only used as examples and not as limitations. Those skilled in the art can adopt alternative configurations according to the embodiments in this specification to implement the applications in this specification. Therefore, the embodiments of this specification are not limited to the embodiments accurately described in the application.

Claims

1. A method for generating a multi-view image, comprising: Obtain a target single-view image of a target object; Input the target single-view image into a pre-trained multi-view image generation model, and output images of m views of the target object, where one view includes images of n domains; Wherein, both m and n are integers greater than 1; and the images of each domain of the target object are consistent in physical properties.

2. The method according to claim 1, wherein: The step of inputting the target single-view image into a pre-trained multi-view image generation model and outputting images of the target object from m perspectives includes: Inputting the target single-view image into the multi-view image generation model, so as to extract key features characterizing the target object from the target single-view image based on the multi-view image generation model; Based on the extracted key features characterizing the target object, determining features for characterizing the interdependencies between the target object at different viewing angles and different domains of the m viewing angles; According to the determined features of the target object at different viewing angles of the m viewing angles and the interdependence between different domains, images of the target object at m viewing angles are generated.

3. The method according to claim 2, wherein: The step of determining features for characterizing the interdependencies between the target object at different viewing angles and between different domains of the m viewing angles based on the extracted key features characterizing the target object includes: Determine, based on the extracted key features characterizing the target object, features characterizing a dependency relationship between different perspectives of the m perspectives of the target object; According to the determined features characterizing the dependency relationship between the target object at different perspectives of the m perspectives, features characterizing the mutual dependency relationship between the target object at different perspectives of the m perspectives and between different domains are determined.

4. The method according to claim 3, wherein: The determining, based on the determined features characterizing the dependency relationship between the target object at different viewpoints of the m viewpoints, features used to characterize the mutual dependency relationship between the target object at different viewpoints of the m viewpoints and between different domains includes: Determine, according to the determined features characterizing the dependency relationship of the target object between different perspectives of the m perspectives, features characterizing the global dependency relationship between different domains of the target object under the same perspective of the m perspectives; According to the determined features of the global dependency relationship between different domains of the target object at the same perspective of the m perspectives, features for characterizing the mutual dependency relationship between different domains of the target object at different perspectives of the m perspectives are determined.

5. The method according to any one of claims 1 to 4, wherein: The multi-view image generation model is obtained through training based on physical constraints, and the physical constraints are used to constrain the generation of images of each domain of the target object to follow physical laws so as to truly and consistently represent the target object.

6. The method according to claim 1, wherein: The physical properties include lighting, geometry, and material properties.

7. The method according to claim 1, wherein: The n is an integer greater than 2, and the n domains are used to characterize the geometric properties and / or physical material properties of the target object.

8. The method according to claim 1, wherein: The n domains include: color, normal, depth, albedo, roughness, and metalness.

9. A method for training a multi-view image generation model, comprising: Obtain a sample single-view image of the sample object; Inputting the sample single-view image into a basic network model, so as to predict, based on the basic network model and in accordance with physical constraints that follow physical laws, images of the sample object at m viewpoints and images of n domains corresponding to each of the images at the m viewpoints of the sample object, wherein both m and n are integers greater than 1, and the physical constraints are used to constrain the images of the sample object at m viewpoints and images of n domains corresponding to each of the images at the m viewpoints of the sample object to have consistency in physical properties; Taking minimizing the difference between the images of the sample object from m perspectives and the preset true values ​​corresponding to each domain image as the training goal, the parameters of the basic network model are updated to obtain a multi-perspective image generation model.

10. The method according to claim 9, wherein: The step of inputting the sample single-view image into a basic network model to predict, based on the basic network model and in accordance with physical constraints that follow physical laws, images of m viewpoints of the sample object and images of n domains corresponding to each of the images of the m viewpoints of the sample object, comprises: Extracting features of the interdependence relationship between the sample object at different perspectives and in different domains of the m perspectives based on the basic network model; Generate an image of n domains of the sample object at each of the m viewing angles according to the extracted features of the interdependence between different domains of the sample object at different viewing angles of the m viewing angles; According to the physical constraints, physically based rendering (PBR) is performed on the images of the n domains at each of the m viewing angles to obtain images of the sample object at the m viewing angles.

11. The method according to claim 9, wherein: The features of the interdependence relationship between the sample object at different perspectives and different domains of the m perspectives are extracted based on the basic network model, including: Extracting features of global dependency relationships between different domains of the sample object at the same viewing angle of the m viewing angles based on the basic network model; According to the extracted features of the global dependency relationship between different domains of the sample object at the same viewing angle of the m viewing angles, features of the mutual dependency relationship between the sample object at different viewing angles and different domains of the m viewing angles are determined.

12. The method according to claim 11, wherein: The parameters of the basic network model include a domain switching parameter; the domain switching parameter is used to determine the domain from which the basic network model needs to extract features of the global dependency relationship.

13. The method according to any one of claims 9 to 12, wherein: The method takes minimizing the difference between the images of the sample object at m perspectives and the preset true values ​​corresponding to the images in each domain as the training target, and updates the parameters of the basic network model to obtain a multi-perspective image generation model, including: Calculating image losses between images of the sample object at m viewing angles and corresponding preset true values; Calculating the domain loss between each domain image of the sample object and the corresponding preset true value; Taking minimizing the image loss and the domain loss as the training goal, the parameters of the basic network model are updated to obtain the multi-view image generation model, wherein the weight of the image loss is less than the weight of the domain loss.

14. A system for generating a multi-view image, comprising: at least one storage medium storing at least one instruction set for generating a multi-view image; At least one processor is communicatively connected to the at least one storage medium, wherein when the at least one processor is running, the at least one instruction set is read, and the method as described in any one of claims 1 to 8 is executed according to the instructions of the at least one instruction set.

15. A training system for a multi-view image generation model, comprising: at least one storage medium storing at least one instruction set for training a multi-view image generation model; At least one processor is communicatively connected to the at least one storage medium, wherein when the at least one processor is running, the at least one instruction set is read, and the method as described in any one of claims 9 to 13 is executed according to the instructions of the at least one instruction set.

Images