Image generation method and device, electronic equipment, computer readable storage medium and computer program product

Abstract

The application provides an image generation method and device, electronic equipment, computer readable storage medium and computer program product. The method comprises: obtaining an image generation prompt word, a plurality of reference images of different perspectives of a target object, and a preset camera perspective parameter; performing three-dimensional reconstruction based on the plurality of reference images of different perspectives to obtain a three-dimensional model of the target object, a reference camera perspective parameter and a perspective feature of each reference image; rendering the three-dimensional model according to the preset camera perspective parameter to obtain a target perspective rendering image; fusing the perspective features of each reference image according to the relative pose parameters of the reference camera perspective parameter and the preset camera perspective parameter to obtain multi-perspective features; and calling a pre-trained image generation model to generate a target image of the target object according to the image generation prompt word, the target perspective rendering image and the multi-perspective features. Through the application, the generation quality of the image can be improved.

Description

Technical Field

[0001] This application relates to artificial intelligence technology, and more particularly to an image generation method, apparatus, electronic device, computer-readable storage medium, and computer program product. Background Technology

[0002] The related technologies for generating new perspective images of specific objects mainly fall into two categories. The first category is based on two-dimensional deep learning models. This method utilizes an image-to-image transformation framework to directly synthesize a new perspective from a reference image. However, due to the lack of three-dimensional geometric perception, this type of method often suffers from problems such as distortion of object structure and loss of identity features when generating new perspectives with large angle changes. The second category is based on three-dimensional reconstruction methods, such as neural radiation fields. This method ensures geometric consistency by reconstructing a three-dimensional scene, but the rendered images are usually weaker than advanced two-dimensional generation models in terms of photorealism and texture detail, and it has stringent requirements on the quantity and quality of the input images. Summary of the Invention

[0003] This application provides an image generation method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can improve the quality of generated images.

[0004] The technical solution of this application embodiment is implemented as follows: This application provides an image generation method, the method comprising: Acquire image generation prompts, multiple reference images of the target object from different perspectives, and preset camera perspective parameters; Based on the multiple reference images from different perspectives, a three-dimensional reconstruction is performed to obtain a three-dimensional model of the target object, reference camera viewpoint parameters and viewpoint features for each reference image; The three-dimensional model is rendered according to the preset camera view parameters to obtain a target view rendering image; Determine the relative pose parameters of the reference camera viewpoint parameters and the preset camera viewpoint parameters for each reference image; Based on the relative pose parameters, the viewpoint features of each reference image are fused to obtain multi-viewpoint features; Based on the image generation prompts, the target view rendering, and the multi-view features, a pre-trained image generation model is invoked to generate a target image of the target object.

[0005] This application provides an image generation apparatus, the apparatus comprising: The acquisition module is used to acquire image generation prompts, reference images of the target object from multiple different perspectives, and preset camera perspective parameters; The rendering module is used to perform 3D reconstruction based on the multiple reference images from different perspectives to obtain a 3D model of the target object, reference camera view parameters and view features of each reference image; and to render the 3D model according to the preset camera view parameters to obtain a target view rendering image. The determining module is used to determine the relative pose parameters of the reference camera viewpoint parameters and the preset camera viewpoint parameters for each reference image; The fusion module is used to fuse the viewpoint features of each reference image according to the relative pose parameters to obtain multi-viewpoint features; The generation module is used to generate a target image of the target object by calling a pre-trained image generation model based on the image, the target view rendering map, and the multi-view features.

[0006] This application provides an electronic device, the electronic device comprising: Memory is used to store executable instructions or computer programs. The processor, when executing computer-executable instructions or computer programs stored in the memory, implements the image generation method provided in the embodiments of this application.

[0007] This application provides a computer-readable storage medium storing a computer program or computer-executable instructions, which, when executed by a processor, implements the image generation method provided in this application.

[0008] This application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, they implement the image generation method provided in this application.

[0009] The embodiments of this application have the following beneficial effects: By acquiring image generation prompts, multiple reference images of the target object from different perspectives, and preset camera viewpoint parameters, 3D reconstruction is first performed based on these reference images to obtain a 3D model of the target object, reference camera viewpoint parameters for each reference image, and viewpoint features. This step accurately recovers the global 3D structure of the object from a small number of images with unknown camera poses, providing a solid geometric foundation for subsequent generation. Subsequently, the 3D model is rendered according to the preset camera viewpoint parameters to obtain a target viewpoint rendering image. This rendering image provides strong prior knowledge of geometric shapes and occlusion relationships for the final image generation, fundamentally ensuring the consistency of the 3D structure of the generated image and effectively avoiding the morphological distortion problems common in traditional 2D methods. Simultaneously, based on the relative pose parameters of the reference camera viewpoint parameters and the preset camera viewpoint parameters for each reference image, the viewpoint features of each reference image are fused to obtain multi-viewpoint features. These multi-viewpoint features serve as rich appearance and texture priors, effectively transferring the details most relevant to the target viewpoint in the reference images (such as material, lighting, and color) to the generation model, ensuring high fidelity of the target object's identity features. Finally, based on image generation prompts, target view renderings, and multi-view features, a pre-trained image generation model is invoked to generate a target image of the target object. Through this collaborative guidance of geometric and appearance priors, the system can focus on generating photorealistic and rich details while maintaining 3D consistency. This approach successfully integrates the structural accuracy of 3D reconstruction with the textural realism of 2D generation, achieving high-quality, high-fidelity new perspective image generation. Attached Figure Description

[0010] Figure 1 This is a schematic diagram of the architecture of the image generation system 100 provided in an embodiment of this application; Figure 2 This is a schematic diagram of the structure of the terminal 400 provided in the embodiments of this application; Figure 3 This is a first flowchart illustrating the image generation method provided in this application embodiment; Figure 4 This is a schematic diagram of the second process of the image generation method provided in the embodiments of this application; Figure 5 This is a schematic diagram of the third process of the image generation method provided in the embodiments of this application; Figure 6 This is a schematic diagram of the fourth process of the image generation method provided in the embodiments of this application; Figure 7 This is a schematic diagram of the fifth process of the image generation method provided in the embodiments of this application; Figure 8 This is a schematic diagram of the sixth process of the image generation method provided in the embodiments of this application; Figure 9 This is a schematic diagram of the seventh process of the image generation method provided in the embodiments of this application; Figure 10 This is a schematic diagram of the eighth process of the image generation method provided in the embodiments of this application; Figure 11 This is a schematic diagram of the ninth process of the image generation method provided in the embodiments of this application; Figure 12 This is a schematic diagram of the tenth process of the image generation method provided in the embodiments of this application; Figure 13 This is a schematic diagram of the eleventh step of the image generation method provided in the embodiments of this application; Figure 14 This is a first schematic diagram of image generation provided in an embodiment of this application; Figure 15 This is a second schematic diagram of image generation provided in the embodiments of this application; Figure 16 This is a schematic diagram illustrating the effect of image generation provided in the embodiments of this application; Figure 17 This is a schematic diagram of image uploading provided in an embodiment of this application; Figure 18 This is a schematic diagram of the spherical coordinate system provided in an embodiment of this application; Figure 19 This is a schematic diagram of the sphere control panel provided in an embodiment of this application; Figure 20 This is a schematic diagram of the structure of the image generation method provided in the embodiments of this application; Figure 21 This is a schematic diagram of the structure of the hybrid view module provided in the embodiments of this application.

[0011] It should be noted that the terms "first" and "second" mentioned above are only used to distinguish between different options and do not represent the degree of superiority or inferiority of the options or their priority in the implementation process. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0013] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0014] In the following description, the terms "first / second / third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first / second / third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0015] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0016] Unless otherwise specified, "at least one" as used below refers to one or more cases, and "multiple" can refer to two or more cases.

[0017] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.

[0018] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0019] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0020] 1) Image generation prompts: These are data streams containing natural language text, used to provide semantic-level constraints and implicit guidance to pre-trained image generation models. These input instructions typically encompass detailed descriptions of the expected context, lighting, artistic atmosphere, and background elements that should be present in the output, serving as global semantic prerequisites for controlling the evolution of low-level noise into a pixel array with realistic visual meaning in the feature space.

[0021] 2) Target Object: In customized visual synthesis tasks, this refers to a specific entity whose core identity features need to be extracted by the computational model, whose appearance attributes need to be preserved, and which needs to be seamlessly embedded into a new prediction scene. The entity's inherent physical uniqueness, such as its surface texture, material color, and local topological structure, is the core visual asset that needs to be strictly mathematically preserved throughout the entire forward and reverse computation process.

[0022] 3) Reference image: refers to a set of pre-provided two-dimensional pixel acquisition samples, which record visual snapshots of a specific entity from multiple different real-world or virtual acquisition nodes.

[0023] 4) Camera perspective parameters: These refer to a set of extrinsic and intrinsic parametric variables used to precisely quantify and define the geometric state of spatial observation in a preset three-dimensional world coordinate system. This set typically includes translation vectors that determine the absolute spatial center coordinates of the observation device, rotation matrices that determine the orientation of the optical axis, and focal length and other underlying data that determine the imaging field of view and distortion mapping relationship. These are the fundamental elements for achieving precise spatial intervention.

[0024] 5) 3D Reconstruction: This is a non-contact reverse engineering spatial calculation process. It aims to analyze the projection rules of two-dimensional pixel sets covering different viewing directions, and use underlying algorithms such as multi-view epipolar geometry or neural implicit radiation field to reverse-engineer and calculate the actual volume occupation, surface shell shape and precise depth information distribution of a specific entity in a virtual digital coordinate system.

[0025] 6) Three-dimensional solid model: It is a low-level spatial geometric representation residing in the computer storage system in the form of digital data structure. It is usually composed of vertex clouds, topologically connected polygonal patches (such as triangular meshes), or implicit scalar functions. It has a clear and discretized three-dimensional boundary range and shape structure, and acts as a geometric computing agent for physical entities in digital space, providing a solid spatial foundation for subsequent shape calculations.

[0026] 7) Viewpoint features: These refer to the high-dimensional dense data tensors extracted from the single-dimensional two-dimensional visual input by a visual encoder built using a deep learning network for each reference image. These tensors highly compressibly represent the statistical distribution characteristics of entity boundary information, illumination response patterns, and local textures captured at this specific single viewing angle within the latent space vector domain.

[0027] 8) Target View Rendering: This refers to the 2D image generated by rendering a 3D proxy model built from a reference image from a specified new camera angle. This image clearly depicts the basic shape, 3D structure, and pose of the object, providing an accurate structural blueprint for subsequent generation processes to ensure that the final result is geometrically correct.

[0028] 9) Relative pose parameters: This is a set of data describing the spatial positional differences between the camera position of each reference image and the camera position of the new perspective. It specifically quantifies the rotation and translation relationships between different observation angles, thereby helping the model determine which original observation information is most critical and effective for synthesizing a new perspective image.

[0029] 10) Multi-view features: These are a series of data extracted from multiple original photos taken from different angles, containing information about the object's appearance. This data carries the subject's identity, such as its color, texture, and unique details. By effectively integrating these features, a rich and consistent visual basis can be provided for the final generation.

[0030] 11) Target Image: This refers to the final synthesized output image that meets all input conditions. This image not only depicts the specified scene and style according to the text instructions, but also ensures that the spatial orientation and three-dimensional shape of its main subject are strictly aligned with the preset viewing angle, and its appearance details faithfully retain the identity shown by the original content, ultimately presenting a high-quality and realistic effect.

[0031] 12) Human-computer interaction interface, used to provide human-computer interaction functions and display images. Examples include graphical user interface (GUI) displays, such as augmented reality (AR) interfaces, virtual reality (VR) interfaces, voice user interface (VUI), interactive projection interfaces (using projection technology to display information on a plane), eye-tracking interfaces (interfaces controlled by detecting the user's gaze), holographic interfaces (three-dimensional holograms formed by projecting images using holographic projection technology, allowing viewing of stereoscopic images without wearing special glasses), multimodal interfaces (interfaces combining multiple interaction methods, such as tactile, visual, and auditory interaction), and brain-machine interface (BMI) interfaces, etc.

[0032] The related technologies for generating new perspective images of specific objects mainly fall into two categories. The first category is based on two-dimensional deep learning models. This method utilizes an image-to-image transformation framework to directly synthesize a new perspective from a reference image. However, due to the lack of three-dimensional geometric perception, this type of method often suffers from problems such as distortion of object structure and loss of identity features when generating new perspectives with large angle changes. The second category is based on three-dimensional reconstruction methods, such as neural radiation fields. This method ensures geometric consistency by reconstructing a three-dimensional scene, but the rendered images are usually weaker than advanced two-dimensional generation models in terms of photorealism and texture detail, and it has stringent requirements on the quantity and quality of the input images.

[0033] Based on the above analysis, the applicant found that the image generation methods of related technologies cannot guarantee the quality of the generated images. In response to the above problems, this application provides an image generation method, apparatus, electronic device, computer-readable storage medium, and computer program product that can improve the quality of generated images.

[0034] The following describes exemplary applications of the electronic devices provided in the embodiments of this application. These electronic devices can be implemented as various types of terminals such as laptops, tablets, desktop computers, set-top boxes, smartphones, smart speakers, smartwatches, smart TVs, and in-vehicle terminals, or as servers. Exemplary applications of the electronic devices as terminals will be described below.

[0035] See Figure 1 , Figure 1 This is a schematic diagram of the architecture of the image generation system 100 provided in the embodiment of this application. In order to support an image generation application, the terminal 400 (exemplarily showing the human-computer interaction interface 411) connects to the server 200 through the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0036] In some embodiments, the terminal 400 is used to acquire the image generation prompt words input on the human-computer interaction interface 411, multiple reference images of the target object from different perspectives, and to acquire preset camera viewpoint parameters. Based on the multiple reference images from different perspectives, it performs 3D reconstruction to obtain a 3D model of the target object, reference camera viewpoint parameters and viewpoint features of each reference image. According to the preset camera viewpoint parameters, it renders the 3D model to obtain a target viewpoint rendering image. According to the relative pose parameters of the reference camera viewpoint parameters of each reference image and the preset camera viewpoint parameters, it fuses the viewpoint features of each reference image to obtain multi-viewpoint features. Based on the image generation prompt words, the target viewpoint rendering image and the multi-viewpoint features, it calls a pre-trained image generation model to generate a target image of the target object and displays the generated target image on the human-computer interaction interface 411.

[0037] In some embodiments, the terminal 400 is used to acquire the following data and send it to the server 200: image generation prompt words input on the human-computer interaction interface 411, multiple reference images of the target object from different perspectives, and acquire preset camera viewpoint parameters; the server 200 is used to perform three-dimensional reconstruction based on the acquired multiple reference images from different perspectives to obtain a three-dimensional model of the target object, reference camera viewpoint parameters and viewpoint features of each reference image, render the three-dimensional model according to the preset camera viewpoint parameters to obtain a target viewpoint rendering image, fuse the viewpoint features of each reference image according to the relative pose parameters of the reference camera viewpoint parameters of each reference image and the preset camera viewpoint parameters to obtain multi-viewpoint features; and, based on the image generation prompt words, the target viewpoint rendering image and the multi-viewpoint features, call a pre-trained image generation model to generate a target image of the target object, send it to the terminal 400, and display the generated target image on the human-computer interaction interface 411.

[0038] In some embodiments, server 200 may be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Terminals and servers can be connected directly or indirectly via wired or wireless communication, which is not limited in this embodiment.

[0039] See Figure 2 , Figure 2 This is a schematic diagram of the structure of the terminal 400 provided in the embodiment of this application. Figure 2 The terminal 400 shown includes at least one processor 410, a memory 450, at least one network interface 420, and a user interface 430. The various components in the terminal 400 are coupled together via a bus system 440. It is understood that the bus system 440 is used to implement communication between these components. In addition to a data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2 The general labeled all buses as Bus System 440.

[0040] Processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0041] User interface 430 includes one or more output devices 431 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0042] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices physically located away from the processor 410.

[0043] The memory 450 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 450 described in this application embodiment is intended to include any suitable type of memory.

[0044] In some embodiments, memory 450 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0045] Operating system 451 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks; The network communication module 452 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc. Presentation module 453 is configured to enable the presentation of information (e.g., a user interface for operating peripheral devices and displaying content and information) via one or more output devices 431 associated with user interface 430 (e.g., a display screen, a speaker, etc.). The input processing module 454 is used to detect and translate one or more user inputs or interactions from one or more input devices 432.

[0046] In some embodiments, the apparatus provided in this application can be implemented in software. Figure 2An image generation apparatus 455 stored in memory 450 is shown. This apparatus can be software in the form of programs and plugins, and includes the following software modules: an acquisition module 4551, a rendering module 4552, a determination module 4553, a fusion module 4554, and a generation module 4555. These modules are logically linked and can therefore be arbitrarily combined or further separated according to their implemented functions. The functions of each module will be described below.

[0047] In some embodiments, the terminal or server can implement the image generation method provided in this application by running various computer-executable instructions or computer programs. For example, computer-executable instructions can be microprogram-level commands, machine instructions, or software instructions. Computer programs can be native programs or software modules in an operating system; they can be native applications (APPs), i.e., programs that need to be installed in the operating system to run; or they can be applets that can be embedded in any APP, i.e., programs that only need to be downloaded to a browser environment to run. In summary, the aforementioned computer-executable instructions can be any form of instruction, and the aforementioned computer programs can be any form of application, module, or plugin.

[0048] The image generation method provided in this application will be described in conjunction with exemplary applications and implementations of the terminals provided in the embodiments of this application.

[0049] The image generation method provided in the embodiments of this application will be described below. As mentioned above, the electronic device implementing the image generation method of the embodiments of this application can be a terminal or a server, or a combination of both. Therefore, the executing entity of each step will not be described again below.

[0050] See Figure 3 , Figure 3 This is a schematic diagram of the first process of the image generation method provided in this application embodiment. Taking the terminal as the execution subject as an example, it will be combined with... Figure 3 The steps shown are explained.

[0051] In step 101, image generation prompts, reference images of the target object from multiple different perspectives, and preset camera perspective parameters are obtained.

[0052] Here, image generation cue words refer to natural language text used to describe the scene, style, or attributes of the desired generated image. The target object is the specific object or entity expected to be presented as the main subject in the generated image. Multiple different viewpoints refer to viewing the target object from multiple significantly different orientations for subsequent 3D reconstruction steps; the combination of these orientations is sufficient to provide the complete information needed to construct a 3D point cloud model of the target object. Reference images are original images taken from different viewing angles that demonstrate the appearance of the target object. Preset camera viewpoint parameters are geometric parameters used to precisely define the viewing angle and position of the target object in the generated image.

[0053] In some embodiments, the user interface can receive text input by the user as image generation prompts, receive multiple image files containing the target object uploaded by the user as reference images, and receive user-specified values ​​for defining the generation perspective as preset camera perspective parameters, thereby completing data acquisition.

[0054] In some embodiments, image generation prompts can be automatically generated based on the content of a reference image, which can then be modified by the user. For example, after identifying the target object in the reference image as "a blue backpack," "a blue backpack" can be automatically generated as the initial image generation prompt. Reference images can also be automatically extracted from a video containing a dynamic display of the target object at preset time intervals or using a keyframe extraction strategy to simplify the user's upload process.

[0055] As an example, if a user wants to place a real-world "blue backpack" in a snowy scene and generate an image from a low, upward-facing angle, the image generation tool could use "a blue backpack in the snow" as the prompt; upload five photos of the blue backpack taken from different angles (such as front, side, back, top, and oblique) as reference images; and obtain preset camera perspective parameters.

[0056] In some embodiments, see Figure 4 , Figure 4 This is a schematic diagram of the second process of the image generation method provided in the embodiments of this application. Figure 3 Step 101, "Obtaining preset camera view parameters," can be done through... Figure 4 Steps 1011 to 1014 are implemented, and the details are explained below.

[0057] In step 1011, the preset horizontal azimuth angle parameters, pitch angle parameters, and camera distance parameters for the target object are obtained.

[0058] Here, the horizontal azimuth parameter defines the horizontal rotation angle of the camera around the target object. The pitch angle parameter defines the vertical viewing angle of the camera relative to the horizontal plane of the target object, distinguishing between a top-down or bottom-up view. The camera distance parameter defines the distance between the camera and the center point of the target object.

[0059] In some embodiments, three independent numerical input controls can be provided, corresponding to the horizontal azimuth, pitch, and camera distance, respectively. In response to a drag operation on the slider, or when a value is detected directly entered into the numerical input controls, a specific value is assigned to each parameter. For example, the horizontal azimuth is set to 90 degrees, the pitch angle to 30 degrees, and the camera distance to 1.5 meters. These three independent values ​​are recorded as input for subsequent calculations.

[0060] In some embodiments, multiple preset parameter combination templates can be provided, such as "golden composition perspective" and "product close-up perspective," allowing users to quickly fill in the horizontal azimuth, pitch, and camera distance parameters by directly selecting a template. Users can also interactively adjust the viewing angle by dragging the virtual camera directly in the 3D preview window, and the corresponding horizontal azimuth, pitch, and camera distance parameters can be calculated in real-time from the virtual camera's position.

[0061] As an example, if you want to observe the target object from the left side, slightly upwards, and at a relatively close distance, you can set the horizontal azimuth angle to 90 degrees, the pitch angle to 20 degrees, and the camera distance to 1.2 units on the interface.

[0062] In step 1012, a preset three-dimensional spherical coordinate system is constructed with the centroid of the target object as the origin.

[0063] Here, the centroid refers to a point in three-dimensional space that represents the geometric center of the target object. The origin is the center point of the coordinate system, the intersection of all coordinate axes, typically with coordinates (0, 0, 0). The default three-dimensional spherical coordinate system is a pre-established coordinate system used to describe the position of a point in three-dimensional space. This system uses radial distance, pitch angle, and azimuth angle to define the position of a point. See also: Figure 18 The subject to be customized is pre-located at the origin of the coordinate system, responding to three scalar parameters: the distance 'a' between the camera and the subject, the horizontal azimuth angle, and the input parameters. and pitch angle .

[0064] In some embodiments, a virtual three-dimensional coordinate space can be created. In this space, the geometric center of the target object is defined as the origin of the coordinate system. Subsequently, a standard spherical coordinate system is established around this origin. The definition of this coordinate system is independent of the specific shape or orientation of the target object, using only the centroid of the target object as the positioning reference.

[0065] As an example, if the target object is a water glass, calculate the geometric center point of all parts of the water glass and mark this center point as coordinates (0, 0, 0) in virtual space. Then, establish a standard mathematical spherical coordinate system centered on this point.

[0066] In step 1013, the horizontal azimuth angle parameter, pitch angle parameter, and camera distance parameter are mapped to a preset three-dimensional spherical coordinate system to obtain three-dimensional spatial coordinates.

[0067] Here, three-dimensional spatial coordinates are coordinates composed of three values, X, Y, and Z, in a three-dimensional Cartesian coordinate system, used to uniquely determine the position of a point in space.

[0068] In some embodiments, the camera distance parameter obtained in step 1011 can be used as the radial distance in a preset three-dimensional spherical coordinate system, the horizontal azimuth parameter as the azimuth, and the pitch angle parameter as the polar angle (or elevation). According to the standard conversion rules from spherical coordinates to Cartesian coordinates, the X, Y, and Z coordinates of the camera in a Cartesian coordinate system with the centroid of the target object as the origin are calculated. These coordinates are the three-dimensional spatial coordinates.

[0069] As an example, the input camera distance parameter is 2, the horizontal azimuth parameter is 90 degrees (π / 2 radians), and the pitch angle parameter is 30 degrees (π / 6 radians). Using the conversion formulas: X = 2 * cos(π / 6) * cos(π / 2) = 0; Y = 2 * cos(π / 6) * sin(π / 2) = 1.732; Z = 2 * sin(π / 6) = 1. The final 3D spatial coordinates are (0, 1.732, 1).

[0070] In step 1014, the three-dimensional spatial coordinates are converted into camera extrinsic parameter matrices using a preset trigonometric function transformation mapping rule, which are then used as preset camera viewpoint parameters.

[0071] In some embodiments, the camera's position vector is first determined based on the three-dimensional spatial coordinates obtained in step 1013. Simultaneously, since the camera needs to always face the origin of the coordinate system (i.e., the centroid of the target object), the camera's viewing direction vector (pointing from the camera position to the origin) can be calculated. Based on the camera's position vector and viewing direction vector, and with a pre-defined "up" direction vector for the camera (usually the positive Y-axis direction of the world coordinate system), a 3x4 matrix is ​​calculated using pre-defined trigonometric transformation mapping rules (such as LookAt transformation or constructing rotation and translation matrices). This 3x4 matrix is ​​the camera extrinsic parameter matrix, serving as the final pre-defined camera viewpoint parameters.

[0072] As an example, the 3D spatial coordinates (0, 1.732, 1) obtained in step 1013 are used as the camera position. Since the camera is looking towards the origin (0, 0, 0), the viewing direction vector is (0, -1.732, -1). Assume the preset "up" direction is (0, 0, 1). Through vector operations such as cross product, the system calculates the three orthogonal basis vectors of the camera, and combined with the camera position, finally generates a unique 3x4 camera extrinsic parameter matrix, for example: [[1, 0, 0, 0], [0, 0.866, -0.5, -1.225], [0, 0.5, 0.866, -2.121]].

[0073] This application embodiment obtains preset horizontal azimuth, pitch, and camera distance parameters for the target object; constructs a preset three-dimensional spherical coordinate system with the centroid of the target object as the origin; maps the horizontal azimuth, pitch, and camera distance parameters to the preset three-dimensional spherical coordinate system to obtain three-dimensional spatial coordinates; and uses preset trigonometric function transformation mapping rules to convert the three-dimensional spatial coordinates into a camera extrinsic parameter matrix, which serves as the preset camera viewpoint parameters. This solution decouples user input from the complex camera extrinsic parameter matrix into intuitive horizontal azimuth, pitch, and camera distance parameters, greatly simplifying the user's control over the generated image viewpoint. By constructing a standardized three-dimensional spherical coordinate system with the centroid of the target object as the origin and automatically converting it using preset trigonometric function transformation mapping rules, it ensures that the user's viewpoint control logic remains consistent and predictable regardless of the target object. This design transforms abstract geometric concepts into concrete and easy-to-understand interactive parameters, enabling precise control of the target object's generated perspective without requiring users to have knowledge of 3D graphics. This solves the technical problems in related technologies where camera perspective settings are not intuitive and difficult to control precisely, resulting in strong randomness in the generated image composition and failure to meet user expectations. It improves the controllability of image generation and the user experience.

[0074] See also Figure 3 The following will be an explanation following step 101 above.

[0075] In step 102, 3D reconstruction is performed based on multiple reference images from different perspectives to obtain a 3D model of the target object, the reference camera perspective parameters and perspective features of each reference image.

[0076] In some embodiments, see Figure 5 , Figure 5 This is a schematic diagram of the third process of the image generation method provided in the embodiments of this application. Figure 3 Step 102 can be achieved through Figure 5 Steps 1021 to 1025 are implemented, and the details are explained below.

[0077] In step 1021, feature extraction is performed on multiple reference images from different perspectives to obtain the initial image features of each reference image.

[0078] In some embodiments, see Figure 6 , Figure 6 This is a schematic diagram of the fourth process of the image generation method provided in the embodiments of this application. Figure 5 Step 1021 can be performed for each reference image. Figure 6 Steps 10211 to 10215 are implemented, and the details are explained below.

[0079] In step 10211, the reference image is divided into multiple image blocks of a preset size.

[0080] Here, the preset size is a predefined two-dimensional size specification used to segment the reference image, such as 16x16 pixels. An image block is a non-overlapping rectangular image region segmented from the reference image according to the preset size.

[0081] In some embodiments, for each received reference image, a grid can be cut in the horizontal and vertical directions starting from the upper left corner of the image, according to the width and height defined by a preset size, until the entire reference image is covered, generating a two-dimensional array composed of multiple image blocks.

[0082] In some embodiments, an overlapping partitioning strategy can be employed, where adjacent image blocks overlap by a preset pixel width when partitioning. This approach preserves richer local contextual information for subsequent processing. Alternatively, the preset size can be dynamically adjusted based on the complexity of the reference image content, using a smaller preset size for detailed areas and a larger preset size for areas with flat backgrounds, achieving adaptive image block partitioning.

[0083] As an example, input a reference image with a resolution of 224x224 pixels and a default size of 14x14 pixels. This reference image can be divided into 16 rows and 16 columns, for a total of 256 image blocks of 14x14 pixels each.

[0084] In step 10212, feature extraction is performed on multiple image blocks to obtain the image block features of each image block.

[0085] Here, image patch features are numerical vectors that represent the visual content of an image patch, calculated from a single image patch through feature extraction operations.

[0086] In some embodiments, each image patch generated in step 10211 can be input into a preset feature extraction model. The feature extraction model performs calculations independently on each image patch, converting the pixel data of the image patch into a high-dimensional feature vector. This process is repeated for all image patches to generate a corresponding image patch feature for each image patch.

[0087] In some embodiments, a multi-scale feature extraction approach can be employed, applying feature extractors with receptive fields of different sizes to each image patch, and fusing the extracted features from multiple different scales to generate a more representative image patch feature. Alternatively, during feature extraction, traditional computer vision features such as color histograms or texture descriptors of the image patch can be introduced, and these features can be concatenated with features extracted by the deep learning model to enrich the dimensionality of the image patch features.

[0088] As an example, a 14x14 pixel image patch can be input into a linear projection layer. The linear projection layer flattens the pixels of the image patch and maps them into a 768-dimensional vector. This 768-dimensional vector is the image patch feature of the image patch.

[0089] In step 10213, the image block features of each image block are concatenated to obtain an image block feature sequence.

[0090] Here, the image patch feature sequence is an ordered set of image patch features of all image patches divided from a single reference image, arranged in a predetermined spatial order.

[0091] In some embodiments, the image block features generated for each image block in step 10212 can be arranged and connected in this order according to the spatial position of the image block in the original reference image (e.g., the raster scanning order from left to right and from top to bottom) to form a one-dimensional sequence structure, which is the image block feature sequence.

[0092] In some embodiments, image patch features can be stitched together using a Z-order curve or a Hilbert curve. This non-linear stitching order better preserves the locality of the two-dimensional space. Additionally, during stitching, a learnable class token can be added at the beginning of the image patch feature sequence to aggregate global information of the entire image in subsequent processing.

[0093] As an example, for a reference image divided into 256 image patches, the system concatenates its 256 768-dimensional image patch features into a sequence of length 256, in the order from the first element in the first row to the 16th element in the 16th row. Each element in the sequence is a 768-dimensional image patch feature.

[0094] In step 10214, the image block feature sequence is positionally encoded to obtain positionally encoded features.

[0095] Here, positional encoding refers to the operation of injecting the spatial location information of each image patch feature in the original reference image into the image patch feature sequence. Positionally encoded features are image patch features that contain both original visual information and spatial location information.

[0096] In some embodiments, a unique positional encoding vector can be generated for each position (from 1 to N) in the image patch feature sequence. The dimension of this positional encoding vector is the same as the dimension of the image patch features. Subsequently, each positional encoding vector is added element-wise to the corresponding image patch feature in the image patch feature sequence to generate a positional encoded feature.

[0097] In some embodiments, learnable positional encoding can be used, where the positional encoding vector itself is used as a model parameter, and the optimal positional representation is automatically learned during training, rather than being generated using a fixed function. Two-dimensional positional encoding can also be used, encoding the positions in the horizontal and vertical directions separately, then fusing the positional encoding information from the two directions and adding it to image patch features to more accurately represent two-dimensional spatial relationships.

[0098] As an example, for a feature sequence of image patches of length 256, 256 different 768-dimensional positional encoding vectors can be generated. Then, the i-th positional encoding vector is added to the i-th image patch feature in the sequence (i ranges from 1 to 256) to obtain a positional encoding feature sequence of length 256.

[0099] In step 10215, attention processing is performed on the position-encoded features to obtain the initial image features of the reference image.

[0100] Here, attention processing is a mechanism that calculates the dependencies between different elements in a sequence and aggregates the sequence information accordingly using weighted sums. Initial image features refer to the final feature representations that, after attention processing, capture the global contextual relationships within the reference image.

[0101] In some embodiments, the positional encoding features generated in step 10214 can be input into a self-attention module. In the self-attention module, each positional encoding feature is correlated with all other positional encoding features by calculating their relevance weights, and then a weighted sum is performed on all positional encoding features based on these weights. This process generates a new feature for each position in the sequence in parallel, and the final output sequence is the initial image feature of the reference image.

[0102] In some embodiments, a multi-head attention mechanism can be employed to perform multiple attention processes in parallel, with each process focusing on a different dependency subspace. The results of these multiple processes are then concatenated and linearly transformed to capture richer feature relationships. Alternatively, a sparse attention mechanism can be used, where each positionally encoded feature is correlated only with other features within a preset range or with global key features during attention processing, thereby reducing computational complexity and improving processing efficiency.

[0103] As an example, a sequence containing 256 positionally encoded features can be input into an attention module. The attention module calculates a 256x256 weight matrix, representing the correlation between any two image patches. This weight matrix is ​​then used to weight and aggregate all positionally encoded features, ultimately outputting a sequence also containing 256 features. This sequence serves as the initial image feature set for the reference image.

[0104] This application's embodiments divide a reference image into multiple image blocks of a preset size; extract features from each image block to obtain image block features for each block; concatenate the image block features of each block to obtain an image block feature sequence; perform positional encoding on the image block feature sequence to obtain positional encoded features; and apply attention processing to the positional encoded features to obtain the initial image features of the reference image. The above scheme successfully transforms the convolutional neural network's image processing paradigm into a sequence-based attention mechanism paradigm by dividing the reference image into multiple image blocks and serializing them. The core advantage of this transformation lies in the fact that by applying attention processing to the positional encoded features, the long-distance dependency between any two image blocks can be directly calculated, thereby capturing the global structure and contextual information of the reference image. This method solves the technical problem that traditional convolutional operations are difficult to efficiently model global dependencies due to limited receptive fields, making it difficult for the extracted initial image features to not only contain local details but also integrate the overall layout and semantic relationships, providing higher-quality and more globally-oriented feature representations for subsequent cross-view feature interactions.

[0105] See also Figure 5 The following will be explained following step 1021 above.

[0106] In step 1022, cross-view feature interaction is performed on the initial image features of multiple reference images to obtain the view features of each reference image.

[0107] Here, cross-view feature interaction refers to a mechanism for information fusion and enhancement processing of initial image features extracted from reference images from multiple different viewpoints. This mechanism establishes associations between features from different viewpoints, enabling the processed features of a single viewpoint to contain contextual and geometric constraint information from other relevant viewpoints.

[0108] In some embodiments, see Figure 7 , Figure 7 This is a schematic diagram of the fifth process of the image generation method provided in the embodiments of this application. Figure 5 Step 1022 can be achieved through Figure 7 Steps 10221 to 10225 are implemented, and the details are explained below.

[0109] In step 10221, the initial image features of each reference image are spatially encoded to obtain single-view enhanced features.

[0110] Here, spatial feature encoding is an operation that performs deep processing on initial image features within a single reference image to enhance their internal spatial relationships and semantic information. Single-view enhanced features are features with stronger representational capabilities obtained after spatial feature encoding.

[0111] In some embodiments, the initial image features of each reference image can be independently input into a self-attention network layer. In this network layer, by calculating the interrelationships between features at different locations within the initial image features and aggregating the information, a feature with the same dimension as the initial image features but containing richer contextual information is output as a single-view enhancement feature.

[0112] In some embodiments, a parallel multi-layer spatial feature encoding structure can be employed, where each encoder layer uses different parameter configurations. Initial image features are simultaneously input into all encoder layers, and the outputs of all layers are then concatenated or weighted averaged to fuse spatial information from different abstraction levels. Another extension is to introduce a loop processing mechanism during the spatial feature encoding process, using the encoder output as the input for the next iteration, performing a fixed number of loops to progressively refine the single-view enhancement features.

[0113] As an example, an initial image feature sequence consisting of 256 feature vectors is input into a standard Transformer encoder layer. After processing by multi-head self-attention and a feedforward neural network, a new sequence consisting of 256 feature vectors is output. This output sequence is the single-view augmentation feature of the reference image.

[0114] In step 10222, a correlation weight matrix is ​​determined based on the single-view enhancement features of each reference image.

[0115] Here, the correlation weight matrix is ​​a two-dimensional matrix, where each element represents the correlation strength between any two reference images from different viewpoints.

[0116] In some embodiments, the single-view augmentation features of each reference image can be compressed into a single view description vector through a global average pooling operation. Then, the dot product or cosine similarity between each pair of view description vectors of all reference images is calculated, and the results are organized into an N×N matrix (N is the number of reference images). This matrix is ​​the relevance weight matrix.

[0117] In some embodiments, before determining the relevance weight matrix, a non-linear projection can be performed on the view description vectors to map them to a new feature space before calculating similarity, in order to capture more complex inter-view relationships. Alternatively, after calculating the similarity scores, a learnable temperature parameter can be introduced to scale the values ​​of the entire matrix, and then the Softmax function can be applied for normalization so that the sum of the weights in each row of the matrix is ​​1.

[0118] As an example, for five reference images, their respective single-view augmentation features can be pooled to obtain five view description vectors. The cosine similarity between each pair of these five vectors is calculated, generating a 5x5 matrix, where the value of the (i, j)th element represents the similarity score between the i-th view and the j-th view.

[0119] In step 10223, the single-view enhancement features of each reference image are weighted and summed according to the correlation weight matrix to obtain the interaction features.

[0120] Here, the interaction feature is a global feature representation that integrates information from all reference images. It is generated by weighting and aggregating the features of each reference image according to its relevance to other images.

[0121] In some embodiments, each row of the relevance weight matrix can be used as a set of weights to perform a weighted summation of the single-view augmentation features of all reference images. For example, using the weights of the i-th row of the matrix, the single-view augmentation features of all reference images are weighted and summed to obtain the preliminary aggregation result for the i-th view. The preliminary aggregation results of all views are then integrated to form the final interactive features.

[0122] In some embodiments, a multi-head weighted summation mechanism can be employed. The single-view augmentation features are split into multiple heads along the channel dimension, and each head is independently weighted using a relevance weight matrix. Finally, the results from all heads are concatenated to capture interaction information from different feature subspaces. Another extension is to additionally concatenate an average feature of all single-view augmentation features after the weighted summation to ensure that the global summary information of the original view is preserved while information interaction is achieved.

[0123] As an example, the first row [w11, w12, w13, w14, w15] of the relevance weight matrix is ​​used as weights to perform a weighted summation of the single-view enhancement features of all five reference images, yielding an intermediate result related to the first viewpoint. This operation is repeated for the remaining four rows of the matrix, resulting in five sets of intermediate results, which together constitute the interaction features.

[0124] In step 10224, the interaction features are split according to the view dimension to obtain the sub-interaction features of the reference image for each view.

[0125] Here, sub-interaction features are the feature parts that are separated from global interaction features and correspond to a single specific perspective.

[0126] In some embodiments, since the interaction feature is an aggregate containing information from all N viewpoints, the interaction feature is indexed and segmented along the dimensions representing different viewpoints, and a corresponding feature block is assigned to each original reference image. This feature block is the sub-interaction feature.

[0127] In some embodiments, a view-specific linear transformation layer is applied to each sub-interaction feature after splitting, allowing each viewpoint branch to personalize and refine the received global information. Alternatively, instead of hard splitting, a learnable routing mechanism can be used to dynamically and "softly" extract and reassemble sub-interaction features from the interaction features based on the single-view enhancement features of each reference image.

[0128] As an example, suppose the interaction feature is a tensor with dimensions (5, C, H, W), where 5 represents the number of viewpoints. Slicing it according to the viewpoint dimension yields five tensors with dimensions (C, H, W), each corresponding to a sub-interaction feature of a reference image.

[0129] In step 10225, the sum of the single-view enhancement features of each reference image and the sub-interaction features of each reference image is used as the viewpoint feature of each reference image.

[0130] Here, the viewpoint features are the final output, which contain both detailed information from a single view and incorporate multiple views. Figure 1 Enhanced feature representation of consistency information.

[0131] In some embodiments, the single-view enhancement features of each reference image can be added element-wise with the corresponding sub-interaction features obtained in step 10224. This residual connection-like operation adds global information obtained from all view interactions as an increment to the original, enhanced single-view information to generate the final view features.

[0132] In some embodiments, a learnable gating unit can be introduced before the addition operation. This gating unit dynamically calculates a weight between 0 and 1 based on the single-view enhancement features and sub-interaction features, controlling the fusion ratio and achieving adaptive feature fusion. Alternatively, after the addition operation, a non-linear activation function and an additional convolutional layer can be connected to further non-linearly transform and integrate the fused features, thereby enhancing their expressive power.

[0133] As an example, the single-view augmentation features (a feature tensor) of the first reference image can be added element-wise to the sub-interaction features (a feature tensor of the same dimension) of the first reference image. The result is the final view feature of the first reference image. Repeat this operation for the other four reference images.

[0134] This application embodiment obtains single-view enhanced features by spatially encoding the initial image features of each reference image; determines a relevance weight matrix based on the single-view enhanced features of each reference image; and obtains interactive features by weighted summation of the single-view enhanced features of each reference image based on the relevance weight matrix. The interactive features are then decomposed according to the viewpoint dimension to obtain sub-interactive features of the reference image for each viewpoint. The sum of the single-view enhanced features and the sub-interactive features of each reference image is used as the viewpoint feature of each reference image. This scheme introduces a cross-viewpoint feature interaction mechanism, constructing a framework that allows for effective communication and fusion of information from all reference images. Specifically, by determining the relevance weight matrix to quantify the correlation between views, and performing weighted summation and information distribution accordingly, each final output viewpoint feature not only contains its own depth information but also implicitly includes the context and geometric constraints from all other relevant viewpoints. This design fundamentally solves the feature inconsistency problem caused by the independent processing of each viewpoint and the lack of information interaction in related technologies. The resulting viewpoint features possess inherent global consistency, providing higher quality, geometrically calibrated inputs for subsequent 3D reconstruction tasks, thereby enabling the generation of more structurally complete and more detailed 3D models.

[0135] See also Figure 5 The following will be explained following step 1022 above.

[0136] In step 1023, spatial camera mapping is performed on the viewpoint features of each reference image to obtain the reference camera viewpoint parameters of each reference image.

[0137] Here, spatial camera mapping refers to a calculation process that decodes or regresses the camera's geometric parameters when the image was captured from image features. Reference camera viewpoint parameters are mathematical representations used to describe the camera's position and orientation in three-dimensional space when each reference image is captured; these are typically camera extrinsic parameter matrices.

[0138] In some embodiments, the viewpoint features of each reference image (typically aggregated into a single feature vector through pooling operations) can be input into a dedicated camera parameter regression network. This regression network consists of several fully connected layers, and its output is constructed into the form of a camera extrinsic parameter matrix, which represents the reference camera viewpoint parameters of the reference image.

[0139] In some embodiments, the regression network not only predicts the camera extrinsic matrix but also the camera intrinsic parameters (such as focal length) to adapt to reference images taken by different devices. A geometric consistency loss function can also be introduced at the end of the regression network for end-to-end training. This function penalizes inconsistencies in all predicted reference camera viewpoint parameters in 3D space, thereby improving overall prediction accuracy.

[0140] As an example, the viewpoint features of a reference image are pooled into a 1024-dimensional vector and input into a multilayer perceptron. The network outputs 12 values, which are reshaped into a 3x4 matrix and used as the reference camera viewpoint parameters for the reference image.

[0141] In step 1024, the viewpoint features of each reference image are analyzed to obtain the depth map information of each reference image.

[0142] Here, depth map information refers to a single-channel image of the same size as the reference image, where the grayscale value of each pixel represents the distance of that point in the original scene from the camera.

[0143] In some embodiments, the viewpoint features of each reference image can be input into a decoder network. This decoder network gradually restores the high-dimensional, abstract viewpoint features to the spatial resolution of the original reference image through a series of upsampling and convolutional layers, and finally outputs a single-channel image, where each pixel value represents depth; this image is the depth map information.

[0144] In some embodiments, the decoder network can employ a U-Net structure, introducing low-level features from the initial feature extraction stage as skip connections during decoding to help recover finer geometric details. Alternatively, after generating the depth map information, a depth refinement network can be connected to smooth or sharpen the initially generated depth map information using constraints of adjacent pixels or across viewpoints.

[0145] As an example, the viewpoint features of a reference image (e.g., with dimensions of 16x16x1024) are input into a decoder consisting of transposed convolutional layers. After multiple upsampling operations, a 224x224x1 depth map is finally output, which is the depth map information.

[0146] In step 1025, a three-dimensional model of the target object is generated based on the depth map information of each reference image and the reference camera viewpoint parameters.

[0147] Here, a three-dimensional model refers to the digital geometric representation of a target object in three-dimensional space, such as a point cloud or a mesh model.

[0148] In some embodiments, for each reference image, each pixel in the depth map can be back-projected into a 3D world coordinate system using its corresponding reference camera viewpoint parameters and depth map information to form a 3D point cloud. Subsequently, the multiple point clouds generated from all reference images are merged in the same coordinate system, and through post-processing such as denoising and redundant point removal, a complete 3D model of the target object is finally formed.

[0149] In some embodiments, after point cloud fusion, a surface reconstruction algorithm (such as Poisson surface reconstruction or Alpha Shapes) can be applied to generate a continuous triangular mesh surface from the fused point cloud, resulting in a smoother and more complete 3D model. Another extension is to assign color information from the original reference image to each point during point cloud fusion, thereby generating a colored 3D model.

[0150] As an example, five local point clouds generated from five reference images can be transformed to a unified world coordinate system based on their respective reference camera viewpoint parameters. These five point clouds are then merged into a dense point cloud containing millions of points, which constitutes the three-dimensional model of the target object.

[0151] This application embodiment extracts features from multiple reference images from different perspectives to obtain initial image features for each reference image. Cross-view feature interaction is then performed on the initial image features of multiple reference images to obtain viewpoint features for each reference image. Spatial camera mapping is then performed on the viewpoint features of each reference image to obtain reference camera viewpoint parameters for each reference image. The viewpoint features of each reference image are then analyzed to obtain depth map information for each reference image. Based on the depth map information and reference camera viewpoint parameters of each reference image, a 3D model of the target object is generated. This approach no longer analyzes each reference image in isolation, but establishes an information-sharing and mutual-calibration channel among all views, so that the viewpoint features relied upon by each subsequent step implicitly contain global geometric constraints. Therefore, the accuracy and cross-view consistency of the reference camera viewpoint parameters and depth map information subsequently parsed from these highly consistent features are significantly enhanced, solving the technical problems of geometric conflicts and registration difficulties caused by independently estimating the parameters of each view in related technologies. Finally, the 3D model generated based on this higher-quality geometric information exhibits superior structural integrity, detail accuracy, and geometric precision compared to traditional methods.

[0152] See also Figure 3 The following will be an explanation following step 102 above.

[0153] In step 103, the three-dimensional model is rendered according to the preset camera view parameters to obtain the target view rendering image.

[0154] In some embodiments, see Figure 8 , Figure 8 This is a schematic diagram of the sixth process of the image generation method provided in the embodiments of this application. Figure 3 Step 103 can be achieved through Figure 8 Steps 1031 to 1034 are implemented, and the details are explained below.

[0155] In step 1031, the three-dimensional model is analyzed to obtain the three-dimensional point cloud spatial distribution of the target object.

[0156] Here, the spatial distribution of three-dimensional point cloud refers to a set of a large number of three-dimensional coordinate points representing the surface of the target object in a three-dimensional coordinate system.

[0157] In some embodiments, if the input 3D model is not in point cloud format (e.g., a triangular mesh model), its surface is sampled to extract the coordinates of all its vertices, or a set of random points with a preset density is generated on its surface. The set of these points constitutes the 3D point cloud spatial distribution of the target object. If the 3D model is already in point cloud format, it is directly used as the 3D point cloud spatial distribution.

[0158] In some embodiments, during the parsing process, the spatial distribution of the 3D point cloud can be downsampled to reduce the number of points using voxel meshes or random sampling, thereby improving the computational efficiency of subsequent rendering steps. Another extension is to attach additional attribute information, such as color values ​​obtained from the original reference image or normal vectors calculated from the 3D model, to each point in the spatial distribution of the 3D point cloud during parsing, for more realistic rendering.

[0159] As an example, a three-dimensional solid model consisting of 100,000 triangular facets is parsed, and all its non-repeating vertices are extracted to form a three-dimensional point cloud spatial distribution containing approximately 50,000 three-dimensional coordinate points.

[0160] In step 1032, the projection plane of the three-dimensional point cloud spatial distribution under the target viewpoint is determined according to the preset camera viewpoint parameters.

[0161] Here, the projection plane refers to a virtual two-dimensional plane defined in three-dimensional space, used to receive the projection results of all points in the three-dimensional point cloud spatial distribution onto this plane.

[0162] In some embodiments, a virtual camera model can be constructed in three-dimensional space using the camera position and orientation information contained in the preset camera viewpoint parameters. Based on the orientation of the camera model, a plane perpendicular to the camera orientation is defined at a preset focal length position in front of it; this plane is the projection plane under the target viewpoint.

[0163] In some embodiments, the size of the projection plane can be dynamically determined. First, an initial projection is performed on the spatial distribution of the 3D point cloud, calculating the bounding boxes of all points on the projection plane. Then, the projection plane is clipped or scaled to just accommodate these bounding boxes to optimize the composition of the rendered image. In addition to standard planar projection, spherical or cylindrical projection is also supported, allowing the generation of renderings with fisheye or panoramic effects by defining projection surfaces of different shapes.

[0164] As an example, based on a given camera extrinsic parameter matrix (as a preset camera viewpoint parameter), the camera is located at coordinates (10, 0, 0), facing the origin (0, 0, 0). Accordingly, a plane perpendicular to the X-axis is defined at position X=1 as the projection plane.

[0165] In step 1033, each three-dimensional coordinate point in the three-dimensional point cloud spatial distribution is mapped to the two-dimensional coordinate system of the projection plane to obtain the two-dimensional coordinate point corresponding to each three-dimensional coordinate point.

[0166] Here, a two-dimensional coordinate point refers to a point in the two-dimensional coordinate system of the projection plane, consisting of a pair of U and V coordinate values, used to represent the projected position of a three-dimensional coordinate point.

[0167] In some embodiments, for each 3D coordinate point in the 3D point cloud spatial distribution, a geometric transformation (usually a perspective projection transformation) defined by preset camera viewpoint parameters is applied. This transformation converts the world coordinates of each 3D coordinate point into camera coordinates, and then calculates the 2D coordinates of that point on the projection plane by dividing by the depth value and other operations. This process is repeated for all 3D coordinate points to obtain a series of corresponding 2D coordinate points.

[0168] In some embodiments, depth culling (Z-culling) can be performed during the mapping process, mapping only 3D coordinate points located within the camera's view frustum and whose depth values ​​lie between a preset near clipping plane and a far clipping plane, to avoid points outside the rendering range or those that are occluded. Another extension is to perform distortion correction on the resulting 2D coordinate points after the geometric transformation, applying a preset lens distortion model (such as radial distortion) to simulate the optical characteristics of a real camera.

[0169] As an example, a three-dimensional coordinate point located at (5, 2, 3) in three-dimensional space is mapped to two-dimensional coordinates (128.5, 96.2) on the projection plane after a perspective projection matrix transformation and subsequent calculation.

[0170] In step 1034, the two-dimensional coordinate points in the two-dimensional coordinate system are rasterized to obtain the target view rendering image.

[0171] Here, rasterization refers to the process of converting discrete two-dimensional coordinate point information into a regular two-dimensional image composed of pixels. The target view rendering image is a two-dimensional image that displays the visual appearance of a three-dimensional model from a preset camera viewpoint in pixel form.

[0172] In some embodiments, firstly, a blank two-dimensional image canvas with a preset resolution (e.g., 512x512 pixels) is created. Then, all the two-dimensional coordinate points generated in step 1033 are traversed, and the coordinate value of each two-dimensional coordinate point is rounded to determine its corresponding pixel position on the image canvas. A preset color value (e.g., white) is then drawn at that pixel position. After all points are drawn, this canvas becomes the target viewpoint rendering image.

[0173] In some embodiments, point splatting can be used for rasterization. Instead of drawing a single pixel at each 2D coordinate point, a graphic with a certain size and shape (such as a circle or Gaussian distribution) is drawn. Its size can be scaled according to the point's depth value in 3D space to simulate the effect of objects appearing larger in the distance and to make the rendering result more continuous. Another extension approach is that if the 3D point cloud spatial distribution contains additional information such as color and normals, this information is used during rasterization to perform simple lighting calculations, generating richer colors for each point instead of uniform color values, to obtain a target viewpoint rendering map with varying brightness and color.

[0174] As an example, create a 512x512 black canvas. For a 2D coordinate point (128.5, 96.2), round its coordinates to (129, 96). At the pixel position in column 129, row 96 of the canvas, change the color from black to white. Repeat this operation for all 2D coordinate points to obtain an image composed of white dots that shows the outline of the target object, i.e., the target view rendering.

[0175] This application embodiment analyzes a 3D model to obtain the 3D point cloud spatial distribution of the target object; determines the projection plane of the 3D point cloud spatial distribution under the target viewpoint based on preset camera viewpoint parameters; maps each 3D coordinate point in the 3D point cloud spatial distribution to the 2D coordinate system of the projection plane to obtain the corresponding 2D coordinate point; and rasterizes the 2D coordinate points in the 2D coordinate system to obtain the target viewpoint rendering image. This scheme simplifies the complex 3D scene rendering process into a direct geometric transformation and rasterization of the 3D point cloud spatial distribution, achieving a lightweight and efficient rendering pipeline. The core advantage of this method is that it does not rely on the complex mesh processing, texture mapping, and lighting calculations of traditional rendering engines, but directly operates on the most basic geometric element—the point—thereby greatly reducing computational overhead and hardware requirements. This method solves the technical problems of high computational cost and slow speed in standard 3D rendering processes in related technologies. It can quickly generate a target viewpoint rendering image that accurately reflects the geometric contour and spatial position of the target object under the target viewpoint without complex GPU acceleration, providing a simple and effective geometric anchor point for subsequent image generation models.

[0176] See also Figure 3 The following will be an explanation following step 103 above.

[0177] In step 104, the relative pose parameters of the reference camera viewpoint parameters and the preset camera viewpoint parameters for each reference image are determined.

[0178] In some embodiments, see Figure 9 , Figure 9 This is a schematic diagram of the seventh process of the image generation method provided in the embodiments of this application. Figure 3 Step 104 can be performed for each reference image. Figure 9 Steps 1041 to 1044 are implemented, and the details are explained below.

[0179] In step 1041, the first pose matrix corresponding to the reference camera viewpoint parameters of the reference image and the second pose matrix corresponding to the preset camera viewpoint parameters are obtained.

[0180] Here, the first pose matrix is ​​a 4x4 homogeneous transformation matrix used to accurately describe the position and pose of the camera in a single reference image in the 3D world coordinate system. The second pose matrix is ​​also a 4x4 homogeneous transformation matrix used to accurately describe the position and pose of the camera at the target generation viewpoint in the same 3D world coordinate system.

[0181] In some embodiments, reference camera viewpoint parameters determined in a previous step for the reference image being processed can be read from memory and constructed into a 4x4 matrix as the first pose matrix. Simultaneously, user-inputted preset camera viewpoint parameters are read and similarly constructed into a 4x4 matrix as the second pose matrix.

[0182] In some embodiments, after obtaining the pose matrix, a matrix verification operation can be performed. This checks whether the rotation portion of the matrix is ​​an orthonormal matrix with a determinant of +1, ensuring that the obtained pose matrix is ​​a valid rigid body transformation matrix and correcting for minor deviations caused by numerical calculation errors. Another extension is that if the obtained camera view parameters are in quaternion plus translation vector format, they are first converted to a 4x4 homogeneous transformation matrix format to unify subsequent calculation processes.

[0183] As an example, the first pose matrix obtained is matrix A, and the second pose matrix is ​​matrix B, where A and B are both 4x4 homogeneous transformation matrices, containing their respective rotation and translation information.

[0184] In step 1042, the product of the inverse of the first pose matrix and the second pose matrix is ​​determined as the relative transformation matrix of the reference image relative to the preset camera viewpoint parameters.

[0185] Here, for a given matrix, its inverse matrix is ​​another matrix whose product is the identity matrix, representing the inverse transformation of the original matrix. The relative transformation matrix is ​​a 4x4 homogeneous transformation matrix that describes the rotation and translation operations required to transform from the coordinate system defined by the first pose matrix to the coordinate system defined by the second pose matrix.

[0186] In some embodiments, firstly, a matrix inversion operation is performed on the first pose matrix obtained in step 1041. Since the pose matrix is ​​an orthogonal matrix, its inverse matrix can be quickly calculated by transposing its rotation part and performing corresponding transformations on its translation part. Then, the obtained inverse matrix is ​​multiplied by the second pose matrix, and the resulting 4x4 matrix is ​​the relative transformation matrix.

[0187] In some embodiments, the data types of the matrices involved in the operation are checked and standardized before matrix multiplication, for example, by converting them to 64-bit floating-point numbers, to maintain higher numerical precision and reduce accumulated errors during inversion and multiplication. Another extension is to use a method based on singular value decomposition (SVD) to compute the pseudo-inverse of the matrix. This method has better numerical stability when the matrix is ​​close to singular or ill-conditioned, and can serve as a robust supplement to standard inversion methods.

[0188] As an example, first calculate the inverse matrix inv(A) of the first pose matrix A. Then, calculate the product of inv(A) and the second pose matrix B to obtain the relative transformation matrix C, i.e., C = inv(A) * B.

[0189] In step 1043, rotation matrix components and translation vector components are extracted from the relative transformation matrix.

[0190] Here, the rotation matrix component refers to the 3x3 submatrix in the relative transformation matrix that describes the relative rotation relationship between the two coordinate systems. The translation vector component refers to the 3x1 vector in the relative transformation matrix that describes the relative displacement relationship between the origins of the two coordinate systems.

[0191] In some embodiments, the 4x4 relative transformation matrix obtained in step 1042 can be parsed in blocks. The 3x3 submatrix in its upper left corner is extracted as the rotation matrix component. The first three elements of its fourth column are extracted to form a 3x1 column vector, which is used as the translation vector component.

[0192] In some embodiments, after extracting the rotation matrix components, they are converted to Euler angles or quaternion representations. This more compact representation is more convenient in certain subsequent processing (such as interpolation or as input to neural networks) and avoids gimbal lock issues. Another extension is to calculate the magnitude (i.e., Euclidean distance) of the translation vector after extracting its components and then normalize the translation vector. Separating direction and distance allows for different weights to be applied to them or for independent analysis in subsequent processing.

[0193] As an example, for a 4x4 relative transformation matrix C, extract its C[0:3, 0:3] part as a 3x3 rotation matrix component R, and extract its C[0:3, 3] part as a 3x1 translation vector component T.

[0194] In step 1044, the relative pose parameters of the reference camera viewpoint parameters and the preset camera viewpoint parameters for each reference image are generated based on the rotation matrix components and translation vector components.

[0195] Here, the relative pose parameter is a numerical vector used to quantitatively describe the geometric relationship between a reference viewpoint and the target generation viewpoint.

[0196] In some embodiments, the 3x3 rotation matrix component extracted in step 1043 can be flattened into a 9-dimensional vector. This 9-dimensional vector is then concatenated with the 3x1 translation vector component to form a 12-dimensional vector. This 12-dimensional vector represents the relative pose parameters of the reference image.

[0197] In some embodiments, instead of directly using the flattened rotation matrix, it can be converted into a 6D continuous rotation representation (e.g., by extracting the first two columns of the rotation matrix using the Gram-Schmidt process). This 6D representation has no redundancy and avoids the discontinuity problems of representations such as Euler angles, making it more suitable as input to neural networks. Finally, this 6D vector is concatenated with the 3D translation vector to form 9-dimensional relative pose parameters. Another extension is to normalize the descriptor after generation, for example, by scaling each element of the translation vector component to the [-1, 1] interval, to eliminate the influence of different scene scales and enhance the generalization ability of the descriptor.

[0198] As an example, a 3x3 rotation matrix [[r11, r12, r13], [r21, r22, r23], [r31, r32, r33]] can be flattened into a vector [r11, r12, ..., r33]. Concatenating this with a 3x1 translation vector [tx, ty, tz] yields 12-dimensional relative pose parameters [r11, ..., r33, tx, ty, tz].

[0199] This application embodiment obtains the first pose matrix corresponding to the reference camera viewpoint parameters of the reference image and the second pose matrix corresponding to the preset camera viewpoint parameters; determines the product of the inverse matrix of the first pose matrix and the second pose matrix as the relative transformation matrix of the reference image relative to the preset camera viewpoint parameters; extracts rotation matrix components and translation vector components from the relative transformation matrix; and generates the relative pose parameters of the reference camera viewpoint parameters and the preset camera viewpoint parameters for each reference image based on the rotation matrix components and translation vector components. The above scheme, through a series of precise matrix operations, transforms two absolute poses defined in the same world coordinate system into a relative pose parameter that directly describes their geometric relationship. Its core advantage lies in the fact that this relative representation eliminates the dependence on the absolute world coordinate system, making the spatial relationship between cameras clear and directly comparable. Regardless of how the entire scene is translated or rotated, the relative transformation between the reference viewpoint and the target viewpoint remains unchanged. This method solves the technical problem in related technologies where the geometric relationship between viewpoints is difficult to quantify and utilize because the camera pose is defined in an arbitrary coordinate system. The generated relative pose parameters provide a standardized, global coordinate system-independent quantitative basis for subsequent viewpoint selection and feature fusion, greatly improving the robustness and consistency of the processing flow.

[0200] See also Figure 3 The following will be an explanation following step 104 above.

[0201] In step 105, the viewpoint features of each reference image are fused according to the relative pose parameters to obtain multi-viewpoint features.

[0202] In some embodiments, see Figure 10 , Figure 10 This is a schematic diagram of the eighth process of the image generation method provided in the embodiments of this application. Figure 3 Step 105 can be achieved through Figure 10 Steps 1051 to 1054 are implemented, and the details are explained below.

[0203] In step 1051, the relative pose parameters are sine and cosine position encoded to obtain pose features.

[0204] Here, sine and cosine position coding refers to a technique that uses sine and cosine functions to map a numerical vector into a higher-dimensional feature space, aiming to inject periodic positional or sequential information into the original numerical value. Pose features refer to the high-dimensional feature vectors used to represent relative pose relationships in the feature space after sine and cosine position coding.

[0205] In some embodiments, for the relative pose parameters of each reference image, each numerical component of the relative pose parameters can be input into a set of sine and cosine functions with different frequencies for calculation. The results of all function calculations are concatenated to form a new vector with a higher dimension than the original relative pose parameters; this new vector is the pose feature.

[0206] In some embodiments, learnable sine and cosine position encoding can be employed. In this approach, the frequencies of the sine and cosine functions are not fixed but serve as trainable parameters, automatically learning the optimal frequency combination during model training to better capture specific patterns in the pose data. Alternatively, a nonlinear transformation can be applied to the relative pose parameters before sine and cosine position encoding, for example, using a small multilayer perceptron, to extract higher-order interaction features. The transformed features are then used for position encoding, thereby enhancing the expressive power of the pose features.

[0207] As an example, for a 12-dimensional relative pose parameter, each component is transformed using 16 sets of sine and cosine functions of different frequencies, resulting in 32 encoded values ​​for each component. Concatenating the encoded values ​​of all components yields a 12 * 32 = 384-dimensional pose feature.

[0208] In step 1052, the matching degree between the pose features and the preset camera view parameters is determined.

[0209] In some embodiments, step 1052 above can be implemented in the following manner: determining the original matching score between the pose features and the preset camera view parameters; generating random noise that follows a standard normal distribution; superimposing the random noise onto the original matching score to obtain a perturbed matching score; and normalizing the perturbed matching score to obtain the matching degree between the pose features and the preset camera view parameters.

[0210] Here, random noise is a numerical value sampled from a preset probability distribution to introduce random perturbation. The perturbation matching score refers to the intermediate score containing randomness obtained by superimposing random noise onto the original matching score. The matching degree refers to the final score after normalization, whose value is within a preset range, and is used to quantitatively represent and compare the degree of matching between different reference images and the target viewpoint.

[0211] In some embodiments, firstly, the pose features of each reference image are input into a pre-defined scoring network (e.g., a multilayer perceptron), and the output of this network is the original matching score. Next, for each original matching score, a random value following a standard normal distribution (mean 0, variance 1) is independently generated as random noise. Subsequently, the generated random noise is added element-wise to the corresponding original matching score to obtain a perturbed matching score. Finally, the perturbed matching scores of all reference images are grouped into a set, and this set is normalized (e.g., by applying the Softmax function), converting each score into a probability value between 0 and 1. This final probability value represents the degree of matching between the pose features and the pre-defined camera viewpoint parameters.

[0212] In some embodiments, when generating random noise, the variance of the random noise is not a fixed value, but rather a learnable parameter or dynamically adjusted according to the training phase. A larger variance is used in the early stages of training to encourage the model to explore more different combinations of reference images. As the training process converges, the variance is gradually reduced to stabilize the selection process, achieving a smooth transition from exploration to utilization. A Softmax function with a temperature parameter can also be used during normalization. By adjusting the temperature parameter, the sharpness of the final matching degree distribution can be controlled. Lower temperatures make the highest-scoring matches more prominent, resulting in stronger determinism in viewpoint selection; while higher temperatures make the matching degree distribution smoother, leading to more diverse selection results.

[0213] As an example, suppose there are three reference images. The original matching scores obtained after the pose features of the three reference images are processed by a scoring network are [2.5, 1.0, 0.5]. The random noise generated from the standard normal distribution is [0.1, -0.2, 0.05]. The perturbed matching score obtained by adding the two is [2.6, 0.8, 0.55]. After normalizing this set of perturbed matching scores using the Softmax function, the final matching degree may be [0.84, 0.09, 0.07].

[0214] This application's embodiments determine the original matching score between pose features and preset camera viewpoint parameters; generate random noise following a standard normal distribution; superimpose the random noise onto the original matching score to obtain a perturbed matching score; and normalize the perturbed matching score to obtain the matching degree between pose features and preset camera viewpoint parameters. The above scheme introduces random noise into the viewpoint selection mechanism during the determination of the matching degree, injecting randomness into the mechanism. Its core advantage lies in the fact that this random perturbation breaks the purely deterministic selection, playing a regularizing role during training. It effectively prevents the model from overfitting when selecting reference images, i.e., over-reliance on a few best-performing views in the current training batch, thus ignoring other equally valuable views. This method solves the technical problems of poor model generalization ability and sensitivity to input changes that may result from deterministic viewpoint selection mechanisms in related technologies. By encouraging the model to explore more diverse viewpoint combinations, it enhances the model's robustness, enabling it to learn more general multi-view fusion strategies, rather than simply memorizing a few specific "optimal" viewpoint combinations.

[0215] In step 1053, a preset number of target reference images are selected from multiple reference images based on the matching degree.

[0216] Here, the preset number is a pre-defined integer value used to specify the number of optimal images to be selected from all reference images. The target reference images are the top few reference images selected from all reference images after being sorted by matching degree, and are most relevant to the target viewpoint.

[0217] In some embodiments, all reference images can be paired with their corresponding matching scores. Then, all reference images are sorted in descending order according to the matching score. Finally, the top K reference images are selected from the sorted list, where K equals a preset number; these K reference images are the selected target reference images.

[0218] In some embodiments, a stochastic top-K selection strategy can be employed. In this approach, instead of strictly selecting the K images with the highest matching scores, a random perturbation (e.g., Gumbel noise) is added to the matching score of each reference image, and then the top K images are selected based on the perturbed scores. This method increases the diversity of selection during training and prevents the model from becoming overly reliant on a few fixed viewpoints. In another embodiment, the preset number K can not be a fixed value but can be dynamically determined. For example, a matching score threshold can be set, and all reference images with matching scores higher than this threshold are selected as target reference images. This allows the number of images participating in the fusion to be adaptively adjusted according to different situations.

[0219] As an example, suppose there are 10 reference images, with a preset quantity of 3. These 10 reference images are sorted according to the calculated matching degree, and then the 1st, 2nd, and 3rd images in the sorted list are selected as the 3 target reference images.

[0220] In step 1054, the viewpoint features of each target reference image are fused to obtain multi-viewpoint features.

[0221] In some embodiments, step 1054 above can be implemented in the following way: normalize the matching degree corresponding to each target reference image to obtain the weight corresponding to each target reference image; and perform weighted fusion of the view features of each target reference image according to the weight corresponding to each target reference image to obtain multi-view features.

[0222] Here, the multi-view feature is a final feature representation that aggregates information from all target reference images. It is generated by weighted fusion of the view features of each target reference image according to their corresponding weights.

[0223] In some embodiments, firstly, the matching degree corresponding to each target reference image is normalized to obtain the weight corresponding to each target reference image. Then, for each target reference image, its corresponding weight (a scalar) is multiplied by its viewpoint feature (a tensor). Finally, all the weighted viewpoint feature tensors obtained after scalar multiplication are summed element-wise, and the final summation result is the multi-viewpoint feature.

[0224] In some embodiments, an additional residual connection can be introduced after weighted fusion. The average (unweighted) features of all target reference image viewpoint features are added to the multi-viewpoint features obtained after weighted fusion. This ensures that even if some viewpoints have extremely low weights, their fundamental information is not completely discarded, thus enhancing the stability of the fusion process. Another extension is to concatenate all weighted viewpoint features along the channel dimension instead of directly using weighted summation, and then input the resulting massive feature set into one or more convolutional layers for processing. This approach allows the learning of a more complex, non-linear fusion strategy, rather than simply linear superposition.

[0225] As an example, suppose there are two target reference images with weights of 0.8 and 0.2, respectively, and their corresponding viewpoint features are tensor A and tensor B. First, calculate the weighted viewpoint features: 0.8 * A and 0.2 * B. Then, add these two results together to obtain the multi-viewpoint feature C = (0.8 * A) + (0.2 * B).

[0226] In some embodiments, the above-mentioned "normalizing the matching degree corresponding to each target reference image to obtain the weight corresponding to each target reference image" can be achieved by: determining the exponential function value of the matching degree corresponding to each target reference image; summing the exponential function values ​​of each target reference image to obtain the normalized denominator; and using the quotient of the exponential function value of each target reference image and the normalized denominator as the weight corresponding to each target reference image.

[0227] Here, the exponential function value refers to the numerical value obtained by applying the natural exponential function to the matching degree corresponding to a target reference image. The normalized denominator is the sum of the exponential function values ​​of all target reference images. The quotient is the result of dividing the exponential function value of a target reference image by the normalized denominator. The weight is a value that, after normalization, represents the relative importance of each target reference image in subsequent fusion steps; the sum of all weights is 1.

[0228] In some embodiments, firstly, for each target reference image, the exponential function value of its corresponding matching degree is calculated. Then, the exponential function values ​​of all target reference images are summed to obtain a normalized denominator. Finally, for each target reference image, its corresponding exponential function value is divided by the normalized denominator, and the quotient is used as the weight corresponding to that target reference image.

[0229] In some embodiments, a global temperature parameter is introduced before calculating the exponential function value. The matching degree corresponding to each target reference image is divided by this temperature parameter before calculating the exponential function value. By adjusting the temperature parameter, the smoothness of the final weight distribution can be controlled: lower temperatures result in a sharper weight distribution, concentrated on a few images with the highest matching degrees; higher temperatures result in a smoother weight distribution. Another extension is to ensure the stability of the numerical calculation by first determining the maximum matching degree among all target reference images before calculating the exponential function value, then subtracting this maximum value from each matching degree before performing the subsequent exponential function value calculation. This operation is mathematically equivalent but can effectively prevent floating-point overflow problems caused by excessively large matching degrees.

[0230] As an example, suppose there are three target reference images with corresponding matching degrees of [2.0, 1.0, 0.1]. First, calculate the exponential function value for each matching degree: exp(2.0) ≈ 7.39, exp(1.0) ≈ 2.72, exp(0.1) ≈ 1.11. Then, sum all the exponential function values ​​to obtain the normalized denominator: 7.39 + 2.72 + 1.11 = 11.22. Finally, calculate the weight corresponding to each target reference image: the weight of the first image is 7.39 / 11.22 ≈ 0.66, the weight of the second image is 2.72 / 11.22 ≈ 0.24, and the weight of the third image is 1.11 / 11.22 ≈ 0.10.

[0231] This application transforms a set of matching scores within an arbitrary range into a probability distribution with a sum of 1 by applying an exponential function transformation and dividing by the sum. Its core advantage lies in the fact that this method not only preserves the original order of matching scores but also amplifies the differences between scores through the characteristics of the exponential function. This allows target reference images with higher matching scores to receive weights far exceeding their linear proportions, thus playing a more dominant role in subsequent feature fusion. This method solves the technical problem in related technologies of how to transform unbounded matching scores into a set of effective weights that can be directly used for weighted summation and effectively highlight key information, ensuring the stability and effectiveness of the weighted fusion process.

[0232] This application's embodiments normalize the matching degree corresponding to each target reference image to obtain the weight corresponding to each target reference image; based on the weight corresponding to each target reference image, the viewpoint features of each target reference image are weighted and fused to obtain multi-viewpoint features. The above scheme achieves an intelligent information aggregation mechanism by performing weighted fusion. Its core advantage lies in the fact that it does not simply average or stitch together the viewpoint features of all selected target reference images, but dynamically adjusts the contribution of each viewpoint feature according to the weight determined by the matching degree between each viewpoint and the target viewpoint. This method solves the technical problem in related technologies of how to effectively integrate multiple information sources while distinguishing their different importance. By giving greater influence to viewpoint features with higher matching degrees, it ensures that the final generated multi-viewpoint features can retain the most relevant and critical visual information of the target viewpoint to the greatest extent, while suppressing secondary or potentially interfering viewpoint information, thereby generating a more accurate, robust, and representative fused feature.

[0233] This application embodiment obtains pose features by performing sine and cosine position encoding on relative pose parameters; determines the matching degree between the pose features and preset camera viewpoint parameters; selects a preset number of target reference images from multiple reference images based on the matching degree; and fuses the viewpoint features of each target reference image to obtain multi-viewpoint features. The above scheme achieves a coarse-to-fine, efficient, and robust viewpoint feature fusion strategy through a multi-stage process. First, sine and cosine position encoding transforms discrete relative pose parameters into continuous and high-dimensional pose features, making them more suitable for deep learning model processing. Next, a sparse selection mechanism based on matching degree is introduced, selecting only a preset number of target reference images most relevant to the target viewpoint for the next stage. The core advantage of this method is that this "select first, then fuse" strategy greatly reduces the computational complexity and information redundancy of subsequent fusion steps, solving the technical problems in related technologies where simply fusing all available viewpoint features results in high computational costs and potential noise interference from irrelevant viewpoint information in the final result. Ultimately, by weighting and fusing only these optimal perspective features, we ensured that the generated multi-perspective features were highly correlated, information-pure, and computationally efficient.

[0234] See also Figure 3 The following will be an explanation following step 105 above.

[0235] In step 106, based on the image generation prompts, the target view rendering map, and the multi-view features, the pre-trained image generation model is invoked to generate the target image of the target object.

[0236] In some embodiments, see Figure 11 , Figure 11This is a ninth flowchart illustrating the image generation method provided in this application embodiment. Figure 3 Step 106 can be achieved through Figure 11 Steps 1061 to 1064 are implemented, and the details are explained below.

[0237] In step 1061, the target view rendering image is encoded to obtain rendering features.

[0238] Here, rendering features refer to high-dimensional features that represent the geometric structure and contour information of the target viewpoint rendering image after processing by the encoder network.

[0239] In some embodiments, the target viewpoint rendering image can be input into a pre-built encoder network (e.g., the encoding part of a convolutional neural network or a visual Transformer). This network transforms the input 2D image into a feature map or feature sequence that is smaller in spatial dimension but deeper in channel dimension through a series of downsampling and feature extraction operations; this output is the rendering feature.

[0240] In some embodiments, a variational autoencoder (VAE) can be used as the encoder for this step. This approach not only yields the rendered features but also a variance describing the distribution of those features, which is useful in some generative models that require random sampling. Another extension is to introduce an image of the same size as the target viewpoint render image, but containing random noise, during the encoding process. The two images are then concatenated along the channel dimension and fed together into the encoder. This increases the randomness of the input, contributing to greater diversity in subsequent generative models.

[0241] As an example, a 512x512 pixel target view rendering map is input into a convolutional encoder. After several convolutional and pooling layers, it outputs a 32x32x512 dimensional feature map as the rendering feature.

[0242] In step 1062, the rendered features are concatenated with preset Gaussian noise to obtain the hidden state features.

[0243] Here, the pre-defined Gaussian noise refers to a feature map that has the same spatial dimension as the rendered features but may have a different number of channels. All its values ​​are independently and randomly sampled from a standard Gaussian distribution. The latent state features refer to the initial latent representation that serves as the starting point of the image generation process, combining deterministic geometric information (from the rendered features) and a random seed (from the Gaussian noise).

[0244] In some embodiments, firstly, a preset Gaussian noise tensor with the same size as the rendering feature space (e.g., 32x32) and a preset number of channels (e.g., 4) is generated. Then, the rendering features are concatenated with this preset Gaussian noise tensor along the channel dimension to form a new feature tensor with more channels. This new tensor is the hidden state feature.

[0245] In some embodiments, the splicing operation is not fixed but employs a learnable gating mechanism. This mechanism dynamically determines the fusion weights of Gaussian noise and rendered features based on their content, achieving adaptive noise injection. Another extension is to avoid direct splicing and instead transform the rendered features and Gaussian noise separately through different linear projection layers, then sum the transformed results element-wise for more flexible fusion of the two types of information.

[0246] As an example, a 32x32x512 dimensional rendering feature is concatenated with a 32x32x4 dimensional preset Gaussian noise tensor in the channel dimension to obtain a 32x32x516 dimensional latent state feature.

[0247] In step 1063, the image-generated prompt words are encoded to obtain text features.

[0248] Here, text features refer to the high-dimensional numerical vector sequence that represents the semantic content of the image-generated prompts in the latent space, obtained by processing the prompts through a text encoder.

[0249] In some embodiments, an image-generated prompt (a string) can be input into a pre-trained text encoder (such as CLIP text encoder or BERT). The encoder first segments the text and converts it into corresponding ID sequences, then processes the ID sequences through a Transformer-based network structure, and finally outputs a vector sequence, where each vector corresponds to the semantic representation of an input token, and the entire sequence is the text feature.

[0250] In some embodiments, when generating prompts from encoded images, one or more negative prompts can be additionally introduced and encoded. In subsequent generation processes, positive text features are used for guidance, and negative text features are used for reverse guidance, to better control the generated content and avoid unwanted elements. Another extension approach is to enhance the generated text features, for example, by performing a non-linear transformation using a small network, or by fusing the outputs of multiple different text encoders to obtain a richer and more robust semantic representation.

[0251] As an example, the image-generated prompt "a red backpack in the snow" is input into a CLIP text encoder, which outputs a vector sequence of length 77 with each element having 768 dimensions, as the text feature.

[0252] In step 1064, based on the latent state features, text features, and multi-view features, a pre-trained image generation model is invoked to generate the target image of the target object.

[0253] In some embodiments, latent state features, text features, and multi-view features can be used as conditional inputs and fed into the backbone network of a pre-trained image generation model (e.g., a Transformer-based diffusion model). This model iteratively denoises or transforms the latent state features at multiple time steps through a series of computational modules (such as self-attention and cross-attention layers). In each iteration, the model uses text features and multi-view features to guide the update direction of the latent state features. After the iteration process reaches a preset number of steps, the finally converged latent state features are fed into a decoder network to transform them from the latent space back into the pixel space, ultimately generating the target image of the target object.

[0254] As an example, an image generation model based on DiT (Diffusion Transformer) receives latent state features as its initial input and interacts with text features and multi-view features respectively through a cross-attention mechanism in each Transformer block. After 28 time steps of iterative denoising, the final latent state features are passed through a VAE decoder to generate a 512x512 pixel target image.

[0255] This application embodiment encodes the target viewpoint rendering image to obtain rendering features; concatenates the rendering features with preset Gaussian noise to obtain latent state features; encodes image generation prompts to obtain text features; and uses a pre-trained image generation model to generate the target image of the target object based on the latent state features, text features, and multi-view features. The above scheme constructs a clear and highly controllable image generation process by explicitly separating and fusing information from three different modalities—geometry (rendering features), randomness (Gaussian noise), and semantics (text features and multi-view features). Its core advantage lies in the explicit concatenation of rendering features (serving as geometric anchors) with Gaussian noise (serving as a source of generation diversity) in step 1062, ensuring that the generation process strictly adheres to the preset viewpoint and contour while also possessing the randomness to generate rich textures and details. This method solves the technical problem in related technologies where relying solely on text or a single image as a condition makes it difficult to simultaneously and accurately control the geometric structure and content diversity of the generated image. By injecting multi-view features and text features as independent conditions, decoupled control of the three dimensions of "identity", "viewpoint" and "scene" of the generated image is achieved, which greatly improves the controllability and fidelity of customized image generation.

[0256] In some embodiments, see Figure 12 , Figure 12 This is a schematic diagram of the tenth process of the image generation method provided in the embodiments of this application. Figure 11 Step 1064 can be achieved through Figure 12 Steps 10641 to 10645 are implemented, and the details are explained below.

[0257] In step 10641, a pre-trained image generation model is invoked to perform a linear mapping on the latent state features to obtain intermediate features.

[0258] Here, intermediate features refer to the intermediate state representations obtained by linearly mapping the latent state features, which are used to prepare for conditional information fusion.

[0259] In some embodiments, the latent state features can be input into a linear projection layer in a pre-trained image generation model. This linear projection layer performs matrix multiplication on each feature vector in the latent state features using a weight matrix, and may add a bias vector. The output of this operation is the intermediate feature, whose dimensions are typically adjusted as needed for subsequent processing.

[0260] In some embodiments, a nonlinear mapping approach can be employed, where the hidden state features are first passed through a linear mapping layer, the result is then input into a nonlinear activation function (e.g., GeLU or SiLU), and finally passed through a second linear mapping layer. This multi-layered structure can extract more complex, nonlinear feature representations as richer intermediate features. Another extension is to use a channel attention mechanism. After linearly mapping the hidden state features, a small network is connected that dynamically generates weights for different channels based on the feature content. The channels of the intermediate features are then weighted to highlight the more important feature dimensions.

[0261] As an example, a 32x32x516 dimensional latent state feature is processed by a 1x1 convolutional layer (an equivalent linear mapping method), which maps 516 input channels to 1024 output channels, ultimately resulting in a 32x32x1024 dimensional intermediate feature.

[0262] In step 10642, attention processing is performed on the intermediate features based on the text features and multi-view features to obtain attention features.

[0263] Here, attention features refer to new features rich in contextual guidance information obtained by incorporating the external conditional information carried by text features and multi-view features into intermediate features through an attention mechanism.

[0264] In some embodiments, the intermediate features can be used as the query vector, and the text features and multi-view features as the key and value vectors, respectively, to perform two independent cross-attention operations. In the first operation, the intermediate features interact with the multi-view features; in the second operation, the intermediate features interact with the text features. Then, the outputs of the two cross-attention operations are residually concatenated (added) with the original intermediate features to obtain the attention features.

[0265] In some embodiments, a gated fusion mechanism can be employed. After calculating the results of interactions with textual features and multi-view features separately, instead of directly adding them, a small neural network dynamically generates two gate weights based on intermediate features. These two weights are used to perform a weighted sum of the two interaction results, allowing the model to adaptively decide whether to prioritize textual guidance or viewpoint guidance in the current step. Alternatively, textual features and multi-view features can be pre-fused. For example, they can be interacted with through an additional attention layer, or they can be simply concatenated along the sequence dimension to form a unified conditional feature. Then, a cross-attention operation between the intermediate feature and this unified conditional feature is performed only once to reduce computational cost and potentially capture the synergistic effect between the two conditions.

[0266] As an example, cross-attention is performed on the 32x32x1024 dimensional intermediate feature (as the query) and the 77x768 dimensional text feature (as the key and value) to obtain the first output. Simultaneously, the same intermediate feature is cross-attention performed on the 256x1024 dimensional multi-view feature (as the key and value) to obtain the second output. These two outputs are then added to the original intermediate feature to obtain the final attention feature.

[0267] In some embodiments, see Figure 13 , Figure 13 This is a schematic diagram of the eleventh step of the image generation method provided in the embodiments of this application. Figure 12 Step 10642 can be achieved through Figure 13 Steps 201 to 204 are implemented, and the details are explained below.

[0268] In step 201, a first attention weighting operation is performed based on intermediate features and multi-view features to obtain visual context features.

[0269] Here, the first attention weighting operation is a type of cross-attention processing. It uses intermediate features as queries and multi-view features as keys and values ​​to calculate which information should be aggregated from the multi-view features at each position in the intermediate features, and the corresponding weights. Visual context features refer to the result obtained after the first attention weighting operation, where the target object's identity and appearance information contained in the multi-view features is weighted and fused into the intermediate features.

[0270] In some embodiments, step 201 above can be implemented as follows: performing a first linear projection transformation on the intermediate features to obtain a first query vector matrix for multi-view features; performing a second linear projection transformation on the multi-view features to obtain a first key-value vector matrix for multi-view features; determining a first product of the transpose of the first query vector matrix and the first key-value vector matrix; normalizing the first product to obtain a first attention weight distribution; and weighting and summing the value vectors in the first key-value vector matrix according to the first attention weight distribution to obtain visual context features.

[0271] Here, the first linear projection transformation refers to performing matrix multiplication on the intermediate features using a pre-defined weight matrix, transforming them into a new feature space. The first query vector matrix is ​​the result obtained after performing the first linear projection transformation on the intermediate features, used to initiate the query in attention calculation. The second linear projection transformation refers to performing matrix multiplication on the multi-view features using two independent pre-defined weight matrices. The first key-value vector matrix is ​​the collective term for the key vector matrix and value vector matrix obtained after performing the second linear projection transformation on the multi-view features. The transpose matrix is ​​the new matrix obtained by interchanging the rows and columns of a matrix. The first product is the result of multiplying the first query vector matrix with the transpose of the key vector part of the first key-value vector matrix; the result is an attention score matrix. The first attention weight distribution is the probability distribution matrix obtained after normalizing the first product, where each element value is between 0 and 1 and the sum of each row is 1.

[0272] In some embodiments, firstly, a first linear projection transformation is performed on the intermediate features to obtain a first query vector matrix. Simultaneously, a second linear projection transformation is performed on the multi-view features to generate a first key-value vector matrix (containing independent key vector and value vector parts). Next, a first product is determined between the first query vector matrix and the transpose of the key vector parts of the first key-value vector matrix. Then, the first product is normalized along its last dimension using the Softmax function to obtain a first attention weight distribution. Finally, based on the first attention weight distribution, the value vector parts of the first key-value vector matrix are weighted and summed; the result is the visual context feature.

[0273] In some embodiments, a multi-head attention mechanism can be employed. This involves dividing intermediate features and multi-view features into multiple "heads" along the channel dimension, and independently performing the complete attention weighting operation (using different linear projection transformations) on each head. Finally, the visual context features calculated from all heads are concatenated and fused through a final linear projection transformation. This approach allows the model to focus on different information in parallel across different representation subspaces. After calculating the first product, a learnable bias term or a term based on relative position encoding can be added. For example, a positional bias can be generated based on the relative positions of the query vector and key vector in the sequence and added to the first product, thus introducing spatial proximity preference into the attention computation.

[0274] As an example, assume the intermediate feature dimension is (N, C) and the multi-view feature dimension is (M, C). Through a first linear projection transformation, a first query vector matrix Q with dimension (N, D) is obtained. Through a second linear projection transformation, the key vector part K (dimension (M, D)) and the value vector part V (dimension (M, D)) of the first key-value vector matrix are obtained. The first product Score = Q * K^T (transpose of K) is calculated, resulting in a score matrix with dimension (N, M). The Score is normalized to obtain the first attention weight distribution W. Finally, the visual context feature Output = W * V with dimension (N, D) is calculated.

[0275] This embodiment of the application obtains a first query vector matrix for multi-view features by performing a first linear projection transformation on the intermediate features; a second linear projection transformation on the multi-view features obtains a first key-value vector matrix for multi-view features; a first product of the transpose of the first query vector matrix and the first key-value vector matrix is ​​determined; the first product is normalized to obtain a first attention weight distribution; and the value vectors in the first key-value vector matrix are weighted and summed according to the first attention weight distribution to obtain the visual context features. This scheme establishes a precise and dynamic information bridge between intermediate features and multi-view features through a standardized scaling dot product attention mechanism. Its core advantage lies in decomposing the complex feature fusion problem into a series of clear, differentiable matrix operations, namely, the interaction of queries, keys, and values. By calculating the first product of the first query vector matrix and the first key-value vector matrix, this method can dynamically calculate the relevance weights of each part of the intermediate features to all parts of the multi-view features, forming a first attention weight distribution. This method addresses the technical challenges of traditional feature fusion techniques that rely on simple concatenation or convolution to model long-range dependencies and have fixed fusion weights that cannot be dynamically adjusted based on content. Ultimately, by weighted summation of the value vectors, it ensures that only the most relevant visual information is aggregated into the visual context features, achieving efficient, accurate, and content-adaptive feature fusion.

[0276] In step 202, a second attention weighting operation is performed based on intermediate features and text features to obtain semantic context features.

[0277] Here, the second attention weighting operation is a type of cross-attention processing. It uses intermediate features as queries and text features as keys and values ​​to calculate which semantic information should be aggregated from the text features at each position in the intermediate features, and the corresponding weights. Semantic context features refer to the result obtained after the second attention weighting operation, where the scene and style description information contained in the text features is weighted and fused into the intermediate features.

[0278] In some embodiments, step 202 above can be implemented as follows: performing a third linear projection transformation on the intermediate features to obtain a second query vector matrix for the text features; performing a fourth linear projection transformation on the text features to obtain a second key-value vector matrix for the text features; determining a second product of the transpose of the second query vector matrix and the second key-value vector matrix; normalizing the second product to obtain a second attention weight distribution; and weighting and summing the value vectors in the second key-value vector matrix according to the second attention weight distribution to obtain semantic context features.

[0279] Here, the third linear projection transformation refers to performing matrix multiplication on the intermediate features using a pre-defined weight matrix. The second query vector matrix is ​​the result obtained after performing the third linear projection transformation on the intermediate features, used in attention calculation to initiate queries on text features. The fourth linear projection transformation refers to performing matrix multiplication on the text features using two independent pre-defined weight matrices. The second key-value vector matrix is ​​the collective term for the key vector matrix and value vector matrix obtained after performing the fourth linear projection transformation on the text features. The second product is the result of multiplying the second query vector matrix with the transpose of the key vector part of the second key-value vector matrix; the result is an attention score matrix reflecting semantic relevance. The second attention weight distribution is the probability distribution matrix obtained after normalizing the second product, where each element's value is between 0 and 1 and the sum of each row is 1.

[0280] In some embodiments, firstly, a third linear projection transformation is performed on the intermediate features to obtain a second query vector matrix. Simultaneously, a fourth linear projection transformation is performed on the text features to generate a second key-value vector matrix (containing independent key vector and value vector parts). Next, a second product is determined of the transpose of the key vector parts in the second query vector matrix and the second key-value vector matrix. Then, the second product is normalized along its last dimension using the Softmax function to obtain a second attention weight distribution. Finally, based on the second attention weight distribution, the value vector parts in the second key-value vector matrix are weighted and summed; the result is the semantic context feature.

[0281] In some embodiments, a multi-head attention mechanism can be employed. This involves dividing the intermediate features and text features into multiple "heads" along the channel dimension, and independently performing the complete attention weighting operation (using different linear projection transformations) on each head. Finally, the semantic context features calculated from all heads are concatenated and fused through a final linear projection transformation. This approach allows the model to focus on different text information in parallel across different semantic representation subspaces. To improve the efficiency of processing long text sequences, memory-optimized attention computation methods can also be used. For example, in determining the second product sum for weighted summation, techniques such as reorganizing the computation order and block computation can avoid instantiating a huge attention score matrix entirely in memory, thereby significantly reducing memory usage when generating prompts from long images.

[0282] As an example, assume the intermediate feature dimension is (N, C) and the text feature dimension is (L, C'). Through the third linear projection transformation, the second query vector matrix Q is obtained, with a dimension of (N, D). Through the fourth linear projection transformation, the key vector part K (dimension (L, D)) and the value vector part V (dimension (L, D)) of the second key-value vector matrix are obtained. The second product Score = Q * K^T (transpose of K) is calculated, resulting in a score matrix of dimension (N, L). The Score is normalized to obtain the second attention weight distribution W. Finally, the semantic context feature Output = W * V is calculated, with a dimension of (N, D).

[0283] This application embodiment obtains a second query vector matrix for text features by performing a third linear projection transformation on the intermediate features; obtains a second key-value vector matrix for text features by performing a fourth linear projection transformation on the text features; determines the second product of the transpose of the second query vector matrix and the second key-value vector matrix; normalizes the second product to obtain a second attention weight distribution; and weights the value vectors in the second key-value vector matrix according to the second attention weight distribution to obtain the semantic context features. The above scheme constructs a dynamic, content-aware semantic information injection channel between intermediate features and text features through a standardized scaling dot product attention mechanism. Its core advantage lies in decomposing the text-guided fusion process into a series of differentiable matrix operations. By calculating the second product of the second query vector matrix and the second key-value vector matrix, this method enables each part of the intermediate features to actively query all words in the text feature sequence and dynamically calculate the importance of each word to the current part, forming a second attention weight distribution. This method solves the technical problem in related technologies where simple concatenation or broadcast addition to fuse text information fails to achieve precise and localized semantic control. Ultimately, by weighted summation of the value vectors, the most relevant semantic information was ensured to be accurately injected into the corresponding image regions, achieving refined guidance from text description to visual content.

[0284] In step 203, residual connection processing is performed on the visual context features, semantic context features, and intermediate features to obtain residual features.

[0285] Here, residual connection processing refers to an operation that adds multiple feature tensors element-wise, aiming to add the transformed information (such as visual context features and semantic context features) as increments to the original information (intermediate features). Residual features are intermediate results obtained by performing residual connection processing on visual context features, semantic context features, and intermediate features, aggregating all information.

[0286] In some embodiments, tensors with identical dimensions—visual context features, semantic context features, and intermediate features—can be aligned, and then element-level addition can be performed on each corresponding position of the three tensors. The final summation result, with the same dimensions as the input tensor, is the residual feature.

[0287] In some embodiments, before addition, three learnable gating weights (scalars or vectors) can be introduced, which are multiplied by the visual context features, semantic context features, and intermediate features, respectively. By training these gating weights, the model can adaptively adjust the contribution ratio of each information source during fusion. Another extension is to add an additional non-linear activation function layer (e.g., GeLU) after performing residual connection processing. This non-linear transformation increases the model's expressive power, allowing for more complex non-linear combinations of features.

[0288] As an example, suppose the visual context features are tensor A, the semantic context features are tensor B, and the intermediate features are tensor C. The residual connection process calculates A + B + C to obtain the residual feature tensor D.

[0289] In step 204, the residual features are normalized to obtain the attention features.

[0290] Here, normalization is a computational process that rescales feature data to adjust the numerical distribution of features to a standard, more stable range, facilitating subsequent neural network computations. Attention features refer to the final output features after normalization; they are the final result of this attention module and are prepared for use in the next computational step of the image generation model.

[0291] In some embodiments, the residual features can be input into a layer normalization module. This module calculates the mean and variance of the residual features along the feature channel dimension, and then uses these two statistics to standardize the residual features so that their distribution approximates a standard normal distribution. Finally, the standardized features are subjected to an affine transformation using two learnable scaling and translation parameters to obtain the final attention features.

[0292] In some embodiments, adaptive normalization can be employed. In this approach, the scaling and translation parameters of the layer normalization module are not fixed, learnable parameters, but are dynamically generated by a small network based on an external condition (e.g., time-step embedding in a diffusion model). This allows the normalization process itself to adaptively adjust according to different generation stages. Another extension is to apply a channel shuffle operation to the residual features before normalization to facilitate information exchange between different channel groups, thereby enhancing the robustness of the feature representation.

[0293] As an example, the residual feature tensor D can be input into a normalization layer. After calculation, a tensor E with the same dimension as tensor D but with a more stable numerical distribution is output. Tensor E is the attention feature.

[0294] This application's embodiments obtain visual context features through a first attention weighting operation based on intermediate features and multi-view features; obtain semantic context features through a second attention weighting operation based on intermediate features and text features; obtain residual features by performing residual connection processing on the visual context features, semantic context features, and intermediate features; and obtain attention features by normalizing the residual features. The above scheme decouples the injection processes of conditional information (visual and semantic) from two different modalities and then unifies and fuses them through residual connections, constructing a structured and parallelized multi-condition control module. Its core advantage lies in allowing the model to calculate the impact of visual consistency (from multi-view features) and semantic consistency (from text features) on the current generation state in parallel and independently, avoiding mutual interference between the two strong conditional information in the early stages of fusion. Subsequently, through residual connections, these two independent adjustment information are aggregated with the original intermediate features, ensuring effective injection of conditional guidance while avoiding the gradient vanishing problem by preserving the original information flow. This method solves the technical problem in related technologies of how to clearly and stably apply multiple heterogeneous conditions to a single generation stream while avoiding information mixing and control conflicts, and significantly improves the generative model's ability to comply with complex conditions from multiple sources.

[0295] See also Figure 12 The following explanation follows step 10642 from the previous text.

[0296] In step 10643, the transmission velocity field of the hidden state features at the current time step is predicted based on the attention features.

[0297] Here, the current time step refers to a discrete point in time representing the current stage of progress during the entire iterative generation process. The transmission velocity field is a vector field with the same dimension as the hidden state features, where each vector indicates the direction and rate at which the corresponding hidden state feature component should move at the current time step.

[0298] In some embodiments, the attention features can be input into a dedicated prediction network module, which may consist of several residual blocks or Transformer blocks. The prediction network module processes the attention features and outputs a tensor with the same dimensions as the hidden state features; this output tensor is the predicted transmission velocity field.

[0299] In some embodiments, not only is the transmission velocity field predicted, but also an acceleration field (i.e., the rate of change of velocity). In subsequent numerical integration iterations, a second-order integration method is used to more accurately simulate the evolution trajectory of the features, potentially achieving the same effect with fewer time steps. Another extension is to adopt the U-Net architecture for the prediction network. By inputting attention features into U-Net and leveraging its multi-scale feature extraction and fusion capabilities, global and local information can be better captured, resulting in a more accurate and smoother transmission velocity field prediction.

[0300] As an example, a 32x32x1024-dimensional attention feature, after being processed by a prediction network, outputs a tensor with the same dimensions as the original hidden state feature (e.g., 32x32x516-dimensional), which serves as the transmission velocity field at the current time step.

[0301] In step 10644, the hidden state features are subjected to multiple time-step numerical integration iterations based on the transmission velocity field until a preset time step length is reached, thereby obtaining the converged hidden state features.

[0302] Here, numerical integration iteration is a computational process that continuously updates the state values ​​through discrete time steps to approximate the solution of a continuous differential equation. The preset time step size refers to the total number of iterations into which the entire generation process is divided. The convergent latent state feature refers to the latent state feature of the final stable state reached after iterative updates through all time steps; theoretically, this feature corresponds to the latent representation of the sharp image.

[0303] In some embodiments, steps 10641 to 10643 can be repeated at each time step in a loop from an initial time step to a preset time step to obtain the transmission velocity field at the current time step. Then, according to a simple numerical integration rule (such as the Euler method), the current hidden state feature is added to the product of the transmission velocity field and a small time increment to obtain the updated hidden state feature, which serves as the input for the next time step. This process continues until the loop is complete.

[0304] In some embodiments, higher-order numerical integration methods are employed, such as second- or fourth-order Runge-Kutta methods. This approach performs multiple velocity field evaluations within each time step, enabling a more accurate approximation of the true integration path. This results in better generation quality with the same preset time step size, or the same quality with fewer time steps. An adaptive step-size strategy can also be used. The increment size of each time step is dynamically adjusted based on the magnitude or rate of change of the transmission velocity field. Smaller step sizes are used for fine integration in regions of rapid feature changes, while larger step sizes are used to accelerate computation in regions of gradual change.

[0305] As an example, in the... Step, based on the predicted transmission velocity field Update hidden state features: { +1} = + *dt, where dt is the time increment. This iteration starts from... Starting with =0, execution continues until... =T-1 (T is the preset time step), finally obtaining As a feature of the hidden state after convergence.

[0306] In step 10645, the converged hidden state features are decoded to obtain the target image of the target object.

[0307] Here, decoding refers to the process of converting the high-dimensional, abstract feature representation (converged latent state features) in the latent space back into a concrete, visualized image in the pixel space.

[0308] In some embodiments, the converged latent state features can be input into a pre-trained decoder network (e.g., the decoder part of a variational autoencoder). This decoder network gradually increases the spatial dimension and reduces the channel dimension of the latent features through a series of transposed convolutions or upsampling layers, and finally outputs a tensor with standard image resolution and color channels, which is the target image of the target object.

[0309] In some embodiments, an image enhancement or super-resolution module is added after decoding. The initial image output from the decoder is post-processed, for example, to increase contrast, sharpen details, or a separate super-resolution network is used to upscale its resolution to a higher level to obtain a better quality final result. Cascaded decoders can also be used. First, a master decoder decodes the converged latent state features into a medium-resolution image. Then, one or more subsequent refinement decoders further enhance details and upscale the resolution based on the output of the previous decoder.

[0310] As an example, a converged latent state feature of 32x32x4 dimensions is input into a VAE decoder. After upsampling and convolution operations, it is finally decoded into a 512x512x3 RGB image, i.e., the target image.

[0311] This application embodiment obtains intermediate features by linearly mapping latent state features using a pre-trained image generation model; attention processing is then applied to the intermediate features based on text features and multi-view features to obtain attention features; the transmission velocity field of the latent state features at the current time step is predicted based on the attention features; numerical integration iterations are performed on the latent state features at multiple time steps based on the transmission velocity field until a preset time step is reached to obtain converged latent state features; the converged latent state features are then decoded to obtain the target image of the target object. The above scheme constructs a clear and efficient multimodal information fusion mechanism by decomposing the conditional injection process into two independent steps: linear mapping and attention processing. Its core advantage lies in the fact that it first maps the main generation object (latent state features) to a prepared, high-dimensional intermediate feature space through step 10641, and then, in step 10642, accurately injects external guiding information from different sources (text and multi-view) into the intermediate features through the powerful association mechanism of cross-attention. This method addresses the technical challenges of aligning different features, causing information interference, and failing to highlight key guiding information when fusing multimodal information simply by splicing or element-wise addition. This decoupled, attention-based fusion approach ensures that semantic (textual) and identity (multi-view) information can be effectively and accurately applied to the generation process, significantly improving the condition compliance and detail fidelity of the final generated image.

[0312] In summary, this embodiment of the application, by acquiring image generation prompts, multiple reference images of the target object from different perspectives, and preset camera viewpoint parameters, first performs 3D reconstruction based on the multiple reference images from different perspectives to obtain a 3D model of the target object, reference camera viewpoint parameters for each reference image, and viewpoint features. This step can accurately recover the global 3D structure of the object from a small number of images with unknown camera poses, providing a solid geometric foundation for subsequent generation. Subsequently, the 3D model is rendered according to the preset camera viewpoint parameters to obtain a target viewpoint rendering image. This rendering image provides a strong priori geometric shape and occlusion relationship for the final image generation, thereby fundamentally ensuring the consistency of the 3D structure of the generated image and effectively avoiding the morphological distortion problem commonly found in traditional 2D methods. At the same time, based on the relative pose parameters of the reference camera viewpoint parameters and the preset camera viewpoint parameters for each reference image, the viewpoint features of each reference image are fused to obtain multi-viewpoint features. These multi-viewpoint features serve as rich appearance and texture priors, effectively transferring the details most relevant to the target viewpoint in the reference images (such as material, lighting, and color) to the generation model, ensuring high fidelity of the target object's identity features. Finally, based on image generation prompts, target view renderings, and multi-view features, a pre-trained image generation model is invoked to generate a target image of the target object. Through this collaborative guidance of geometric and appearance priors, the system can focus on generating photorealistic and rich details while maintaining 3D consistency. This approach successfully integrates the structural accuracy of 3D reconstruction with the textural realism of 2D generation, achieving high-quality, high-fidelity new perspective image generation.

[0313] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0314] Image customization is a core capability of modern generative models, aiming to synthesize personalized images that meet the specific needs of a subject and scene based on user-provided reference images and natural language instructions. Image customization methods in related technologies are primarily based on two-dimensional conditions, utilizing a single reference image and text prompts as input. These methods maintain consistency in subject identity through methods such as two-dimensional feature alignment. However, due to a lack of explicit understanding of the underlying three-dimensional structure of objects, these methods often exhibit geometric drift and inconsistencies in identity features when the target perspective changes significantly, making them difficult to apply in scenes requiring precise three-dimensional structure and camera geometry. Another related technology is three-dimensional perceptual image synthesis, such as Neural Radiation Field (NeRF). These methods generate images from new perspectives by reconstructing the three-dimensional geometric representation of objects, ensuring excellent three-dimensional consistency. However, they typically require a large number of input images with precise camera pose information, placing stringent data requirements on them; simultaneously, the photorealistic quality and detail richness of the generated images often fall short of top-tier two-dimensional generative models, and they are difficult to flexibly combine with text instructions for creative scene editing. In summary, the relevant technologies face a core contradiction in achieving high-fidelity, controllable viewpoint object image customization: two-dimensional methods generate high-quality images but lack three-dimensional consistency, while three-dimensional methods can guarantee three-dimensional consistency but lack sufficient generation quality and application flexibility.

[0315] Image customization aims to synthesize a new image that retains the identity of a specific subject while conforming to the scene description, based on a user-provided reference image and natural language instructions. Currently, both open-source and closed-source models exist. The core technical solutions mainly include the following approaches: 2D condition-driven customization methods, which rely on a single-viewpoint reference image and feature alignment techniques (feature injection or feature stitching) to attempt to maintain the subject's identity in the output image. Instruction-based editing methods demonstrate powerful zero-shot customization capabilities, but typically only operate within a single-viewpoint system and lack explicit camera control interfaces.

[0316] In order to support multi-view input and explicitly control the camera viewpoint and preserve the 3D structure, this application introduces 3D perception customization. The generation methods most related to 3D perception customization include: 3D perception object manipulation: mainly focuses on editing the pose or viewpoint of existing visual elements in the image, rather than synthesizing a completely new scene context. New viewpoint synthesis: can generate consistent multi-view appearance maps from a small number of observations [6], but usually does not have the ability to process complex text instructions to change the scene context. Viewpoint controllable customization model: although it supports camera parameter adjustment, it is often limited by single image reference, making it difficult to capture the complete 3D structure of the subject, and the effect is not good. Optimization-based customization scheme: obtains 3D consistency by optimizing each specific scene, but the computational cost is high and the efficiency is extremely low, and the effect is also not good.

[0317] The related technologies suffer from the following technical problems: Lack of precise camera control: Most 2D editing models and image customization models do not provide explicit geometric interfaces, making it impossible to render based on the precise camera pose (such as azimuth and pitch) specified by the user. This results in unpredictable orientation and size of the generated objects. Inefficient integration of multi-view evidence: When users can provide multiple subject photos, the models in these technologies lack an effective mechanism to dynamically aggregate visual evidence distributed across different viewpoints, often leading to information redundancy or underutilization. Geometric drift and identity inconsistency: Camera-controlled image customization methods in these technologies, such as CustomNet, only support single-image input and do not introduce explicit 3D modeling and rendering processes. When users attempt to drastically change the generation viewpoint, the model struggles to infer the underlying 3D structure of the subject, resulting in geometric distortion and identity inconsistencies in the generated objects.

[0318] To address the aforementioned technical problems, this application proposes a geometry-guided generative rendering framework for 3D-aware image customization. The core idea is to complementaryly combine an explicit, 3D-consistent rendering process with a powerful, learning-driven generative model. Specifically, it includes the following steps: Geometric grounding prior construction: This process receives a multi-view set containing an arbitrary number of reference images, user-specified target camera viewpoint parameters, and a text instruction describing the target scene. Utilizing a pre-trained 3D perception model, such as a Visual Geometry Grounded Transformer (VGGT), it reconstructs an explicit 3D surrogate model of the target object (corresponding to the 3D model mentioned above) from the multi-view reference images and extracts geometrically aware multi-view features. Simultaneously, it estimates the camera pose parameters for each reference image. This fundamentally eliminates the geometric drift problem in viewpoint transformations and provides precise camera control.

[0319] Target view rendering cues generation: Using the 3D proxy model obtained in the previous step and the user-specified target camera view parameters, a rendering map of the target view (corresponding to the target view rendering map mentioned above) is generated through explicit geometric rendering (such as rasterization or ray detection). This rendering map (e.g., an RGB image or depth map of the target view) serves as a structured anchor point strictly aligned with the camera view, providing strong geometric constraints for subsequent generation processes.

[0320] Multi-view feature fusion: A lightweight "Mixture-of-Views (MoV)" module is introduced. Based on the relative pose parameters between the preset camera view parameters from the target camera's perspective and the reference camera view parameters from each reference camera's perspective, a geometrically determined routing score is calculated, and the Top-K reference views with the richest information are selected accordingly. Then, the features of these selected views (corresponding to the view features mentioned above) are weighted and fused to obtain multi-view features. This allows for the dynamic and sparse aggregation of the most relevant view features based on geometric relationships, significantly improving processing efficiency while maintaining identity consistency.

[0321] Generative rendering based on Rectified Flow: A Diffusion Transformer (DiT) based on Rectified Flow is used as the generator. The target viewpoint rendering image, the fused multi-viewpoint features, and the encoded text instructions (corresponding to the image generation prompts mentioned above) are input into the DiT model as conditions. The DiT model predicts a time-varying velocity field and uses an Ordinary Differential Equation (ODE) solver to numerically integrate from random noise, ultimately generating a target image that meets all conditions (viewpoint, identity, scene).

[0322] Through the above-described approach, this embodiment of the application collaboratively injects explicit 3D geometric constraints (i.e., 3D stereo model) and implicit appearance and texture constraints (through fused multi-view features) into a powerful generative model. This achieves precise control of the viewpoint while maintaining the 3D consistency and identity characteristics of the subject, generating high-quality, highly realistic customized images. It enables an efficient workflow that supports "customization upon shooting" without costly scene-by-scene optimization, and significantly outperforms existing commercial and open-source benchmarks in terms of structural integrity and 3D consistency metrics.

[0323] The image generation method provided in this application supports three-dimensional-aware image customization, which is an image generation task. Its input includes multi-view reference images of a specific subject, target camera viewpoint parameters (automatically calculated based on input horizontal azimuth, pitch, and camera distance parameters), and natural language scene editing instructions. The output is an image that conforms to the instructions and maintains the consistency of the subject's identity from a specified viewpoint (determined by the target camera parameters). This aims to solve the problems of lack of three-dimensional consistency in two-dimensional image customization methods and insufficient generation quality and flexibility in three-dimensional synthesis methods in related technologies. Specifically, this application aims to achieve a customization method that, with only an arbitrary number (or even a small number) of multi-view reference images, can generate high-quality images that maintain a high degree of consistency between the subject's identity and three-dimensional geometric structure, while possessing photorealistic qualities, following user-specified camera viewpoints and text instructions.

[0324] As an example, see Figure 14 , Figure 14 This is a first schematic diagram of image generation provided in an embodiment of this application. For example... Figure 14 As shown, the left image displays the backpack from five different perspectives, while the right image shows the backpack generated under different natural language scene editing commands. For example, Figure 5 The generated images on the right, from left to right, depict the following natural language scenes: 1) A blue-patterned backpack rests on a thick layer of powdery white snow in the cold morning light; 2) A blue-patterned backpack is placed on a dense patch of green moss at the base of an old tree in a misty forest; 3) A blue-patterned backpack is placed on ripples of fine sand in a vast desert, where the sun shines brightly and the sky is clear and azure; 4) A backpack presented as part of an impressionist oil painting, with visible thick brushstrokes and a textured canvas background. Specifically, for the images generated on the right, the camera view parameters are the same for images 1 and 2; the camera view parameters are the same for images 3 and 4, but the scenes in each of the four images are different. The camera view parameters are different for images 1 and 3.

[0325] Here, the camera view parameters (referring to camera extrinsic parameters) are represented by a 3*4 dimension matrix, used to accurately describe the coordinate transformation relationship of the camera relative to the subject in 3D space. This precisely defines the camera's current position, thereby indirectly defining the object's orientation and size. To improve the convenience of user input, parameter mapping based on a spherical coordinate system is supported.

[0326] As an example, see Figure 18 , Figure 18 This is a schematic diagram of the spherical coordinate system provided in an embodiment of this application. For example... Figure 18 As shown, the subject to be customized is preset to be located at the origin of the coordinate system, responding to three scalar parameters: the distance 'a' between the camera and the subject, the horizontal azimuth angle, and the input parameters. and pitch angle That is, the above spherical coordinates can be automatically calculated and converted into a standardized 3×4 extrinsic parameter matrix through a preset trigonometric function transformation formula.

[0327] This application proposes a geometry-guided generative rendering framework to address the problems of related image customization methods, which generally rely on a single image of the same identity, making it difficult to effectively utilize multi-view input, maintain cross-view 3D consistency, and lack precise camera pose control. A "shoot-to-customize" workflow is constructed: multi-view reference images of the user-input subject are obtained, target camera pose parameters (horizontal angle, pitch angle, distance, etc.) are specified, and scene instructions are input to generate high-quality customized images. The generated results exhibit stable performance in terms of subject detail fidelity, target view alignment accuracy, and scene semantic consistency, thus supporting controllable, multi-view consistent image customization. The core technology lies in combining explicit geometric constraints with powerful diffusion generation capabilities through "generative rendering." 1) Explicit geometric anchoring: Instead of letting the generative model learn perspective changes through implicit association, it first uses a 3D proxy model to render "rough" geometric cues (such as RGB images) in the target perspective as anchor points for generation.

[0328] 2) Dynamic view selection logic: The MoV module is based on the camera pose descriptor to realize the sparse aggregation of dynamic input multi-view features, which effectively solves the problem of redundant input multi-view information and promotes the 3D consistency of customized objects.

[0329] 3) Diffusion Generation: Text instructions, 3D models, and multi-view features are simultaneously injected into the DiT model to ensure that the synthesized result is precisely controlled in three dimensions: "what (identity)," "where (viewpoint)," and "environment (scene)." This application's embodiment uses paired data and the Rectified Flow paradigm to train a DiT model capable of achieving the above functions.

[0330] This application's embodiments utilize a flow-matching-based generative modeling method to construct a deterministic linear transport path between noise samples and target clean samples (referring to real data provided during training, such as a real image). It characterizes the continuous evolution of samples over time by learning a time-varying velocity field, thereby completing generation via Ordinary Differential Equation (ODE) integration. The correction flow is modeled using a deterministic flow of the explicit velocity field (noise-clean sample). In the algorithm implementation, the training phase constructs intermediate states based on time interpolation and regresses the target velocity; the inference phase initializes with noise and obtains generated samples through numerical integration along the learned velocity field.

[0331] This application implements a highly efficient customization process of "shooting and customizing", which is mainly reflected in the following application scenarios: by simply acquiring a few photos taken from different angles around a physical object, the object can be specified in any novel scene (such as "on the desert surface" or "on the grassland under the sunset") via text, and the viewing angle of the product can be precisely controlled.

[0332] As an example, see Figure 17 , Figure 17 This is a schematic diagram of image uploading provided in an embodiment of this application. For example... Figure 17 As shown, the top-level image displays five images of the uploaded shoes, each corresponding to a different perspective. Figure 17 The mid-view image shows the front view of the shoe. Figure 17 The image below shows the right side of the shoe.

[0333] To enable interactivity of the target viewpoint parameters, a virtual 3D sphere control panel can be provided, such as... Figure 19 As shown, Figure 19 This is a schematic diagram of the sphere control panel provided in an embodiment of this application. Figure 19 As shown, the three parameters "azimuth", "pitch" and "distance" are represented by different symbols. Assuming that the object is located in a spherical coordinate system, the user does not need to understand the complex 3×4 matrix. By simply adjusting the three parameters "azimuth", "pitch" and "distance", the camera's extrinsic parameters, i.e., the camera pose matrix, can be automatically calculated.

[0334] In some embodiments, after obtaining the uploaded reference image, the set target camera viewpoint parameters, and the text prompts controlling the scene context, a customized result, i.e., a target image, can be generated for the target object. The image generation method provided in this application embodiment can be applied to scenarios such as product display, creative asset creation, and IP generation.

[0335] As an example, see Figure 15 , Figure 15 This is a second schematic diagram of image generation provided in an embodiment of this application. For example... Figure 15 As shown, there are 4 different sets of images. The images between the groups (such as the images in the first group and the images in the second group) have different camera viewpoint parameters, and the images within the group (such as the left and right images in the first group) generate different prompt words.

[0336] The structure of the image generation method provided in the embodiments of this application will be described below. See also Figure 20 , Figure 20 This is a schematic diagram of the structure of the image generation method provided in the embodiments of this application. For example... Figure 20 As shown, firstly, reference images from multiple perspectives, target camera viewpoint parameters, and text instructions (i.e., image generation prompts) are acquired. The VCGT module performs 3D reconstruction on the images from multiple perspectives, obtaining a 3D geometric proxy of the object for rendering (corresponding to the 3D model mentioned above), viewpoint features corresponding to the images from multiple perspectives, and reference camera viewpoint parameters for each perspective. Next, the rendering module renders the 3D geometric proxy based on preset camera viewpoint parameters, generating a target viewpoint rendering image as a preliminary image cue conforming to the target viewpoint. This rendering image, along with initial noise, passes through an encoder and serves as basic input to the core generator (a DiT structure composed of multiple frozen bi-blocks and single-blocks). During generation, the output is precisely controlled through two parallel conditional streams: on one hand, text instructions are processed by a text encoder to provide semantic guidance for the generated content; on the other hand, viewpoint features extracted from multiple perspectives are fed into a trainable Mixture-of-Views (MoV) module. This module intelligently selects and fuses the most useful view information and then injects it into modules at different levels of the generator to ensure consistency between the subject identity and the 3D structure. Finally, the feature stream guided by multiple conditions (corresponding to the multi-view features mentioned above) is synthesized by the decoder to produce a high-quality customized image (corresponding to the target image mentioned above) that both follows the text instructions and conforms to the specified target camera viewpoint, while maintaining the 3D consistency of the subject. Meanwhile, in Figure 20 The code is marked with a freeze flag and a trainable flag. During the training process, only the module corresponding to the trainable flag needs to be trained.

[0337] In some embodiments, to efficiently process multi-view evidence, the Mixture-of-Views (MoV) module can calculate the relative pose descriptor between the target camera view parameters and the camera view parameters of each reference view. (Corresponding to the relative pose parameters mentioned above). Based on the relative pose descriptor After sinusoidal frequency embedding, a learnable lightweight MLP is used to calculate scores. Only the viewpoint features of the top k most relevant viewpoints (with the highest scores) are selected, and the viewpoint features of the top k viewpoints are weighted and fused using weights calculated by a normalization function to obtain multi-viewpoint features. This design is similar to an expert hybrid mechanism, which reduces computational redundancy and prevents irrelevant viewpoint features from contaminating the generation process. Finally, the weighted and fused multi-viewpoint features are injected into the DiT model through a cross-attention mechanism.

[0338] As an example, see Figure 21 , Figure 21 This is a schematic diagram of the structure of the hybrid view module provided in an embodiment of this application. Figure 21 As shown, firstly, the initial "hidden state" sequence representing the current generated target view (obtained by fusing the target view rendering image with noise features) is split into two paths after purification processing by the backbone DiT block. One path serves as the basis information directly to the final residual summation node, while the other path serves as the query vector for use by the top cross-attention layer. Simultaneously, in the reference feature injection branch, a lightweight router receives "camera tokens" representing the relative geometric pose differences between the target view and the reference set, and calculates the relevance score accordingly. Based on this, it performs sparse filtering of the top k view features, selecting only a few views with the closest spatial geometric relationship from the multi-view candidate pool. These are then assembled into concise and highly valuable "selected multi-view features" as keys and values ​​according to the calculated weights. Finally, the top-level cross-attention mechanism uses the target query vector to accurately retrieve these "routed" relevant reference features, extracts the detail increments that maintain appearance consistency, and seamlessly superimposes them back into the backbone feature stream through the final residual summation, thereby outputting a high-quality, feature-free, and fully updated hidden state to the next layer of the network.

[0339] In some embodiments, the importance score of the viewpoint feature corresponding to the reference image of each viewpoint can be determined by formula (1).

[0340] (1) in, These are relative pose parameters. It is a geometric feature vector after frequency encoding. Used to transform high-dimensional geometric features Mapped to a scalar value (score). It is a temperature hyperparameter used to scale the output distribution. The smaller the value, the steeper the distribution of the subsequent normalization results; The larger the value, the smoother the distribution. It is the first The original importance score of the viewpoint features corresponding to the reference images from each viewpoint.

[0341] Then, noise or irrelevant references can be filtered out using formula (2), retaining only the highest-scoring ones. Each score is then converted into a probability distribution (weights).

[0342] (2) in, This is the total number of reference images. It is the original importance score of all reference images. Select the largest value from the above set. The index corresponding to each score is used to reduce subsequent calculations and filter out irrelevant background. It is the set of indexes of the selected optimal viewpoint features. This is the normalized attention weight. It's a probability value between 0 and 1, and the sum of the weights of the k selected items is 1. It represents the attention weight of the k selected items. The actual contribution ratio of each reference item to the current goal.

[0343] Next, the selected perspective features can be scaled according to the weights calculated earlier using formula (3), and then spliced ​​together to form a huge context feature matrix, i.e., multi-view features.

[0344] (3) in, It is the first Viewpoint features of a reference image These are the weighted features. Using the scalar weights calculated earlier... Multiply by the characteristic matrix This achieves geometrically guided feature decay (features with smaller weights have a weaker impact on subsequent features). It is to select The weighted feature matrices are concatenated along the sequence dimension to obtain the multi-view features. . It is a multi-view feature The dimensional shape, where, It refers to the number of channels, which is the feature depth of each feature vector. It is the total length of the sequence.

[0345] Finally, using formula (4) and the cross-attention mechanism, based on multi-view features... The current target is hidden. Perform targeted information extraction and status updates.

[0346] (4) in, , , These are the three essential elements of the attention mechanism. Here, From hidden state Provide; and key (Key, ) and Value All of them have multi-perspective features supply. The output of the attention mechanism. It represents the feature increment or update amount extracted from multiple views / references. It's a residual connection. It incorporates the new information extracted through attention. Added to the target's original hidden state This process completes an iterative update of the target features. This design greatly enhances the stability of network training and the efficiency of gradient backpropagation.

[0347] In some embodiments, a corrected flow training framework can be employed for training on noisy samples. With target sample Construct linear transport paths between them and learn the rate field that changes over time. And using the loss function shown in formula (5), based on the rate field Update the parameters of the image generation model.

[0348] (5) in, It is the time step, a value randomly selected between 0 and 1, representing the time step from the noise. ( =0) to the real image ( A time point on a linear path (=1). These are intermediate state samples, essentially noisy images. and real images A linear mixture. When When it's close to 0, it looks more like noise; when... As it approaches 1, it looks more like a real image. This is the image of the "current location" provided to the model for prediction. It generates prompts based on the acquired images. Indicates starting from the origin Directly to the destination The "real" or "ideal" direction and speed. Because the path is a straight line, this speed remains constant throughout the entire path. This is the "standard answer" used to evaluate the model's prediction results. It is the square of the L2 norm, representing the rate of prediction by the model. Compared to the actual rate The difference between the predictions is zero if the model's predictions are perfectly accurate; the square of the L2 norm is also zero. The more inaccurate the predictions, the larger this value becomes. This value is the "penalty score" or "error" for a single prediction.

[0349] The image generation method provided in this application is compared with eight other related methods in terms of five preset performance metrics. These five metrics are: Frechet Inception Distance (FID), Kernel Inception Distance (KID), DINO Image Similarity (DINO-I), CLIP Image Similarity (CLIP-I), and CLIP Text Similarity (CLIP-T). FID is one of the most commonly used metrics for evaluating the performance of generative adversarial networks and other generative models. It measures the realism and diversity of the generated image by comparing the difference in feature distributions extracted by the Inception-v3 network between the generated image and the real image. A lower FID value indicates higher quality and closer relevance to the real image distribution. KID uses a kernel method to calculate the distance between feature distributions. Compared to FID, KID is considered more stable with small sample sizes and has better relevance to human perception. Similar to FID, it is also used to measure the quality of the generated image. A lower KID value indicates better generated image quality. DINO-I uses features extracted by DINO (a self-supervised learning model) to calculate the similarity between two images. DINO features are particularly sensitive to the structure and geometry of objects. Therefore, a high DINO-I value means that the subject in the generated image is highly consistent with the reference image in terms of contour, pose, and structure, making it ideal for evaluating identity preservation capabilities in 3D consistency tasks. A higher DINO-I value indicates better identity preservation. CLIP-I uses the CLIP (a powerful vision-language pre-trained model) image encoder to extract features and calculates the cosine similarity between the features of the generated image and the reference image. CLIP features focus more on semantic and appearance levels. A high CLIP-I value indicates that the generated image is similar to the reference image in terms of content, style, color, and texture. A higher CLIP-I value indicates better identity and appearance preservation. CLIP-T leverages the powerful cross-modal capabilities of the CLIP model to extract visual features from the generated image and text features from the input text instructions, and then calculates the cosine similarity between these two features. This score directly measures the extent to which the generated image follows the description of the text instructions. The higher the CLIP-T value, the better the alignment between the image and the text, meaning it better meets the instructions.

[0350] Table 1

[0351] As shown in Table 1, "FID" and "KID" are used to evaluate the visual fidelity of the generated image, with lower values ​​being better; "DINO-I" and "CLIP-I" are used to measure the consistency of the generated image with the reference image in terms of identity, with higher values ​​being better; "CLIP-T" evaluates the degree of conformity between the generated image and the text instruction, with higher values ​​being better. The data shows that although Method 3 achieved the lowest scores in FID (152.22) and KID (4.32), the method proposed in this application embodiment, while maintaining a competitive FID (157.03), achieved significantly better results in the core indicators for measuring identity preservation, DINO-I (71.64) and CLIP-I (84.39). This indicates that the image generation method provided in this application embodiment can most effectively maintain the consistency of the subject's identity from different perspectives while achieving high-quality image generation.

[0352] The image generation method provided in this application embodiment is compared with eight other related methods in terms of four preset three-dimensional geometric performance indicators. The four performance indicators are ChamferDistance (CD), Volume Intersection over Union (VIoU), Relative Rotation Error (RRE), and Camera Center Error (CCE).

[0353] Chamfer distance (CD) is a metric used to measure the difference between two point clouds. It calculates the average distance from each point in one point cloud to the nearest point in another, and vice versa, combining these two values ​​to evaluate the similarity between the two point clouds. In this evaluation, it was used to compare the geometric differences between the 3D point cloud reconstructed from the generated image and the real 3D point cloud. The lower the CD value, the more accurate the reconstructed 3D geometry and the better the 3D consistency. Volume intersection-union ratio (VIoU) is calculated by voxelizing the point clouds and then calculating the ratio of the intersection to the union of the reconstructed 3D model and the real 3D model in voxel space, thus evaluating the degree of volumetric overlap between the two. This is a commonly used metric to measure the overall structural integrity and accuracy of a 3D model. The higher the VIoU value, the greater the volumetric overlap between the reconstructed 3D model and the real model, and the better the geometric consistency. Relative rotation error (RRE) is used to evaluate the deviation between the orientation of an object in the generated image and the rotation angle specified by the target camera. It is calculated by comparing the angular difference between the camera rotation matrix estimated from the generated image and the target camera rotation matrix. A lower RRE value indicates a more precise alignment between the object's pose and the specified viewpoint in the generated image, resulting in greater controllability of the viewpoint. Camera Center Error (CCE) is used to evaluate the deviation between the object's position in the generated image and the specified position of the target camera. It calculates the Euclidean distance between the estimated camera center position from the generated image and the target camera center position. A lower CCE value indicates a more precise alignment between the object's spatial position and the specified viewpoint in the generated image, resulting in greater controllability of the viewpoint.

[0354] Table 2

[0355] As shown in Table 2, "CD" and "VIoU" are used to measure the geometric quality of the 3D model reconstructed from the multi-view generation results, while "RRE" and "CCE" evaluate the alignment accuracy between the subject pose and the target camera in the generated image. The data shows that the method proposed in this application achieves optimal performance across all four metrics. Specifically, this application achieves the lowest value of 0.0454 for the CD metric reflecting geometric consistency and the highest value of 0.0842 for the VIoU metric; simultaneously, it achieves the lowest value of 27.79 degrees for the RRE metric evaluating viewpoint control accuracy and the lowest value of 0.50 for the CCE metric. These data significantly outperform all other related methods, strongly demonstrating the significant technical advantages of the image generation method provided in achieving 3D geometric consistency and precise viewpoint control.

[0356] Finally, the image generation effect of the image generation method provided in this application embodiment is compared with that of five other related technologies (methods 1-5). See [link to documentation]. Figure 16 , Figure 16 This is a schematic diagram illustrating the effect of image generation provided in an embodiment of this application. For example... Figure 16 As shown, five reference images from different perspectives are provided for each object in three cases. The model is then required to generate customized images based on text instructions (e.g., "boots in the forest," "school bus on the road," "cake on the grass") from three new target perspectives. The bottom of the images shows a realistic rendering under the target perspective parameters, serving as a benchmark for evaluating the pose and contour accuracy of the generated results. Figure 16 As can be seen, Method 1 performed poorly in all cases, generating objects whose size and pose deviated significantly from the target, and even causing model crashes. While Methods 2 and 3 could generate images that matched the theme, the object poses deviated significantly from the realistic rendering, exhibiting viewpoint drift. For example, the boots generated by Method 3 faced completely differently from the realistic rendering. Methods 4 and 5 still exhibited instability. For instance, the boots generated by Method 4 were inconsistent in size, and the school bus background generated by Method 5 was rather monotonous, with objects appearing slightly floating from certain viewpoints.

[0357] In comparison, the image generation method provided in this application produces objects with the highest alignment to realistic renderings in terms of contour, pose, and spatial position. Whether it's the orientation of the boots, the side angle of the school bus, or the arrangement of the cake, all are closest to the realistic rendering baseline, demonstrating superior viewpoint control. The color, wear marks, and shoelace details of the boots, as well as the window layout of the school bus, maintain a high degree of consistency across the three images, thanks to its effective multi-view feature fusion mechanism.

[0358] In summary, Figure 16 This demonstrates intuitively that, compared to other advanced image customization models, the image generation model proposed in this application has significant advantages in handling 3D perception and viewpoint controllable generation tasks. It can more accurately follow the geometric constraints of the target camera, while better maintaining the consistency of the subject across viewpoints, resulting in geometrically more accurate and identity-coherent generated results.

[0359] Compared to 2D-driven methods in related technologies, the image generation method provided in this application supports multi-view input, significantly enhancing the 3D coherence of customized objects. Furthermore, this application provides an explicit camera extrinsic parameter interface, enabling precise control over the viewpoint of the generated image. Additionally, this application supports a "shoot-to-customize" workflow. Users only need to take a few photos of the product with their mobile phone to precisely embed the product into any marketing scenario (such as "snowy" or "beach at sunset") within seconds using the image generation method provided in this application. Due to its precise viewpoint control capability, the generated image perfectly aligns with the composition angle required for the advertising space, greatly reducing the material production cost for brand marketing.

[0360] The following continues to describe the exemplary structure of the image generation apparatus 455 provided in the embodiments of this application as a software module. In some embodiments, such as Figure 2 As shown, the software module stored in the image generation device 455 of the memory 450 may include: The acquisition module 4551 is used to acquire image generation prompts, reference images of the target object from multiple different perspectives, and preset camera perspective parameters.

[0361] The rendering module 4552 is used to perform 3D reconstruction based on multiple reference images from different perspectives to obtain a 3D model of the target object, the reference camera view parameters and view features of each reference image; and to render the 3D model according to the preset camera view parameters to obtain a target view rendering image.

[0362] The determination module 4553 is used to determine the relative pose parameters of the reference camera viewpoint parameters and the preset camera viewpoint parameters for each reference image.

[0363] The fusion module 4554 is used to fuse the viewpoint features of each reference image based on the relative pose parameters to obtain multi-viewpoint features.

[0364] The generation module 4555 is used to generate target images of target objects by calling a pre-trained image generation model based on image-generated prompts, target view rendering maps, and multi-view features.

[0365] In some embodiments, the acquisition module 4551 is further configured to acquire preset horizontal azimuth angle parameters, pitch angle parameters, and camera distance parameters for the target object; construct a preset three-dimensional spherical coordinate system with the centroid of the target object as the origin; map the horizontal azimuth angle parameters, pitch angle parameters, and camera distance parameters to the preset three-dimensional spherical coordinate system to obtain three-dimensional spatial coordinates; and use preset trigonometric function transformation mapping rules to convert the three-dimensional spatial coordinates into a camera extrinsic parameter matrix as preset camera viewpoint parameters.

[0366] In some embodiments, the rendering module 4552 is further configured to: extract features from multiple reference images from different perspectives to obtain initial image features for each reference image; perform cross-perspective feature interaction on the initial image features of multiple reference images to obtain perspective features for each reference image; perform spatial camera mapping on the perspective features of each reference image to obtain reference camera perspective parameters for each reference image; parse the perspective features of each reference image to obtain depth map information for each reference image; and generate a three-dimensional model of the target object based on the depth map information and reference camera perspective parameters of each reference image.

[0367] In some embodiments, the rendering module 4552 is further configured to perform the following processing for each reference image: dividing the reference image into multiple image blocks of a preset size; extracting features from each of the multiple image blocks to obtain image block features for each image block; concatenating the image block features of each image block to obtain an image block feature sequence; performing position encoding on the image block feature sequence to obtain position encoded features; and performing attention processing on the position encoded features to obtain initial image features of the reference image.

[0368] In some embodiments, the rendering module 4552 is further configured to perform spatial feature encoding on the initial image features of each reference image to obtain single-view enhanced features; determine a relevance weight matrix based on the single-view enhanced features of each reference image; perform weighted summation on the single-view enhanced features of each reference image based on the relevance weight matrix to obtain interaction features; split the interaction features according to the view dimension to obtain sub-interaction features of the reference image for each view; and sum the single-view enhanced features of each reference image and the sub-interaction features of each reference image as the view feature of each reference image.

[0369] In some embodiments, the rendering module 4552 is further configured to analyze the three-dimensional model to obtain the three-dimensional point cloud spatial distribution of the target object; determine the projection plane of the three-dimensional point cloud spatial distribution under the target view according to the preset camera view parameters; map each three-dimensional coordinate point in the three-dimensional point cloud spatial distribution to the two-dimensional coordinate system of the projection plane to obtain the two-dimensional coordinate point corresponding to each three-dimensional coordinate point; and perform rasterization processing on the two-dimensional coordinate points in the two-dimensional coordinate system to obtain the target view rendering map.

[0370] In some embodiments, the determining module 4553 is further configured to perform the following processing for each reference image: obtain a first pose matrix corresponding to the reference camera view parameters of the reference image, and a second pose matrix corresponding to the preset camera view parameters; determine the product of the inverse matrix of the first pose matrix and the second pose matrix as the relative transformation matrix of the reference image relative to the preset camera view parameters; extract rotation matrix components and translation vector components from the relative transformation matrix; and generate the relative pose parameters of the reference camera view parameters and the preset camera view parameters for each reference image based on the rotation matrix components and translation vector components.

[0371] In some embodiments, the fusion module 4554 is further configured to perform sine and cosine position encoding on the relative pose parameters to obtain pose features; determine the matching degree between the pose features and the preset camera view parameters; select a preset number of target reference images from multiple reference images based on the matching degree; and fuse the view features of each target reference image to obtain multi-view features.

[0372] In some embodiments, the fusion module 4554 is further configured to determine the original matching score between the pose features and the preset camera view parameters; generate random noise that follows a standard normal distribution; superimpose the random noise onto the original matching score to obtain a perturbed matching score; and normalize the perturbed matching score to obtain the matching degree between the pose features and the preset camera view parameters.

[0373] In some embodiments, the fusion module 4554 is further configured to normalize the matching degree corresponding to each target reference image to obtain the weight corresponding to each target reference image; and to perform weighted fusion of the viewpoint features of each target reference image according to the weight corresponding to each target reference image to obtain multi-viewpoint features.

[0374] In some embodiments, the fusion module 4554 is further configured to determine the exponential function value of the matching degree corresponding to each target reference image; sum the exponential function values ​​of each target reference image to obtain a normalized denominator; and use the quotient of the exponential function value of each target reference image and the normalized denominator as the weight corresponding to each target reference image.

[0375] In some embodiments, the generation module 4555 is further configured to encode the target view rendering image to obtain rendering features; concatenate the rendering features with preset Gaussian noise to obtain latent state features; encode the image generation prompt words to obtain text features; and, based on the latent state features, text features, and multi-view features, call a pre-trained image generation model to generate a target image of the target object.

[0376] In some embodiments, the generation module 4555 is further configured to call a pre-trained image generation model to perform linear mapping on the latent state features to obtain intermediate features; perform attention processing on the intermediate features based on text features and multi-view features to obtain attention features; predict the transmission velocity field of the latent state features at the current time step based on the attention features; perform numerical integration iteration on the latent state features for multiple time steps based on the transmission velocity field until a preset time step length is reached to obtain converged latent state features; and decode the converged latent state features to obtain the target image of the target object.

[0377] In some embodiments, the generation module 4555 is further configured to perform a first attention weighting operation based on intermediate features and multi-view features to obtain visual context features; perform a second attention weighting operation based on intermediate features and text features to obtain semantic context features; perform residual connection processing on visual context features, semantic context features and intermediate features to obtain residual features; and perform normalization processing on residual features to obtain attention features.

[0378] In some embodiments, the generation module 4555 is further configured to perform a first linear projection transformation on the intermediate features to obtain a first query vector matrix for the multi-view features; perform a second linear projection transformation on the multi-view features to obtain a first key-value vector matrix for the multi-view features; determine a first product of the transpose of the first query vector matrix and the first key-value vector matrix; normalize the first product to obtain a first attention weight distribution; and perform a weighted summation of the value vectors in the first key-value vector matrix according to the first attention weight distribution to obtain visual context features.

[0379] In some embodiments, the generation module 4555 is further configured to perform a third linear projection transformation on the intermediate features to obtain a second query vector matrix for the text features; perform a fourth linear projection transformation on the text features to obtain a second key-value vector matrix for the text features; determine a second product of the transpose of the second query vector matrix and the second key-value vector matrix; normalize the second product to obtain a second attention weight distribution; and perform a weighted summation of the value vectors in the second key-value vector matrix according to the second attention weight distribution to obtain semantic context features.

[0380] This application provides a computer program product, which includes a computer program or computer-executable instructions stored in a computer-readable storage medium. The processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the image generation method described in this application.

[0381] This application provides a computer-readable storage medium storing computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the processor will execute the image generation method provided in this application. For example, ... Figure 3 The image generation method is shown.

[0382] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0383] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0384] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0385] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0386] In summary, this embodiment of the application, by acquiring image generation prompts, multiple reference images of the target object from different perspectives, and preset camera viewpoint parameters, first performs 3D reconstruction based on the multiple reference images from different perspectives to obtain a 3D model of the target object, reference camera viewpoint parameters for each reference image, and viewpoint features. This step can accurately recover the global 3D structure of the object from a small number of images with unknown camera poses, providing a solid geometric foundation for subsequent generation. Subsequently, the 3D model is rendered according to the preset camera viewpoint parameters to obtain a target viewpoint rendering image. This rendering image provides a strong priori geometric shape and occlusion relationship for the final image generation, thereby fundamentally ensuring the consistency of the 3D structure of the generated image and effectively avoiding the morphological distortion problem commonly found in traditional 2D methods. At the same time, based on the relative pose parameters of the reference camera viewpoint parameters and the preset camera viewpoint parameters for each reference image, the viewpoint features of each reference image are fused to obtain multi-viewpoint features. These multi-viewpoint features serve as rich appearance and texture priors, effectively transferring the details most relevant to the target viewpoint in the reference images (such as material, lighting, and color) to the generation model, ensuring high fidelity of the target object's identity features. Finally, based on image generation prompts, target view renderings, and multi-view features, a pre-trained image generation model is invoked to generate a target image of the target object. Through this collaborative guidance of geometric and appearance priors, the system can focus on generating photorealistic and rich details while maintaining 3D consistency. This approach successfully integrates the structural accuracy of 3D reconstruction with the textural realism of 2D generation, achieving high-quality, high-fidelity new perspective image generation.

[0387] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. An image generation method, characterized in that, The method includes: Acquire image generation prompts, multiple reference images of the target object from different perspectives, and preset camera perspective parameters; Based on the multiple reference images from different perspectives, a three-dimensional reconstruction is performed to obtain a three-dimensional model of the target object, reference camera viewpoint parameters and viewpoint features for each reference image; The three-dimensional model is rendered according to the preset camera view parameters to obtain a target view rendering image; Determine the relative pose parameters of the reference camera viewpoint parameters and the preset camera viewpoint parameters for each reference image; Based on the relative pose parameters, the viewpoint features of each reference image are fused to obtain multi-viewpoint features; Based on the image generation prompts, the target view rendering, and the multi-view features, a pre-trained image generation model is invoked to generate a target image of the target object.

2. The method according to claim 1, characterized in that, The process of obtaining preset camera view parameters includes: Obtain preset horizontal azimuth angle parameters, pitch angle parameters, and camera distance parameters for the target object; Construct a preset three-dimensional spherical coordinate system with the centroid of the target object as the origin; The horizontal azimuth parameter, the pitch angle parameter, and the camera distance parameter are mapped to the preset three-dimensional spherical coordinate system to obtain three-dimensional spatial coordinates; Using a preset trigonometric function transformation mapping rule, the three-dimensional spatial coordinates are converted into a camera extrinsic parameter matrix, which is used as a preset camera viewpoint parameter.

3. The method according to claim 1, characterized in that, The 3D reconstruction based on the multiple reference images from different perspectives yields a 3D model of the target object, reference camera viewpoint parameters for each reference image, and viewpoint features, including: Feature extraction is performed on the multiple reference images from different perspectives to obtain the initial image features of each reference image; Cross-view feature interaction is performed on the initial image features of multiple reference images to obtain the view features of each reference image; Spatial camera mapping is performed on the viewpoint features of each reference image to obtain the reference camera viewpoint parameters of each reference image; The viewpoint features of each reference image are analyzed to obtain the depth map information of each reference image; A three-dimensional model of the target object is generated based on the depth map information of each reference image and the viewpoint parameters of the reference camera.

4. The method according to claim 3, characterized in that, The step of extracting features from the multiple reference images from different viewpoints to obtain initial image features for each reference image includes: For each of the reference images, the following processing is performed: The reference image is divided into multiple image blocks of a preset size; Feature extraction is performed on each of the multiple image blocks to obtain the image block features of each image block; The image block features of each image block are concatenated to obtain an image block feature sequence; The image block feature sequence is position-encoded to obtain position-encoded features; Attention processing is applied to the location-encoded features to obtain the initial image features of the reference image.

5. The method according to claim 3, characterized in that, The step of performing cross-view feature interaction on the initial image features of multiple reference images to obtain the viewpoint features of each reference image includes: Spatial feature encoding is performed on the initial image features of each of the reference images to obtain single-view enhanced features; A correlation weight matrix is ​​determined based on the single-view enhancement features of each of the reference images; Based on the relevance weight matrix, the single-view enhancement features of each reference image are weighted and summed to obtain the interaction features; The interaction features are split according to the view dimension to obtain the sub-interaction features of the reference image for each view; The sum of the single-view enhancement features of each reference image and the sub-interaction features of each reference image is used as the viewpoint feature of each reference image.

6. The method according to claim 1, characterized in that, The step of rendering the three-dimensional model according to the preset camera viewpoint parameters to obtain a target viewpoint rendering image includes: The three-dimensional model is analyzed to obtain the three-dimensional point cloud spatial distribution of the target object; Based on the preset camera viewpoint parameters, determine the projection plane of the three-dimensional point cloud spatial distribution under the target viewpoint; Each three-dimensional coordinate point in the spatial distribution of the three-dimensional point cloud is mapped to the two-dimensional coordinate system of the projection plane to obtain the two-dimensional coordinate point corresponding to each three-dimensional coordinate point; The two-dimensional coordinate points in the two-dimensional coordinate system are rasterized to obtain a target view rendering image.

7. The method according to claim 1, characterized in that, The step of determining the relative pose parameters of the reference camera viewpoint parameters and the preset camera viewpoint parameters for each reference image includes: For each of the reference images, the following processing is performed: Obtain the first pose matrix corresponding to the reference camera viewpoint parameters of the reference image, and the second pose matrix corresponding to the preset camera viewpoint parameters; The product of the inverse of the first pose matrix and the second pose matrix is ​​determined as the relative transformation matrix of the reference image relative to the preset camera viewpoint parameters. Extract the rotation matrix components and translation vector components from the relative transformation matrix; Based on the rotation matrix components and the translation vector components, the relative pose parameters of the reference camera viewpoint parameters and the preset camera viewpoint parameters for each reference image are generated.

8. The method according to claim 1, characterized in that, The step of fusing the viewpoint features of each reference image based on the relative pose parameters to obtain multi-viewpoint features includes: The relative pose parameters are coded by sine and cosine to obtain pose features; Determine the degree of matching between the pose features and the preset camera viewpoint parameters; Based on the matching degree, a predetermined number of target reference images are selected from the plurality of reference images; The viewpoint features of each target reference image are fused to obtain multi-viewpoint features.

9. The method according to claim 8, characterized in that, Determining the matching degree between the pose features and the preset camera viewpoint parameters includes: Determine the original matching score between the pose features and the preset camera viewpoint parameters; Generate random noise that follows a standard normal distribution; The random noise is superimposed on the original matching score to obtain the perturbed matching score; The perturbation matching score is normalized to obtain the matching degree between the pose feature and the preset camera viewpoint parameters.

10. The method according to claim 8, characterized in that, The process of fusing the viewpoint features of each target reference image to obtain multi-viewpoint features includes: The matching degree corresponding to each target reference image is normalized to obtain the weight corresponding to each target reference image; Based on the weight corresponding to each target reference image, the viewpoint features of each target reference image are weighted and fused to obtain multi-viewpoint features.

11. The method according to claim 10, characterized in that, The step of normalizing the matching degree corresponding to each target reference image to obtain the weight corresponding to each target reference image includes: Determine the exponential function value of the matching degree corresponding to each of the target reference images; The exponential function values ​​of each target reference image are summed to obtain a normalized denominator; The quotient of the exponential function value and the normalized denominator of each target reference image is used as the weight corresponding to each target reference image.

12. The method according to claim 1, characterized in that, The step of generating a target image of the target object by calling a pre-trained image generation model based on the image-generated prompt words, the target view rendering map, and the multi-view features includes: The target view rendering image is encoded to obtain rendering features; The rendering features are concatenated with preset Gaussian noise to obtain hidden state features; The image is used to generate prompts, which are then encoded to obtain text features. Based on the hidden state features, the text features, and the multi-view features, a pre-trained image generation model is invoked to generate a target image of the target object.

13. The method according to claim 12, characterized in that, The step of generating a target image of the target object by calling a pre-trained image generation model based on the latent state features, the text features, and the multi-view features includes: The hidden state features are linearly mapped by a pre-trained image generation model to obtain intermediate features. Based on the text features and the multi-view features, attention processing is performed on the intermediate features to obtain attention features; Based on the attention features, predict the transmission velocity field of the hidden state features at the current time step; The hidden state features are iterated by numerical integration over multiple time steps based on the transmission velocity field until a preset time step length is reached, thereby obtaining the converged hidden state features. The converged hidden state features are decoded to obtain the target image of the target object.

14. The method according to claim 13, characterized in that, The step of performing attention processing on the intermediate features based on the text features and the multi-view features to obtain attention features includes: A first attention weighting operation is performed based on the intermediate features and the multi-view features to obtain visual context features; A second attention weighting operation is performed based on the intermediate features and the text features to obtain semantic context features; The visual context features, the semantic context features, and the intermediate features are subjected to residual connection processing to obtain residual features; The residual features are normalized to obtain the attention features.

15. The method according to claim 14, characterized in that, The step of performing a first attention weighting operation based on the intermediate features and the multi-view features to obtain visual context features includes: A first linear projection transformation is performed on the intermediate features to obtain a first query vector matrix for the multi-view features; A second linear projection transformation is performed on the multi-view features to obtain a first key-value vector matrix for the multi-view features; Determine the first product of the transpose of the first query vector matrix and the first key value vector matrix; The first product is normalized to obtain the first attention weight distribution; Based on the first attention weight distribution, the value vectors in the first key-value vector matrix are weighted and summed to obtain the visual context features.

16. The method according to claim 14, characterized in that, The second attention weighting operation based on the intermediate features and the text features to obtain semantic context features includes: A third linear projection transformation is performed on the intermediate features to obtain a second query vector matrix for the text features; A fourth linear projection transformation is performed on the text features to obtain a second key-value vector matrix for the text features; Determine the second product of the second query vector matrix and the transpose of the second key value vector matrix; The second product is normalized to obtain the second attention weight distribution; Based on the second attention weight distribution, the value vectors in the second key-value vector matrix are weighted and summed to obtain semantic context features.

17. An image generation apparatus, characterized in that, The device includes: The acquisition module is used to acquire image generation prompts, reference images of the target object from multiple different perspectives, and preset camera perspective parameters; The rendering module is used to perform 3D reconstruction based on the multiple reference images from different perspectives to obtain a 3D model of the target object, reference camera view parameters and view features of each reference image; and to render the 3D model according to the preset camera view parameters to obtain a target view rendering image. The determining module is used to determine the relative pose parameters of the reference camera viewpoint parameters and the preset camera viewpoint parameters for each reference image; The fusion module is used to fuse the viewpoint features of each reference image according to the relative pose parameters to obtain multi-viewpoint features; The generation module is used to generate a target image of the target object by calling a pre-trained image generation model based on the image, the target view rendering map, and the multi-view features.

18. An electronic device, characterized in that, The electronic device includes: Memory is used to store executable instructions or computer programs. A processor, configured to execute computer-executable instructions or computer programs stored in the memory, implements the image generation method according to any one of claims 1 to 16.

19. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, the image generation method according to any one of claims 1 to 16 is implemented.

20. A computer program product comprising computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, the image generation method according to any one of claims 1 to 16 is implemented.

Images