A controllable 3D graphics content generation method and system based on multi-image fusion

Abstract

This invention discloses a controllable 3D graphic content generation method and system based on multi-image fusion, comprising: receiving multiple input images for controlling 3D generation; analyzing the correlation between the multiple input images and the 3D spatial representation using a matching algorithm to obtain a feature sequence encoded by each input image; constructing a hybrid conditional feature sequence containing comprehensive semantics of multiple images based on the feature sequences using a fusion algorithm; inputting the hybrid conditional feature sequence into a generation model, wherein an enhancement mechanism is introduced to adaptively adjust the feature correlation to obtain a fused and enhanced conditional feature sequence; inputting the fused and enhanced conditional feature sequence into a 3D generation decoder to output the final 3D graphic content. This invention enables flexible control from the whole to the part under multi-image conditions, and based on the automatic alignment capability from 2D to 3D, significantly improves generation efficiency while ensuring generation quality.

Description

Technical Field

[0001] This invention belongs to the field of computer graphics technology, specifically relating to a method and system for generating controllable 3D graphic content based on multi-image fusion. Background Technology

[0002] In recent years, with the development of deep learning, 3D graphics content generation technology has made significant progress in the fields of computer graphics and artificial intelligence. Traditional manual modeling methods often rely on professionals to perform complex geometric modeling and texture creation, which is time-consuming and labor-intensive. In contrast, neural network-based automated generation methods can directly synthesize 3D objects from external conditions, greatly lowering the threshold for content production. Against this backdrop, a series of end-to-end 3D native generation models have been proposed. For example, methods such as LRM, CLAY, and TRELLIS, by modeling in a 3D latent space, can directly output intermediate representations such as sparse voxels, latent fields, or Gaussian points during the inference stage, and then decode them into renderable meshes or volume rendering results. These methods excel in geometric consistency and cross-view stability, and have a fast inference speed, typically completing a 3D generation within tens of seconds, making them an important research direction in current 3D generation.

[0003] While native 3D generative models have made progress in efficiency and stability, they generally only support a single input condition, such as a reference image or a text prompt. In practical applications, users often need to use multiple images simultaneously to control different parts of a 3D object, for example, when multiple different reference images are needed to control different regions of a 3D object. However, existing methods lack this multi-condition joint driving capability, resulting in significant shortcomings in the combination of global and local inputs. Furthermore, these methods have not established fine-grained local control mechanisms, making it difficult to achieve independent editing and flexible manipulation of local regions.

[0004] To enhance controllability, some studies have proposed controllable generation frameworks based on the Score Distillation Sampling (SDS) method. For example, Progressive3D and A3D methods use a pre-trained 2D diffusion model as a prior and introduce local region constraints during optimization, thereby enabling local modifications to the 3D model. The advantage of these methods is that they can achieve a certain degree of fine-grained control, allowing users to adjust and optimize specific regions. However, these optimization-based frameworks have significant drawbacks: First, the generation efficiency is extremely low, requiring several minutes or even hours of iterative optimization for each local edit or overall generation, making it difficult to meet the needs of real-time or interactive applications; second, SDS methods typically rely on manual selection of editing regions in 3D space, which is complex and lacks automation capabilities; finally, due to the instability of the optimization process, these methods still suffer from insufficient cross-view consistency, easily leading to structural distortion or texture misalignment.

[0005] Another related category of work attempts to achieve 3D generation through forward inference. For example, SDFusion uses the signed distance function (SDF) as the generation space and introduces multimodal conditions such as images, text, or point clouds for control; MaskedLRM proposes a local editing mechanism based on a large-scale reconstruction model, enabling forward inference-based geometric modifications under single-image conditions. The advantage of these methods is their high inference efficiency, no longer relying on time-consuming iterative optimization, and the ability to complete generation in a short time. However, these methods also have significant limitations. First, they are mainly geared towards the generation of geometric shapes, lacking complete modeling and control of appearance information such as texture and material, and the generated results often lack visual detail. Second, they generally cannot handle joint input from multiple images, making it difficult to generate multi-region, multi-style fused 3D objects under complex conditions. Finally, due to the lack of an effective 2D-to-3D alignment mechanism, local regions selected by the user in a 2D image often cannot be accurately mapped to corresponding parts in 3D space, requiring manual selection of the corresponding 3D region.

[0006] Overall, existing 3D graphics content generation technologies still have significant shortcomings in terms of efficiency, controllability, and multi-condition utilization. While controllable generation methods based on score distillation sampling can achieve some local control, their reliance on long-term optimization makes them insufficient for high-efficiency generation. Forward inference methods offer speed advantages but are limited to geometric generation, lacking appearance control and unable to simultaneously utilize multiple images to drive different local regions. As for native 3D generation models like TRELLIS, while exhibiting excellent inference efficiency and cross-view consistency, they can only accept single input conditions and lack the ability to fuse multiple conditions and achieve fine-grained local control. Therefore, existing technologies cannot simultaneously achieve high efficiency, accuracy, and flexibility, failing to meet the comprehensive demands of multi-image control, local editing, and high-quality appearance generation in practical applications. There is an urgent need for a new 3D graphics content generation scheme that can effectively fuse multiple conditional images within a unified framework, achieving flexible control from the overall to the local level, and possessing automated 2D-to-3D alignment and feature fusion capabilities to generate high-quality 3D graphics content that better meets user expectations. Summary of the Invention

[0007] In view of the above, the purpose of this invention is to provide a controllable three-dimensional graphic content generation method and system based on multi-image fusion, so as to achieve flexible control from the whole to the part under multi-image conditions, and have an automated two-dimensional to three-dimensional alignment mechanism, so as to significantly improve the generation efficiency of three-dimensional graphic content while ensuring the generation quality.

[0008] To achieve the above-mentioned objectives, the present invention provides the following technical solution:

[0009] In a first aspect, the present invention provides a controllable 3D graphics content generation method based on multi-image fusion, comprising the following steps:

[0010] Receive multiple input images for controlling 3D generation;

[0011] The correlation between the multiple input images and the three-dimensional spatial representation is analyzed by a matching algorithm to obtain the feature sequence of each input image after encoding.

[0012] Based on the feature sequence, a hybrid conditional feature sequence containing comprehensive semantics of multiple images is constructed using a fusion algorithm;

[0013] The hybrid conditional feature sequence is input into the generation model, where an enhancement mechanism is introduced to adaptively adjust the feature association relationship in order to obtain the fused and enhanced conditional feature sequence.

[0014] The fused and enhanced conditional feature sequence is input into the 3D generation decoder, which outputs the final 3D graphic content.

[0015] Preferably, after receiving multiple input images, they are divided into global control images and local control images. A three-dimensional voxel set is generated based on the global control images, and the user defines the mask region of the local control images to mark the details or style parts that need to be controlled in three-dimensional space, thus obtaining a masked local control image.

[0016] Preferably, the step of analyzing the correlation between the multiple input images and the three-dimensional spatial representation using a matching algorithm to obtain the feature sequence encoded for each input image includes:

[0017] A forward matching process is performed between a 3D voxel set and a masked local control image: Feature sequences and local index sets of the masked regions are extracted from each local control image; the correlation between each voxel in the 3D voxel set and the feature maps of the local control images is calculated; the correlations are aggregated based on the local index sets to obtain the forward matching strength of each voxel; and the covered voxel set that is semantically aligned with the masked region is selected based on the forward matching strength. Specifically, the mask is first downsampled to the feature maps of the local control images. Figure 1 To achieve consistent resolution, feature sequence indices belonging to the mask region are matched from each local control image based on the downsampled mask, and a set of local indexes and encoded feature sequences of the local control image are constructed. The correlation between each voxel in the three-dimensional voxel set and the feature map of the local control image is calculated through cross-modal attention. The correlation is aggregated based on the local index set to obtain the positive matching strength between each voxel and the mask region. Voxels with positive matching strength greater than or equal to the positive matching threshold are constructed into a set of covered voxels that are semantically aligned with the mask region.

[0018] Back-matching is performed based on a 3D voxel set and the global control image: The correlation between each voxel in the remaining uncovered voxel set and the feature map of the global control image is calculated, and the correlations are aggregated to obtain the back-matching strength. Based on the back-matching strength, feature sequences and a global index set are extracted from the global control image. Specifically, for the global control image, the difference between the 3D voxel set and the covered voxel set is taken as the uncovered voxel set. Cross-modal attention is used to calculate the correlation between each voxel in the 3D voxel set and the feature map of the global control image. Only the correlations between each voxel corresponding to the uncovered voxel set and the feature map of the global control image are aggregated to obtain the back-matching strength. The feature maps of the global control image corresponding to voxels with back-matching strength greater than or equal to the back-matching threshold are selected. Based on the feature sequence index of the feature maps, the feature sequence of the global control image and its global index set are constructed.

[0019] Preferably, the step of constructing a hybrid conditional feature sequence containing comprehensive semantics of multiple images based on the feature sequence using a fusion algorithm includes:

[0020] Based on the local index set, the corresponding feature sequence is extracted from the feature map of the local control image of each mask. Based on the global index set, the corresponding feature sequence is extracted from the feature map of the global control image. The feature sequences of all the extracted images are concatenated to incorporate the global and local semantic information into a unified cross-modal attention computing framework, forming a hybrid conditional feature sequence containing the comprehensive semantics of multiple images.

[0021] Preferably, the step of inputting the hybrid conditional feature sequence into the generation model, wherein an enhancement mechanism is introduced to adaptively adjust the feature association relationship to obtain the fused and enhanced conditional feature sequence, includes:

[0022] The hybrid conditional feature sequence, along with local and global index sets, is input into the generative model. Based on the local and global index sets, the voxel regions affected by each image feature sequence corresponding to the hybrid conditional feature sequence in 3D space are determined. An enhancement matrix is ​​constructed to assign enhancement intensity parameters to the corresponding voxel regions of the local and global control images. The enhancement matrix is ​​applied to the cross-modal attention map during the generative model's inference process. The attention is weighted by element-wise multiplication, enabling the generative model to output the fused and enhanced conditional feature sequence.

[0023] Preferably, during the generative model inference process, a smooth transition strategy is introduced at the region boundary, so that the enhancement intensity parameter corresponding to the local control image is set according to the ratio of the overall image area to the mask area selected by the user, ensuring that small areas are more controlled, while large areas maintain a smooth transition.

[0024] Preferably, the step of inputting the fused and enhanced conditional feature sequence into the 3D generation decoder and outputting the final 3D graphic content includes:

[0025] The enhanced conditional feature sequence is fused and decoded with the positions of each voxel in the three-dimensional voxel set using a three-dimensional generative decoder to generate three-dimensional graphic content including triangular meshes, neural radiation fields, or three-dimensional Gaussian point representations.

[0026] Secondly, embodiments of the present invention also provide a controllable three-dimensional graphic content generation system based on multi-image fusion, which is implemented using the above-mentioned controllable three-dimensional graphic content generation method based on multi-image fusion, including: a multi-image input module, a three-dimensional semantic perception alignment module, a multi-condition feature fusion module, a local attention enhancement module, and a three-dimensional generation decoding module.

[0027] The multi-image input module is used to receive multiple input images for controlling 3D generation;

[0028] The three-dimensional semantic perception alignment module is used to analyze the correlation between the multiple input images and the three-dimensional spatial representation through a matching algorithm to obtain the feature sequence encoded by each input image;

[0029] The multi-condition feature fusion module is used to construct a hybrid condition feature sequence containing comprehensive semantics of multiple images based on the feature sequence through a fusion algorithm;

[0030] The local attention enhancement module is used to input the hybrid conditional feature sequence into the generation model, wherein an enhancement mechanism is introduced to adaptively adjust the feature association relationship in order to obtain the fused and enhanced conditional feature sequence.

[0031] The 3D generation and decoding module is used to input the fused and enhanced conditional feature sequence into the 3D generation and decoding module and output the final 3D graphic content.

[0032] Thirdly, embodiments of the present invention also provide an electronic device, including a memory and one or more processors, wherein the memory is used to store a computer program, and the processor is used to implement the above-described method for generating controllable three-dimensional graphics content based on multi-image fusion when executing the computer program.

[0033] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a computer, implements the above-described method for generating controllable three-dimensional graphic content based on multi-image fusion.

[0034] Compared with the prior art, the beneficial effects of the present invention include at least the following:

[0035] This invention utilizes a 3D semantic-aware alignment mechanism and a multi-condition feature fusion mechanism to fuse multiple conditional images within a unified framework, achieving refined control from the overall to the local. It can automatically complete precise semantic alignment from 2D to 3D and stable multi-condition fusion, significantly improving the controllability and ease of use of 3D generation. Simultaneously, a local attention enhancement mechanism effectively resolves the problem of multi-region feature conflicts, ensuring visual consistency and detail quality of the generated results. The entire process is highly integrated; with a typical input of three conditional images, the average generation time is less than 20 seconds, and it can output complete 3D graphic content containing both geometric and appearance information, demonstrating high efficiency and practicality. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 This is a flowchart illustrating the controllable 3D graphic content generation method based on multi-image fusion provided in an embodiment of the present invention.

[0038] Figure 2 This is a schematic diagram of the structure of a controllable 3D graphics content generation system based on multi-image fusion provided in an embodiment of the present invention. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.

[0040] While current controllable generation methods based on score distillation sampling can achieve a certain degree of local control, they suffer from low generation efficiency and rely on manual selection of 3D regions. Forward inference methods, although faster, can only generate geometric structures, lacking texture and material modeling, and cannot support joint control of multiple images. Therefore, the inventive concept of this invention is to address the shortcomings of existing 3D graphics content generation methods in terms of efficiency, controllability, and multi-condition utilization. This invention provides a controllable 3D graphics content generation method and system based on multi-image fusion. First, a cross-modal attention mechanism combining forward and reverse approaches achieves 3D semantic perception alignment, automatically establishing a high-precision semantic correspondence from 2D to 3D. Second, a multi-condition feature fusion mechanism integrates the global image and multiple masked local images, achieving multi-level control from the overall to the detailed. Subsequently, to address multi-condition conflicts, a local attention enhancement mechanism is used to introduce an adjustable enhancement matrix during the inference process, precisely controlling the intensity and range of each condition's influence. Finally, 3D generation decoding is performed based on refined conditional features, generating complete 3D graphics content in a single step. This method combines high efficiency, controllability, and the flexibility and comprehensiveness of multi-condition utilization.

[0041] like Figure 1 As shown in the figure, the embodiment provides a controllable 3D graphics content generation method based on multi-image fusion, including the following steps:

[0042] S1 receives multiple input images used to control the generation of 3D images.

[0043] This step is used to receive multiple conditional image inputs, including a global control image and several local control images. The global control image is used to determine the overall shape and structure of the 3D object, while the local control images provide local details or style features through user-specified selection areas.

[0044] In this embodiment, the input includes a global control image. and Zhang Local Control Image The user controls each local image. Provide binary mask The length that needs to be controlled in the annotation Width Size of the region. Global control image. A geometry generator using a pre-trained latent diffusion model Predicting a three-dimensional voxel set:

[0045] ,

[0046] in, This represents a three-dimensional set of voxels whose positions are determined by the global control image, serving as a three-dimensional spatial reference for subsequent semantic alignment of forward and reverse matching. Indicates the first Individual location, express The total number.

[0047] S2, the correlation between the multiple input images and the three-dimensional spatial representation is analyzed by the matching algorithm to obtain the feature sequence after encoding each input image.

[0048] This step establishes an automated semantic correspondence between a 2D input image and a 3D voxel space. First, a cross-modal attention mechanism is used to calculate the correlation between 2D conditional features and 3D voxels. Based on the attention distribution, local image regions are mapped to the best-matching 3D positions, completing forward alignment. For voxels not covered by local features, the system supplements them using global image features and uses the cross-modal attention mechanism to find corresponding contextual semantics in the 2D image, thus achieving comprehensive semantic coverage and consistency. By combining forward and reverse alignment, the tedious manual annotation or selection of 3D regions required in traditional methods is avoided, enabling 2D regions to be efficiently and accurately mapped to 3D space.

[0049] S2.1, 3D semantic-aware alignment (forward matching).

[0050] After obtaining the three-dimensional voxel set generated from the global control image Next, semantic alignment for forward matching is performed using local control images. The local control images are mask regions provided by the user. Clearly define the details or style elements you wish to control in 3D space. To map the 2D mask onto the patch space of the local control image, first downsample the mask to match the local control image. Encoded feature map Consistent resolution The downsampled mask is obtained. Therefore, the set of effective feature sequence indices for local control of the image can be defined as follows:

[0051] ,

[0052] in, Represents a local index set. Indicates the patch index for linear expansion. and These correspond to the row and column positions in the two-dimensional mask, respectively. Through this process, the feature sequence indexes belonging to the masked region can be accurately selected from each local image, and the feature sequence after encoding the local control image can be obtained, providing effective input for subsequent positive cross-modal attention.

[0053] To establish the 2D-to-3D correspondence, an appearance generator based on a pre-trained latent diffusion model is used. The internal cross-modal attention mechanism yields the corresponding attention map. Specifically, the three-dimensional voxel set... and the corresponding local control image Feature map Input to After entering the network, it will first go through the forward network. The three-dimensional voxel set and local control images Feature map Mapping to query vector and key-value pairs :

[0054] ,

[0055] Wherein, the index of the query vector Index of voxel position Correspondingly, the index of the key-value pair With local index set Index in correspond.

[0056] Next, attention calculations are used to obtain each voxel. With local control image Feature map Correlation between :

[0057] ,

[0058] Among them, superscript Indicates transpose. Indicates the feature dimension.

[0059] Subsequently, based on the index set of the local mask selection area Calculate the positive matching strength between each voxel and the region. :

[0060] ,

[0061] Set the positive matching threshold ,like Then it is considered that voxels Semantically aligned with local regions:

[0062] ,

[0063] in, Indicates the first A set of covered voxels is used for subsequent reverse matching.

[0064] This process allows user-specified two-dimensional regions to be accurately projected into three-dimensional space, enabling directional control over these local features during subsequent generation. The technical benefit of this step is that it transforms the user's intuitive two-dimensional input into explicit three-dimensional voxel-level constraints, thereby significantly reducing the cost of manually selecting three-dimensional regions.

[0065] S2.2, 3D semantic-aware alignment (reverse matching).

[0066] After completing the positive matching of a local region, some voxels may still remain uncovered. Let the set of uncovered voxels be denoted as . :

[0067] ,

[0068] in, This represents the total number of voxels that have been covered. The difference operation of sets.

[0069] To ensure that all selected feature maps cover all voxel regions, the appearance generator of the pre-trained latent diffusion model is used again. The internal cross-modal attention mechanism yields the corresponding attention map. Specifically, the three-dimensional voxel set... and the corresponding global control image Corresponding feature map Input to After entering the network, it will first go through the forward network. The three-dimensional voxel set and global control image Corresponding feature map Mapping to query vector and key-value pairs :

[0070] ,

[0071] Wherein, the index of the query vector Index of voxel position Correspondingly, the index of the key-value pair Global index set with global control image Index in correspond.

[0072] Next, attention calculations are used to obtain each voxel. With global control image Feature map Correlation between :

[0073] ,

[0074] Among them, superscript Indicates transpose. Indicates the feature dimension.

[0075] Subsequently, after obtaining the corresponding correlations, only the unmatched voxels were processed. Aggregation is performed to obtain the feature map of the global control image for these unmatched voxels. Reverse matching strength:

[0076] ,

[0077] By setting a reverse matching threshold The most significant portion of the encoded feature sequence of the global control image is selected and used to form a global index set. :

[0078] ,

[0079] in, This represents the feature sequence index in the feature map of the global control image.

[0080] S3. Based on the feature sequence, a hybrid conditional feature sequence containing comprehensive semantics of multiple images is constructed using a fusion algorithm.

[0081] This step unifies global features and multiple local features into a conditional representation through a feature fusion mechanism. While maintaining overall contextual consistency, it preserves the independent representation of multiple local regions, thus providing finely controlled input conditions for subsequent 3D generation. Compared to methods that rely on only a single image, this approach can fully utilize multi-source image information to achieve multi-level control from the global to the local level.

[0082] In this embodiment, the index sets obtained from the forward and reverse matching are unified with the corresponding image feature sequences into the same conditional space. Specifically, for the index sets of the local control images obtained in steps S2.1 and S2.2... With the set of indices of the global control image , respectively corresponding to the positive region and uncovered areas By concatenating tokens from these feature sequences from different sources and discarding all tokens not in the index set, a complete hybrid conditional feature sequence containing comprehensive semantics from multiple images is formed. Specifically, it is obtained through the following formula:

[0083] ,

[0084] in, Represents local control image Corresponding feature map The The feature sequence corresponding to each index Represents global control image Corresponding feature map The The feature sequence corresponding to each index This indicates a concatenation operation. The purpose of this is to incorporate both global and local semantic information into a unified cross-modal attention computation framework, enabling subsequent steps to simultaneously consider both fine-grained local constraints and global consistency within a holistic attention graph, thereby achieving the organic fusion of multi-conditional information.

[0085] S4, the hybrid conditional feature sequence is input into the generation model, wherein an enhancement mechanism is introduced to adaptively adjust the feature association relationship in order to obtain the fused and enhanced conditional feature sequence.

[0086] This step aims to improve the stability and detail fidelity of fusion when multiple local images may have conflicting conditions. When different local regions are adjacent or partially overlap in 3D space, feature interference and visual inconsistencies can easily occur. To address this, a region enhancement mechanism is introduced during the cross-modal attention process. This mechanism assigns higher attention weights to the voxels corresponding to the target region while suppressing interference signals from non-target regions, making features more focused on the user-specified local region. Furthermore, a smooth transition strategy is introduced at multiple region boundaries to ensure natural connections between features, effectively avoiding boundary blurring and local feature overlap. This allows multiple local conditions to maintain independent expression while achieving harmonious overall fusion.

[0087] In the embodiment, the mixed conditional feature sequence is obtained through the above steps. Afterwards, and three-dimensional voxel set Input pre-trained latent diffusion model appearance generator This yields conditional feature outputs, but the results may show a mixture of features from multiple images. Therefore, to control the influence range of each conditional image in three-dimensional space, this invention... An enhancement matrix is ​​introduced into the internal cross-modal attention mechanism. , acting on Attention generated during reasoning Above.

[0088] For each local control image Assign a strength enhancement parameter This is used to adjust its contribution to 3D generation. Specifically, for the ... Zhang's local control image, its feature map index set is: The corresponding three-dimensional voxel set is Then for all , Position assignment:

[0089] ,

[0090] Similarly, for global control images Using its feature map index set With uncovered voxel set For all , Assignment:

[0091] ,

[0092] The remaining uncovered elements remain as Ultimately, the enhanced attention is:

[0093] ,

[0094] in, This represents element-wise multiplication. Subsequently, enhanced attention is used. Replace the original Internal attention Furthermore, all other values ​​are retained from the pre-trained model's inference process. Finally, the fused and enhanced conditional feature sequence is obtained by inference with the trained model. .

[0095] This mechanism can adjust the control intensity of each conditional image on its corresponding 3D region: smaller Values ​​promote smooth integration between regions, and larger values... This value strengthens local independence, achieving more decoupled control. By default, each region... The ratio of the overall image area to the area of ​​the mask selected by the user is used to ensure stronger control over small areas while maintaining a smooth transition in large areas.

[0096] S5 inputs the fused and enhanced conditional feature sequence into the 3D generation decoder and outputs the final 3D graphic content.

[0097] This step generates complete 3D image content guided by fusion conditions and alignment mechanisms. Based on a structured 3D latent representation, fused and aligned conditional features are injected into the generative model to generate the corresponding 3D shape and appearance. The decoded 3D graphic content can be output in various forms, including triangular meshes, implicit volume rendering representations, or Gaussian point clouds, to meet the needs of different application scenarios. It can not only generate geometric structures but also synthesize corresponding textures and material information, thereby outputting high-quality 3D objects with complete appearances.

[0098] In this embodiment, the conditional features generated in step S4 are... and corresponding voxel positions Input them together into the 3D generation and decoding module:

[0099] ,

[0100] Among them, decoder Three formats can be used, each corresponding to a different 3D graphic content. The system employs triangular meshes, neural radiation fields (NeRF), and 3D Gaussian points (GS) to meet diverse application requirements. The final generated 3D result maintains a high degree of consistency with the input image in both overall shape and detailed appearance, naturally integrating local features from different images within a unified 3D space, achieving meticulous control from global structure to local style.

[0101] In summary, the controllable 3D graphic content generation method based on multi-image fusion provided by this invention achieves flexible control from the overall to the local level under multi-image conditions through the synergistic effect of each step. Unlike existing solutions that rely on manual editing or single conditions, the framework provided by this invention can achieve automated 2D to 3D alignment and stable multi-condition fusion while ensuring high efficiency. The generated 3D graphic content can better meet the user's expectations in terms of both geometry and appearance.

[0102] Based on the same inventive concept, such as Figure 2 As shown, this embodiment of the invention also provides a controllable three-dimensional graphic content generation system 200 based on multi-image fusion, including: a multi-image input module 210, a three-dimensional semantic perception alignment module 220, a multi-condition feature fusion module 230, a local attention enhancement module 240, and a three-dimensional generation decoding module 250.

[0103] The multi-image input module 210 is used to receive multiple input images for controlling the 3D generation.

[0104] The 3D semantic perception alignment module 220 is used to analyze the correlation between the multiple input images and the 3D spatial representation through a matching algorithm to obtain the feature sequence encoded by each input image.

[0105] The multi-condition feature fusion module 230 is used to construct a hybrid condition feature sequence containing comprehensive semantics of multiple images based on the feature sequence through a fusion algorithm.

[0106] The local attention enhancement module 240 is used to input the hybrid conditional feature sequence into the generation model, wherein an enhancement mechanism is introduced to adaptively adjust the feature association relationship to obtain the fused and enhanced conditional feature sequence.

[0107] The 3D generation and decoding module 250 is used to input the fused and enhanced conditional feature sequence into the 3D generation and decoding module and output the final 3D graphic content.

[0108] Based on the same inventive concept, embodiments of the present invention also provide an electronic device, including a memory and one or more processors, wherein the memory is used to store a computer program, and the processor is used to implement the above-described method for generating controllable three-dimensional graphics content based on multi-image fusion when executing the computer program.

[0109] Based on the same inventive concept, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a computer, implements the above-described method for generating controllable 3D graphic content based on multi-image fusion.

[0110] It should be noted that the controllable 3D graphics content generation system, electronic device, and computer-readable storage medium based on multi-image fusion provided in the above embodiments all belong to the same inventive concept as the controllable 3D graphics content generation method based on multi-image fusion. For details of their specific implementation process, please refer to the embodiments of the controllable 3D graphics content generation method based on multi-image fusion, which will not be repeated here.

[0111] The specific embodiments described above illustrate the technical solution and beneficial effects of the present invention in detail. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A controllable 3D graphic content generation method based on multi-image fusion, characterized in that, Includes the following steps: The system receives multiple input images for controlling 3D generation, divides them into global control images and local control images, generates a 3D voxel set based on the global control images, and allows the user to define the mask region of the local control images to mark the details or style parts that need to be controlled in 3D space, thus obtaining the masked local control images. The correlation between the multiple input images and the 3D spatial representation is analyzed using a matching algorithm to obtain the feature sequence encoded by each input image, including: forward matching: extracting the feature sequence of the mask region and the local index set from each local control image; calculating the correlation between each voxel in the 3D voxel set and the feature map of the local control image; aggregating the correlation based on the local index set to obtain the forward matching strength of each voxel; and filtering out the covered voxel set that is semantically aligned with the mask region based on the forward matching strength; and reverse matching: calculating the correlation between each voxel in the remaining uncovered voxel set and the feature map of the global control image; aggregating the correlation to obtain the reverse matching strength; and extracting the feature sequence and global index set from the global control image based on the reverse matching strength. Based on the feature sequence, a hybrid conditional feature sequence containing comprehensive semantics of multiple images is constructed using a fusion algorithm; The hybrid conditional feature sequence is input into the generation model, where an enhancement mechanism is introduced to adaptively adjust the feature association relationship in order to obtain the fused and enhanced conditional feature sequence. The fused and enhanced conditional feature sequence is input into the 3D generation decoder, which outputs the final 3D graphic content.

2. The controllable 3D graphics content generation method based on multi-image fusion according to claim 1, characterized in that, The construction of a hybrid conditional feature sequence containing comprehensive semantics of multiple images based on the feature sequence using a fusion algorithm includes: Based on the local index set, the corresponding feature sequence is extracted from the feature map of the local control image of each mask. Based on the global index set, the corresponding feature sequence is extracted from the feature map of the global control image. The feature sequences of all the extracted images are concatenated to incorporate the global and local semantic information into a unified cross-modal attention computing framework, forming a hybrid conditional feature sequence containing the comprehensive semantics of multiple images.

3. The controllable 3D graphics content generation method based on multi-image fusion according to claim 1 or 2, characterized in that, The step of inputting the hybrid conditional feature sequence into the generation model, wherein an enhancement mechanism is introduced to adaptively adjust the feature association relationship to obtain the fused and enhanced conditional feature sequence, includes: The hybrid conditional feature sequence, along with local and global index sets, is input into the generative model. Based on the local and global index sets, the voxel regions affected by each image feature sequence corresponding to the hybrid conditional feature sequence in 3D space are determined. An enhancement matrix is ​​constructed to assign enhancement intensity parameters to the corresponding voxel regions of the local and global control images. The enhancement matrix is ​​applied to the cross-modal attention map during the generative model's inference process. The attention is weighted by element-wise multiplication, enabling the generative model to output the fused and enhanced conditional feature sequence.

4. The controllable 3D graphics content generation method based on multi-image fusion according to claim 3, characterized in that, During the generative model inference process, a smooth transition strategy is introduced at the region boundaries. The enhancement intensity parameter corresponding to the local control image is set according to the ratio of the overall image area to the mask area selected by the user, ensuring that small areas are more controlled while large areas maintain a smooth transition.

5. The controllable 3D graphics content generation method based on multi-image fusion according to claim 1, characterized in that, The process of inputting the fused and enhanced conditional feature sequence into the 3D generation decoder and outputting the final 3D graphic content includes: The enhanced conditional feature sequence is fused and decoded with the positions of each voxel in the three-dimensional voxel set using a three-dimensional generative decoder to generate three-dimensional graphic content including triangular meshes, neural radiation fields, or three-dimensional Gaussian point representations.

6. A controllable 3D graphics content generation system based on multi-image fusion, implemented using the controllable 3D graphics content generation method based on multi-image fusion as described in any one of claims 1 to 5, characterized in that, include: The module includes a multi-image input module, a 3D semantic perception alignment module, a multi-condition feature fusion module, a local attention enhancement module, and a 3D generation and decoding module. The multi-image input module is used to receive multiple input images for controlling 3D generation; The three-dimensional semantic perception alignment module is used to analyze the correlation between the multiple input images and the three-dimensional spatial representation through a matching algorithm to obtain the feature sequence encoded by each input image; The multi-condition feature fusion module is used to construct a hybrid condition feature sequence containing comprehensive semantics of multiple images based on the feature sequence through a fusion algorithm; The local attention enhancement module is used to input the hybrid conditional feature sequence into the generation model, wherein an enhancement mechanism is introduced to adaptively adjust the feature association relationship in order to obtain the fused and enhanced conditional feature sequence. The 3D generation and decoding module is used to input the fused and enhanced conditional feature sequence into the 3D generation and decoding module and output the final 3D graphic content.

7. An electronic device comprising a memory and one or more processors, the memory being used to store a computer program, characterized in that, The processor is used to implement the controllable three-dimensional graphic content generation method based on multi-image fusion as described in any one of claims 1 to 5 when executing a computer program.

8. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by a computer, it implements the controllable three-dimensional graphic content generation method based on multi-image fusion as described in any one of claims 1 to 5.

Images