Three-dimensional texture generation method, apparatus, device, medium, and program product
By utilizing projection overlap information to construct an attention bias matrix in 3D texture generation, the problem of inconsistent texture generation in existing technologies is solved, achieving high-quality geometric alignment and reducing computational costs.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING ZITIAO NETWORK TECH CO LTD
- Filing Date
- 2026-04-13
- Publication Date
- 2026-06-05
AI Technical Summary
Existing 3D texture generation methods lack explicit geometric-pixel alignment constraints, resulting in inconsistencies between the generated textures and the input images, which affects the fidelity of 3D reconstruction and is difficult to resolve through fine-tuning training.
By acquiring projection overlap information between 2D and 3D, an attention bias matrix is constructed and injected into the cross-attention layer of the pre-trained 3D generative model to guide the generative model to focus on the correct image regions and avoid texture blurring.
It achieves geometric alignment between 3D textures and input images, improving generation quality, reducing model computational costs, and eliminating the need to retrain the model.
Smart Images
Figure CN122156427A_ABST
Abstract
Description
Technical Field
[0001] It relates to the field of computer technology, specifically to three-dimensional texture generation methods, devices, equipment, media, and program products. Background Technology
[0002] With the development of Artificial Intelligence Generated Content (AIGC) technology, the demand for generating three-dimensional (3D) textures using generative models is increasing. Therefore, there is a need to provide a 3D texture generation method to meet these requirements. Summary of the Invention
[0003] A method, apparatus, device, medium, and program product for generating three-dimensional textures, to solve the problem of generating three-dimensional textures.
[0004] In a first aspect, a three-dimensional texture generation method includes: acquiring a first image and a camera projection configuration of the first image, and acquiring a first three-dimensional representation, wherein the first image corresponds to a first object, the camera projection configuration corresponds to the first image, and the first three-dimensional representation is used to characterize the geometry and spatial occupancy of the first object; using the camera projection configuration, obtaining overlap information between the first image and the first three-dimensional representation; injecting the overlap information into a first attention module of a first model to generate a three-dimensional texture, wherein the three-dimensional texture corresponds to the first object.
[0005] In a second aspect, a three-dimensional texture generation apparatus includes: an acquisition module for acquiring a first image and a camera projection configuration of the first image, and acquiring a first three-dimensional representation, wherein the first image corresponds to a first object, the camera projection configuration corresponds to the first image, and the first three-dimensional representation is used to characterize the geometry and spatial occupancy of the first object; an overlap information acquisition module for acquiring overlap information between the first image and the first three-dimensional representation using the camera projection configuration; and a texture generation module for injecting the overlap information into a first attention module of a first model to generate a three-dimensional texture, wherein the three-dimensional texture corresponds to the first object.
[0006] Thirdly, an electronic device includes: a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to perform the three-dimensional texture generation method of the first aspect or any corresponding embodiment described above.
[0007] Fourthly, a computer-readable storage medium storing computer instructions for causing a computer to perform the three-dimensional texture generation method of the first aspect or any corresponding embodiment thereof.
[0008] Fifthly, a computer program product includes computer instructions for causing a computer to execute the three-dimensional texture generation method of the first aspect or any corresponding embodiment thereof.
[0009] In some cases, methods, apparatuses, devices, media, and program products for generating 3D textures utilize the camera projection configuration of a first image to obtain overlap information between the first image and a first 3D representation. This overlap information is used to characterize the correspondence between voxels in 3D space and pixel regions in a 2D image. Subsequently, the overlap information is injected into the first attention module of a first model, guiding the first model to focus on relevant image regions based on the overlap information. This ensures accurate acquisition of image texture information from the corresponding viewpoint, avoiding texture blurring or artifacts and guaranteeing that the first model generates high-quality 3D textures. Simultaneously, by directly adjusting the first attention module of the pre-trained first model, geometric alignment between the generated texture and the input image can be ensured without retraining the model, reducing the computational cost of the model. Attached Figure Description
[0010] To more clearly illustrate the technical solutions in specific implementation methods or related technologies under certain circumstances, the accompanying drawings used in the description of specific implementation methods or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0011] Figure 1 These are schematic diagrams based on application scenarios under certain conditions; Figure 2 This is a schematic diagram of the first type of 3D texture generation method under certain circumstances; Figure 3 This is a schematic diagram of the second process for generating 3D textures under certain circumstances; Figure 4 This is a schematic diagram of the third process for generating 3D textures under certain circumstances; Figure 5 These are specific schematic diagrams generated based on three-dimensional textures under certain conditions; Figure 6 This is a schematic diagram of the process flow for each view under certain circumstances; Figure 7 This is a structural block diagram of a 3D texture generation device under certain conditions; Figure 8 These are schematic diagrams of the hardware structure of electronic devices under certain circumstances. Detailed Implementation
[0012] To make the objectives, technical solutions, and advantages clearer in some cases, the technical solutions in some cases will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments, not all embodiments. Based on the embodiments in some cases, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this solution.
[0013] It is understood that before using the technical solutions disclosed in the various embodiments in certain situations, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in certain situations and their authorization should be obtained in accordance with relevant laws and regulations through appropriate means.
[0014] For example, upon receiving a user's proactive request, a prompt message can be sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware such as electronic devices, applications, servers, or storage media that perform the operation based on the prompt message.
[0015] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.
[0016] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the specific implementation method. Other methods that comply with relevant laws and regulations may also be applied to this implementation method.
[0017] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.
[0018] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this document, "multiple" means two or more, unless otherwise explicitly specified.
[0019] With the development of Artificial Intelligence Generated Content (AIGC) technology, the demand for generating three-dimensional (3D) textures using generative models is increasing. Related 3D texture generation methods mainly rely on image-conditional 3D generation models, such as flow matching-based models or diffusion-based models.
[0020] When generating textures, the aforementioned 3D generative models typically take one or more reference images as input, inject the image features into the generated 3D latent representation (such as voxels or three planes) through a cross-attention mechanism, and then decode to generate fine geometry and textures.
[0021] However, when injecting image features, the model typically relies on global cross-attention, lacking explicit geometric-pixel alignment constraints. Specifically, this manifests as the model's inability to associate voxels in 3D space with their corresponding pixel regions in the 2D image. This results in generated textures that, while stylistically similar, may differ in detail from the input image; for example, a logo in the input image might be on the left, but the generated texture might show it on the right. Furthermore, texture confusion is prone to occur with symmetrical objects; for instance, the texture of the object's front side might be incorrectly pasted onto the object's back side. This negatively impacts the fidelity of 3D reconstruction, and these problems are difficult to resolve through fine-tuning.
[0022] Based on this, by utilizing known 3D representations, camera poses, and geometric projections, projection overlap information between 2D and 3D images is obtained. An attention bias matrix is then constructed to inject this overlap information into the cross-attention layer of a pre-trained 3D generative model. This forces the 3D generative model to focus on the "correct" image regions, guiding the 3D texture generation process and preventing texture blurring. Furthermore, this method directly adjusts the cross-attention layer of the pre-trained 3D generative model, ensuring geometric alignment between the generated texture and the input image without retraining the model, thus reducing computational costs.
[0023] As an optional application scenario, such as Figure 1 As shown, application 101 is installed in electronic device 110, and user 130 can interact with application 101 through electronic device 110 and / or access device of electronic device 110.
[0024] For example, application 101 can be any application that provides services related to image generation. For instance, application 101 could be a 3D object texture generation application. Figure 1 In the application scenario shown, if application 101 is active, electronic device 110 can display the interface 102 of application 101. Interface 102 may include various pages that application 101 can provide, such as image display pages, interactive pages, settings pages, etc.
[0025] In some embodiments, electronic device 110 is communicatively connected to server 120 to provide services to application 101. Electronic device 110 may be a mobile terminal, fixed terminal, or portable terminal, etc., including but not limited to mobile phones, desktop computers, laptop computers, multimedia tablets, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices, or any combination thereof. In some embodiments, electronic device 110 may also support any type of interface, and server 120 may be various types of computing systems or servers capable of providing computing power, including but not limited to mainframes, edge computing nodes, computing devices in cloud environments, etc.
[0026] It should be noted that, Figure 1 This is merely an example of an application scenario and does not limit the scope of protection of this invention.
[0027] The embodiments of the present invention will now be described with reference to the accompanying drawings. It should be understood that the pages shown in the drawings are merely examples, and various page designs are possible in practice. The various graphic elements on the page may have different arrangements and different visual representations; one or more elements may be omitted or replaced, and one or more other elements may also be present, without any limitation in the embodiments of the present invention. Furthermore, the embodiments described below primarily pertain to electronic device 110. It should be understood that the actions described relative to electronic device 110 can be performed by application 101 on electronic device 110, or can be performed by application 101 in conjunction with its server (e.g., server 120).
[0028] In some cases, an embodiment of a three-dimensional texture generation method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.
[0029] In some cases, a three-dimensional texture generation method can be used to generate textures for three-dimensional models, which can be deployed in the aforementioned electronic devices to perform three-dimensional texture generation. Figure 2 This is a flowchart of 3D texture generation methods in some cases, such as... Figure 2 As shown, the process includes the following steps: Step S201: Obtain the first image and the camera projection configuration of the first image, and obtain the first three-dimensional representation.
[0030] The first image corresponds to the first object, the camera projection configuration corresponds to the first image, and the first three-dimensional representation is used to characterize the geometry and spatial occupancy of the first object.
[0031] The first image can be a multi-view image representation of the first object; the first object can be various three-dimensional objects in the actual scene, such as cups, furniture, trees, toys, etc., without specific limitations.
[0032] The camera projection configuration represents the camera pose and geometric projection of the first image. The camera pose represents the position and geometric projection of the first image. The position and orientation of the camera in the world coordinate system can be represented as: Geometric projection represents the process by which a 3D point in the world coordinate system is transformed into a 2D pixel in the first image through camera pose changes and camera intrinsic parameters. Point cloud diagrams can be used for this purpose. Represented in the form of .
[0033] In some cases, the first image The first image can be obtained by taking pictures of the first object from multiple perspectives, or it can be an image of the first object taken from a local image library from multiple perspectives, or it can be an image of the first object taken from an online public database from multiple perspectives. There is no specific limitation on the method of obtaining the first image here.
[0034] Accordingly, the camera projection configuration of the first image corresponds to the first image. Based on the given first image, the corresponding camera pose and geometric projection can be obtained through the feedforward reconstruction method.
[0035] As described above, the first three-dimensional representation is used to characterize the geometry and spatial occupancy of the first object; that is, the first three-dimensional representation is actually the spatial representation of the first object in the world coordinate system.
[0036] In some cases, the geometry and spatial occupancy of a first object can be modeled using a structured latent representation of a 3D generative model, resulting in a first 3D representation of the first object, which can be represented as a sparse set of voxels. .
[0037] Step S202: Using camera projection configuration, obtain the overlap information between the first image and the first three-dimensional representation.
[0038] The overlap information is used to characterize the degree of spatial overlap between each image feature in the first image and each voxel in the first three-dimensional representation. This overlap information may include no overlap and overlap.
[0039] By combining the camera projection configuration, information such as the number of points and area of each image block in the first image projected onto each voxel in the first 3D representation can be obtained. The presence or absence of overlap between the first image and the first 3D representation can be determined by the number of projected points or the projected area. For example, if the number of projected points exceeds a preset value, overlap between the first image and the first 3D representation is determined; otherwise, no overlap is found. The number of projected points can be used to characterize the overlap information between the two. Similarly, if the projected area exceeds a preset area value, overlap between the first image and the first 3D representation is determined; otherwise, no overlap is found. The projected area can be used to characterize the overlap information between the two.
[0040] By combining the camera projection configuration, the ray intersection depth between each image patch in the first image and each voxel in the first 3D representation can also be obtained. The ray intersection depth is used to determine whether there is overlap between the first image and the first 3D representation. For example, if the ray intersection depth is lower than a preset depth value, it is determined that there is overlap between the first image and the first 3D representation; otherwise, there is no overlap. This ray intersection depth can be used to characterize the overlap information between the two.
[0041] By combining the camera projection configuration, image features of each voxel in the first 3D representation projected onto the first image can be obtained. These image features are then compared with the corresponding image patch features in the first image to calculate a feature similarity. If the feature similarity exceeds a preset value, it is determined that there is overlap between the first image and the first 3D representation; otherwise, there is no overlap. This feature similarity can be used to characterize the overlap information between the two.
[0042] Step S203: Inject the overlapping information into the first attention module of the first model to generate a three-dimensional texture, which corresponds to the first object.
[0043] The first model represents the 3D texture generation model; the first attention module is used to build the association between different modalities to instruct the 3D texture generation process of the first model. For example, the first attention module can be a cross-attention layer, using cross-attention to guide texture generation.
[0044] When the first model performs 3D texture generation, the overlap information can be injected into the first attention module of the first model using methods such as an attention bias matrix or a preset overlap threshold. This modifies the attention information of the first attention module, and the modified attention information constrains the first model to select the most relevant image texture information according to the overlap information, ensuring that the image texture information is geometrically aligned with the first object, so that the texture generation can replicate the detailed positions of the first image. Simultaneously, for the same region, the first model can automatically focus on the viewpoint with the most overlap information for texture generation, avoiding conflicts between information from different viewpoints.
[0045] In some specific cases, the overlap between the first image and the first 3D representation, represented by the overlap information, can be used to construct an attention bias matrix. The overlap information is then injected into the first attention module of the first model using the attention bias matrix.
[0046] In other specific scenarios, an overlap threshold can be set on the first attention module. The overlap between the first image and the first 3D representation, as represented by the overlap information, is filtered according to this threshold, retaining image tokens with overlap values in the top-K. Subsequently, the first model can generate 3D textures based on the retained image tokens.
[0047] In some 3D texture generation methods, the overlap information between the first image and its 3D representation is obtained by utilizing the camera projection configuration of the first image. This overlap information is used to characterize the correspondence between voxels in 3D space and pixel regions in the 2D image. Subsequently, this overlap information is injected into the first attention module of the first model, guiding the model to focus on relevant image regions based on the overlap information. This ensures accurate acquisition of image texture information from the corresponding viewpoint, avoiding texture blurring or artifacts and guaranteeing high-quality 3D texture generation. Furthermore, by directly adjusting the first attention module of the pre-trained first model, geometric alignment between the generated texture and the input image can be ensured without retraining the model, reducing computational costs.
[0048] In some cases, a 3D texture generation method is provided, which can be used to generate textures for 3D models using generative models. These generative models can be deployed in the aforementioned electronic devices to perform 3D texture generation. Figure 3 It is a flowchart based on three-dimensional texture generation methods in some cases, such as Figure 3 As shown, the process includes the following steps: Step S301: Obtain a first image and its camera projection configuration, and obtain a first three-dimensional representation. The first image corresponds to a first object, the camera projection configuration corresponds to the first image, and the first three-dimensional representation is used to characterize the geometry and spatial occupancy of the first object. For details, please refer to the relevant descriptions of the steps in the embodiments shown above; they will not be repeated here.
[0049] Step S302: Using camera projection configuration, obtain the overlap information between the first image and the first three-dimensional representation.
[0050] Specifically, step S302 includes: Step S3021: Extract the two-dimensional token of the first image, which corresponds to the first image block in the first image.
[0051] The first image includes multiple image blocks, which are fixed-size rectangular sub-regions cropped from the entire first image. For example, an N×M first image can be cut into multiple P×P small squares, each of which is an image block.
[0052] The first image block is any one of the multiple image blocks; the 2D token represents the two-dimensional spatial structure of the first image block.
[0053] By segmenting the first image into blocks, multiple first image blocks are obtained. Each first image block is used as a basic unit to preserve the spatial relationship of pixels within the block. That is, the image features of the first image block are directly used as two-dimensional tokens.
[0054] Step S3022: Using camera projection configuration, obtain the first three-dimensional point set, which corresponds to the two-dimensional token.
[0055] The first 3D point set is a set of 3D points of the first image block corresponding to the 2D token. Specifically, using the camera projection configuration, each voxel in the first 3D representation is projected onto the first image to obtain the image block corresponding to each voxel in the first image. Then, the first 3D point set corresponding to each first image block is obtained by backprojecting the depth map or point cloud map.
[0056] Step S3023: Obtain the number of points in the first three-dimensional point set projected into the first voxel to obtain the first point count.
[0057] Among them, the first voxel corresponds to the first three-dimensional representation, the first number of points corresponds to the first image block, and the overlapping information includes the first number of points.
[0058] The first point count represents the average number of projected points of the first voxel observed from all visible viewpoints. Specifically, the number of three-dimensional points falling within the first voxel is obtained by projecting points from the first three-dimensional point set onto the first voxel.
[0059] Next, count all 3D points that fall within the first voxel to obtain the total number of 3D points within the first voxel. Combine the ratio of the number of 3D points falling within the first voxel to the total number of 3D points within the first voxel to obtain the first point count.
[0060] In some optional cases, step S3023 above includes: Step a1: Obtain the number of points in the first 3D point set projected onto the first voxel to obtain the second point count, which corresponds to the first image block.
[0061] Step a2: Normalize the second point number to obtain the first point number.
[0062] The first image block corresponds to a first three-dimensional point set. The intersection of each three-dimensional point in the first three-dimensional point set with the first voxel is performed to obtain the number of three-dimensional points projected from each three-dimensional point in the first three-dimensional point set into the first voxel. This number of three-dimensional points is the second number of points corresponding to the first image block.
[0063] For example, the first image block The corresponding first set of three-dimensional points is Calculate the first voxel With the first three-dimensional point set intersection The first image patch is obtained. Projected onto voxels The second dot inside.
[0064] To eliminate density differences between different voxels, the number of second points within each first voxel is normalized to obtain the normalized number of first points. This ensures that the number of first points has good adaptability under different resolutions and geometric densities, thus improving robustness.
[0065] In some optional cases, step a2 above includes: Step a21: Obtain the number of the second image blocks. The number of the second points corresponding to the second image blocks is greater than 0.
[0066] Step a22: Normalize the number of second image blocks to obtain the number of first points.
[0067] The second image block is an image block in the first image. The number of second points corresponding to the second image block is greater than 0, that is, the set of three-dimensional points corresponding to the second image block overlaps with the first voxel.
[0068] The number of second image blocks with a second point count greater than 0 is obtained by detecting the number of second points projected onto the first voxel by the three-dimensional point set corresponding to each second image block.
[0069] The summation of the second point sets corresponding to each second image patch yields the total number of 3D points projected onto the first voxel. This total number of 3D points is then used to normalize the second point count, yielding the corresponding first point count. The specific normalization process can be expressed as follows:
[0070] in, Indicates the first dot; Indicates falling into voxels The second point; This represents the total number of three-dimensional points.
[0071] By normalizing the number of second image patches to unify the standardized density of different voxels, the density difference is avoided from affecting the accurate determination of the number of projection points. This allows for precise measurement of the relative importance of different viewpoints, facilitating the automatic selection of the best observation viewpoint during multi-view fusion and helping to overcome texture blurring and misalignment problems.
[0072] In some optional cases, step S302 may include: using a camera projection configuration to obtain the projected area of the first image relative to a first three-dimensional representation, or the ray intersection depth, wherein the overlap information includes the projected area or the ray intersection depth.
[0073] All point clouds corresponding to the first 3D representation are mapped onto the image pixel plane of the first image through camera projection configuration, resulting in 2D pixels of the first 3D representation on the image pixel plane. The 2D pixels are fitted to a contour, and the pixel area enclosed by this contour is obtained. This pixel area is used as the projected area to characterize the overlap information between the first image and the first 3D representation.
[0074] Starting from the camera's optical center, a camera ray is generated pointing towards the image pixels. This camera ray intersects with the point cloud corresponding to the first 3D representation, and the intersection point is obtained. The depth value of this intersection point in the camera coordinate system is the ray intersection depth. This ray intersection depth is used to characterize the overlap information between the first image and the first 3D representation. The smaller the ray intersection depth, the more likely there is an overlap between the first image and the first 3D representation.
[0075] It supports using information such as the number of projection points, projection area, and ray intersection depth to characterize overlap information, making it easy to flexibly select the appropriate overlap information according to actual needs.
[0076] Step S303: Inject the overlapping information into the first attention module of the first model to generate a three-dimensional texture, which corresponds to the first object.
[0077] Specifically, step S303 includes: Step S3031: Obtain the first deviation between the second point count and the first point count. The second point count is used to characterize the first point count before normalization.
[0078] The first deviation represents the difference in points between the second and first points. As described above, the first point is obtained by normalizing the second point. The difference between the second and first points is the first deviation between the second and first points.
[0079] Step S3032: Obtain a first bias value using the first deviation and the first coefficient. The first bias value is positively correlated with the first deviation.
[0080] Each first image patch has a corresponding second number of points. The difference between the second number and the first number of points for each first image patch is calculated sequentially to obtain the maximum deviation. Combining the first deviation, the maximum deviation, and the first coefficient, a first bias value is obtained, which is used to control the texture generation of the first model.
[0081] The first bias value is positively correlated with the first deviation. Specifically, if the first deviation is greater than 0, a positive bias is applied to encourage the first model to focus on it; if the first deviation is less than or equal to 0, the first bias value is set to 0 or a negative value to suppress the first model's focus. The first coefficient represents the hyperparameter of the control strength; for example, the first coefficient can be set to 5.
[0082] Specifically, the method for obtaining the first bias value can be expressed as follows:
[0083] Where B represents the first bias value; Indicates the first coefficient; k This represents the k-th first image block.
[0084] Step S3033: Inject the first bias value into the first attention module of the first model to generate a three-dimensional texture.
[0085] A bias matrix is constructed by taking the first bias value corresponding to each image patch. This bias matrix is then fused into the attention module of the first model to obtain attention information carrying the first bias value. This attention information is used to instruct the texture generation process of the first model to obtain the corresponding 3D texture.
[0086] In some optional cases, the first bias value is injected into the first attention module of the first model in the following ways: by summing or multiplying the first bias value with the corresponding attention score.
[0087] The first bias value is summed with the corresponding attention score, and then injected into the first attention module of the first model. Specifically, this can be expressed as:
[0088] in, This represents the attention information of the first attention module; This represents the bias matrix formed by the first bias value; The query vector representing the three-dimensional token corresponding to the voxel; This represents the feature key-value pairs of the first image.
[0089] The first bias value is multiplied by the corresponding attention score, and then injected into the first attention module of the first model. Specifically, this can be expressed as:
[0090] in, This represents the attention information of the first attention module; This represents the bias matrix formed by the first bias value; The query vector representing the three-dimensional token corresponding to the voxel; This represents the feature key-value pairs of the first image.
[0091] Here, the first bias value can be injected into the first attention module by addition or multiplication, so that the first model can be modified directly in the inference stage. Geometric alignment control of the generated result can be achieved without retraining the model. Therefore, there is no need to collect new data for fine-tuning training, and it can be directly applied to all first models, which greatly reduces the application threshold and computing cost.
[0092] In some cases, the 3D texture generation method projects the first 3D point set corresponding to the 2D token onto the first voxel, obtains the first point number projected into the first voxel, and uses the first point number to represent the overlap information. This allows the overlap information to characterize the texture overlap degree under different viewpoints, making it easier to automatically focus on the viewpoint with the most overlap information, avoiding texture blurring or misalignment caused by information conflicts between different viewpoints, and improving the texture consistency of multiple viewpoints.
[0093] Based on the deviation between the second point count and the first point count, a corresponding first bias value is obtained and injected into the first attention module. This first bias value guides the first model to focus on the correct image region and suppresses the first model from focusing on the wrong image region, ensuring that the 3D texture can accurately replicate the detailed position of the first image. This achieves pixel-level texture alignment, eliminates feature drift in 3D texture generation, and improves the generation quality of 3D texture.
[0094] In some cases, a 3D texture generation method is provided, which can be used to generate textures for 3D models using generative models. These generative models can be deployed in the aforementioned electronic devices to perform 3D texture generation. Figure 4 This is a flowchart based on 3D texture generation methods under certain conditions, such as... Figure 4 As shown, the process includes the following steps: Step S401: Obtain a first image and its camera projection configuration, and obtain a first three-dimensional representation. The first image corresponds to a first object, the camera projection configuration corresponds to the first image, and the first three-dimensional representation is used to characterize the geometry and spatial occupancy of the first object. For details, please refer to the relevant descriptions of the steps in the embodiments shown above; they will not be repeated here.
[0095] Step S402: Using the camera projection configuration, obtain the overlap information between the first image and the first 3D representation. For details, please refer to the relevant descriptions of the corresponding steps in the embodiments shown above; they will not be repeated here.
[0096] Step S403: Inject the overlapping information into the first attention module of the first model to generate a three-dimensional texture, which corresponds to the first object.
[0097] Specifically, step S403 includes: Step S4031: Use overlap information to filter the first image block in the first image.
[0098] As described above, the overlap information is used to characterize the degree of overlap between the first image and the first three-dimensional representation. By pre-setting an overlap threshold for filtering the first image blocks, the degree of overlap represented by each overlap information is compared with the overlap threshold, and the first image blocks with an overlap degree less than the overlap threshold are filtered out, resulting in the first image blocks with an overlap degree greater than or equal to the overlap threshold.
[0099] Step S4032: The filtered first image block is injected into the first attention module to generate a three-dimensional texture.
[0100] The filtered first image blocks are injected into the first attention module, allowing the first attention module to acquire image tokens from each of the first image blocks. The image tokens from each first image block are then used to instruct the first model on the 3D texture prediction process for the first object, resulting in the 3D texture corresponding to the first object.
[0101] In some cases, the provided 3D texture generation method filters the first image patch in the first image based on overlap information to obtain a first image patch with acceptable overlap information. This filtered first image patch is then directly injected into the first attention module for 3D texture generation. Therefore, the first image patch can be directly used during the inference stage to perform geometric alignment control of the 3D texture, ensuring accurate acquisition of image texture information during 3D texture generation and improving the quality of the generated 3D texture.
[0102] As a specific application example in some situations, such as Figure 5As shown, based on the obtained 3D representation of the object and the corresponding 2D image features and camera pose of the input image, the number of points projected from each 2D token in the image features onto each voxel in the 3D representation is analyzed. This number of points represents the spatial overlap between the image block and the voxel. Then, the number of points is normalized to obtain the average point count (APC). The deviation between the normalized average point count and the number of points before normalization is calculated. If the deviation indicates an overlap higher than the APC, a positive bias is applied; otherwise, if the deviation indicates an overlap lower than or equal to the APC, the bias is set to 0. The obtained bias value is injected into the cross-attention layer of the 3D generative model to guide the 3D generative model in performing the 3D texture generation process.
[0103] For a 3D voxel corresponding to a 3D representation, when multiple viewpoints exist, the bias value is obtained for each viewpoint. For example... Figure 6 As shown, if the APC is 60, in viewpoint 1, the number of points projected onto the corresponding voxel by the image patch corresponding to the 2D Token is 100. Since 100 > 60, a positive bias is given, i.e., the bias value > 0. In viewpoint 2, the number of points projected onto the corresponding voxel by the image patch corresponding to the 2D Token is 20. Since 20 < 60, the bias value is set to 0. Subsequently, the corresponding bias values for each viewpoint are injected into the cross-attention layer of the 3D generative model to guide the 3D generative model in generating 3D textures.
[0104] In some cases, a three-dimensional texture generation apparatus is also provided for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.
[0105] In some cases, a three-dimensional texture generation apparatus is provided, such as Figure 7 As shown, it includes: The acquisition module 701 is used to acquire a first image and a camera projection configuration of the first image, and to acquire a first three-dimensional representation. The first image corresponds to a first object, the camera projection configuration corresponds to the first image, and the first three-dimensional representation is used to characterize the geometry and spatial occupancy of the first object.
[0106] The overlap information acquisition module 702 is used to obtain the overlap information between the first image and the first three-dimensional representation by utilizing the camera projection configuration.
[0107] The texture generation module 703 is used to inject overlapping information into the first attention module of the first model to generate a three-dimensional texture, which corresponds to the first object.
[0108] In some optional cases, the overlap information acquisition module 702 includes: The token extraction unit is used to extract a two-dimensional token from the first image, and the two-dimensional token corresponds to a first image block in the first image.
[0109] The three-dimensional point set acquisition unit is used to obtain a first three-dimensional point set by utilizing camera projection configuration, and the first three-dimensional point set corresponds to a two-dimensional token.
[0110] The projection unit is used to obtain the number of points in the first three-dimensional point set projected into the first voxel to obtain the first point count. The first voxel corresponds to the first three-dimensional representation, and the first point count corresponds to the first image block. The overlap information includes the first point count.
[0111] In some alternative implementations, the projection unit includes: The projection point acquisition subunit is used to obtain the number of points in the first three-dimensional point set projected into the first voxel, and obtain the second point count, which corresponds to the first image block.
[0112] The normalization sub-unit is used to normalize the second point number to obtain the first point number.
[0113] In some optional cases, the normalization processing subunit is specifically used to obtain the number of the second image block, where the number of the second point corresponding to the second image block is greater than 0; and to normalize the number of the second point using the number of the second image block to obtain the number of the first point.
[0114] In some optional cases, the texture generation module 703 includes: The point deviation unit is used to obtain the first deviation between the second point and the first point. The second point is used to characterize the first point before normalization.
[0115] The bias value acquisition unit is used to obtain a first bias value using a first deviation and a first coefficient, wherein the first bias value is positively correlated with the first deviation.
[0116] The first injection unit is used to inject the first bias value into the first attention module of the first model to generate a three-dimensional texture.
[0117] In some optional cases, the first injection unit is used to sum or multiply the first bias value with the corresponding attention score.
[0118] In some optional cases, the texture generation module 703 includes: The filtering unit is used to filter the first image block in the first image using overlap information.
[0119] The second injection unit is used to inject the filtered first image block into the first attention module to generate a three-dimensional texture.
[0120] In some cases, the provided three-dimensional texture generation apparatus can execute the three-dimensional texture generation method provided in the above embodiments, and has the corresponding functional modules and beneficial effects of the method.
[0121] By utilizing the camera projection configuration of the first image, overlap information between the first image and the first 3D representation is obtained. This overlap information is used to characterize the correspondence between voxels in 3D space and pixel regions in the 2D image. Subsequently, the overlap information is injected into the first attention module of the first model, guiding the first model to focus on relevant image regions based on the overlap information. This ensures accurate acquisition of image texture information from the corresponding viewpoint, avoiding texture blurring or artifacts and guaranteeing that the first model generates high-quality 3D textures. Simultaneously, by directly adjusting the first attention module of the pre-trained first model, geometric alignment between the generated texture and the input image can be ensured without retraining the model, reducing the computational cost of the model.
[0122] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.
[0123] Figure 8 This is a schematic diagram of the structure of an electronic device provided in certain situations.
[0124] The following is a detailed reference. Figure 8 This diagram illustrates a structural schematic suitable for implementing an electronic device in certain situations. The electronic device may include a processor (e.g., a central processing unit, graphics processor, etc.) 801, which can perform various appropriate actions and processes based on a program stored in read-only memory (ROM) 802 or a program loaded from memory 808 into random access memory (RAM) 803. RAM 803 also stores various programs and data required for the operation of the electronic device. The processor 801, ROM 802, and RAM 803 are interconnected via bus 804. An input / output interface 805 is also connected to bus 804.
[0125] Typically, the following devices can be connected to the input / output interface 805: input devices 806 including, for example, a touchscreen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 807 including, for example, a liquid crystal display, speaker, vibrator, etc.; memory devices 808 including, for example, magnetic tape, hard disk, etc.; and communication devices 809. Communication device 809 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 8 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown, and more or fewer devices may be implemented or have instead.
[0126] In particular, the processes described in the flowchart above can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 809, or installed from memory 808, or installed from ROM 802. When the computer program is executed by processor 801, it performs the functions defined in the three-dimensional texture generation method in some cases.
[0127] Figure 8 The electronic devices shown are merely examples and should not be construed as limiting their functionality or scope of use in any situation.
[0128] In some cases, a computer-readable storage medium is also provided, in which the above-described methods can be implemented in hardware, firmware, or implemented as recordable on a storage medium, or implemented as computer code originally stored on a remote storage medium or a non-transitory machine-readable storage medium and subsequently stored on a local storage medium after being downloaded via a network. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code that, when accessed and executed by the computer, processor, or hardware, implements the three-dimensional texture generation method shown in the above embodiments.
[0129] Some of the above solutions can be applied as computer program products, such as computer program instructions. When executed by a computer, these instructions, through the operation of the computer, can invoke or provide the aforementioned methods and / or technical solutions. Those skilled in the art should understand that the forms in which computer program instructions exist in computer-readable media include, but are not limited to, source files, executable files, and installation package files. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instruction; the computer compiling the instruction and then executing the corresponding compiled program; the computer reading and executing the instruction; or the computer reading and installing the instruction and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.
[0130] Although embodiments in some cases have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the above description, and such modifications and variations all fall within the scope defined by the appended claims.
Claims
1. A method for generating three-dimensional textures, comprising: A first image and its camera projection configuration are obtained, and a first three-dimensional representation is obtained. The first image corresponds to a first object, the camera projection configuration corresponds to the first image, and the first three-dimensional representation is used to characterize the geometry and spatial occupancy of the first object. Using the camera projection configuration, overlap information between the first image and the first three-dimensional representation is obtained; The overlapping information is injected into the first attention module of the first model to generate a three-dimensional texture, which corresponds to the first object.
2. The method according to claim 1, wherein obtaining the overlap information between the first image and the first three-dimensional representation using the camera projection configuration includes: Extract a two-dimensional token from the first image, the two-dimensional token corresponding to a first image block in the first image; Using the camera projection configuration, a first three-dimensional point set is obtained, and the first three-dimensional point set corresponds to the two-dimensional token; The number of points in the first three-dimensional point set projected onto the first voxel is obtained to obtain the first point count. The first voxel corresponds to the first three-dimensional representation, and the first point count corresponds to the first image block. The overlap information includes the first point count.
3. The method according to claim 2, wherein obtaining the number of points in the first three-dimensional point set projected onto the first voxel to obtain the first point count comprises: The number of points in the first three-dimensional point set projected onto the first voxel is obtained to obtain the second point count, which corresponds to the first image block. The second point number is normalized to obtain the first point number.
4. The method according to claim 3, wherein the normalization process of the second point number to obtain the first point number includes: Obtain the number of the second image patch, where the number of the second point corresponding to the second image patch is greater than 0; The second number of points is obtained by normalizing the number of the second image blocks.
5. The method according to claim 2, wherein injecting the overlapping information into the first attention module of the first model to generate a three-dimensional texture comprises: Obtain the first deviation between the second point count and the first point count, where the second point count is used to characterize the first point count before normalization. A first bias value is obtained by using the first deviation and the first coefficient, and the first bias value is positively correlated with the first deviation; The first bias value is injected into the first attention module of the first model to generate the three-dimensional texture.
6. The method according to claim 5, wherein the first bias value is injected into the first attention module of the first model in a manner comprising: The first bias value is summed or multiplied with the corresponding attention score.
7. The method according to claim 1, wherein injecting the overlapping information into the first attention module of the first model to generate a three-dimensional texture comprises: The first image block in the first image is filtered using the overlapping information; The filtered first image patch is injected into the first attention module to generate a three-dimensional texture.
8. A three-dimensional texture generation apparatus, comprising: The acquisition module is used to acquire a first image and a camera projection configuration of the first image, and to acquire a first three-dimensional representation, wherein the first image corresponds to a first object, the camera projection configuration corresponds to the first image, and the first three-dimensional representation is used to characterize the geometry and spatial occupancy of the first object; An overlap information acquisition module is used to obtain overlap information between the first image and the first three-dimensional representation using the camera projection configuration; The texture generation module is used to inject the overlapping information into the first attention module of the first model to generate a three-dimensional texture, which corresponds to the first object.
9. An electronic device, characterized in that, include: A memory and a processor are communicatively connected, the memory stores computer instructions, and the processor executes the computer instructions to perform the three-dimensional texture generation method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing a computer to perform the three-dimensional texture generation method according to any one of claims 1 to 7.
11. A computer program product, characterized in that, Includes computer instructions for causing a computer to perform the three-dimensional texture generation method according to any one of claims 1 to 7.