Scale and structure consistency image generation method fusing three-dimensional geometric prior

Abstract

The invention discloses a scale and structure consistency image generation method fusing three-dimensional geometric prior, and belongs to the technical field of computer vision and image generation. The method comprises the following steps: acquiring a scene image and depth information corresponding to the scene image, reconstructing a scene point cloud containing pixel space coordinates and color information, acquiring a colored point cloud of an object to be inserted, placing the colored point cloud at a specified position of the scene through rigid body transformation, and unifying a coordinate system with the scene point cloud; projecting the transformed object point cloud to an image plane through a camera model, determining a visible point according to the depth, and constructing an insertion region mask to remove pixels of a region corresponding to a scene; then constructing a multi-channel geometric structure priori graph containing color, depth and other information, and splicing the multi-channel geometric structure priori graph with the cleared scene image to form a diffusion type generation model input; and finally, generating an image under geometric prior guidance, ensuring that the inserted object is consistent with the original scene in the aspects of position, scale, perspective and the like, solving the problem of insufficient geometric consistency of the existing method, and realizing image content insertion with an accurate structure and real vision.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and image generation technology, specifically relating to an image generation method that integrates three-dimensional geometric priors and scale and structural consistency. Background Technology

[0002] With the rapid development of generative models (such as generative adversarial networks and diffusion models), image generation and content editing tasks have made significant progress in terms of visual realism, semantic consistency, and scene aesthetics. In recent years, controllable generation methods based on text, edge maps, semantic segmentation maps, and other conditions have been able to accomplish tasks such as style transfer, object editing, and region completion, and have become the mainstream technical approach in the field of image generation.

[0003] However, existing controllable generation methods mainly rely on two-dimensional semantic or edge priors, lacking explicit representations of real three-dimensional structures. This makes it difficult to meet the geometric consistency requirements of tasks such as "inserting external objects into real-world scene images." When objects are complex, diverse in shape, or require precise placement, two-dimensional priors cannot provide accurate three-dimensional position, rotation, scale, and visibility information. This results in the insertion results failing to maintain consistency with the real scene in terms of position, scale, occlusion, and perspective, easily leading to problems such as floating, misalignment, deformation, or edge ghosting.

[0004] Furthermore, while diffusion-based image generation models excel in detail realism, their control signals largely originate from two-dimensional image space, making it impossible to directly model spatial relationships determined by three-dimensional point clouds. Without explicit three-dimensional geometric constraints, the generation model cannot determine the positional boundaries of inserted objects based on real-world spatial coordinates, nor can it guarantee consistency between the inserted region and the scene in terms of physical scale, orientation, and depth, thus affecting the credibility and usability of the generated results.

[0005] Therefore, how to utilize the RGB-D information of the scene and the 3D point cloud data of external objects to provide precise spatial geometric constraints for the image generation process, so that the position, scale, contour, and occlusion of the inserted object in the generated image are highly consistent with the real 3D scene, is an important problem that current image insertion and controllable generation technologies urgently need to solve. Summary of the Invention

[0006] The main objective of this invention is to overcome the shortcomings and deficiencies of the prior art and provide a method for generating images that integrates three-dimensional geometric priors and scale and structural consistency, so that the spatial position, appearance structure and visual consistency of the inserted object in the image are matched with the real scene.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] In a first aspect, the present invention provides a method for generating scale- and structure-consistent images that integrates three-dimensional geometric priors, comprising the following steps:

[0009] Acquire scene images and corresponding depth information, reconstruct scene point clouds based on camera device intrinsic parameters, and obtain scene 3D data containing spatial coordinates and color information of each pixel.

[0010] Obtain the colored point cloud data of the object to be inserted, perform a three-dimensional rigid body transformation on the colored point cloud data according to the given three-dimensional position and rotational orientation, place the object to be inserted in the specified three-dimensional position and orientation in the scene, and make the transformed object point cloud and the scene point cloud in the same coordinate system.

[0011] Using the intrinsic and extrinsic parameters of the camera device, the transformed object point cloud is projected onto the scene image plane to obtain a two-dimensional projection map of the object containing RGB color information under the current camera view. Visible points are determined based on the point cloud depth occlusion relationship, and only the object point cloud visible in the current view is retained.

[0012] The masking area of ​​the target object in the scene image is determined based on the projection coordinates, and the scene image pixels within the masking area are cleared.

[0013] A geometric prior channel is constructed based on the projection results. The geometric prior channel includes one or more of the following: color projection map, depth projection map, visibility mask, occupancy mask, and contour map. Multiple geometric prior channels are combined into a multi-channel structural prior map.

[0014] The multi-channel structure prior image and the cleaned scene image are stitched together along the channel dimension, provided that the pixel coordinates are strictly aligned, to form the input image of the image generation model.

[0015] Guided by the multi-channel structure prior map, the insertion result image is generated through a diffusion generation model, so that the objects in the output image are consistent with the real 3D scene in terms of position, shape contour, depth relationship, occlusion relationship and physical scale.

[0016] As a preferred technical solution, the internal parameters of the camera device include the focal length parameters of the camera in the x-axis and y-axis directions of the image, and the coordinate parameters of the principal point on the camera imaging plane; in the scene 3D point cloud data, the 3D spatial coordinate information corresponding to each pixel is used to characterize the position of the pixel in the real 3D scene, and the color information is used to characterize the actual color of the pixel in the scene.

[0017] As a preferred technical solution, the three-dimensional rigid body transformation operation includes translation and rotation operations on the object point cloud. The translation operation moves the object point cloud to a preset three-dimensional position in the scene, and the rotation operation adjusts the object point cloud to a preset orientation. During the transformation, the geometry and size of the object point cloud itself remain unchanged. In the colored three-dimensional point cloud data of the object to be inserted, the coordinate information of each point represents the position of the point in the object's own coordinate system, and the color information represents the object surface color corresponding to the point.

[0018] As a preferred technical solution, the object point cloud projection adopts perspective projection, which maps the object point cloud in three-dimensional space onto a two-dimensional image plane according to the set projection rules; during the visible point selection process, if multiple object point cloud projection points correspond to the same pixel position on the image plane, the projection point that is closer to the camera in three-dimensional space is selected as the final visible point of that pixel position.

[0019] As a preferred technical solution, when constructing a multi-channel geometric prior map, the coordinates of the selected visible points on the two-dimensional image plane are mapped to discrete pixel positions in the image. At each pixel position, information such as the object point cloud color, depth, visibility, or whether it is occupied by an object is recorded, forming a single type of geometric prior channel. If the distribution of projection points on the image plane after the object point cloud is projected is relatively sparse, neighborhood expansion, pixel interpolation, or other image filling methods are used to supplement complete color, depth, or occupancy information in the pixel areas around the projection points, forming a continuous two-dimensional geometric information representation. The image filling methods include fixed-range neighborhood diffusion, weighted projection filling, Gaussian scattering filling, and other methods that can achieve pixel information expansion.

[0020] As a preferred technical solution, the channel dimension stitching operation merges the color channels of the cleaned scene image with the geometric information channels of the multi-channel geometric structure prior image to form a multi-channel input image containing scene background information and object geometric constraint information.

[0021] As a preferred technical solution, the diffusion generation model gradually reduces noise interference in the input image through a built-in noise prediction network. While preserving scene background information and object geometric constraint information, it generates a clear synthetic image containing the object to be inserted. During the generation process, geometric prior constraints ensure that the object to be inserted matches the original scene in terms of perspective imaging effect and size ratio, avoiding situations such as floating objects, abnormal sizes, or perspective distortion.

[0022] Secondly, the present invention provides a scale and structure consistency image generation system that integrates three-dimensional geometric priors, applied to the aforementioned scale and structure consistency image generation method that integrates three-dimensional geometric priors, including a data acquisition module, a point cloud transformation module, a projection processing module, a mask construction module, a geometric prior generation module, a channel stitching module, and an image generation module;

[0023] The data acquisition module is used to acquire scene images and depth information corresponding to the scene images, reconstruct scene point clouds based on camera intrinsic parameters, and obtain scene 3D data containing spatial coordinates and color information of each pixel.

[0024] The point cloud transformation module is used to acquire colored point cloud data of the object to be inserted, perform three-dimensional rigid body transformation on the colored point cloud data according to the given three-dimensional position and rotational attitude, place the object to be inserted in the specified three-dimensional position and orientation in the scene, and make the transformed object point cloud and the scene point cloud in the same coordinate system.

[0025] The projection processing module is used to project the transformed object point cloud onto the scene image plane using the intrinsic and extrinsic parameters of the camera device, to obtain a two-dimensional projection map of the object containing RGB color information under the current camera view, and to determine the visible points based on the point cloud depth occlusion relationship, retaining only the object point cloud visible in the current view.

[0026] The masking construction module is used to determine the masking area of ​​the target object in the image based on the projection coordinates, and to clear the scene image pixels within the masking area.

[0027] The geometric prior generation module is used to construct geometric prior channels based on the projection results. The geometric prior channels include one or more of the following: color projection map, depth projection map, visibility mask, occupancy mask, and contour map. Multiple geometric prior channels are combined into a multi-channel structural prior map.

[0028] The channel stitching module is used to stitch the multi-channel structure prior image and the cleaned scene image according to the channel dimension, under the premise of strict alignment of pixel coordinates, to form the input image of the image generation model.

[0029] The image generation module is used to generate an inserted result image through a diffusion generation model under the guidance of the multi-channel structure prior image, so that the objects in the output image are consistent with the real three-dimensional scene in terms of position, shape contour, depth relationship, occlusion relationship and physical scale.

[0030] Thirdly, the present invention provides an electronic device, the electronic device comprising:

[0031] At least one processor; and,

[0032] A memory communicatively connected to the at least one processor; wherein,

[0033] The memory stores computer program instructions that can be executed by the at least one processor, which enables the at least one processor to perform the scale and structure consistency image generation method that integrates three-dimensional geometric priors.

[0034] Fourthly, the present invention provides a computer-readable storage medium storing a program, which, when executed by a processor, implements the aforementioned method for generating scale- and structurally consistent images by fusing three-dimensional geometric priors.

[0035] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0036] Existing image generation and content insertion methods often rely on 2D semantic tags, edge maps, or depth maps for control, lacking precise 3D spatial geometric constraints. When inserting external objects into real-world scene images, the generation model cannot ensure consistency between the inserted object and the scene in terms of position, scale, rotation, occlusion, and perspective, easily leading to distortions such as geometric misalignment, scale abnormalities, edge ghosting, or "floating" effects. This invention achieves structurally accurate and visually realistic image insertion through multi-dimensional techniques. Specifically, firstly, it achieves pixel-level structural consistency based on the geometric constraints of real 3D point clouds, where the projection position, contour, and pose of the inserted object are directly determined by 3D geometry, without relying on internal model estimation, thus effectively avoiding misalignment, rotation errors, or perspective distortion. Secondly, it uses an external rectangular mask to clear conflict areas, preventing structural conflicts caused by background overlap. Simultaneously, it utilizes the RGB texture of the object's point cloud to construct a projection color prior, further enhancing the consistency and realism of the generated results. Furthermore, multi-channel structural priors provide key layout information such as depth, visibility, and occlusion, significantly improving the physical credibility of scene fusion. This solution can flexibly adapt to different object categories and different scenarios, and can achieve accurate image insertion without additional annotation. While maintaining visual realism, it can precisely control the position, structure and proportion of the inserted object, and has strong versatility and application value. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is a flowchart of the method for generating scale and structural consistency images that integrates three-dimensional geometric priors according to an embodiment of the present invention;

[0039] Figure 2 This is a block diagram of the scale and structure consistency image generation system that integrates three-dimensional geometric priors, as described in an embodiment of the present invention.

[0040] Figure 3 This is a structural diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0041] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0042] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.

[0043] like Figure 1 As shown, this embodiment provides a method for generating scale- and structure-consistent images by fusing three-dimensional geometric priors, including the following steps:

[0044] S1. Scene Data Acquisition and Point Cloud Reconstruction: Scene images and depth maps are acquired using a single RGB-D camera, and scene point clouds are reconstructed based on camera intrinsic parameters to obtain 3D scene data containing the spatial coordinates and color information of each pixel.

[0045] Furthermore, obtaining the scene's 3D data, which includes the spatial coordinates and color information of each pixel, specifically involves:

[0046] The RGB image and depth map of the scene are acquired using a depth camera RGB-D, and each pixel is mapped to a three-dimensional space based on the camera intrinsic parameter matrix K to obtain a scene point cloud containing coordinate and color information.

[0047] The intrinsic parameter matrix K is represented as follows:

[0048] ;

[0049] The formula for the scene point cloud is as follows:

[0050] .

[0051] S2. Target object point cloud and pose transformation: Obtain the colored point cloud data of the object to be inserted, and perform rigid body transformation on the object point cloud according to the given 3D position and rotational orientation (including translation vector t and rotation matrix R), placing it in the specified 3D position and orientation in the scene, while making the transformed object point cloud and scene point cloud in the same coordinate system.

[0052] Furthermore, the RGB point cloud representation of the object to be inserted is as follows:

[0053]

[0054] After performing a rigid body transformation on the object's point cloud, placing it at a specified position in the scene, and mapping it to the scene coordinate system, the actual point cloud representation of the target object in the scene is as follows:

[0055] .

[0056] S3. Object point cloud projection based on camera model: Using the intrinsic and extrinsic parameters of the scene camera, the transformed object point cloud is projected onto the scene image plane to obtain a two-dimensional projection map of the object from the current camera viewpoint, including the object's RGB color information.

[0057] Furthermore, the transformed point cloud By performing perspective projection, we obtain the coordinates of the two-dimensional image:

[0058] ;

[0059] in For point clouds Coordinates in the camera coordinate system.

[0060] Furthermore, it also includes the visibility determination step, specifically:

[0061] The visible point is determined based on the depth relationship of the projection points. When the same pixel location corresponds to multiple projection points, the point with the smaller depth value (i.e., closer to the camera) is selected as the final visible point of that pixel. If it is necessary to further improve the accuracy of visibility, the local geometric relationship of the point cloud (such as the direction of the normal vector) can be used to help determine whether it is within the camera's visible range.

[0062] S4. Construction of Insertion Region Mask and Scene Image Clearing Based on Projection Results: Generate the minimum bounding rectangle region of the object on the image plane based on the projection point set, and construct the insertion region mask; clear, fill or empty the original scene pixels in the region to avoid background conflicts or structural overlaps during the generation stage.

[0063] S5. Multi-channel geometric prior map construction: Construct geometric prior channels based on the projection results. These channels may include one or more of the following: color projection map, depth projection map, visibility mask, occupancy mask, and contour map. Combine multiple channels into a multi-channel structural prior map to explicitly represent the true geometric structure and spatial position of objects in the scene.

[0064] Furthermore, the specific correction for S5 is as follows:

[0065] Based on the projection results obtained in step S3, the two-dimensional coordinates of the visible points are... The discrete pixel locations are mapped to the image plane, and the color, depth, visibility, or occupancy information of the object point cloud is written at the corresponding pixel to generate a two-dimensional structure prior map. For cases where the same pixel location corresponds to multiple projection points, the visibility judgment in step S3 is used to maintain the consistency of occlusion relationships.

[0066] For cases where projection points are sparsely distributed, color, depth, or occupancy information can be supplemented in the pixel regions surrounding the projection points through neighborhood expansion, interpolation, or other image filling methods to form a continuous two-dimensional structural representation. The filling methods are not limited to fixed neighborhood diffusion, weighted projection, Gaussian scattering, or other methods that can achieve pixel expansion.

[0067] The structural prior map may include one or more of the following channels: color projection map, visibility mask, occupancy mask, depth projection map, and contour map. These channels are aligned by pixel coordinates and combined to form a multi-channel geometric structural prior map. It is used to explicitly express the projection position, outline shape and spatial relationship of the target object in the scene.

[0068] S6. Input Construction for Image Generation Based on Geometric Priors: The multi-channel structural prior image and the cleaned scene image are stitched together along the channel dimension, ensuring strict alignment of pixel coordinates, to form the input image for the image generation model. This method allows the image generation model to directly receive explicit structural constraints such as the geometric position, contour shape, depth relationships, and color cues of the target object during the feature extraction stage.

[0069] Furthermore, step S6 specifically involves:

[0070] Scene image after mask removal With structural prior diagrams Stitching along the channel dimension while ensuring strict alignment of pixel coordinates:

[0071]

[0072] This step forms the input for the diffusion-based image generation model. It ensures that the generation model can directly acquire geometric information such as object position, shape contours, depth relationships, and color priors during the input phase.

[0073] S7. Structural Consistency-Driven Diffusion Image Generation: Under the condition of including geometric prior guidance, the insertion result image is generated through a diffusion generation model, so that the objects in the output image are consistent with the real 3D scene in terms of position, shape contour, depth relationship, occlusion relationship and physical scale, thus obtaining a synthetic image with strong structural consistency and high visual realism.

[0074] Furthermore, step S7 specifically includes:

[0075] Input image The input is a diffusion-based image generation model, which progressively denoises the image through a noise prediction network to generate the final synthetic image after inserting the target object. .

[0076] The model generates results under the constraints of structural priors, ensuring that the inserted objects are consistent with the real scene in terms of position, scale, outline, perspective, and occlusion relationships.

[0077] It should be noted that, for the sake of simplicity, the aforementioned method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously.

[0078] Based on the same idea as the scale and structure consistency image generation method incorporating 3D geometric priors in the above embodiments, this invention also provides a scale and structure consistency image generation system incorporating 3D geometric priors. This system can be used to execute the above-described scale and structure consistency image generation method incorporating 3D geometric priors. For ease of explanation, the structural schematic diagram of the embodiment of the scale and structure consistency image generation system incorporating 3D geometric priors only shows the parts related to the embodiments of this invention. Those skilled in the art will understand that the illustrated structure does not constitute a limitation on the device, and may include more or fewer components than illustrated, or combine certain components, or have different component arrangements.

[0079] Please see Figure 2 In another embodiment of this application, a scale and structure consistency image generation system 100 that integrates three-dimensional geometric priors is provided. The system includes a data acquisition module 101, a point cloud transformation module 102, a projection processing module 103, a mask construction module 104, a geometric prior generation module 105, a channel stitching module 106, and an image generation module 107.

[0080] The data acquisition module 101 is used to acquire scene images and depth information corresponding to the scene images, reconstruct scene point clouds based on camera intrinsic parameters, and obtain scene 3D data containing spatial coordinates and color information of each pixel.

[0081] The point cloud transformation module 102 is used to acquire colored point cloud data of the object to be inserted, perform three-dimensional rigid body transformation on the colored point cloud data according to the given three-dimensional position and rotational posture, place the object to be inserted in a specified three-dimensional position and orientation in the scene, and make the transformed object point cloud and the scene point cloud in the same coordinate system.

[0082] The projection processing module 103 is used to project the transformed object point cloud onto the scene image plane using the intrinsic and extrinsic parameters of the camera device, to obtain a two-dimensional projection map of the object containing RGB color information under the current camera view, and to determine the visible points based on the point cloud depth occlusion relationship, retaining only the object point cloud visible in the current view.

[0083] The masking construction module 104 is used to determine the masking area of ​​the target object in the image according to the projection coordinates, and to clear the scene image pixels in the masking area.

[0084] The geometric prior generation module 105 is used to construct a geometric prior channel based on the projection result. The geometric prior channel includes one or more of the following: color projection map, depth projection map, visibility mask, occupancy mask, and contour map. Multiple geometric prior channels are combined into a multi-channel structural prior map.

[0085] The channel stitching module 106 is used to stitch the multi-channel structure prior image and the cleaned scene image according to the channel dimension, under the premise of strict alignment of pixel coordinates, to form the input image of the image generation model.

[0086] The image generation module 107 is used to generate an inserted result image through a diffusion generation model under the guidance of the multi-channel structure prior image, so that the objects in the output image are consistent with the real three-dimensional scene in terms of position, shape contour, depth relationship, occlusion relationship and physical scale.

[0087] It should be noted that the scale and structure consistency image generation system of the present invention, which integrates three-dimensional geometric priors, corresponds one-to-one with the scale and structure consistency image generation method of the present invention. The technical features and beneficial effects described in the above embodiments of the scale and structure consistency image generation method, which integrates three-dimensional geometric priors, are all applicable to the embodiments of scale and structure consistency image generation, which integrates three-dimensional geometric priors. For details, please refer to the description in the embodiments of the present invention, which will not be repeated here.

[0088] Furthermore, in the above embodiments of the scale and structure consistency image generation system integrating three-dimensional geometric priors, the logical division of each program module is merely illustrative. In practical applications, the above functions can be assigned to different program modules as needed, for example, for the sake of corresponding hardware configuration requirements or software implementation convenience. That is, the internal structure of the scale and structure consistency image generation system integrating three-dimensional geometric priors can be divided into different program modules to complete all or part of the functions described above.

[0089] Please see Figure 3 In one embodiment, an electronic device is provided for implementing a method for generating scale- and structure-consistent images by fusing three-dimensional geometric priors. The electronic device 200 may include a first processor 201, a first memory 202, and a bus, and may also include a computer program stored in the first memory 202 and executable on the first processor 201, such as a scale- and structure-consistent image generation program 203 that fuses three-dimensional geometric priors.

[0090] The first memory 202 includes at least one type of readable storage medium, including flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the first memory 202 can be an internal storage unit of the electronic device 200, such as the portable hard drive of the electronic device 200. In other embodiments, the first memory 202 can also be an external storage device of the electronic device 200, such as a plug-in portable hard drive, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the electronic device 200. Furthermore, the first memory 202 can include both internal storage units and external storage devices of the electronic device 200. The first memory 202 can be used not only to store application software and various types of data installed on the electronic device 200, such as the code of the scale and structural consistency image generation program 203 that integrates three-dimensional geometric priors, but also to temporarily store data that has been output or will be output.

[0091] In some embodiments, the first processor 201 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The first processor 201 is the control unit of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It executes programs or modules stored in the first memory 202 and calls data stored in the first memory 202 to perform various functions of the electronic device 200 and process data.

[0092] Figure 3 Only electronic devices with components are shown; it will be understood by those skilled in the art that... Figure 3 The structure shown does not constitute a limitation on the electronic device 200, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0093] The scale and structure consistency image generation program 203, which integrates three-dimensional geometric priors and is stored in the first memory 202 of the electronic device 200, is a combination of multiple instructions. When run in the first processor 201, it can achieve the following:

[0094] Scene images and depth maps are acquired using an RGB-D camera, and scene point clouds are reconstructed based on camera intrinsic parameters to obtain 3D scene data containing spatial coordinates and color information for each pixel.

[0095] Obtain the colored point cloud data of the object to be inserted, perform a three-dimensional rigid body transformation on the colored point cloud data according to the given three-dimensional position and rotational orientation, place the object to be inserted in the specified three-dimensional position and orientation in the scene, and make the transformed object point cloud and the scene point cloud in the same coordinate system.

[0096] Using the intrinsic and extrinsic parameters of the RGB-D camera, the transformed object point cloud is projected onto the scene image plane to obtain a two-dimensional projection map of the object containing RGB color information from the current camera viewpoint. Visible points are determined based on the point cloud depth occlusion relationship, and only the object point cloud visible in the current viewpoint is retained.

[0097] The masking area of ​​the target object in the image is determined based on the projection coordinates, and the scene image pixels within the masking area are cleared.

[0098] A geometric prior channel is constructed based on the projection results. The geometric prior channel includes one or more of the following: color projection map, depth projection map, visibility mask, occupancy mask, and contour map. Multiple geometric prior channels are combined into a multi-channel structural prior map.

[0099] The multi-channel structure prior image and the cleaned scene image are stitched together along the channel dimension, provided that the pixel coordinates are strictly aligned, to form the input image of the image generation model.

[0100] Guided by the multi-channel structure prior map, the insertion result image is generated through a diffusion generation model, so that the objects in the output image are consistent with the real 3D scene in terms of position, shape contour, depth relationship, occlusion relationship and physical scale.

[0101] Furthermore, if the modules / units integrated in the electronic device 200 are implemented as software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0102] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0103] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0104] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A method for generating scale- and structure-consistent images by integrating three-dimensional geometric priors, characterized in that, Includes the following steps: Acquire scene images and corresponding depth information, reconstruct scene point clouds based on camera device intrinsic parameters, and obtain scene 3D data containing spatial coordinates and color information of each pixel. Obtain the colored point cloud data of the object to be inserted, perform a three-dimensional rigid body transformation on the colored point cloud data according to the given three-dimensional position and rotational orientation, place the object to be inserted in the specified three-dimensional position and orientation in the scene, and make the transformed object point cloud and the scene point cloud in the same coordinate system. Using the intrinsic and extrinsic parameters of the camera device, the transformed object point cloud is projected onto the scene image plane to obtain a two-dimensional projection map of the object containing RGB color information under the current camera view. Visible points are determined based on the point cloud depth occlusion relationship, and only the object point cloud visible in the current view is retained. The masking area of ​​the target object in the image is determined based on the projection coordinates, and the scene image pixels within the masking area are cleared. A geometric prior channel is constructed based on the projection results. The geometric prior channel includes one or more of the following: color projection map, depth projection map, visibility mask, occupancy mask, and contour map. Multiple geometric prior channels are combined into a multi-channel structural prior map. The multi-channel structure prior image and the cleaned scene image are stitched together along the channel dimension, provided that the pixel coordinates are strictly aligned, to form the input image of the image generation model. Guided by the multi-channel structure prior map, the insertion result image is generated through a diffusion generation model, so that the objects in the output image are consistent with the real 3D scene in terms of position, shape contour, depth relationship, occlusion relationship and physical scale.

2. The method for generating scale- and structure-consistent images by fusing three-dimensional geometric priors as described in claim 1, characterized in that, The internal parameters of the camera device include the focal length parameters of the camera in the x-axis and y-axis directions of the image, and the coordinate parameters of the principal point on the camera imaging plane; in the scene 3D point cloud data, the 3D spatial coordinate information corresponding to each pixel is used to characterize the position of the pixel in the real 3D scene, and the color information is used to characterize the actual color of the pixel in the scene.

3. The method for generating scale- and structure-consistent images by fusing three-dimensional geometric priors as described in claim 1, characterized in that, The three-dimensional rigid body transformation operation includes translation and rotation operations on the object point cloud. The translation operation moves the object point cloud to a preset three-dimensional position in the scene, and the rotation operation adjusts the object point cloud to a preset orientation. The geometric shape and size of the object point cloud remain unchanged during the transformation process. In the colored 3D point cloud data of the object to be inserted, the coordinate information of each point represents the position of the point in the object's own coordinate system, and the color information represents the surface color of the object corresponding to that point.

4. The method for generating scale- and structure-consistent images by fusing three-dimensional geometric priors as described in claim 1, characterized in that, The object point cloud projection adopts perspective projection, which maps the object point cloud in three-dimensional space onto a two-dimensional image plane according to the set projection rules. In the visible point selection process, if multiple object point cloud projection points correspond to the same pixel position on the image plane, the projection point that is closer to the camera in three-dimensional space is selected as the final visible point of that pixel position.

5. The method for generating scale- and structure-consistent images by fusing three-dimensional geometric priors as described in claim 1, characterized in that, When constructing a multi-channel geometric prior map, the coordinates of the selected visible points on the two-dimensional image plane are mapped to the discrete pixel positions of the image. At each pixel position, information such as the color, depth, visibility, or whether it is occupied by an object in the corresponding object point cloud is recorded to form a single type of geometric prior channel. If the distribution of projection points on the image plane after the object point cloud is projected is relatively sparse, neighborhood expansion, pixel interpolation, or other image filling methods can be used to supplement complete color, depth, or occupancy information in the pixel area around the projection points to form a continuous two-dimensional geometric information representation. The image filling methods include fixed-range neighborhood diffusion, weighted projection filling, Gaussian scattering filling, and other methods that can achieve pixel information expansion.

6. The method for generating scale- and structure-consistent images by fusing three-dimensional geometric priors as described in claim 1, characterized in that, The channel dimension stitching operation merges the color channels of the cleaned scene image with the geometric information channels of the multi-channel geometric prior image to form a multi-channel input image containing scene background information and object geometric constraint information.

7. The method for generating scale- and structure-consistent images by fusing three-dimensional geometric priors as described in claim 1, characterized in that, The diffusion generation model gradually reduces noise interference in the input image through a built-in noise prediction network, and generates a clear synthetic image containing the object to be inserted while preserving scene background information and object geometric constraint information. During the generation process, geometric prior constraints are used to ensure that the object to be inserted matches the original scene in terms of perspective imaging effect and size ratio, so as to avoid the situation of floating objects, abnormal size or perspective distortion.

8. A scale- and structure-consistent image generation system integrating three-dimensional geometric priors, characterized in that, The method for generating scale- and structure-consistent images based on the fusion of three-dimensional geometric priors, as described in any one of claims 1-7, includes a data acquisition module, a point cloud transformation module, a projection processing module, a mask construction module, a geometric prior generation module, a channel stitching module, and an image generation module. The data acquisition module is used to acquire scene images and depth information corresponding to the scene images, reconstruct scene point clouds based on camera intrinsic parameters, and obtain scene 3D data containing spatial coordinates and color information of each pixel. The point cloud transformation module is used to acquire colored point cloud data of the object to be inserted, perform three-dimensional rigid body transformation on the colored point cloud data according to the given three-dimensional position and rotational attitude, place the object to be inserted in the specified three-dimensional position and orientation in the scene, and make the transformed object point cloud and the scene point cloud in the same coordinate system. The projection processing module is used to project the transformed object point cloud onto the scene image plane using the intrinsic and extrinsic parameters of the camera device, to obtain a two-dimensional projection map of the object containing RGB color information under the current camera view, and to determine the visible points based on the point cloud depth occlusion relationship, retaining only the object point cloud visible in the current view. The masking construction module is used to determine the masking area of ​​the target object in the image based on the projection coordinates, and to clear the scene image pixels within the masking area. The geometric prior generation module is used to construct geometric prior channels based on the projection results. The geometric prior channels include one or more of the following: color projection map, depth projection map, visibility mask, occupancy mask, and contour map. Multiple geometric prior channels are combined into a multi-channel structural prior map. The channel stitching module is used to stitch the multi-channel structure prior image and the cleaned scene image according to the channel dimension, under the premise of strict alignment of pixel coordinates, to form the input image of the image generation model. The image generation module is used to generate an inserted result image through a diffusion generation model under the guidance of the multi-channel structure prior image, so that the objects in the output image are consistent with the real three-dimensional scene in terms of position, shape contour, depth relationship, occlusion relationship and physical scale.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores computer program instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the scale and structure consistency image generation method according to any one of claims 1-7.

10. A computer-readable storage medium storing a program, characterized in that, When the program is executed by the processor, it implements the scale and structure consistency image generation method according to any one of claims 1-7.

Images