Image generation method and device
By acquiring the image to be processed and the target's 3D bounding box, and fusing the mask region to generate appearance and position features, the problem of insufficient accuracy and consistency in autonomous driving image generation is solved, improving the image generation quality and the generalization performance of the autonomous driving dataset.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-29
- Publication Date
- 2026-04-07
AI Technical Summary
In existing technologies, the generation of images related to autonomous driving suffers from insufficient 3D spatial positioning accuracy, poor consistency, and insufficient generalization performance, which affects the performance of downstream tasks.
By acquiring the image to be processed, the reference image of the target object, and the target's 3D bounding box, the mask region is fused to generate appearance and location features. These features are then processed using an image generation network to improve the accuracy and consistency of image generation.
It improves the accuracy and consistency of generated images, enhances the quality and generalization performance of autonomous driving datasets, and improves the performance of downstream tasks.
Smart Images
Figure CN121811005A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing technology, and in particular to an image generation method and apparatus. Background Technology
[0002] The development of autonomous driving applications requires extensive verification and testing of autonomous vehicles to ensure their safety. With the rapid development of AI generative models, AI can be used for image editing and generation to improve image quality and thus enhance the performance of downstream tasks. However, current technologies suffer from insufficient 3D spatial positioning accuracy, poor consistency, and inadequate generalization performance, which negatively impact the performance of downstream tasks. Summary of the Invention
[0003] To address the aforementioned problems in the prior art, this invention discloses an image generation method and apparatus that can improve the accuracy and consistency of generated images, enhance image generation quality, and thereby improve the quality and generalization performance of autonomous driving datasets. The technical solution disclosed in this invention is as follows: According to one aspect of the disclosed embodiments of the present invention, an image generation method is provided, comprising: The process involves acquiring an image to be processed, a reference image of the target object, and a target 3D bounding box; the image to be processed includes multiple frames, and the target 3D bounding box is used to indicate the location of the editing region in the image to be processed. The masked region corresponding to the target 3D bounding box is fused with the image to be processed to obtain an updated image with a mask and a masked image; the updated image is used to determine the edited region, and the masked image is used to identify the boundary of the edited region; Based on the reference image, the updated image, and the mask image, a first appearance feature corresponding to the target object is generated; Based on the multi-view image features corresponding to the reference image, a second appearance feature corresponding to the target object is generated; The depth information corresponding to the target 3D bounding box is extracted to generate the position features corresponding to the target object; The first appearance feature, the second appearance feature, and the position feature are input into an image generation network for image generation processing to obtain the target image corresponding to the image to be processed.
[0004] Optionally, the target 3D bounding box includes the azimuth information of the target object, and generating the second appearance feature corresponding to the target object based on the multi-view image features corresponding to the reference image includes: From the multi-view image features corresponding to the reference image, determine the view image features corresponding to the azimuth information; The second appearance feature is obtained by matching the viewpoint image features with the projection area of the target 3D bounding box in the corresponding image to be processed.
[0005] Optionally, the step of extracting features from the depth information corresponding to the target 3D bounding box to generate the position features corresponding to the target object includes: Project and fuse multiple faces of the target's 3D bounding box to obtain a depth feature map; Multi-scale feature extraction is performed on the depth feature map to obtain multi-scale depth features; The location features are obtained by matching the multi-scale depth features in both the spatial and temporal dimensions.
[0006] Optionally, generating the first appearance feature corresponding to the target object based on the reference image, the updated image, and the mask image includes: The reference image and the updated image are merged to obtain a merged image; The merged image and the mask image are fused and feature extracted to obtain the first appearance sub-feature; Feature extraction and fusion processing are performed on the reference image and the updated image respectively to obtain the second appearance sub-feature; The first appearance sub-feature and the second appearance sub-feature are fused to obtain the first appearance feature.
[0007] Optionally, the target 3D bounding box includes a first target 3D bounding box and a second target 3D bounding box. The reference image is obtained by segmenting the target object from the image to be processed. The step of fusing the mask region corresponding to the target 3D bounding box with the image to be processed to obtain an updated image with a mask and a mask image includes: Projection processing is performed on the first target 3D bounding box and the second target 3D bounding box respectively to obtain the first mask region corresponding to the first target 3D bounding box and the second mask region corresponding to the first target 3D bounding box; The first mask region and the second mask region are superimposed on the image to be processed to obtain the updated image; Based on the second mask region, the image to be processed is binarized to obtain the mask image.
[0008] Optionally, the step of extracting features from the depth information corresponding to the target 3D bounding box to generate the position features corresponding to the target object includes: Feature extraction is performed on the depth information corresponding to the three-dimensional bounding box of the second target to generate the position features corresponding to the target object.
[0009] Optionally, the target 3D bounding box includes a first target 3D bounding box, the reference image is used to update the appearance of the target object in the image to be processed, and the step of fusing the mask region corresponding to the target 3D bounding box with the image to be processed to obtain an updated image with a mask and a mask image includes: The first target's three-dimensional bounding box is projected to obtain the first mask region; The first mask region is superimposed on the image to be processed to obtain the updated image; Based on the first mask region, the image to be processed is binarized to obtain the mask image.
[0010] Optionally, the target 3D bounding box includes a second target 3D bounding box, the target object is the object to be inserted into the image to be processed, and the step of fusing the mask region corresponding to the target 3D bounding box with the image to be processed to obtain an updated image with a mask and a mask image includes: The second target's three-dimensional bounding box is projected to obtain the second mask region; The second mask region is superimposed on the image to be processed to obtain the updated image; Based on the second mask region, the image to be processed is binarized to obtain the mask image.
[0011] Optionally, the method further includes: Obtain the image to be processed and the 3D bounding box of the first target; The mask region corresponding to the three-dimensional bounding box of the first target is fused with the image to be processed to obtain the updated image and the mask image; the edited region is the region that needs to be deleted and repaired. Based on the preset blank features, the updated image, and the mask image, background features are generated; The depth information corresponding to the target 3D bounding box is used to extract features to generate the position features; The background features and the location features are input into the image generation network for image generation processing to obtain the target image corresponding to the image to be processed.
[0012] According to another aspect of the disclosed embodiments of the present invention, an image generation apparatus is provided, comprising: The first acquisition module is used to acquire the image to be processed, the reference image of the target object, and the target 3D bounding box; the image to be processed includes multiple frames, and the target 3D bounding box is used to indicate the position of the editing area in the image to be processed; The first fusion module is used to fuse the mask region corresponding to the target 3D bounding box with the image to be processed to obtain an updated image with a mask and a mask image; the updated image is used to determine the editing region, and the mask image is used to identify the boundary of the editing region; The first appearance feature generation module is used to generate a first appearance feature corresponding to the target object based on the reference image, the updated image and the mask image; The second appearance feature generation module is used to generate a second appearance feature corresponding to the target object based on the multi-view image features corresponding to the reference image. The location feature generation module is used to extract features from the depth information corresponding to the target three-dimensional bounding box and generate the location features corresponding to the target object. The image generation module is used to input the first appearance feature, the second appearance feature and the position feature into the image generation network for image generation processing to obtain the target image corresponding to the image to be processed.
[0013] According to another aspect of the disclosed embodiments of the present invention, an electronic device for image generation is provided, including a processor and a memory, wherein the memory stores at least one instruction, the at least one instruction being loaded and executed by the processor to implement the image generation method described in any of the preceding claims.
[0014] According to another aspect of the embodiments disclosed in this invention, a computer-readable storage medium is provided, wherein at least one instruction is stored therein, the at least one instruction being loaded and executed by a processor to implement the image generation method described in any of the preceding claims.
[0015] According to another aspect of the disclosed embodiments of the present invention, a computer program product containing instructions is provided, which, when run on a computer, causes the computer to perform the image generation method described in any of the above embodiments of the present invention.
[0016] The image generation method provided by this invention has the following technical effects: The image generation method provided by this invention first acquires an image to be processed, a reference image of the target object, and a target 3D bounding box. The image to be processed includes multiple frames, and the target 3D bounding box is used to indicate the location of the edit region in the image to be processed. The mask region corresponding to the target 3D bounding box is fused with the image to be processed to obtain an updated image with a mask and a mask image. The updated image is used to determine the edit region, and the mask image is used to identify the boundary of the edit region. Based on the reference image, the updated image, and the mask image, a first appearance feature corresponding to the target object is generated. Based on the multi-view image features corresponding to the reference image, a second appearance feature corresponding to the target object is generated. Feature extraction is performed on the depth information corresponding to the target 3D bounding box to generate the position feature corresponding to the target object. The first appearance feature, the second appearance feature, and the position feature are input into an image generation network for image generation processing to obtain the target image corresponding to the image to be processed. This method can improve the accuracy and consistency of the generated image, improve the image generation quality, and thus enhance the quality and generalization performance of autonomous driving datasets.
[0017] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description
[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0019] Figure 1 This is a flowchart illustrating an image generation method according to an exemplary embodiment; Figure 2 This is a schematic diagram illustrating a process for generating a first appearance feature according to an exemplary embodiment; Figure 3 This is a schematic diagram illustrating a process for generating a second appearance feature according to an exemplary embodiment; Figure 4 This is a schematic diagram illustrating a process for generating location features according to an exemplary embodiment; Figure 5 This is a block diagram of an image generation apparatus according to an exemplary embodiment; Figure 6 This is a block diagram illustrating a terminal electronic device for image generation according to an exemplary embodiment; Figure 7 This is a block diagram illustrating a server electronic device for image generation according to an exemplary embodiment. Detailed Implementation
[0020] To enable those skilled in the art to better understand the technical solutions disclosed in this invention, the technical solutions in the disclosed embodiments will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.
[0021] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention disclosed herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.
[0022] Please see Figure 1 , Figure 1 This is a flowchart illustrating an image generation method according to an exemplary embodiment. This specification provides the operational steps of the method as described in the embodiments or flowcharts, but based on conventional or non-inventive labor, more or fewer operational steps may be included. The order of steps listed in the embodiments is merely one possible execution order among many and does not represent the only execution order. In actual system or server products, the method can be executed sequentially or in parallel (e.g., in a parallel processor or multi-threaded processing environment) as shown in the embodiments or drawings. Specifically, as... Figure 1 As shown, the above method may include: S101: Obtain the image to be processed, the reference image of the target object, and the target's 3D bounding box.
[0023] In one specific embodiment, the image to be processed can be any image that needs to be edited. The image to be processed can include multiple frames. Specifically, each frame can include a target object and / or environmental information surrounding the target object. The target object can include vehicles, pedestrians, obstacles, etc. Optionally, the multiple frames can form a video. The reference image can be an image that includes only a portion of the target object. The reference image can be used to provide the appearance of the target object to guide the image editing process. The target 3D bounding box can be used to indicate the location of the editing area in the image to be processed. Specifically, the target 3D bounding box can be used to define the position, size, and azimuth of the target object in 3D space to ensure geometric consistency during the image editing process.
[0024] In one specific embodiment, the reference image and the target 3D bounding box can be determined according to the editing task for the object to be processed. The editing task may include a repositioning task for the target object in the image to be processed (i.e., moving it from the initial position to the updated position), an appearance replacement task, a deletion task (deleting the target object in the image to be processed), and an insertion task (inserting the target object at a specified position in the image to be processed), etc.
[0025] Specifically, in the case of editing tasks as relocation tasks, the reference image can be obtained by segmenting the target object from the image to be processed. The target 3D bounding box can include a first target 3D bounding box and a second target 3D bounding box. Specifically, the first target 3D bounding box can include the initial position information of the target object in the image to be processed, and the second target 3D bounding box can include the updated position information of the target object in the image to be processed.
[0026] Specifically, when the editing task is an appearance replacement task, the reference image can be a pre-provided appearance image of the target object that needs to be replaced. The reference image can be used to update and replace the appearance of the target object in the image to be processed. The target 3D bounding box can include the first target 3D bounding box, which can include the position information of the target object in the image to be processed.
[0027] Specifically, when the editing task is an insertion task, the target object is the object in the image to be inserted and processed, the reference image is the appearance image of the object to be inserted, the target 3D bounding box may include a second target 3D bounding box, and the second target 3D bounding box may include the position information of the object to be inserted.
[0028] Specifically, when the editing task is a deletion task, no reference image is needed; only the first target 3D bounding box is required, which indicates the location of the target object to be deleted. The above editing task needs to maintain consistency between appearance attributes and scene dynamics.
[0029] S103: The mask region corresponding to the target 3D bounding box is fused with the image to be processed to obtain an updated image with a mask and a mask image.
[0030] In one specific embodiment, the updated image can be used to determine the editing region, and the mask image can be used to identify the boundary of the editing region. Specifically, the updated image can be a mask region portion corresponding to the target 3D bounding box and a background image in the image to be processed excluding the mask region portion. The mask image can be a binarized image, where all pixel values within the mask region can be 1 (representing the editing region), and all pixel values outside the mask region can be 0 (representing the background region).
[0031] In some embodiments, when the editing task is the aforementioned repositioning task, fusing the mask region corresponding to the target 3D bounding box with the image to be processed to obtain an updated image with a mask and a mask image may include: Projection processing is performed on the three-dimensional bounding box of the first target and the three-dimensional bounding box of the second target respectively to obtain the first mask region corresponding to the three-dimensional bounding box of the first target and the second mask region corresponding to the three-dimensional bounding box of the first target; The first mask region and the second mask region are superimposed on the image to be processed to obtain the updated image; Based on the second mask region, the image to be processed is binarized to obtain the mask image.
[0032] In one specific embodiment, when the editing task is the aforementioned repositioning task, it is necessary to segment the target object from the image to be processed to obtain a reference image to preserve the original appearance, as well as the target object's 3D bounding box in the image to be processed (i.e., the aforementioned first target 3D bounding box), used to indicate the initial position of the target object, and a pre-determined 3D bounding box (i.e., the aforementioned second target 3D bounding box), used to indicate the updated position and orientation of the target object. Specifically, the first and second target 3D bounding boxes are respectively subjected to 2D projection, and the resulting projection area is the core mask range, which is superimposed on each frame image, occluding the corresponding areas while keeping the remaining background areas unchanged, to obtain the aforementioned updated image. Specifically, based on the 2D projection of the first and second target 3D bounding boxes, a binary mask with the same size as the image to be processed is generated (mask area is 1, background area is 0).
[0033] In some embodiments, when the editing task is the appearance replacement task described above, fusing the mask region corresponding to the target 3D bounding box with the image to be processed to obtain an updated image with a mask and a mask image may include: The masked region corresponding to the target's 3D bounding box is fused with the image to be processed to obtain an updated image with a mask and a masked image, including: Project the three-dimensional bounding box of the first target to obtain the first mask region; The first mask region is overlaid on the image to be processed to obtain the updated image; Based on the first mask region, the image to be processed is binarized to obtain the mask image.
[0034] In one specific embodiment, when the editing task is the aforementioned appearance replacement task, a 3D bounding box of the target object in the image to be processed (i.e., the aforementioned first target 3D bounding box) is needed to indicate the position of the target object, as well as pre-provided replacement appearance information (i.e., reference image), such as texture, color, shape, etc. Specifically, only a 2D projection of the first target 3D bounding box is needed. The resulting projection area is the core mask range, which is superimposed on each frame image to occlude the corresponding area, while the remaining background area remains unchanged, thus obtaining the aforementioned updated image. Specifically, based on the 2D projection of the first target 3D bounding box, a binary mask with the same size as the image to be processed is generated (mask area is 1, background area is 0).
[0035] In some embodiments, when the editing task is the insertion task described above, fusing the mask region corresponding to the target 3D bounding box with the image to be processed to obtain an updated image with a mask and a mask image may include: The second target's 3D bounding box is projected to obtain the second mask region; The second mask region is overlaid on the image to be processed to obtain the updated image; Based on the second mask region, the image to be processed is binarized to obtain the mask image.
[0036] In one specific embodiment, when the editing task is the aforementioned insertion task, a reference image of the target object to be inserted into the image to be processed and a 3D bounding box (i.e., the second target 3D bounding box) of the target object to be inserted into the image to be processed are required in advance. Specifically, only a 2D projection of the second target 3D bounding box is needed. The resulting projection area is the core mask range, which is superimposed on each frame image to occlude the corresponding area, while the remaining background area remains unchanged, thus obtaining the aforementioned updated image. Specifically, based on the 2D projection of the second target 3D bounding box, a binary mask with the same size as the image to be processed is generated (the mask area is 1, and the background area is 0).
[0037] S105: Based on the reference image, the updated image, and the mask image, generate the first appearance feature corresponding to the target object.
[0038] Optional, such as Figure 2As shown, based on the reference image, the updated image, and the mask image, the first appearance feature corresponding to the target object is generated as follows: S201: Merge the reference image and the updated image to obtain a merged image.
[0039] Specifically, the pasting area is determined based on the projection area, and the reference image is pasted into the projection area of the updated image to obtain the merged image, ensuring that the size and position of the target object match.
[0040] S203: Perform fusion and feature extraction processing on the merged image and the mask image to obtain the first appearance sub-feature.
[0041] Specifically, the pasted image and the mask image are stitched together along the channels, and feature extraction processing is performed on the stitched image to obtain the first appearance sub-feature.
[0042] S205: Perform feature extraction and fusion processing on the reference image and the updated image respectively to obtain the second appearance sub-features.
[0043] Specifically, based on a cross-attention mechanism, the features extracted from the reference image and the updated image can be fused to obtain second-order appearance features. These features can then be used to constrain the semantically correct appearance of the generated images during subsequent image generation.
[0044] S207: The first appearance feature and the second appearance feature are fused to obtain the first appearance feature.
[0045] S107: Generate the second appearance feature corresponding to the target object based on the multi-view image features corresponding to the reference image.
[0046] In one specific embodiment, the reference image is essentially a single-view two-dimensional image, which only contains appearance information of the object from one angle. It is difficult to cover the multi-view changes that may occur in autonomous driving videos. Therefore, multi-view features can be generated by SV3D model, which is essentially a three-dimensional appearance representation of the object inferred from a single two-dimensional image.
[0047] Optional, such as Figure 3 As shown, the second appearance feature of the target object generated based on the multi-view image features corresponding to the reference image may include: S301: Determine the viewpoint image features corresponding to the azimuth information from the multi-view image features corresponding to the reference image.
[0048] In one specific embodiment, the SV3D model, through its built-in 3D reconstruction network (composed of an encoder and decoder), generates M virtual views of the object from preset fixed perspectives, using a reference object as input. These views are not complete images, but rather feature-level views containing the object's 3D structure (such as the object's contour, surface texture, depth distribution, etc. from that perspective). The intermediate features fm of each block in the SV3D model are extracted as 3D priors. Specifically, the M preset fixed perspectives can be set according to actual application requirements, such as 0°, 3°, 6°, etc. Specifically, the target 3D bounding box includes azimuth information. Based on the azimuth information of the corresponding image frame editing position, the perspective closest among the M fixed perspectives is selected to obtain the corresponding perspective image features.
[0049] S303: Match the viewpoint image features with the projection area of the target 3D bounding box in the corresponding image to be processed to obtain the second appearance feature.
[0050] Specifically, the 3D information fusion module adjusts the viewpoint image features to the size and position of the projection area of the corresponding 3D bounding box of the image frame, thereby aligning the SV3D features with the 3D space of the image frame and ensuring that the multi-viewpoint features are consistent with the current viewpoint.
[0051] In the above embodiments, a 3D information fusion module is applied to overcome the limitations of single-view reference images by fusing multi-view features, so that the replacement object remains consistent under different viewpoints, thereby improving the quality of the subsequently generated images and ensuring the fidelity of appearance.
[0052] S109: Extract features from the depth information corresponding to the target's 3D bounding box to generate the position features corresponding to the target object.
[0053] Specifically, depth-sensing projection is performed on the target's 3D bounding box, projecting each face of the target's 3D bounding box independently onto the image plane, preserving depth information, which can improve the accuracy of position control, and feature extraction is performed to obtain position features.
[0054] Optional, such as Figure 4 As shown, the above-mentioned feature extraction of depth information corresponding to the target's 3D bounding box to generate the positional features of the target object may include: S401: Project and fuse multiple faces of the target's 3D bounding box to obtain a depth feature map.
[0055] In one specific embodiment, each face (front, back, left, right, top, and bottom) of the target 3D bounding box is processed separately, generating a single-channel image for each face. The pixel value is the depth of the corresponding point. That is, all 3D points (x, y, z) of a face are projected onto 2D pixel coordinates (u, v), and the depth value z is assigned to the pixel intensity. The single-channel depth images of the six faces are stitched together along the channel dimension to form a 6-channel depth feature map.
[0056] S403: Perform multi-scale feature extraction on the depth feature map to obtain multi-scale depth features.
[0057] Specifically, a residual structure with L layers is used to extract multi-scale features fk (k=0,…,L-1) of the deep feature map, corresponding to different layers of U-Net.
[0058] S405: Match multi-scale depth features in both spatial and temporal dimensions to obtain location features.
[0059] Specifically, the temporal and spatial adaptation network modules input the data into the spatial and temporal residual modules of U-Net to guide the target's position adjustment. Specifically, the spatial adaptation module processes the position feature fk using 2D convolution (Conv2d), layer normalization (LayerNorm), and SiLU activation function before inputting it into the spatial residual module of the video U-Net; the temporal adaptation module permutes the frame and channel dimensions of the position feature fk, processes it using 3D convolution (Conv3d), and then injects it into the temporal residual module of the video U-Net.
[0060] In the above embodiments, the spatial and temporal adaptation modules enable the input features to adapt to the spatial resolution and temporal dimension of U-Net, ensuring the accuracy of the position at different scales and maintaining motion consistency between image frames.
[0061] In some embodiments, when the editing task is a repositioning task, the above-mentioned feature extraction of the depth information corresponding to the target 3D bounding box to generate the position features corresponding to the target object may include: Feature extraction is performed on the depth information corresponding to the three-dimensional bounding box of the second target to generate the position features of the target object.
[0062] Specifically, when the editing task is a relocation task, the position features are related to the 3D bounding box (i.e., the second target 3D bounding box) used to indicate the updated position of the target object. Only the depth information corresponding to the second target 3D bounding box needs to be extracted to obtain the position features of the target object.
[0063] S111: Input the first appearance feature, the second appearance feature, and the position feature into the image generation network for image generation processing to obtain the target image corresponding to the image to be processed.
[0064] In one specific embodiment, the image generation network can be a diffusion network. The diffusion network integrates location and appearance features, achieving multimodal fusion through cross-attention and residual connections. Starting from random noise, it gradually generates high-quality, temporally consistent images through a denoising process. Optionally, the image generation network can be trained on a pre-defined diffusion network. Specifically, through random occlusion and reconstruction tasks, the network is forced to learn to recover occluded objects using reference images (appearance cues) and target 3D bounding boxes (location cues), and to learn to delete objects using background inpainting data.
[0065] In an optional embodiment, when the editing task is the deletion task described above, the method may further include: Obtain the image to be processed and the 3D bounding box of the first target; The mask region corresponding to the 3D bounding box of the first target is fused with the image to be processed to obtain the updated image and the mask image; the edit region is the region that needs to be deleted and repaired. Background features are generated based on preset blank features, updated images, and mask images; Feature extraction is performed on the depth information corresponding to the target's 3D bounding box to generate positional features; Background and location features are input into an image generation network for image generation processing to obtain the target image corresponding to the image to be processed.
[0066] In one specific embodiment, when the editing task is the aforementioned deletion task, only the first target 3D bounding box is needed to indicate the location of the target object in the image to be processed, i.e., the location to be deleted. No reference image is required, and the deletion task does not require appearance cues; it only needs to focus on background restoration. This is equivalent to replacing the appearance features of the object with the background features of the scene. The network can be guided to ignore the object to be deleted and focus on background restoration through a cross-attention mechanism. The deletion task does not need to generate a new object appearance; it needs to guide the model to remove objects in a specified region and fill the background. Therefore, the second appearance sub-feature in the aforementioned task is replaced with a preset blank feature without appearance meaning to avoid the model mistakenly generating objects.
[0067] Specifically, the deletion task does not require a reference image, therefore it does not trigger SV3D to generate multi-view features. In this task, the 3D information fusion module only relies on depth features to repair the background based on depth information, ensuring that the repaired area is consistent with the 3D geometry of the scene (such as depth continuity and road texture continuity). The process of obtaining the updated image and mask image, generating position features, and image generation processing can be found in the detailed steps of the aforementioned related processes, and will not be repeated here.
[0068] As can be seen from the technical solutions provided in the embodiments of this specification above, this specification first obtains the image to be processed, a reference image of the target object, and a target 3D bounding box; the image to be processed includes multiple frames, and the target 3D bounding box is used to indicate the position of the editing area in the image to be processed; the mask area corresponding to the target 3D bounding box is fused with the image to be processed to obtain an updated image with a mask and a mask image; the updated image is used to determine the editing area, and the mask image is used to identify the boundary of the editing area; based on the reference image, the updated image, and the mask image, a first appearance feature corresponding to the target object is generated; based on the multi-view image features corresponding to the reference image, a second appearance feature corresponding to the target object is generated; the depth information corresponding to the target 3D bounding box is used to extract features to generate the position features corresponding to the target object; the first appearance feature, the second appearance feature, and the position features are input into an image generation network for image generation processing to obtain the target image corresponding to the image to be processed. Therefore, by using 3D position control, multi-scale appearance information, and a unified task framework, the core deficiencies of traditional methods in terms of position accuracy, appearance consistency, multi-task versatility, and 3D information utilization are systematically solved. This improves image editing accuracy and consistency, enhances image generation quality, and thus increases the diversity and generalization of autonomous driving data, thereby improving the performance of downstream tasks (such as BEV target detection).
[0069] This invention also provides an image generation apparatus, such as... Figure 5 As shown, the device includes: The first acquisition module 510 is used to acquire the image to be processed, the reference image of the target object, and the target three-dimensional bounding box; the image to be processed includes multiple frames, and the target three-dimensional bounding box is used to indicate the position of the editing area in the image to be processed; The first fusion module 520 is used to fuse the mask region corresponding to the target 3D bounding box with the image to be processed to obtain an updated image with a mask and a mask image; the updated image is used to determine the editing region, and the mask image is used to identify the boundary of the editing region; The first appearance feature generation module 530 is used to generate a first appearance feature corresponding to the target object based on the reference image, the updated image and the mask image; The second appearance feature generation module 540 is used to generate a second appearance feature corresponding to the target object based on the multi-view image features corresponding to the reference image. The first position feature generation module 550 is used to extract features from the depth information corresponding to the target three-dimensional bounding box and generate position features corresponding to the target object. The first image generation module 560 is used to input the first appearance feature, the second appearance feature and the position feature into an image generation network for image generation processing to obtain a target image corresponding to the image to be processed.
[0070] Optionally, the target 3D bounding box includes the azimuth information of the target object, and the second appearance feature generation module includes: A viewpoint image feature determination unit is used to determine the viewpoint image features corresponding to the azimuth information from the multi-viewpoint image features corresponding to the reference image. The matching unit is used to match the viewpoint image features with the projection area of the target three-dimensional bounding box in the corresponding image to be processed to obtain the second appearance feature.
[0071] Optionally, the location feature generation module includes: The first fusion unit is used to project and fuse multiple faces of the target 3D bounding box to obtain a depth feature map; A multi-scale feature extraction unit is used to perform multi-scale feature extraction on the depth feature map to obtain multi-scale depth features; The first location feature determination unit is used to match the multi-scale depth features in the spatial and temporal dimensions respectively to obtain the location features.
[0072] Optionally, the first appearance feature generation module includes: The merging unit is used to merge the reference image and the updated image to obtain a merged image; The second fusion unit is used to fuse the merged image and the mask image and perform feature extraction processing to obtain the first appearance sub-feature; The third fusion unit is used to perform feature extraction and fusion processing on the reference image and the updated image respectively to obtain the second appearance sub-feature; The fourth fusion unit is used to fuse the first appearance sub-feature and the second appearance sub-feature to obtain the first appearance feature.
[0073] Optionally, the target 3D bounding box includes a first target 3D bounding box and a second target 3D bounding box, the reference image is obtained based on segmentation of the target object from the image to be processed, and the first fusion module includes: The first projection unit is used to project the first target three-dimensional bounding box and the second target three-dimensional bounding box respectively to obtain the first mask region corresponding to the first target three-dimensional bounding box and the second mask region corresponding to the first target three-dimensional bounding box. The first overlay unit is used to overlay the first mask region and the second mask region with the image to be processed to obtain the updated image; The first mask image determination unit is used to perform binarization processing on the image to be processed based on the second mask region to obtain the mask image.
[0074] Optionally, the location feature generation module includes: The second position feature determination unit is used to extract features from the depth information corresponding to the three-dimensional bounding box of the second target and generate the position features corresponding to the target object.
[0075] Optionally, the target 3D bounding box includes a first target 3D bounding box, the reference image is used to update the appearance of the target object in the image to be processed, and the first fusion module further includes: The second projection unit is used to project the three-dimensional bounding box of the first target to obtain the first mask region. The second overlay unit is used to overlay the first mask region with the image to be processed to obtain the updated image; The second mask image determination unit is used to perform binarization processing on the image to be processed based on the first mask region to obtain the mask image.
[0076] Optionally, the target 3D bounding box includes a second target 3D bounding box, the target object is the object to be inserted into the image to be processed, and the first fusion module further includes: The third projection unit is used to project the three-dimensional bounding box of the second target to obtain the second mask region. The third overlay unit is used to overlay the second mask region with the image to be processed to obtain the updated image; The third mask image determination unit is used to perform binarization processing on the image to be processed based on the second mask region to obtain the mask image.
[0077] Optionally, the device further includes: The second acquisition module is used to acquire the image to be processed and the three-dimensional bounding box of the first target. The second fusion module is used to fuse the mask region corresponding to the three-dimensional bounding box of the first target with the image to be processed to obtain the updated image and the mask image; the edited region is the region that needs to be deleted and repaired. The background feature generation module is used to generate background features based on preset blank features, the updated image, and the mask image; The second location feature generation module is used to extract features from the depth information corresponding to the target three-dimensional bounding box and generate the location features; The second image generation module is used to input the background features and the location features into the image generation network for image generation processing to obtain the target image corresponding to the image to be processed.
[0078] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.
[0079] Figure 6 This is a block diagram illustrating an electronic device for image generation according to an exemplary embodiment. The electronic device may be a terminal, and its internal structure diagram may be as follows: Figure 6 As shown, the electronic device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage medium. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements an image generation method. The display screen can be a liquid crystal display (LCD) or an e-ink display. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the device's casing, or an external keyboard, touchpad, or mouse.
[0080] Figure 7 This is a block diagram illustrating an electronic device for image generation according to an exemplary embodiment. The electronic device may be a server, and its internal structure diagram may be as follows: Figure 7As shown, the electronic device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage medium. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements an image generation method.
[0081] Those skilled in the art will understand that Figure 6 or Figure 7 The structures shown are merely block diagrams of some structures related to the disclosed solutions of this invention, and do not constitute a limitation on the electronic devices to which the disclosed solutions of this invention are applied. Specific electronic devices may include more or fewer components than those shown in the figures, or combine certain components, or have different component arrangements.
[0082] In an exemplary embodiment, an electronic device for image generation is also provided, including a processor and a memory, wherein the memory stores at least one instruction, which is loaded and executed by the processor to implement the image generation method disclosed in the present invention.
[0083] In an exemplary embodiment, a computer-readable storage medium is also provided, which stores at least one instruction, which is loaded and executed by a processor to implement the image generation method disclosed in this invention.
[0084] In an exemplary embodiment, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform the image generation method disclosed in this invention.
[0085] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.
[0086] Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles disclosed herein and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the following claims.
[0087] It should be understood that the present invention is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the present invention is limited only by the appended claims.
Claims
1. An image generation method, characterized in that, include: Acquire the image to be processed, the reference image of the target object, and the target's 3D bounding box; The image to be processed includes multiple frames, and the target 3D bounding box is used to indicate the location of the editing region in the image to be processed. The masked region corresponding to the target 3D bounding box is fused with the image to be processed to obtain an updated image with a mask and a masked image; the updated image is used to determine the edited region, and the masked image is used to identify the boundary of the edited region; Based on the reference image, the updated image, and the mask image, a first appearance feature corresponding to the target object is generated; Based on the multi-view image features corresponding to the reference image, a second appearance feature corresponding to the target object is generated; The depth information corresponding to the target 3D bounding box is extracted to generate the position features corresponding to the target object; The first appearance feature, the second appearance feature, and the position feature are input into an image generation network for image generation processing to obtain the target image corresponding to the image to be processed.
2. The method according to claim 1, characterized in that, The target 3D bounding box includes the azimuth information of the target object, and the step of generating the second appearance feature corresponding to the target object based on the multi-view image features corresponding to the reference image includes: From the multi-view image features corresponding to the reference image, determine the view image features corresponding to the azimuth information; The second appearance feature is obtained by matching the viewpoint image features with the projection area of the target 3D bounding box in the corresponding image to be processed.
3. The method according to claim 1, characterized in that, The step of extracting features from the depth information corresponding to the target 3D bounding box to generate the position features corresponding to the target object includes: Project and fuse multiple faces of the target's 3D bounding box to obtain a depth feature map; Multi-scale feature extraction is performed on the depth feature map to obtain multi-scale depth features; The location features are obtained by matching the multi-scale depth features in both the spatial and temporal dimensions.
4. The method according to claim 1, characterized in that, The step of generating the first appearance feature corresponding to the target object based on the reference image, the updated image, and the mask image includes: The reference image and the updated image are merged to obtain a merged image; The merged image and the mask image are fused and feature extracted to obtain the first appearance sub-feature; Feature extraction and fusion processing are performed on the reference image and the updated image respectively to obtain the second appearance sub-feature; The first appearance sub-feature and the second appearance sub-feature are fused to obtain the first appearance feature.
5. The method according to any one of claims 1 to 4, characterized in that, The target 3D bounding box includes a first target 3D bounding box and a second target 3D bounding box. The reference image is obtained based on segmentation of the target object from the image to be processed. The process of fusing the mask region corresponding to the target 3D bounding box with the image to be processed to obtain an updated image with a mask and a mask image includes: Projection processing is performed on the first target 3D bounding box and the second target 3D bounding box respectively to obtain the first mask region corresponding to the first target 3D bounding box and the second mask region corresponding to the first target 3D bounding box; The first mask region and the second mask region are superimposed on the image to be processed to obtain the updated image; Based on the second mask region, the image to be processed is binarized to obtain the mask image.
6. The method according to claim 5, characterized in that, The step of extracting features from the depth information corresponding to the target 3D bounding box to generate the position features corresponding to the target object includes: Feature extraction is performed on the depth information corresponding to the three-dimensional bounding box of the second target to generate the position features corresponding to the target object.
7. The method according to any one of claims 1 to 4, characterized in that, The target 3D bounding box includes a first target 3D bounding box. The reference image is used to update the appearance of the target object in the image to be processed. The process of fusing the mask region corresponding to the target 3D bounding box with the image to be processed to obtain an updated image with a mask and a mask image includes: The first target's three-dimensional bounding box is projected to obtain the first mask region; The first mask region is superimposed on the image to be processed to obtain the updated image; Based on the first mask region, the image to be processed is binarized to obtain the mask image.
8. The method according to any one of claims 1 to 4, characterized in that, The target 3D bounding box includes a second target 3D bounding box, the target object is the object to be inserted into the image to be processed, and the process of fusing the mask region corresponding to the target 3D bounding box with the image to be processed to obtain an updated image with a mask and a mask image includes: The second target's three-dimensional bounding box is projected to obtain the second mask region; The second mask region is superimposed on the image to be processed to obtain the updated image; Based on the second mask region, the image to be processed is binarized to obtain the mask image.
9. The method according to claim 1, characterized in that, The method further includes: Obtain the image to be processed and the 3D bounding box of the first target; The mask region corresponding to the three-dimensional bounding box of the first target is fused with the image to be processed to obtain the updated image and the mask image; the edited region is the region that needs to be deleted and repaired. Based on the preset blank features, the updated image, and the mask image, background features are generated; The depth information corresponding to the target 3D bounding box is used to extract features to generate the position features; The background features and the location features are input into the image generation network for image generation processing to obtain the target image corresponding to the image to be processed.
10. An image generation apparatus, characterized in that, The device includes: The first acquisition module is used to acquire the image to be processed, the reference image of the target object, and the target 3D bounding box; the image to be processed includes multiple frames, and the target 3D bounding box is used to indicate the position of the editing area in the image to be processed; The first fusion module is used to fuse the mask region corresponding to the target 3D bounding box with the image to be processed to obtain an updated image with a mask and a mask image; the updated image is used to determine the editing region, and the mask image is used to identify the boundary of the editing region; The first appearance feature generation module is used to generate a first appearance feature corresponding to the target object based on the reference image, the updated image and the mask image; The second appearance feature generation module is used to generate a second appearance feature corresponding to the target object based on the multi-view image features corresponding to the reference image. The location feature generation module is used to extract features from the depth information corresponding to the target three-dimensional bounding box and generate the location features corresponding to the target object. The image generation module is used to input the first appearance feature, the second appearance feature and the position feature into the image generation network for image generation processing to obtain the target image corresponding to the image to be processed.