Image processing using scene maps

Abstract

A system and method for image processing includes retrieving an original image, where the original image includes an object; and generating a scene graph for the original image. A modified image is created using the generated scene graph by comparing the scene graph with at least one candidate scene graph that describes different relationships between the objects in the original image. And after determining different posture relationships and image blocks between the original image and the new modified image, rendering the modified image according to the different relationships defined in the at least one candidate scene graph.

Description

TECHNICAL FIELD

[0001] The present invention relates generally to creating new images or modifying images using a scene graph. BACKGROUND

[0002] Due to the continuous improvement of computational photography technology, modern mobile devices are capable of generating high-quality images. This has given rise to the demand for image modification functions by users. For example, a user can wish to enhance the visual appeal of an image because an image taken by a smartphone can not have visual appeal due to factors such as lighting conditions, color balance, or composition. Image processing can include correcting defects such as red-eye, blur, noise, or adding creative effects such as filters, text overlays, or other graphical elements. In other cases, image processing can include rearranging image elements, or adding or deleting image elements.

[0003] A neural processing unit (NPU) is a microprocessor specialized for accelerating machine learning algorithms, typically by operating on a predictive model such as an artificial neural network (ANN) or a random forest (RF). NPUs can be integrated into mobile devices, making it feasible to utilize many AI-based photo enhancement algorithms (e.g., image inpainting, glare removal, deblurring, and denoising). As a result, techniques have been developed to utilize recorded videos and dynamic images to train robotic NPUs for object processing. SUMMARY

[0004] One general aspect includes a computer-implemented method for rendering an image. The computer-implemented method further includes: accessing an original image, wherein the original image contains objects; generating a scene graph for the original image, wherein the scene graph describes relationships between the objects in the original image; generating one or more candidate scene graphs, wherein each of the one or more candidate scene graphs has one or more alternative object relationships for one or more objects in the original image; rendering a modified image using the candidate scene graphs by creating objects based on the objects in the original image and the alternative object relationships in the candidate scene graphs, including: relocating at least a portion of one or more objects in the original image in the modified image; generating image regions in the modified image where objects in the original image have changed. Implementation may include the computer-implemented method, wherein generating the scene graph may include: generating the scene graph of the original image using a neural model for scene graph extraction. Implementation may include the computer-implemented method described in any of the above embodiments, wherein generating the scene graph may include: determining object bounding boxes for each object in the image; segmenting each object in the image. The implementation can include the computer implementation method described in any of the above embodiments, wherein each candidate scene graph can include a scene graph for accessing object bounding boxes associated with objects in the candidate scene graph, the bounding boxes being obtained from the neural model. The implementation can include the computer implementation method described in any of the above embodiments, wherein the rendering can include: predicting object layout using a graph neural network. The implementation can include the computer implementation method described in any of the above embodiments, wherein the rendering can include: predicting object poses in the modified image based on the at least one candidate scene graph, wherein the predicted object poses may differ from the object poses in the original image if necessary, based on object relationships in the modified image. The implementation can include the computer implementation method described in any of the above embodiments, wherein generating image patches includes: filling the modified image with regions where objects in the original image are located in new positions in the modified image, wherein the new positions are different from the positions of the objects in the original image, based on object relationships in the modified image. The implementation method may include the computer implementation method described in any of the above embodiments, wherein generating image blocks for the modified image includes: repositioning objects in the original image in the modified image by using an image inpainting neural model to fill the regions of holes formed in the original image.Implementations can include the computer-implemented method of any of the above embodiments, further comprising fusing the generated image region with the bounding box edges into the hole in the original image to create the modified image in the generation module with style-based loss. Implementations of the described technology can include hardware, a method or process, or computer software on a computer-accessible medium.

[0005] Another general aspect includes a user device. The user device includes a storage medium that can include computer instructions. The device also includes an image capture system and a display device. The device also includes one or more processors communicatively coupled with the storage medium, wherein the one or more processors execute the instructions to cause the system to: capture an original image, the original image including objects in the image; generate a scene graph for the original image, wherein the scene graph describes relationships between the objects in the original image and locations of the objects in the original image; generate one or more candidate scene graphs, wherein each candidate scene of the one or more candidate scenes has one or more alternative object relationships for one or more of the objects in the original image; render a modified image with the candidate scene graph by creating objects from the objects in the original image and the alternative object relationships in the candidate scene graph, including: repositioning at least a portion of one or more objects in the original image in the modified image; generating an image region in the modified image where an object in the original image is changed.

[0006] Implementations can include a user device, wherein the one or more processors execute instructions to generate a scene graph includes generating a scene graph of the original image using a neural model for scene graph extraction. Implementations can include the device of any of the above embodiments, wherein the one or more processors execute instructions to generate a scene graph by determining an object bounding box for each object in the original image and segmenting each object in the original image. Implementations can include the device of any of the above embodiments, wherein each candidate scene graph can include a scene graph for accessing object bounding boxes associated with objects in the candidate scene graph, the bounding boxes obtained from the neural network. Implementations can include the device of any of the above embodiments, wherein the one or more processors execute instructions to predict object layouts using a graph neural model. Implementations can include the device of any of the above embodiments, wherein the one or more processors execute instructions to predict object poses of the modified image from the candidate scene graph, wherein the predicted object poses are different from object poses in the original image according to object relationships in the modified image. Implementations can include the device of any of the above embodiments, wherein the one or more processors execute instructions to generate image patches to fill areas in the original image where objects are in new locations in the modified image, wherein the new locations are different from locations of the objects in the original image as necessary according to object relationships in the modified image. Implementations can include the device of any of the above embodiments, wherein the one or more processors execute instructions to generate image patches for the modified image by repositioning objects in the original image in the modified image using an image inpainting neural model for holes formed in the original image. Implementations can include the device of any of the above embodiments, wherein the one or more processors execute instructions to fuse generated image regions with bounding box edges into holes in the original image to create the modified image with style-based loss in the generation module.

[0007] Yet another aspect includes a non-transitory computer-readable medium storing computer instructions for rendering an image. The non-transitory computer-readable medium storing computer instructions causes one or more processors to perform the steps of: accessing an original image, wherein the image contains objects; generating a scene graph for the original image, wherein the scene graph describes relationships between the objects in the original image; generating one or more candidate scene graphs, wherein each of the one or more candidate scenes has one or more alternative object relationships for one or more of the objects in the original image; rendering a modified image with the candidate scene graphs by creating objects from the objects in the original image and the alternative object relationships in the candidate scene graphs, including: repositioning at least a portion of one or more objects in the original image in the modified image; generating an image area in the modified image where an object in the original image has changed.

[0008] Implementations can include a non-transitory computer-readable medium, where the instructions cause the one or more processors to perform the step of generating a scene graph can include generating a scene graph of the original image using a neural model for scene graph extraction. Implementations can include the non-transitory computer-readable medium of any of the preceding embodiments, where the instructions cause the one or more processors to perform the step of generating a scene graph can include determining an object bounding box for each object in the original image; and segmenting each object in the original image. Implementations can include the non-transitory computer-readable medium of any of the preceding embodiments, where each candidate scene graph can include a scene graph for accessing object bounding boxes associated with objects in the candidate scene graph, the bounding boxes obtained from the neural model. Implementations can include the non-transitory computer-readable medium of any of the preceding embodiments, where the instructions cause the one or more processors to perform the rendering step can include predicting object layouts using a graph neural network based on the neural model. Implementations can include the non-transitory computer-readable medium of any of the preceding embodiments, where the instructions cause the one or more processors to perform the rendering step by predicting object poses in the modified image from the candidate scene graph, where the object poses differ from object poses in the original image as necessary according to object relationships in the modified image. Implementations can include the non-transitory computer-readable medium of any of the preceding embodiments, where rendering a modified image can further include the instructions causing the one or more processors to perform the step of generating image patches, where objects in the original image are in new locations in the modified image, where the new locations differ from locations of the objects in the original image as necessary according to object relationships in the modified image. Implementations can include the non-transitory computer-readable medium of any of the preceding embodiments, where the generating image patches for the original image forming holes in the modified image includes repositioning objects in the original image in the modified image using an image inpainting neural model. Implementations can include the non-transitory computer-readable medium of any of the preceding embodiments, where the instructions cause the one or more processors to perform the step of fusing generated image regions with bounding box edges into holes in the original image to create the modified image with style-based losses in the generation module. Implementations of the described technology can include hardware, a method or process, or computer software located on a computer-readable medium.

[0009] This summary introduces a selection of concepts, which will be further described below in the detailed description. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended for use in determining the scope of the claimed subject matter. The claimed subject matter is not limited to implementations that solve any or all of the disadvantages noted in the background. BRIEF DESCRIPTION OF DRAWINGS

[0010] Aspects of the application are illustrated by way of example and not limitation in the figures of the accompanying drawings, in which like references indicate similar, or like, elements.

[0011] Figure 1 An application of image processing techniques is shown; Figure 2 is a flowchart of an advanced method according to image processing techniques; Figure 3 is a flowchart of one embodiment of a method for processing one or more images and one embodiment for implementing steps 230, 240 and 250 in Figure 2 Figure 4 Candidate bounding boxes in the original and modified images are shown; Figure 5A and Figure 5B show scene graphs of the original and modified images, respectively; Figure 6A is a block diagram of predicting a new object layout from a given candidate scene graph using a GNN-based model; Figure 6B is a sequence of images showing image inpainting; Figure 7A is a flowchart of another embodiment of a method for processing one or more images and one embodiment for implementing steps 230, 240 and 250 in Figure 2 Figure 7B , Figure 7C and Figure 7D show user interfaces for the method in Figure 7A Figure 8A is a flowchart of another embodiment of a method for processing one or more images and one embodiment for implementing steps 230, 240 and 250 in Figure 2 Figure 8B show user interfaces for the method in Figure 8A Figure 9 is a flowchart of another embodiment of a method for processing one or more images and one embodiment for implementing steps 230, 240 and 250 in Figure 2 ​​​​​a flowchart of one embodiment of steps 230, 240, and 250 in FIG. 2; Figure 10 is a block diagram of a mobile processing device that can be used to implement various embodiments of the technology described herein; Figure 11 is a block diagram of an example of a network processing device that can be used to implement various embodiments of the technology described herein. DETAILED DESCRIPTION

[0012] The present invention and embodiments relate to image processing techniques that enable end users to modify existing images or generate new images from existing images by processing objects in the images. The disclosed embodiments include a method that takes an original image and creates a scene graph for the image, where the image contains objects. When a modification is to be made to the image, the scene graph, which describes relationships between objects in the image, is compared to one or more candidate scene graphs that describe different relationships between at least some of the objects in the original image. The technique then selects a different arrangement of objects in the image and renders a new, modified image according to the different relationships defined in the candidate scene graph.

[0013] One application of image processing is to generate one or more new images from an original image and a user's intent for changes to the image. Image processing has various applications in current mobile devices, including providing additional functionality for photo albums on the device, etc. Image processing can also be used to generate "goal" images for training algorithms for robots. Modern robot training methods employ imitation learning and reinforcement learning, among other ways, in which control algorithms are not pre-programmed but are trained using existing video and image data. The algorithms automatically learn to process by exploring the action search space according to knowledge acquired from human actions. Generating a goal image with a scene graph (e.g., "apple on table") from an original image ("apple under table") can provide useful information for training algorithms. For example, some modern robot control algorithms based on machine learning are trained in an end-to-end fashion: only the source and goal images are provided, and the algorithm needs to learn the intermediate operations to accomplish a particular task. Using the techniques described herein, generating new images from scene graphs extracted from human instructions can be used to train these types of algorithms. Although the techniques introduced herein will be described with examples of creating new or "modified" images, it should be apparent that the techniques have applications beyond such uses, including the goal images for robot training discussed above.

[0014] Figure 1 An application of image processing is shown. Figure 1A first image 110 is shown, which in this embodiment includes a rider 112 astride a horse 114. The image can include a background 116, which includes the ground 118 on which the horse 114 is standing. The rider 112 in the image 110 is holding the reins of the horse 114, which are tied to the bridle. After processing 100 by the techniques disclosed herein, in one embodiment, the pose and position of the rider 112 are both repositioned, and the rider 112 is now standing on the ground 118 next to the horse 114 and holding the reins.

[0015] Figure 2 is a flowchart of an advanced method according to the image processing techniques. The method can be performed on a mobile device, a network processing device, and / or a service host processing device, where the service host processing device includes one or more servers (e.g., the server shown), which can be operated by a mobile device user, or by a service provider that provides the image processing techniques disclosed herein as a service to mobile device users or network device users. Figure 11 The flow is initiated in 210. In embodiments, the user selects an image to be processed to initiate the flow. In other embodiments, the processing device initiates the flow automatically according to image criteria (e.g., images taken at a particular location, or images of a particular subject matter).

[0016] In 220, one or more images are accessed by the method. The images can be stored on the mobile device, on a network device, or on a service host. In embodiments described herein, the images can be captured by the mobile device, and are therefore described as being stored on the mobile device. In embodiments herein, the images can be accessed by retrieving the images from a storage device directly from an image capture system, such as a camera. In 230, the accessed one or more images are analyzed, and a scene graph is generated for the images. As discussed herein, the scene graph generated for the accessed images is compared to an input scene graph and / or a user-modified input scene graph to perform image processing. The process of generating a scene graph in 230 is described below in connection with Figure 3 and Figure 5A / Figure 5BThe modified image is created in 240 based on analysis of the scene graph generated for the original image and one or more candidate scene graphs for the new modified image based on the original image. The process includes creating one or more candidate scene graphs that modify the position and / or relationship of objects and comparing the scene graph to the one or more candidate scene graphs. The one or more candidate scene graphs describe different relationships between at least some of the objects in the original image. In 250, the processed image is rendered. Step 250 can include rendering one or more modified images or alternative images. The modified images described herein can be images in which one or more objects are not repositioned, but are refined or removed from the image. The alternative images described herein include images in which one or more objects in the image have been repositioned and / or changed in pose according to the determined relationships between the images and / or user input from a user interface that enables the user to manipulate the scene graph of the image, or to select one or more alternative images, or both. In 260, the one or more modified images or alternative images are output to a storage device or other output device.

[0017] While the techniques herein will be discussed in terms of processing of one image, it should be recognized that multiple images can be processed in parallel or serially at each step.

[0018] Figure 3 One embodiment of a method for processing one or more images is shown, as well as one embodiment for implementing steps 230, 240, and 250 in Figure 2 In 310, object instances in the image are segmented and bounding boxes are determined. Bounding box prediction and image segmentation employ classical deep learning techniques of computer vision. Figure 4 Candidate bounding boxes in image 110 are shown: box 402 for the person, box 404 for the horse; box 406 for the reins, and box 408 for the ground. In an embodiment, step 310 is performed by a neural network.

[0019] At 310, any of a variety of mature neural networks can be used to predict the image bounding boxes. For example, a Region-Based Convolutional Neural Network (R-CNN) is a method that uses a selective search to extract a fixed number of regions from an image. The fixed number of region proposals generated using a known selective search algorithm are classified. These candidate region proposals are warped into squares and fed into a convolutional neural network that generates a 4096-dimensional feature vector as output. The convolutional neural network (CNN) acts as a feature extractor, and the output dense layer is composed of features extracted from the image that are fed into a support vector machine to classify the presence of an object in the candidate region proposal. Another alternative is Fast R-CNN. Fast R-CNN is similar to the R-CNN algorithm, but instead of feeding the region proposals into the CNN, the input image is fed into the CNN to generate a convolutional feature map. The proposed regions are identified from the convolutional feature map and warped into squares, which are reshaped to a fixed size by using a Region of Interest (RoI) pooling layer to be fed into a fully connected layer. The proposed regions are predicted for class and the offset values of the bounding box from the RoI feature vector.

[0020] Another alternative is Faster R-CNN, where the image is provided as input to a CNN to generate a convolutional feature map, but instead of using a selective search algorithm on the feature map to identify region proposals, a separate network is used to predict the region proposals. Yet another alternative is a You Only Look Once (YOLO) object detection, where a single convolutional network predicts the bounding boxes and class probabilities of these boxes by taking an image and dividing the image into an SxS grid (with m bounding boxes within each grid). For each of these bounding boxes, the network outputs the class probability and offset values of the bounding box, and values above a threshold are selected and used to localize the object within the image. Any of the above neural networks can be used to implement step 310.

[0021] The image segmentation in step 310 can be performed by any of a variety of neural models for image segmentation, including FastFCN, Gated-SCNN, DeepLab, and MaskRCNN. For example, FastFCN is based on a Fully Convolutional Network (FCN) architecture and employs dilated convolutions to capture multi-scale contextual information. FastFCN achieves high accuracy while being computationally efficient, making it suitable for real-time applications. Gated Shape CNN (Gated SCNN) uses a gated convolutional neural network to selectively suppress irrelevant feature maps and enhance important ones. This helps the model focus on relevant information and achieve better segmentation results. DeepLab is a series of models for semantic image segmentation. DeepLab combines the use of atrous (dilated) convolutions, spatial pyramid pooling, and fully connected conditional random fields, resulting in state-of-the-art performance on various segmentation benchmarks. MaskRCNN extends the Faster R-CNN model by adding a mask branch that, in addition to predicting bounding boxes and class labels, can also predict object masks. This enables the model to perform both object detection and segmentation, making it very useful for image editing. Any of the above methods for obtaining image segmentation can be used to move object blocks to new locations while keeping the rest of the image unchanged.

[0022] Returning to Figure 3 In 315, the input image is analyzed by a scene graph extraction technique based on a graph neural network to generate a scene graph for the input image. The scene graph is a structured representation of the image, with nodes in the scene graph corresponding to object bounding boxes and their object classes, and edges corresponding to pairwise relationships between objects. An example of a scene graph is shown in Figure 5A and Figure 5B The scene graph generated in 315 has the highest relevance to the image evaluated in 210. Once objects are detected by image segmentation in 310, scene graph generation begins, at which point objects are classified into different categories or classes. For example, a deep learning model is trained on a large dataset of labeled images to assign each object a label representing its semantic class. Relationships between objects are then determined by analyzing their spatial arrangement, relative position, and interactions in the scene. For example, it is recognized that a “person” is “holding” a “rein,” or that a “saddle” is “on” a “horse.” Graph construction is achieved by connecting objects and their relationships to nodes representing the objects and edges representing relationships between the objects. Each node can contain information such as object label, bounding box coordinates, and possibly other attributes such as color or size. As Figure 5A and Figure 5BAs shown, edges between nodes capture these relationships, often labeled with descriptive phrases such as “on,” “in front of,” etc.

[0023] In one embodiment, the scene graph extraction employs a graph convolutional neural network (GNN)-based neural model to obtain a scene graph from the original image, such as by iterative message passing to generate the scene graph. The method of generating the scene graph by iterative message passing generates a visual representation of the scene from the objects and their relationships by iteratively passing messages between objects in the scene, refining the predicted relationships at each iteration. The GNN is used to encode the visual features of the objects and their pairwise relationships. The model also incorporates contextual information such as object co-occurrence statistics and spatial layout constraints to improve the accuracy of the predicted relationships. The iterative message passing method enables the model to capture complex and subtle relationships between objects, resulting in a more accurate generation of the scene graph.

[0024] In 320, one or more candidate scene graphs of the modified image are generated, and the scene graph generated for the input image in 315 is compared to the one or more candidate scene graphs. Generating the candidate scene graphs involves moving and / or changing the position and / or pose of one or more objects in the input image. Each candidate scene graph presents an alternative arrangement and / or object relationship of the objects in the input image. In one embodiment, one candidate scene graph is selected so that the arrangement of the objects in the input image is changed according to the arrangement defined in the candidate scene graph, providing the basis for creating an alternative image of the input image. Various selection methods can be employed to select one or more candidate scene graphs to generate the alternative image. For example, the candidate scene graphs can be scored according to the number of objects that match the objects in the obtained scene graph that can be rearranged according to the candidate scene graph, and the highest scoring candidate scene graph is selected for use in 320. The candidate scene graphs can include one or more objects that are common to the scene graph obtained based on the input image, and can represent alternative arrangements of the objects. The GNN-based model can be used to create the candidate scene graphs. In other embodiments discussed below, the user can be presented with one or more candidate scene graphs through a user interface, providing the user with different alternatives for the alternative image. For example, in one embodiment, the candidate scene graphs are presented to the user through a user interface, and the user can select one of the candidate scene graphs to be used in 320 to create the alternative image. Figure 4 、 Figure 5A and Figure 5B , Figure 5A The scene graph obtained from the image 110 shown depicts a rider astride a horse, holding the horse’s reins, which are tied to the horse’s bridle. Figure 5A and Figure 5B show the scene graphs of the original image and the modified image, respectively.

[0025] In 320, the method can compute that the "best" replacement image based on the objects and their locations in the original image is the jockey standing next to the horse, holding the reins to hold the horse (as shown in image 120, whose scene graph is shown in Figure 5B As shown in Figure 5A and Figure 5B The objects identified and classified in block 402 (for the person), block 404 (for the horse), block 406 (for the horse's halter), and block 408 (for the ground) are reordered from Figures 5A to 5B The jockey block 402 has been repositioned, with the jockey pose adjusted to 402a; the horse block 404 has been repositioned, with the horse pose adjusted to 404a; the horse's halter block 406 has been repositioned, with the pose adjusted to 406a.

[0026] To generate the image shown in Figure 5B In 325, the object layout is predicted from the selected candidate scene graph. Initially, the candidate scene graph is parsed to understand the relationships and attributes of each object, which are used in the rendering process to render each object in its correct position and with its correct properties. As shown in Figure 6A A GNN-based network is used to predict a new object layout from a given candidate scene graph. The GNN-based network learns and extracts candidate graph embeddings, which encode the layout of objects in an image. The GNN-based network is trained using pairs of scene graphs and images with annotated bounding boxes from a publicly available dataset, such as Visual Genome. Visual Genome is a dataset that connects the relationships between objects in an image that need further inference. To learn and predict new scene graph-based object layouts using the GNN-based network, a dataset of scene graphs and their corresponding object layouts is needed, where each scene graph represents a scene containing a set of objects and their relationships, and each object layout should represent the position and size of the objects in the scene. The GNN learns the spatial relationships between objects in a scene and uses this information to predict object positions. The GNN is trained on a dataset of scene graphs and object layouts. During training, the GNN learns to map each scene graph to its corresponding object layout by passing messages between nodes and updating node representations at each iteration.

[0027] In the context of object layout prediction, candidate graphs represent objects and their relationships in an image and learn abstract graph embeddings that encode spatial relationships between objects. GNNs learn to encode spatial relationships between objects in abstract graph embeddings in the process of iteratively updating node representations. Such embeddings can be used to predict the position and size of objects in an image. After training and evaluation, the GNN model can be used to predict new object layouts from input scene graphs. Given a new scene graph, the GNN will output a predicted object layout that can be used to visualize the scene.

[0028] At 330, any new poses of people and objects are predicted using a neural model, such as an attention-based model or a convolutional network, based on the relevance of object poses and relationships. As shown in Figure 5A and Figure 5B poses when their relationships in a scene graph change. For example, the scene graph “person on horse” indicates that the person in the corresponding image is in a riding pose, while the scene graph “person standing next to horse” indicates that the person in the corresponding image is in a standing pose. The neural model is trained on a paired pose label and image dataset, where scene graphs can be obtained based on images. The model predicts different object or human poses with different object relationships. Both CNNs and attention-based neural networks can capture the relevance between poses and object relationships in an image by leveraging their ability to learn spatial relationships and patterns in data. In the context of object pose and relationship prediction, the goal is to predict the position, orientation, and relationships between objects in an image.

[0029] CNNs can learn to capture spatial relationships by using convolutional filters that scan the image at different positions and scales. These filters detect patterns and features related to object poses and relationships, such as edges, corners, and textures. By stacking multiple convolutional layers, CNNs can learn increasingly complex features, capturing more abstract concepts, such as the constituent parts of objects and their configurations. Attention-based models can capture spatial relationships by selectively focusing on different parts of an image. Attention mechanisms enable the model to focus on specific regions of the image that are most relevant to predicting object poses and relationships. By focusing on objects and their relationships, attention-based models can learn to capture more nuanced details about object layouts.

[0030] In addition to capturing spatial relationships, CNNs and attention-based models can leverage their ability to capture temporal relationships in sequences of images. Overall, both CNNs and attention-based models can capture the relevance between poses and object relationships in an image by leveraging their ability to learn spatial and temporal relationships in data.

[0031] In step 335, image patches are generated to repair image locations where objects have been removed or moved to new positions with new poses. Image patches involving scene graph changes can be generated using image generation models such as those based on generative adversarial networks (GANs) or diffusion-based models. Figure 6 illustrates image inpainting according to the technique described in this paper. To generate image patches, regions in the image where objects have been removed need to be identified and marked as holes. Figure 6B In the initial image 610, a rider image is shown. Figure 1 Riding on horse 114 like that. (and) Figure 1 Similarly, target image 640 shows a rider standing next to horse 114 and holding the reins. Since the rider object is moving, position 625 of the rider on horse 114 is marked as a hole. Image portions not involving scene graph changes are directly copied from the original image without any alterations. Next, the GAN generates image patches of the object in its new position and pose based on the new layout, thus performing image inpainting. Figure 6B In Figure 620, rider 112a is shown standing in a new posture, holding the reins of horse 114. In GAN-based models, image patches can be generated by training a generator network within the GAN, producing small patches instead of the entire image. This can be achieved by modifying the input and output sizes of the generator network. During training, the generator is trained to generate realistic image patches from a vector of random noise. The discriminator is trained to distinguish real image patches from fake patches generated by the generator. By training the generator and discriminator networks adversarially, the generator learns to generate realistic image patches, making it difficult for the discriminator to distinguish them from real patches. A similar approach can be used to generate image patches in diffusion-based models. Instead of using a generator network, this model applies a diffusion process to a given input image to generate new image patches. The diffusion process progressively adds noise to the input image over a series of time steps, and at each time step, a new image patch is generated by sampling from the distribution of a partially diffused image. The diffusion process can be repeated multiple times to generate multiple image patches from the same input image. The quality of the generated image patches can be improved by using diffusion-based models with more diffusion steps or by employing more complex diffusion processes.

[0032] In version 340, a pre-trained image instigation model is used to fill any holes left after moving objects according to the new layout. Specifically, an image with holes is passed to the image instigation model, which then generates a reasonable reconstruction for the missing areas. The image instigation model creates image regions for the modified image, in which objects or parts of objects from the original image have been removed. Figure 6BIn the middle, the hole 625 can be conceptually represented as the area 635 where the rider 112 occludes the horse and the area 645 where the rider occludes the background. The inpainting model can generate multiple possible reconstruction results for the one or more missing areas. The best reconstruction result is selected according to the visual quality or similarity to the original area, and the reconstructed area is merged with the rest of the image. Similar to the image generation in step 335, the image inpainting module used to fill the hole in step 340 can also be based on existing GAN-based generative models or diffusion-based models. Figure 6B In the middle, the hole 625 can be conceptually represented as the area 635 where the rider 112 occludes the horse and the area 645 where the rider occludes the background. The inpainting model can generate multiple possible reconstruction results for the one or more missing areas. The best reconstruction result is selected according to the visual quality or similarity to the original area, and the reconstructed area is merged with the rest of the image. Similar to the image generation in step 335, the image inpainting module used to fill the hole in step 340 can also be based on existing GAN-based generative models or diffusion-based models.

[0033] In 345, the generated image patch is merged with the original image at the boundary box edge using a style-based loss in the generation module. The generation module refines the edge area of the image patch and can be based on some loss, such as a style-based discriminator loss function. The boundary box edge or boundary of the generated image patch is extracted and a style-based loss is calculated to measure the similarity of the generated image patch to the original image in terms of style or visual appearance. There are multiple ways to calculate the style-based loss, such as using a deep convolutional neural network to extract features and comparing them using a perceptual loss or Gram matrix difference, etc. Subsequently, an optimization algorithm is used to iteratively adjust the pixel values within the boundary box area of the original image to merge this area with the generated image patch. After the optimization process is complete, the pixel values within the boundary box area of the original image are replaced with the optimized values. In the rendering sub-steps 335, 340 and 345 described above, rendering the image patch and image inpainting includes calculating the color and brightness of each pixel in the image (image patch or inpainting area) according to the light, object material, camera position, etc.

[0034] In 350, the image is output to a display, storage device or for further processing.

[0035] Figure 7A An alternative method to the method in Figure 3 is shown. In the method in Figure 7A the steps with the same reference numerals as in Figure 3 are exactly the same flow steps as in Figure 3 .

[0036] In Figure 7A , after step 315, a scene graph modification interface is provided to the user in 710. An example of the scene graph interface is shown in Figure 7B and Figure 7C . In Figure 7BIn this interface, users can select objects from a scene graph and use the interface to change the arrangement or relationships of objects by repositioning them and / or altering the relationships between them. In one embodiment, users can use a mobile device's touchscreen interface (such as...). Figure 7B and Figure 7C (As shown) You can change the relationships and links of objects by clicking object icons 1002 and 1004 and using a text interface such as the on-screen keyboard. Figure 7B In the text, the figures are "sitting" "on" "a saddle". Figure 7C In the game, the character is now "standing" next to the "horse," and the relationships and connections have changed to reflect this change. It should be understood that in... Figure 7B and Figure 7C In the example, the interface can take various graphical user interface forms and support input from various input / output devices. Although this interface is shown as a mobile phone touchscreen, it can be implemented with various processing devices, displays, and interaction methods (such as traditional mice and keyboards). Figure 7D As shown, this interface can be a voice interface (activated by pressing the microphone icon 795), allowing users to interact with the scene graph using natural language. The user may be prompted to "describe how you wish to change the scene" in natural language. In 720, user interaction with the scene graph is received. User interaction can alter the presented scene graph by rearranging objects or changing object relationships. In 740, instead of comparing the generated scene graph with the original scene graph obtained from the image, the changes between the two scene graphs are identified, and the generated scene graph is compared with the scene graph modified by the user. Subsequent steps of the method follow... Figure 3 The same applies. It should be noted that even when presenting the interface to the user in step 710, the user can choose to ignore the interaction. In this case, step 320 is executed instead of steps 720 and 740 above. The method then proceeds from... Figure 3 Continue with step 320.

[0037] Figure 8A This demonstrates an alternative to image processing techniques. Figure 8A In the method, with Figure 3 The steps in the figure that have the same reference numerals are the same as those in the figure. Figure 3 The steps are exactly the same as those in the process. Figure 8A In this method, the user is given the opportunity to choose between two or more alternative images that the system may generate. Therefore, this method... Figure 3steps 310, 315, and 320. Following step 320, in 810, the user is presented with an interface containing different alternative image options for the user to select from. In 820, user input is received selecting one of the alternative images, and in 830, the object layout is arranged according to the user selection using the graph neural network-based model according to the candidate scene graph and the user selection. Figure 8B An example interface is shown. Another alternative interface can employ a similar interface to that shown in Figure 8B , and employ natural language input as shown in Figure 7C . As shown in Figure 7A , the user can choose to ignore the interaction in step 810, in which case steps 820 and 830 are replaced by step 325, which continues the method from step 320 in Figure 3 .

[0038] Figure 9 Another alternative method is shown. In the method of Figure 9 , the steps with the same reference numerals as in Figure 3 , Figure 7A , and Figure 8A are exactly the same flow steps as in Figure 3 , Figure 7A , and Figure 8A . In Figure 9 , the method of Figure 8A and Figure 7A is combined, both pre-processing one or more scene graphs and selecting the final generated image.

[0039] Figure 10 One example of a typical architecture of a mobile device is depicted. Mobile processing device 1000 has memory 1010, physical connectors 1020, system processor 1040, neural processor 1042, input / output (I / O) controller 1050, cellular radio channel 1070, other communication channels 1062, and power controller 1090. Each of these components is connected by one or more system buses (not shown).

[0040] The memory 1010 coupled to processor 1040 and neural processor 1042 includes an operating system 1012, an image processing application 1014, and other applications 1013 for the mobile processing device. The memory 1010 can be various types of non-volatile and volatile memory, containing instructions for implementing the applications described herein. The operating system 1012 handles different operations of the mobile processing device 1000 and may include a user interface for operations such as making and receiving calls, sending text messages, and checking voicemail. Applications 1013 can be any type of program, such as a camera application for photos and / or videos, a contacts application, a calendar application, a media player, an internet browser, games, etc.

[0041] The non-volatile memory 1015 can be any form of computer-readable medium storing instructions that cause the system processor 1040 and the neural processor to perform the methods described herein. The non-volatile memory 1015 may also include stored versions of applications 1013 and 1014, image data 1016 (e.g., photos and videos), and candidate scene graphs stored in scene graph data structure 1018.

[0042] Operating system 1012 manages the hardware of mobile processing device 1000, including the display / touchscreen (occupying...) Figure 7B / Figure 7C The mobile processing device 1000 includes hardware such as a display area 750, a camera 1006, a speaker 1004, and a microphone 1002. The operating system 1012 also manages the software (i.e., applications 1013 and 1014) on the mobile processing device 1000 to perform user-requested tasks and process incoming data. This is achieved through the operating system's control and allocation of memory 1010, system tasks, system resources, file systems, etc. The system processor 1040 performs operations on the mobile processing device according to this control and allocation. The neural processor 1042 may be a dedicated processor that works in conjunction with image processing applications to implement the aforementioned CNN and GNN components. The NPU includes control and arithmetic logic components necessary for executing machine learning algorithms and accelerating the execution of the machine learning tasks described herein.

[0043] The power controller 1090 of the mobile processing device 1000 distributes the power of the mobile processing device power supply 1092 to the circuits used to operate the mobile processing device 1000 and its various mobile processing device components with different features.

[0044] In addition, the physical connector 1020 can be used to connect the mobile processing device 1000 to an external power source, such as an AC adapter or a powered docking station.

[0045] The mobile processing device 1000 also includes a cellular wireless channel 1070 for receiving and transmitting data, such as phone, text, email, web data, etc. Cellular wireless communication can be implemented through any standard network protocol for mobile device communication (i.e., GSM, PCS, D-AMPS, UMTS, CDMA, WCDMA, LTE, etc.). The mobile processing device 1000 can also include other communication channels 1062 for receiving and transmitting data, such as Wi-Fi, Bluetooth, etc. The mobile processing device 1000 can have other functional elements 1064 for communication, such as GPS. The communication medium for operation of the mobile processing device 1000 is not limited to the media described and can include any other communication medium known in the art.

[0046] It should be recognized that any suitable processing device (mobile processing device or other processing device) can implement the present technology. Thus, while Figure 10 A mobile device is shown, but similar to Figure 10 The components shown can also be provided in a general purpose processing device, such as a desktop computer, laptop, or server.

[0047] The cellular wireless channel 1070 can be a combination of a traditional 2G / 3G / 4G subsystem and a 5G communication subsystem.

[0048] The content shown in the memory 1010 as part of the image processing application 1014 also includes a selected image analyzer 1030 including a CNN 1031 that can include instructions for the system processor and neural processor to implement step 310 in the methods described herein; a scene graph generator 1032 and GNN 1033 that can include instructions for the system processor and neural processor to implement step 315 in the methods described herein; a scene graph comparator 1034 and CNN 1035 that can include instructions for the system processor and neural processor to implement steps 320 and 325 (and 740) in the methods described herein; a candidate scene graph generator 1036 and GNN 1037 that can include instructions for the system processor and neural processor to generate a candidate scene graph; a pose prediction module 1038 and GNN 1039 that can include instructions for the system processor and neural processor to implement step 330 in the methods described herein; an image patch generator 1041 and CNN 1043 that can include instructions for the system processor and neural processor to implement step 335; an image inpainting module 1045 and CNN 1047 that can include instructions for the system processor and neural processor to implement step 340 in the methods described herein. A renderer 1047 can also be included to cause the system processor to implement step 345. As shown below Figure 11 One or more of these image processing applications can be run on a service host processing device, as shown.

[0049] Figure 11 is a block diagram of a network processing device 1100 that can be used to implement various embodiments of network processing devices (including service hosts and the like) on which the methods described herein can run. Particular network devices can use all of the components shown or only a subset of the components, and levels of integration can vary from device to device. For example, a network device might be a stand-alone device with only a CPU 1610, or a device might be a software modem integrated with a host processor that includes the CPU 1610, NPU 1710, and storage such as the memory 1620 and file storage 1630.

[0050] The network device 1100 can include a central processing unit (CPU) 1610, an NPU 1710, a memory 1620, a mass storage device 1630, and an I / O interface 1660 connected to a bus 1670. The bus 1670 can be one or more buses from among any of several types of buses, including a memory bus or a memory controller, a peripheral bus, etc. The network interface 1650 enables the network processing device to communicate over the network 1680 with other processing devices (e.g., the processing devices described herein). The I / O interface can connect to a keyboard, mouse, touchscreen, display device, and / or image capture device.

[0051] The CPU 1610 can include any type of electronic data processor. The memory 1620 can include any type of system memory such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), read-only memory (ROM), a combination of

[0052] The mass storage device 1630 can include any type of storage device used to store data, programs and other information and make that data, programs and other information accessible via the bus 1670. For example, the mass storage device 1630 can include one or more of a solid state drive, a hard disk drive, a disk drive, an optical drive and the like. One or more stored applications (e.g., the applications 1013 and 1014 described above) can be maintained in the mass storage device and accessed and loaded by the CPU and NPU as necessary for execution.

[0053] It will be appreciated that the execution of the methods in Figure 3 、 Figure 7A 、 Figure 8A and Figure 9 may be shared between the mobile processing device and the network processing device 1100. In embodiments, any combination of steps 220, 230, 240, 250 and 260 can be performed in whole or in part on each such device, with data being transferred between the devices to share processing load.

[0054] For purposes of this document, it is to be understood that the figures shown herein are not necessarily drawn to scale.

[0055] For purposes of this document, reference to "a" or "one" embodiment shall not be construed to mean an exclusively singular implementation or configuration. Rather, "a" or "one" embodiment open up the use of both singular and plural aspects of the disclosed subject matter unless otherwise indicated by the context.

[0056] For purposes of this document, a connection between a first and a second device can be a direct connection or an indirect connection (e.g., through one or more other devices). In some cases, a first device can be said to be connected or coupled to a second device when the first device is directly connected to the second device or when the first device is indirectly connected to the second device through one or more intervening devices. When a first device is said to be directly connected to a second device, there are no intervening devices between the first device and the second device. Two devices are in "communication" if they are directly or indirectly connected and are capable of transferring electronic signals to one another.

[0057] While the application has been described with reference to particular features and embodiments, it will be apparent that various modifications and combinations can be made to the described embodiments without departing from the scope of the application. Accordingly, the description and drawings are to be regarded in an illustrative rather than a restrictive sense, and all such modifications of the application as can reasonably and equitably be expected within the scope of the application are intended to be embraced therein.

[0058] The techniques described herein can be implemented using hardware, software, or a combination of both. The software used to implement the techniques described herein can be stored on one or more processor-readable storage devices, such as the storage devices described above. The software can be programmed to cause one or more processors to perform the functions described herein. Processor-readable storage devices can include volatile and nonvolatile media, removable and non-removable media, and computer-readable media. By way of example, and not limitation, computer-readable media can include computer-readable storage media and communication media. Computer-readable storage media can be implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules, or other data. Examples of computer-readable storage media include RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by a computer. Computer-readable media does not include a propagated signal, modulated data signal, or a carrier wave.

[0059] Communication media typically embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The term "modulated data signal" refers to a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as RF and other wireless media. Combinations of the any of the above can also be included within the scope of computer-readable media.

[0060] In alternative embodiments, portions of the software can be replaced by dedicated hardware logic components. Illustrative types of hardware logic components that can be used include, but are not limited to, Field-programmable Gate Arrays (FPGAs), Application-specific Integrated Circuits (ASICs), Application-specific Standard Products (ASSPs), System-on-a-chip (SOCs), Complex Programmable Logic Devices (CPLDs), dedicated computer systems, etc. In one embodiment, software (stored in a storage device) implementing one or more embodiments is used to program one or more processors. The one or more processors can be in communication with one or more computer-readable media / storage devices, peripheral devices, and / or communication interfaces.

[0061] It should be understood that the present subject matter can be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and fully convey the scope of the subject matter to those skilled in the art. Indeed, the subject matter is intended to cover alternatives, modifications and equivalents of these embodiments, including practices which fall within the scope and spirit of the subject matter as defined by the following claims. Furthermore, in the following detailed description of the subject matter, numerous specific details are set forth in order to provide a thorough understanding of the subject matter. However, it will be clear to those of ordinary skill in the art that the subject matter can be practiced without these specific details.

[0062] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0063] The description of the application presented for the purposes of illustration and description, but is not intended to be exhaustive or limited to the application as disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the application. The aspects of the application are chosen and described in order to best explain the principles of the application and its practical application, and to enable others skilled in the art to understand the application with various modifications as suitable for the particular use contemplated.

[0064] For the purposes of this document, each flow associated with the disclosed technology can be performed in series or in parallel with one or more computing devices. Each step in the flow can be performed by the same or different computing devices used in other steps, and each step need not necessarily be performed by a single computing device.

[0065] While the subject matter has been described above in terms of specific aspects and / or methods, it is to be understood that the subject matter defined by the appended claims is not necessarily limited to the specific features or acts described, nor must the described aspects and methods be performed in the order stated. Rather, the intended scope of the subject matter is to be given by the appended claims, and their equivalents.

Claims

1. A computer-based method for rendering an image, characterized in that, include: Access the original image, wherein the original image contains objects; A scene graph is generated for the original image, wherein the scene graph of the original image describes the relationships between the objects in the original image; Generate one or more candidate scene graphs, wherein each of the one or more candidate scenes has one or more alternative object relationships with respect to one or more objects in the original image; Creating objects based on the relationship between the objects in the original image and the alternative objects in the candidate scene graph, and rendering a modified image using the candidate scene graph, includes: repositioning at least a portion of one or more objects in the original image in the modified image; and generating image regions in the modified image where the objects in the original image have changed.

2. The computer implementation method according to claim 1, characterized in that, The generation of the scene graph includes: generating a scene graph of the original image using a neural network for scene graph extraction.

3. The computer implementation method according to claim 1 or 2, characterized in that, The process of generating the scene graph includes: determining the object bounding box of each object in the original image; and segmenting each object in the original image.

4. The computer implementation method according to claim 2 or 3, characterized in that, Each candidate scene graph includes a scene graph for accessing object bounding boxes associated with objects in the candidate scene graph, the bounding boxes being obtained from the neural network.

5. The computer implementation method according to claim 4, characterized in that, The rendering includes: using a graph neural network to predict object layout.

6. The computer implementation method according to any one of claims 1 to 5, characterized in that, The rendering includes: predicting the pose of objects in the modified image based on the at least one candidate scene graph, wherein the predicted object pose is different from the object pose in the original image based on the object relationships in the candidate scene graph.

7. The computer implementation method according to claim 6, characterized in that, Generating an image block includes: creating objects from the original image in the modified image at new locations or in new poses, wherein the new locations or poses differ from the locations of the objects in the original image based on the object relationships in the modified image.

8. The computer implementation method according to any one of claims 1 to 7, characterized in that, The generated image patch includes: repositioning objects in the original image in the modified image using an image inpainting neural model, and filling the regions of holes formed in the original image in the modified image.

9. The computer implementation method according to any one of claims 1 to 8, characterized in that, Also includes: The generated image regions with bounding box edges are fused into the holes in the original image to create the modified image in the generation module using style-based loss.

10. A user equipment, characterized in that, include: Storage media, including computer instructions; Image capture system; Display devices; One or more processors communicatively coupled to the storage medium, wherein the one or more processors execute the instructions to cause the system to: Capture a raw image, the raw image including the objects in the image; A scene graph is generated for the original image, wherein the scene graph of the original image describes the relationships between the objects in the original image and the positions of the objects in the original image; Generate one or more candidate scene graphs, wherein each of the one or more candidate scenes has one or more alternative object relationships with respect to one or more objects in the original image; Creating objects based on the relationship between the objects in the original image and the alternative objects in the candidate scene graph, and rendering a modified image using the candidate scene graph, includes: repositioning at least a portion of one or more objects in the original image in the modified image; and generating image regions in the modified image where the objects in the original image have changed.

11. The user equipment according to claim 10, characterized in that, The one or more processors executing instructions to generate a scene graph include: generating a scene graph of the original image using a neural model for scene graph extraction.

12. The user equipment according to claim 10 or 11, characterized in that, The one or more processors execute instructions to generate a scene map by determining the object bounding box of each object in the original image and segmenting each object in the original image.

13. The user equipment according to claim 11 or 12, characterized in that, Each candidate scene graph includes a scene graph for accessing object bounding boxes associated with objects in the candidate scene graph, the bounding boxes being obtained from the neural model.

14. The user equipment according to claim 13, characterized in that, The one or more processors execute instructions to use a graph neural network to predict object layout.

15. The user equipment according to any one of claims 10 to 14, characterized in that, The one or more processors execute instructions to predict object poses in the modified image based on the candidate scene graph, wherein the predicted object poses differ from the object poses in the original image based on object relationships in the modified image.

16. The user equipment according to claim 15, characterized in that, The one or more processors execute instructions to generate image patches, wherein objects in the original image are in new positions or poses in the modified image, and wherein, based on object relationships in the modified image, the new positions differ from the positions or poses of the objects in the original image.

17. The user equipment according to any one of claims 10 to 16, characterized in that, The one or more processors execute instructions to generate image patches, thereby repositioning objects in the original image in the modified image using an image inpainting neural model, and filling in areas of holes formed in the original image in the modified image.

18. The user equipment according to any one of claims 10 to 17, characterized in that, The one or more processors execute instructions to merge generated image regions with bounding box edges into holes in the original image, in order to create the modified image in the generation module using style-based loss.

19. A non-transitory computer-readable medium, characterized in that, The non-transitory computer-readable medium stores computer instructions for rendering an image, which, when executed by one or more processors, cause the one or more processors to perform the following steps: Access the original image, wherein the image contains objects; A scene graph is generated for the original image, wherein the scene graph describes the relationships between the objects in the original image; Generate one or more candidate scene graphs, wherein each of the one or more candidate scenes has one or more alternative object relationships with respect to one or more objects in the original image; Creating objects based on the relationship between the objects in the original image and the alternative objects in the candidate scene graph, and rendering a modified image using the candidate scene graph, includes: repositioning at least a portion of one or more objects in the original image in the modified image; and generating image regions in the modified image where the objects in the original image have changed.

20. The non-transitory computer-readable medium according to claim 19, characterized in that, The instructions cause the one or more processors to perform the generation of the scene graph, including: generating a scene graph of the original image using a neural model for scene graph extraction.

21. The non-transitory computer-readable medium according to claim 19 or 20, characterized in that, The instructions that cause the one or more processors to execute the generation of the scene graph include: determining the object bounding box of each object in the original image; and segmenting each object in the original image.

22. The non-transitory computer-readable medium according to claim 20 or 21, characterized in that, Each candidate scene graph includes a scene graph for accessing object bounding boxes associated with objects in the candidate scene graph, the bounding boxes being obtained from the neural model.

23. The non-transitory computer-readable medium according to claim 22, characterized in that, The instructions that cause the one or more processors to perform the rendering include: predicting object layout using a graph neural network.

24. The non-transitory computer-readable medium according to any one of claims 19 to 23, characterized in that, The instructions cause the one or more processors to perform the rendering by predicting object poses in the modified image based on the candidate scene graph, wherein the object poses may differ from the object poses in the original image, depending on object relationships in the modified image.

25. The non-transitory computer-readable medium according to claim 24, characterized in that, Creating the modified image further includes: the instructions causing the one or more processors to execute the generated image block, wherein objects in the original image are located in new positions in the modified image, wherein, depending on the object relationships in the modified image, the new positions may differ from the positions of the objects in the original image if necessary.

26. The non-transitory computer-readable medium according to any one of claims 19 to 25, characterized in that, The instructions cause the one or more processors to generate image blocks by: repositioning objects in the original image in the modified image using an image inpainting neural model, and filling in the regions of holes formed in the original image in the modified image.

27. The non-transitory computer-readable medium according to any one of claims 19 to 26, characterized in that, Also includes: The instructions cause the one or more processors to perform the step of fusing generated image regions with bounding box edges into holes in the original image, in order to create the modified image in the generation module using style-based loss.

Images