Computer-implemented method, device and computer program product for augmenting a first image with image data from a second image

Abstract

A method for inserting an object from a 2D image into another environment while preserving the spatial relationship of the object in the 2D image intact is provided. An apparatus comprising circuitry configured to perform the method and a computer program product adapted to perform the method are also provided.

Description

Technical Field

[0001] This disclosure relates to the field of image analysis, and more specifically, to a method for enhancing a first image using image data from a second image. This disclosure also relates to an apparatus including a circuit system configured to perform the method. Furthermore, this disclosure relates to a computer program product suitable for performing the method. Background Technology

[0002] In image enhancement methods, it is often desirable to identify and label which objects are present in an image. However, machines struggle to determine why these objects are placed in the way they are currently positioned.

[0003] Users might want to find an image in a catalog that inspires their room design, and they want to use image augmentation to see how the objects in that image would look in their room. Typically, images can be augmented to place a data model of an object within the image, allowing the user to visualize how a scene with the added model would look. However, machines struggle to convert between a 2D image at one scale and a 3D environment at another scale while preserving the original intent of the 2D image.

[0004] Therefore, there is room for improvement in the fields of image analysis and image enhancement. Summary of the Invention

[0005] In view of the foregoing, the object of the present invention is to overcome or mitigate at least some of the problems discussed above. In particular, the object of this disclosure is to provide a method for inserting objects from a 2D image into another environment while maintaining the spatial relationships of the objects in the 2D image intact. Other and / or alternative objects of the invention will be apparent to the reader of this disclosure.

[0006] According to a first aspect of the present invention, a computer-implemented method is provided for enhancing a first image using image data from a second image. The method includes: capturing a first image depicting a first scene using an image capture device; identifying key objects in the first image; receiving a second image depicting a second scene; identifying a plurality of objects in the second image; determining a first 3D coordinate space of the first image; determining a second 3D coordinate space of the second image; using the first 3D coordinate space to determine object coordinates of the key objects in the first image; using the second 3D coordinate space to determine object coordinates of the identified plurality of objects in the second image; identifying relationships between the identified plurality of objects in the second image based on the determined object coordinates; matching one of the plurality of objects in the second image with the key object in the first image; receiving a plurality of 3D models from a database, the plurality of 3D models corresponding to at least some of the plurality of objects in the second image; inserting the plurality of 3D models into the first image while maintaining the identified relationships between the corresponding plurality of objects in the second image and the matched object in the second image, and replicating the relationships for the key object in the first image using the first 3D coordinate space, thereby rendering the enhanced image.

[0007] In the context of this specification, the term "inserting multiple 3D models into a first image" should be understood as adding a visual representation of a 3D model to a first image.

[0008] For example, if the first image is a 2D image—and this is merely one possible implementation—the 3D model is converted into a 2D image, and the visual representation of that 2D image, such as pixels, polygons, or vectors, replaces the pixels, polygons, or vectors of the first image, which correspond to determining where they should be placed to give the impression that the 3D scene corresponding to the 2D first image now includes objects of the 3D model. This is achieved by using a first 3D coordinate space to copy the identified relationships between multiple objects in the second image corresponding to key objects in the first image and a matching object in the second image. This may include rotating or resizing the 3D model before converting it to a 2D image.

[0009] If the first image is a 3D image, this is just one possible implementation, and the 3D model can be directly inserted into the first image, possibly involving the conversion of the correct type of 3D model.

[0010] In either case, by inserting objects from the second image into the first image, the first image is enhanced into an enhanced image that includes objects from both images.

[0011] In the context of this specification, the term "relationship" should be understood as a spatial relationship, such as object A being on top of or above object B and / or object B being next to object C, etc.

[0012] The first and second 3D coordinate spaces can extend along the X, Y, Z planes / directions. The 3D coordinate spaces can be determined by applying algorithms to the images. It's important to note that many algorithms exist that can be applied to calculate and determine 3D coordinate spaces. For example, the 3D coordinate spaces can be calculated by applying plane detection algorithms, RANSAC algorithms, Hough algorithms, etc., to the first and / or second images.

[0013] The objects in the second image [i.e., image data of the corresponding objects in the second scene] can be segmented (extracted, distinguished, etc.) using any known algorithm, such as one or more algorithms using edge features, binary patterns, orientation patterns, gradient features, spatiotemporal features.

[0014] The step of identifying objects in an image can be accomplished using an image search algorithm. In the context of this specification, the term "image search algorithm" should be understood as any known method of searching a database for images of [objects] similar to the object in the image and classifying the object using the results [e.g., tags / categories of similar images found in the database]. This database can be a specialized database or a database found using the Internet. At the time of this disclosure, examples of known commercial image search algorithms include Google Images, TinEye, and Alibaba's Pailitao.

[0015] The method presented is an improved approach for enhancing images while maintaining their realism. By preserving the relationships between objects in the second image, a more realistic view of the enhanced image can be achieved. This results in a relatively low-complexity image enhancement method achieved through the use of relationships.

[0016] By receiving a 3D model of the identified object, the 3D model can be directly rendered into the first image. This allows for low-complexity processing of adding image data belonging to an object in the second image to the first image. Furthermore, 3D models are easier to insert into a 3D environment, and because 3D models can be rotated and / or scaled, for example, more realistic enhanced images are produced.

[0017] According to the implementation method, the step of inserting multiple 3D models further includes replacing the key object in the first image with a 3D model corresponding to an object that matches one of the objects in the second image.

[0018] Replacing key objects is beneficial because it may require less complex transformations between coordinate spaces when replicating relationships. Furthermore, a specific version of the key object (e.g., a specific model of a bed, chair, etc.) can be rendered into the first object, replacing the corresponding key object (e.g., a bed, chair, etc. from another model) that originally existed in the first image. This can potentially produce an improved, enhanced image.

[0019] According to an implementation, the step of inserting multiple 3D models includes replacing at least one identified object in a first image with at least a portion of the multiple 3D models.

[0020] Replacing identified objects is beneficial because it may require less complex transformations between coordinate spaces when copying relationships. Furthermore, replacement allows identified objects from the second image to be inserted into optimal positions in the first image that are otherwise occupied, or it can ensure a reasonable number of each type of object is present in the resulting enhanced image. Replacement can be of different object types, or object recognition can be used to replace objects in the first image with corresponding objects from the second image. The replaced object can be a key object in the first image or any other object.

[0021] According to the implementation method, the step of identifying key objects in the first image includes: identifying multiple objects in the first image, and selecting one of the multiple objects as a key object according to a predetermined priority.

[0022] Predetermined priorities are beneficial because selection and subsequent insertion become more accurate. For example, it may be known that a particular type of furniture (such as a bed or table) is a significant focal point of a room and can therefore be used as a logically simple key object to replicate relationships from that key object to other objects. Alternatively, or additionally, if the focus is on the object with the largest and least mobility, it may be easier to replicate relationships, and such an object may have a higher priority. Alternatively, or additionally, if the focus is on the object with the most relationships, it may be easier to replicate relationships, and such an object may have a higher priority. All these criteria, and many more, may influence each other simultaneously or sequentially.

[0023] According to the implementation method, the matching steps include: performing object recognition of key objects in the first image, and matching the recognized objects with the metadata of the 3D models corresponding to the identified multiple objects.

[0024] Object recognition is beneficial because relationships are often more meaningful and logical when they relate to the type of an object rather than, for example, its size or location. This allows for better and more reliable subsequent insertions. Metadata can be the simplest way to match objects identified in a first image with multiple objects identified in a second image; however, alternatives also include performing object recognition on multiple objects identified in the second image. Object recognition encompasses conventional algorithms for data and image processing, as well as machine learning methods such as neural networks.

[0025] According to an implementation, the step of identifying multiple objects in a second image includes: segmenting the second image into different parts using a computer vision algorithm, and identifying the multiple objects and free space in the depicted second scene.

[0026] Image segmentation is beneficial because it allows each object in an image to be identified individually. Identified empty spaces can also be useful when multiple 3D models are inserted into a first image, for example, to fill empty spaces or, for example, to maintain a certain amount of empty space in front of a television set.

[0027] According to the implementation, the relationship between the plurality of objects identified in the second image includes at least one of the following: one object is supported by another object; one object is close to another object; and one object is to the side or in front of another object.

[0028] Different relationships are beneficial because they connect objects in a relational way that allows multiple relationships to be linked simultaneously. They also readily adapt to different environments or coordinate spaces.

[0029] According to the implementation, the relationship between the multiple objects identified in the second image includes orientation information and / or relative rotation, and the step of inserting at least a portion of the multiple objects in the second image into the first image includes adjusting the orientation information to fit the first image, such as an object being on a specific side of another object, for example, an object being to the left or in front of another object, and the relative rotation being an object facing a specific direction relative to other objects, a floor, a ceiling, or a wall.

[0030] Orientational information is beneficial because it is easier to manipulate while preserving the logic behind the relationships. For example, if there is no space to the left of an object, placing the object that was originally on the left at the same distance as the one on the right, rather than moving other objects around it, may produce a more realistic enhanced image. Furthermore, if, for example, a key object is rotated compared to its corresponding object in the second image, a more realistic enhanced image may be produced if the orientational information is rotated in the same way.

[0031] According to the implementation, the relationships between multiple objects identified in the second image include relative and absolute distances to walls, floors, or ceilings in the second scene and / or relative and absolute distances between objects in the second 3D coordinate space; and the step of inserting multiple 3D models into the first image includes: when the relationships are copied for key objects in the first image using the first 3D coordinate space, prioritizing the relative and absolute distances according to metadata corresponding to the categories of the 3D models.

[0032] Prioritizing different types of distances is beneficial because different types of objects are affected by different logics. For example, maintaining the absolute distance between a chair and a table might be more important, while maintaining the relative distance between a sofa and a coffee table is more logical. Thus, metadata influences how different types of distances are prioritized.

[0033] According to an implementation, the step of inserting multiple 3D models into a first image may include: when copying the relationship for key objects in the first image using a first 3D coordinate space, prioritizing directional information or distance based on metadata corresponding to the category of the 3D model.

[0034] Prioritizing orientation information or distance based on metadata corresponding to the category of a 3D model is beneficial because different types of objects are affected by different logics. For example, maintaining the distance between a shelf and a painting may be more important than maintaining their orientation. According to implementation methods, prioritizing different relational data types based on metadata includes user preferences and / or machine learning methods to determine which relational data types to prioritize when the 3D model's category is transformed between a first 3D coordinate space and a second 3D coordinate space.

[0035] In the context of this specification, the term "different relational data types" should be understood as information such as the relative orientation and / or rotational direction of an object, or distances such as relative or absolute distances to other objects or to a ceiling, floor, or wall.

[0036] It is beneficial to prioritize relational data types based on user preferences and / or machine learning methods, as it can be difficult to create a predetermined priority that will be objectively applied to all instances. Thus, it may be helpful to incorporate external intelligence, such as information previously added to the database or user settings, to improve the resulting enhanced image.

[0037] According to the implementation method, the step of identifying the relationship between multiple objects identified in the second image includes: using machine learning to identify interior design rules, and incorporating these interior design rules into the relationship.

[0038] Incorporating interior design rules into relationships is beneficial because it produces a more realistic, subsequently enhanced image.

[0039] According to an embodiment, the step of capturing a first image depicting a first scene using an image capture device includes using a mixed reality device as the image capture device, and wherein the method further includes: using the mixed reality device to display an augmented image, and updating the augmented image in real time as the field of view of the mixed reality device changes.

[0040] Real-time updates to augmented images are beneficial because they allow users to look around as they would in real life, resulting in more immersive and realistic augmented images.

[0041] According to an embodiment, the method further includes the step of storing the enhanced image in a memory.

[0042] The storage step is beneficial because it allows users to easily compare different layouts and designs and more easily distribute them to interested parties.

[0043] According to a second aspect, a computer program product is provided, the computer program product including a computer-readable storage medium having instructions adapted to perform the method of the first aspect when executed by a device with processing capabilities.

[0044] According to a third aspect, an apparatus is provided for displaying a first image enhanced using image data from a second image. The apparatus includes a circuit system configured to: capture a first image depicting a first scene using an image capture device; identify key objects in the first image; receive a second image depicting a second scene; identify a plurality of objects in the second image; determine a first 3D coordinate space of the first image; determine a second 3D coordinate space of the second image; determine object coordinates of the key objects in the first image using the first 3D coordinate space; determine object coordinates of the identified plurality of objects in the second image using the second 3D coordinate space; identify relationships between the identified plurality of objects in the second image based on the determined object coordinates; match one of the plurality of objects in the second image with the key object in the first image; receive a plurality of 3D models from a database, the 3D models corresponding to at least some of the plurality of objects in the second image; insert the plurality of 3D models into the first image while maintaining the identified relationships between the corresponding plurality of objects in the second image and the matched object in the second image, and copy the relationships for the key objects in the first image using the first 3D coordinate space, thereby rendering the enhanced image; and displaying the enhanced image.

[0045] The second and third aspects can generally have the same features and advantages as the first aspect.

[0046] Generally, unless otherwise expressly defined herein, all terms used in this disclosure shall be interpreted according to their ordinary meaning in the art. Unless otherwise expressly stated, all references to the word "comprising" do not exclude other elements or steps, and the indefinite article "a / an / the [element, device, component, means, step, etc.]" shall be interpreted openly to refer to at least one instance of the stated element, device, component, means, step, etc. Unless expressly stated otherwise, the steps of any method disclosed herein need not be performed in the exact order disclosed. Attached Figure Description

[0047] Referring to the accompanying drawings, the above and additional objects, features, and advantages of the invention will be better understood from the following illustrative and non-limiting detailed description of preferred embodiments of the invention, wherein the same reference numerals will be used for similar elements, wherein:

[0048] Figure 1 This is a schematic diagram based on the first image of the embodiment;

[0049] Figure 2 This is a schematic diagram of the second image according to the embodiment;

[0050] Figure 3 This is a schematic diagram of an enhanced image according to an embodiment;

[0051] Figure 4 This is a schematic diagram of the first scenario according to the implementation method;

[0052] Figure 5 This is a schematic diagram illustrating different types of relationships between multiple objects according to an implementation method;

[0053] Figure 6 This is a schematic diagram illustrating the relationship between multiple objects in the second image according to an embodiment; and

[0054] Figure 7 This is a schematic flowchart of a method for enhancing a first image according to an embodiment. Detailed Implementation

[0055] The invention will now be described more fully with reference to the accompanying drawings, in which presently preferred embodiments of the invention are shown. However, the invention may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided to be thorough and complete and to fully convey the scope of the invention to those skilled in the art.

[0056] The following will describe the use of the second image 300[ Figure 2 Image data to enhance the first image 100 [ Figure 1The method is a computer implementation on any device with processing circuitry, such as a computer with a CPU. The method is described as a series of steps in a specific sequence; however, it should be clear from the following disclosure that the described sequence is merely one possible implementation. Furthermore, some steps may be skipped, occur simultaneously, and / or be performed using different units or circuits.

[0057] In short, the method includes: identifying objects 106 and 206 in two images 100 and 300; identifying relationships between at least objects 206 in the second image 300; and using the identified relationships to realistically insert at least some of the objects 206 from the second image 300 into the first image 100; thereby rendering an enhanced image 400. Figure 3 ].

[0058] The following disclosure will focus on an implementation where first image 100 depicts a first scene in a user's home and second image 300 depicts a second scene 200 representing a more desirable setting for the user. Second image 300 could, for example, be an image from a catalog depicting an ideal room created by a professional designer. Second image 300 could also be a room in a previous home that the user wants to essentially recreate in a new house without furniture. In this case, the size and fixed features of the new house may make such a reconstruction non-trivial. In either case, the inventors have realized that the exact items in second image 200 may be less important than the relationships between them (i.e., the use of free space, which types of furniture are assembled together, and the use of decorative items).

[0059] Therefore, from the perspective of requiring less attention to objects than existing methods and needing to make fewer purchases by utilizing objects and features already present in the first scene, the enhanced image 400 generated by this method can be used to decorate a more desirable room in a more efficient way. The results are further customized specifically for the user's room and situation, thus producing more accurate and realistic results.

[0060] It should be noted that although the described embodiments will focus on furniture for clarity, the inventive concepts set forth in the appended claims can be implemented in a variety of ways (e.g., factory optimization or urban planning).

[0061] Figure 1 A schematic diagram of a first image 100 according to an embodiment is disclosed. The first image 100 depicts a first scene captured by an image capture device 102. The first scene is a room in a user's home and can be captured by the user to set the scene in a way that the user is at least curious about how it can be changed. The image capture device 102 can be any suitable device such as a camera, smartphone, scanning device, or mixed reality headset.

[0062] The first image 100 can be two-dimensional, three-dimensional, or a set of two-dimensional images forming a three-dimensional impression. It can depict a scene from the real world or be generated by a computer.

[0063] The first image 100 is processed by the processing circuit in such a way that, for example, it is processed directly by the image capture device 102 or is processed by transmitting it to a separate unit first using a communication means known to those skilled in the art.

[0064] Key objects 106 are identified in the first image 100, corresponding to objects in the image that are of the most important relation to the essence of the room. For Figure 1 In the example, the sofa is key object 106. In the bedroom, the bed would typically be key object 106.

[0065] Multiple objects in the first image 100 can be identified, and one can be selected as a key object 106. Identifying objects in the first image 100 can be accomplished using an image search algorithm. An object can be selected as the key object 106 based on a predetermined priority. This predetermined priority can be a predetermined list, such as based on user preferences.

[0066] Predetermined priorities are beneficial because selection and subsequent insertion become more accurate. For example, it might be known that a particular type of furniture (such as a bed or table) is a key element of the room, where its relationship to other objects is logical and easy to replicate.

[0067] Alternatively, the key object 106 can be identified using a method independent of other objects without regard to other objects in the first image 100.

[0068] Key objects 106 can be determined based on their size, location, and / or type. For example, a bed might generally be considered a key object 106 in any room where it exists; a large object might be identified as a key object 106, either because of its imposing presence or because of the difficulty of moving it; alternatively, an object located in the center of a room or frame could be chosen as a key object 106 simply because the user seems to consider it important. All these considerations, and more, can be combined using different weights and priorities for different criteria.

[0069] If we focus on the largest, least mobile object, it might be easier to replicate relationships, and such an object might have a higher priority. Alternatively, or otherwise, if we focus on the object with the most relationships, it might be easier to replicate relationships, and such an object might have a higher priority. All these criteria, and many more, can influence each other simultaneously or sequentially.

[0070] In one implementation, for example, if the key object 106 cannot be used because it cannot be matched with object 206 in the second image 300 or because the relationship of the key object 106 cannot be replicated, a secondary key object is selected to replace the key object 106. Preferably, the secondary key object is selected as the next best object in the first image 100 according to any of the priorities determined above.

[0071] In another embodiment, during the insertion step S24, the secondary key object is used in conjunction with the key object 106, requiring the device performing the method to copy the relationship between the secondary key object and the key object 106.

[0072] A first 3D coordinate space is determined for the first image 100, and this first 3D coordinate space is used to determine the object coordinates of the key object 106. The first 3D coordinate space allows the object in the two-dimensional first image 100 to be processed in three dimensions, which results in a more accurate enhanced image 400.

[0073] The first 3D coordinate space can be determined using image processing of the first image 100. If the first image 100, captured using, for example, an AR / VR device, is three-dimensional, the 3D coordinate space can be extracted directly from the metadata of the first image 100. If the first image 100 is a two-dimensional capture of a three-dimensional first scene, the 3D coordinate space can be determined, for example, by analyzing the shadows or relative sizes of objects in the first image 100. If the first image 100 is a set of two-dimensional images from different angles, different angles can be used to determine the 3D coordinate space. Any known and suitable method can be used to determine the 3D coordinate space.

[0074] The first 3D coordinate space can extend along the X, Y, Z planes / directions. The 3D coordinate space can be determined by applying an algorithm to the first image 100. It is important to note that many algorithms exist that can be applied to calculate and determine the 3D coordinate space. For example, the 3D coordinate space can be calculated by applying plane detection algorithms, RANSAC algorithms, Hough algorithms, etc., to the first image 100.

[0075] Figure 2 A schematic diagram of a second image 300 according to an embodiment is disclosed. The second image 300 depicts a second scene 200 that may be realistic or unrealistic. Compared to the first scene, the second scene 200 may be a room that users prefer, such that an enhanced version of the first scene with elements desired from the second scene 200 [e.g., object 206 and / or the relationship between object 206] can be displayed as an enhanced image 400. Hereinafter, the second image 300 will be described as an image from a furniture catalog, assuming that images in a professionally curated catalog are likely to be desired by a large number of users.

[0076] The second image 300 is processed by the processing circuit in such a way as, for example, by direct processing by the image capture device 102 or by being processed by first transmitting it to a separate unit using a communication means known to those skilled in the art.

[0077] Multiple objects 206 are identified in the second image 300. Any known algorithm can be used to segment [extract, distinguish, etc.] the objects in the second image 300, such as algorithms using one or more of edge features, binary patterns, orientation patterns, gradient features, spatiotemporal features, etc.

[0078] An image search algorithm can be used to identify objects in the second image 300. This can be done after the image has been segmented, or it can be done before the image has been segmented.

[0079] Computer vision algorithms can be used to identify multiple objects 206 in the second image 300 to segment the second image 300 into different parts and identify free space and the multiple objects 206 in the depicted second scene 200.

[0080] It is not necessary to identify all objects in the second image 300, and some selections can be made regarding the identified objects; for example, windows may be ignored.

[0081] A second 3D coordinate space is determined for the second image 300. The second 3D coordinate space is used to determine the object coordinates of the identified plurality of objects 206. The second 3D coordinate space allows the objects 206 in the two-dimensional second image 300 to be processed in three dimensions, thereby more accurately identifying their relationships, which will be described in more detail below.

[0082] The second 3D coordinate space can be determined using image processing of the second image 300. If the second image 300 is three-dimensional, the 3D coordinate space can be extracted directly from the metadata of the second image 300. If the second image 300 is a two-dimensional capture of a three-dimensional second scene 200, the 3D coordinate space can be determined, for example, by analyzing the shadows or relative sizes of objects in the second image 300. If the second image 300 is a set of two-dimensional images from different angles, the 3D coordinate space can be determined using different angles.

[0083] A second 3D coordinate space can be determined for the second image 300. By determining the second 3D coordinate space, the object coordinates can be determined for the object 206 identified in the second 3D coordinate space. The second 3D coordinate space can be determined for the first 3D coordinate space as described above.

[0084] Relationships between the multiple objects 206 identified in the second image 300 are determined based on the object coordinates. These relationships are used to ensure that the objects 206 are inserted into the first image 100 in a realistic and desired manner. Assuming that the objects 206 in the second scene 200 are arranged relative to each other in a specific way, it is desirable to replicate these relationships.

[0085] The relationships between the multiple objects 206 identified in the second image 300 can include: an object being supported by another object; an object being close to another object; and / or an object being to the side or in front of another object. Similar relationships related to the floor, ceiling, and / or walls of a room can be identified. These can be combined with the relationships between objects.

[0086] This is a non-exhaustive list of relationships related to furniture; however, depending on the implementation, other relationships may be more or less related.

[0087] Different relationships are beneficial because they connect objects in a relational way that allows multiple relationships to be linked simultaneously. They also readily adapt to different environments or coordinate spaces.

[0088] A typical example of a link is a vase placed on a plate on a table, which is next to a sofa.

[0089] Relationships can also include orientation information, such as an object being to the left of another object and / or facing a specific direction. This can be used to correctly orient certain objects or to rotate or mirror objects that would otherwise not fit into the augmented image 400.

[0090] In order to maintain the identified relationship between multiple objects 206 in the second image 300 and a matching object in the second image 300 when inserting the corresponding multiple 3D models into the first image 100, the orientation information of the relationship can be changed in a manner corresponding to the difference in orientation and rotation of the key object 106 in the first image 100 compared to the matching object among the multiple objects 206 in the second image 300.

[0091] For example, if a key object is rotated compared to its corresponding object in the second image, a more realistic enhanced image may be produced if the orientation information is rotated in the same way. When the bedside table of the second image 300 is inserted into the first image 100, if it is placed on the left side of the bed, it can be kept on the left side of the corresponding bed (e.g., as key object 106). However, if there is no space to the left of the object, a more realistic enhanced image may be produced if the object originally located on the left is placed at the same distance as on the right, rather than moving other surrounding objects. In this case, the distance of the relationship is given higher priority than the orientation information.

[0092] Relationships may include: relative and absolute distances to walls, floors, or ceilings in the second scene and / or relative and absolute distances between objects 206 in the second 3D coordinate space.

[0093] Prioritizing different types of distances is beneficial because different types of objects are affected by different logics. For example, maintaining an absolute distance between a chair and a table might be more important, while maintaining a relative distance between a sofa and a coffee table is more logical. Thus, the type of object influences how different types of distances are prioritized.

[0094] Identifying relationships can include using machine learning to identify interior design rules and combining these rules with relationships. Machine learning can include any machine learning algorithm such as decision trees, neural networks, and / or classifiers.

[0095] Interior design rules can be any suitable exploration of furniture placement in a room; such as having a chair in front of a table or free space in front of a television; or putting similar or complementary color schemes together. This produces a more realistic and / or more appealing enhanced image.

[0096] In one implementation, identifying the relationship includes receiving a predetermined relationship that is stored as metadata related to the second image 300.

[0097] One of the multiple objects 206 in the second image 300 matches a key object 106 in the first image 100. The match allows the two images 100 and 300 to have a connection between them, which will facilitate the insertion step S24.

[0098] The matched object is preferably an object of a similar type to the key object 106, for example, both are like... Figure 1 and Figure 2 Sofas in the image. However, they don't have to match exactly; for example, a sofa can be matched with a chair. Matching is done based on the object's function or basic type [e.g., seating furniture] rather than the object's specific identity.

[0099] The matching can be based on a metric similar to the identification of the key object 106 in the first image 100. This could include matching the most concentrated or largest object, rather than matching objects of the same type.

[0100] Matching may include performing object recognition on key objects 106 in the first image 100 and matching the recognized objects with metadata of a plurality of identified objects 206. In this case, the metadata includes information related to the type of the already identified objects.

[0101] Object recognition is beneficial because relationships are often more meaningful and logical if they relate to the type of the object rather than, for example, its size or location. This allows for better and more reliable subsequent insertions. Metadata is the simplest way to match objects identified in the first image 100 with a plurality of objects 206 identified in the second image 300; however, alternatives also include performing object recognition on the plurality of objects 206 identified in the second image 300. Object recognition includes conventional algorithms for data and image processing as well as machine learning methods such as neural networks.

[0102] In one implementation, more than one of the plurality of objects 206 in the second image 300 matches a key object 106 in the first image 100. In this case, the relationship between all matching objects is replicated in the enhanced image 400. This can also be combined with implementations having several key objects 106 and / or secondary key objects.

[0103] The processing circuit receives multiple 3D models 304 from the database, whereby the multiple 3D models 304 correspond to at least some of the multiple objects 206 in the second image 300.

[0104] 3D models 304 are easier to insert into 3D environments and produce more realistic enhanced images 400 because they may be rotated and / or scaled, for example. Furthermore, 3D models 304 can be easily rendered into enhanced images 400 in high quality using any known re-rendering algorithm.

[0105] Furthermore, the database including 3D model 304 may also include metadata about the 3D model 304 corresponding to the identified object 206. This data may include, for example, the type of furniture or, for example, variations in color.

[0106] The database is preferably a previously established database populated with data and models related to the implementation method. In the implementation method, if object 206 is not found in the second scenario 200, the database can be updated.

[0107] There may be more than one 3D model 304 corresponding to the object 206 identified in the second image 300. In this case, one is selected based on user preference, user choice, machine learning, availability, and / or data related to previous selections. In the case of availability, if one 3D model 304 corresponds to an item that is easier to obtain than another [e.g., has more stock, is in a more suitable price range for the user, or other items have been discontinued], the easier-to-obtain item can be automatically selected. In the case of data related to previous selections, the 3D model 304 that was previously favored by the user can be automatically selected.

[0108] A subgroup of multiple objects 206 in the second image 300 can be selected as objects to be included in the enhanced image 400. Therefore, only 3D models 304 corresponding to some of the multiple objects 206 in the second image 300 can be received.

[0109] Subgroups of object 206 can be selected based on user preferences, user choices, machine learning, availability, and / or data related to previous selections.

[0110] The selection of subgroups of objects 206 can also be at least semi-random, such that at least one of the objects 206 in the group was added by chance. Different objects may have different probabilities of being included in a subgroup, which may be affected by any or all of the selection criteria in the preceding paragraphs. The user can re-randomize the subgroups until a satisfactory enhanced image 400 is obtained.

[0111] Similar randomization can be applied to which of the multiple suitable 3D models 304 are received. These can also be re-randomized to allow for enhancement of different 3D models 304 in the image 400.

[0112] The 3D model 304 can be used in conjunction with image segmentation. Image segmentation can then be improved to extract only or almost only image data related to the identified object 206, and to extract all or almost all image data related to the identified object 206. Therefore, the received 3D model 304 is used to determine which part of the second image 300 to extract image data from in order to avoid extracting any environment related to the object 206. This can improve the enhanced image 400 by efficiently increasing the realism of the image.

[0113] Figure 3 A schematic diagram of an enhanced image 400 according to an embodiment is shown. The enhanced image 400 illustrates its use with the second image 300. Figure 2 The 3D model 304 corresponding to object 206 in the image is enhanced from the first image 100. Figure 1 The resulting image 400 represents a more desirable room, where desirability is established based on the teachings of the second image 300. The enhanced image 400 can be presented to the user in 2D or 3D form, for example, displayed on a screen or in a mixed reality device.

[0114] Multiple 3D models 304 are inserted into the first image 100, while maintaining the relationships identified between the multiple objects 206 in the second image 300 and a matching object in the second image 300, and the relationships are copied for the key object 106 in the first image 100 using the first 3D coordinate space, thereby rendering the enhanced image 400.

[0115] Some 3D models 304 replace objects such as lamps and coffee tables from the first image 100 in the enhanced image 400. Other objects such as shelves and plants are added compared to the first image 100.

[0116] Note that, based on the teachings of the second image 300, the single coffee table in the first image 100 is replaced by two coffee tables in the enhanced image 400. The relationship and / or metadata of the coffee tables allow the processing circuitry to identify them as objects of the same type as the coffee table in the first image 100, and therefore they can be replaced. Thus, the number of objects does not need to be the same to establish a matching of item types.

[0117] The relationship of the bowl being positioned on top of the coffee table exists in both the first image 100 and the second image 300. Therefore, even if the coffee table has been replaced, this relationship is still replicated in the enhanced image 400.

[0118] In the second image 300, the small potted plant has a relationship where it is positioned on top of a shelf. Therefore, when inserted into the first image 100, this relationship is replicated, resulting in the enhanced image 400, where the plant supported by the shelf in the first image 300 has been replaced by a plant supported by the shelf in the enhanced image 400. Replicating relationships does not only involve directly copying them, as equivalence relationships can sometimes be achieved using different combinations of items. In this case, the relationship is that the plant is supported by something near the wall and above the sofa.

[0119] The lamp's position relative to the sofa is the same as in the first image 100, but mirrored compared to the second image 300. Reproduction of relationships involves more than just direct copying; sometimes rotation or mirroring of distances can still preserve the general concept of the relationship. In this case, the relationship of the lamp next to the sofa is reproduced.

[0120] The stool is most closely related to the armchair in the second image 300. However, since the armchair is not inserted into the enhanced image 400, the stool can simply be inserted into the free space in the enhanced image 400 that is in a similar position to the key object 106, as the processing circuit has determined that this best preserves the relationship of the stool.

[0121] Even if another painting is available at the same location in the second image 300, the painting in the first image 100 remains in the enhanced image 400. This could be a result of randomness or spatial constraints, but it could also be excluded from replacement because its furniture type is identified as a painting. The processing circuitry can be pre-programmed or learned through machine learning to typically avoid replacing paintings.

[0122] Although Figure 3Not shown, but the step of inserting multiple 3D models 304 in S24 may also include replacing the key object 106 in the first image 100 with a 3D model 304 corresponding to an object that matches one of the objects in the second image 300.

[0123] Replacing the key object 106 is beneficial because it may require less complex transformations between coordinate spaces when copying relationships. Furthermore, the key object 106 may be just as important to the second image 300 as it is to the first image 100, and by replacing it, the desirability of the second image 300 can be more fully transferred to the enhanced image 400.

[0124] The step of inserting multiple 3D models 304 in S24 may include replacing the object identified in the first image 100 with at least a portion of the multiple 3D models 304.

[0125] This can include using and about Figure 2 The matching method described in the text is the same.

[0126] Replacing the identified objects is beneficial because when copying relationships, it may require less complex transformations between coordinate spaces. Furthermore, replacement can allow the identified objects 206 of the second image 300 to be inserted into optimal positions otherwise occupied in the first image 100, or it can ensure a reasonable number of each type of object is present in the resulting enhanced image 400. The replacement can be of different object types, or object recognition can be used to replace objects in the first image 100 with corresponding objects from the second image 300.

[0127] The step of inserting multiple 3D models 304 into the first image 100 in S24 may include: when replicating the relationship for key objects 106 in the first image 100 using a first 3D coordinate space, prioritizing relative or absolute distances based on metadata corresponding to the categories of the 3D models 304.

[0128] Prioritizing different types of distances is beneficial because different types of objects are affected by different logics. For example, preserving the absolute distance between a chair and a table might be more important, while preserving the relative distance between a sofa and a coffee table is more logical. Therefore, metadata influences how different types of distances are prioritized.

[0129] Metadata may include identifiers of the furniture types involved in the 3D model 304, which may be used for prioritization based on, for example, user preferences and / or machine learning, such as making the painting use the absolute distance to the floor, while the sofa uses the relative distance to other furniture.

[0130] Prioritizing relative or absolute distances based on metadata may include: when the category based on 3D model 304 is transformed between a first 3D coordinate space and a second 3D coordinate space, user preferences and / or machine learning methods determine which of the relative or absolute distances to prioritize.

[0131] It is beneficial to determine which distances to prioritize based on user preferences and / or machine learning methods, as it can be difficult to create a predetermined priority order that will be objectively applied to all instances. Therefore, some external intelligence, including access to information previously added to the database (such as user settings), can improve the resulting enhanced image 400.

[0132] Figure 4 A schematic diagram of a first scenario according to an implementation method is shown. Figure 4 This illustrates a user using a mixed reality device [i.e., VR or AR headset] as an image capture device 102 to capture a first image 100 depicting a first scene in S02.

[0133] The processing circuitry can be configured to display the augmented image 400 using a mixed reality device, and to update the augmented image 400 in real time as the field of view of the mixed reality device changes.

[0134] Thus, if the user turns around, the augmented image 400 will change, and the 3D model 304 corresponding to the object 206 from the second image 300 will be inserted into the new first image 100. For example, a shelf can... Figure 4 The image capture device 102 is outside the field of view, but still within the room. Once the shelf comes into view and becomes part of the first image 100, the enhanced image 400 can show the potted plant on top of the shelf from the second image 300.

[0135] Real-time updates to the enhanced image 400 are beneficial because they allow users to look around as they would in real life, resulting in a more immersive and realistic enhanced image 400.

[0136] As the field of view of the mixed reality device changes, information from the first image 100 can be stored in memory. This can then be used to make the augmented image 400 more realistic by, for example, ensuring the correct number of objects of each furniture type are present in the room; or moving objects already present in the first scene to different locations based on relationships identified in the second image 300. Using the shelf example as described above, once a shelf is identified as existing in the room, it is no longer inserted into the augmented image 400, nor is the shelf already there removed; and existing shelves can be moved instead of inserting new shelves from the second scene 200.

[0137] The second image 300 can be pre-selected so that the enhanced image 400 is created once the image capture device 102 is activated. Alternatively, the second image 300 to be used as a desirability reference can be automatically selected based on a priority list and / or machine learning. Preferably, this selection is notified at least by a key object 106 identified in the first image 100, thereby causing the relevant second image 300 to be selected. For example, if a toilet is identified in the first image 100, the second image 300 depicting a second scene 200 as a bathroom is automatically selected.

[0138] If a user is not satisfied with the second image 300 after seeing the enhanced image 400, they can select a new second image 300, for example, randomly or by using a menu.

[0139] Now we will combine Figures 1 to 3 and Figure 7 Let's take an example to illustrate the image capture device 102.

[0140] Image capture device 102 may be a device for displaying a first image 100, which is enhanced with image data from a second image 300. The device includes a circuit system configured to: capture, in S02, the first image 100 depicting a first scene; identify, in S04, key objects 106 in the first image 100; receive, in S06, the second image 300 depicting a second scene 200; identify, in S08, a plurality of objects 206 in the second image 300; determine, in S10, a first 3D coordinate space of the first image 100; determine, in S12, a second 3D coordinate space of the second image 300; use the first 3D coordinate space to determine, in S14, the object coordinates of the key objects 106 in the first image 100; and use the second 3D coordinate space to determine, in S16, the identified objects in the second image 300. The object coordinates of multiple objects 206; identifying the relationship between the multiple objects 206 identified in the second image 300 based on the determined object coordinates; matching one of the multiple objects 206 in the second image 300 with a key object 106 in the first image 100; receiving multiple 3D models 304 from the database, the 3D models corresponding to at least some of the multiple objects 206 in the second image; inserting the multiple 3D models 304 into the first image 100 while maintaining the identified relationship between the corresponding multiple objects 206 in the second image 300 and the matched object in the second image 300, and copying the relationship for the key object 106 in the first image 100 using a first 3D coordinate space, thereby rendering the enhanced image 400; and displaying the enhanced image 400.

[0141] Figure 5This is a schematic diagram illustrating different types of relationships between multiple objects 206 according to the implementation method. Directional relationships are represented by single arrows, proximity relationships by double arrows, and support relationships by dashed lines.

[0142] Furthermore, examples of different room relationships are given according to the implementation of the relationship representation. For example, a kitchen is shown as having a table away from the wall, a chair next to the table, and other furniture close to the wall. A bedroom is shown as having a bed next to one wall, bedside tables on either side of the bed, an entertainment unit on the opposite wall, and a television on the entertainment unit.

[0143] Figure 6 A schematic diagram showing the relationships between a plurality of objects 206 in the second image 300 according to an embodiment is provided. Directional relationships are indicated by single arrows, proximity relationships by double arrows, and support relationships by dashed lines.

[0144] Most of the relationships shown are related to key object 106. Generally, key object 106 will have the most relationships; however, this is not always the case.

[0145] A shelf is behind the sofa and supports the potted plant. A footrest is in front of the sofa and near one of the coffee tables. A stool is next to the armchair. The freestanding plants have no defined relationship to the sofa and can therefore be added to fill any empty space. The painting is simply supported by the wall and has no other relation to the sofa.

[0146] Figure 7 This is a schematic flowchart of a method for enhancing a first image 100 according to an embodiment. The method includes multiple steps, some of which are optional. These steps can be performed by any number of devices and units, including one. Although a specific order of the steps is shown, these steps can be performed in a different order. For example, the receiving step S22 can be performed at any time after step S08, which identifies multiple objects 206 in the second image 300.

[0147] The first step shown is the capture step S02, which includes capturing a first image 100 depicting the first scene using the image capture device 102. The captured image 100 can be transmitted or processed by the image capture device 102.

[0148] Next, we illustrate identification step S04, which includes identifying key objects 106 in the first image 100. This step S04 can be performed by any processing circuitry. Image processing and / or metadata may be used during this step.

[0149] Then comes the receiving step S06, which includes receiving a second image 300 depicting the second scene 200. This step S06 can be performed by a receiver or an image capture device 102. The second image 300 can be selected by a user.

[0150] The next step is identification step S08, which includes identifying a plurality of objects 206 in the second image 300. This step S08 can be performed by any processing circuitry. Image processing and / or metadata may be used during this step.

[0151] The next step is determination step S10, which includes determining the first 3D coordinate space of the first image 100. This step S10 can be performed by any processing circuit. Image processing and / or metadata may be used during this step.

[0152] The next step is determination step S12, which includes determining the second 3D coordinate space of the second image 300. This step S12 can be performed by any processing circuitry. Image processing and / or metadata may be used during this step.

[0153] The next step is determination step S14, which involves using a first 3D coordinate space to determine the object coordinates of the key object 106 in the first image 100. This step S14 can be performed by any processing circuitry. Image processing and / or metadata may be used during this step.

[0154] The next step is determination step S16, which involves using a second 3D coordinate space to determine the object coordinates of the plurality of objects 206 identified in the second image 300. This step S16 can be performed by any processing circuitry. Image processing and / or metadata may be used during this step.

[0155] The next step is identification step S18, which involves identifying the relationships between the plurality of objects 206 identified in the second image 300 based on the determined object coordinates. This step S18 can be performed by any processing circuitry. Image processing and / or metadata may be used during this step.

[0156] The next step is matching step S20, which involves matching one of the plurality of objects 206 in the second image 300 with a key object 106 in the first image 100. This step S20 can be performed by any processing circuitry. Image processing and / or metadata may be used during this step.

[0157] The next step is receiving step S22, which includes receiving a plurality of 3D models 304 from a database, the plurality of 3D models 304 corresponding to at least some of the plurality of objects 206 in the second image 300. The database is preferably pre-populated.

[0158] The next step is insertion step S24, which includes: inserting multiple 3D models 304 into the first image 100 while maintaining the identified relationships between the corresponding multiple objects 206 in the second image 300 and a matching object in the second image 300; replicating the relationships for key objects 106 in the first image 106 using a first 3D coordinate space; thereby rendering the enhanced image 400. The enhanced image 400 can then be displayed or transmitted for display by a monitor such as a virtual reality headset or a standby reality headset.

[0159] The method may also include step S26 of storing the enhanced image 400 in memory. The storage step S26 is advantageous because it allows users to easily compare different layouts and designs and more easily distribute them to interested parties. The memory can be any short-term or long-term computer storage medium known in the art.

[0160] The method can be implemented using a computer program product including a computer-readable storage medium having instructions adapted to perform the method when executed by a device with processing capabilities.

[0161] Those skilled in the art will recognize that the invention is by no means limited to the preferred embodiments described above. Rather, many modifications and variations are possible within the scope of the appended claims.

[0162] Furthermore, by studying the accompanying drawings, the disclosure, and the appended claims, a person skilled in the art can understand and implement variations of the disclosed embodiments when practicing the claimed invention. The fact that certain measures are described in mutually different dependent claims does not mean that combinations of these measures cannot be used advantageously.

[0163] The systems and methods disclosed above can be implemented as software, firmware, hardware, or a combination thereof. In a hardware implementation, the task division between functional units mentioned in the above description does not necessarily correspond to the division of physical units; instead, a physical component may have multiple functions, and a task may be performed collaboratively by multiple physical components. Some or all components may be implemented as software executed by a digital signal processor or microprocessor, or as hardware or application-specific integrated circuits. Such software may be distributed on a computer-readable medium, which may include computer storage media (or non-transitory media) and communication media (or transient media). As is well known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other storage technologies, CD-ROM, digital multifunction disc, DVD or other optical disc storage devices, magnetic tape cassettes, magnetic tape, disk storage devices or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible by a computer. Furthermore, as is well known to those skilled in the art, communication media typically embody computer-readable instructions, data structures, program modules, or other data in the form of modulated data signals such as carrier waves or other transmission mechanisms, and include any information transmission medium.

Claims

1. A computer-implemented method for enhancing a first image using image data from a second image, the method comprising: Use an image capture device to capture a first image depicting the first scene; Identify key objects in the first image; Receive a second image depicting the second scene; Identify multiple objects in the second image; Determine the first 3D coordinate space of the first image; Determine the second 3D coordinate space of the second image; The first 3D coordinate space is used to determine the object coordinates of key objects in the first image; The second 3D coordinate space is used to determine the object coordinates of the multiple objects identified in the second image; The spatial relationships between multiple objects identified in the second image are determined based on the determined object coordinates; Match one of the multiple objects in the second image with the key object in the first image; Receive multiple 3D models from the database, the multiple 3D models corresponding to at least some of a plurality of objects in the second image; as well as The plurality of 3D models are inserted into the first image, while maintaining the spatial relationship between the plurality of objects in the second image corresponding to the plurality of 3D models and a matching object in the second image, and the spatial relationship is copied for the key object in the first image using the first 3D coordinate space, thereby establishing the spatial relationship between the plurality of 3D models and the key object and rendering the enhanced image.

2. The method according to claim 1, wherein, The step of inserting the plurality of 3D models further includes replacing the key object in the first image with a 3D model corresponding to an object that matches one of the objects in the second image.

3. The method according to claim 1 or 2, wherein, The step of identifying key objects in the first image includes: identifying multiple objects in the first image, and selecting one of the multiple objects as the key object according to a predetermined priority.

4. The method according to claim 1, wherein, The step of inserting the plurality of 3D models includes replacing at least one identified object in the first image with at least a portion of the plurality of 3D models.

5. The method according to claim 1, wherein, The matching steps include: performing object recognition of key objects in the first image, and matching the recognized objects with the metadata of the 3D models corresponding to the identified objects.

6. The method according to claim 1, wherein, The step of identifying multiple objects in the second image includes: segmenting the second image into different parts using a computer vision algorithm, and identifying the multiple objects and free space in the depicted second scene.

7. The method according to claim 1, wherein, The spatial relationships between the multiple objects identified in the second image include at least one of the following: an object is supported by another object; an object is close to another object; and an object is to the side or in front of another object.

8. The method according to claim 1, wherein, The spatial relationships between the multiple objects identified in the second image include orientation information indicating that one object is on a particular side of another object and / or indicating that one object faces a particular direction relative to other objects, the floor, the ceiling, or the wall, and wherein the step of inserting at least a portion of the multiple objects in the second image into the first image includes adjusting the orientation information to fit the first image.

9. The method according to claim 1, wherein, The spatial relationships between the multiple objects identified in the second image include: relative and absolute distances to the walls, floor, or ceiling of the second scene and / or relative and absolute distances between objects in the second 3D coordinate space; and The step of inserting the plurality of 3D models into the first image includes: when copying the spatial relationship for key objects in the first image using the first 3D coordinate space, prioritizing the relative distance or the absolute distance according to metadata corresponding to the category of the 3D model.

10. The method according to claim 1, in, The spatial relationships between the multiple objects identified in the second image include orientation information indicating that one object is on a particular side of another object and / or indicating that one object faces a particular direction relative to other objects, the floor, the ceiling, or the wall. The step of inserting at least a portion of the multiple objects in the second image into the first image includes adjusting the orientation information to fit the first image. The spatial relationships between the multiple objects identified in the second image include: the relative and absolute distances to the walls, floors, or ceilings of the second scene and / or the relative and absolute distances between objects in the second 3D coordinate space; The step of inserting the plurality of 3D models into the first image includes: when copying the spatial relationships for key objects in the first image using the first 3D coordinate space, prioritizing the relative distances or absolute distances according to metadata corresponding to the categories of the 3D models, and The step of inserting the plurality of 3D models into the first image includes: when copying the spatial relationship for key objects in the first image using the first 3D coordinate space, prioritizing the direction information, the relative distance, or the absolute distance according to the metadata corresponding to the category of the 3D model.

11. The method according to claim 10, wherein, Prioritizing the orientation information, relative distance, or absolute distance based on metadata includes user preferences and / or machine learning methods to determine which of the orientation information, relative distance, or absolute distance should be prioritized when the category of the 3D model is transformed between the first 3D coordinate space and the second 3D coordinate space.

12. The method according to claim 1, wherein, The step of identifying spatial relationships between multiple objects identified in the second image includes: using machine learning to identify interior design rules and incorporating the interior design rules into the spatial relationships.

13. The method according to claim 1, wherein, The step of capturing a first image depicting a first scene using an image capture device includes using a mixed reality device as the image capture device, and wherein the method further includes: using the mixed reality device to display the enhanced image, and updating the enhanced image in real time as the field of view of the mixed reality device changes.

14. A computer program product comprising a computer-readable storage medium having instructions adapted to perform the method according to any one of claims 1 to 13 when executed by a device having processing capabilities.

15. An apparatus for displaying a first image enhanced using image data from a second image, the apparatus comprising a circuit system configured to: Use an image capture device to capture a first image depicting the first scene; Identify key objects in the first image; Receive a second image depicting the second scene; Identify multiple objects in the second image; Determine the first 3D coordinate space of the first image; Determine the second 3D coordinate space of the second image; The first 3D coordinate space is used to determine the object coordinates of key objects in the first image; The second 3D coordinate space is used to determine the object coordinates of the multiple objects identified in the second image; The spatial relationships between multiple objects identified in the second image are determined based on the determined object coordinates; Match one of the multiple objects in the second image with a key object in the first image; Receive multiple 3D models from the database, the multiple 3D models corresponding to at least some of a plurality of objects in the second image; The plurality of 3D models are inserted into the first image, while maintaining the spatial relationship between the plurality of objects in the second image corresponding to the plurality of 3D models and a matching object in the second image, and the spatial relationship is copied for the key object in the first image using the first 3D coordinate space, thereby establishing the spatial relationship between the plurality of 3D models and the key object and rendering the enhanced image; as well as The enhanced image is displayed.

Images