Electronic device for processing image, and operation method therefor

Abstract

Provided is a method comprising the steps of: acquiring an input image; acquiring a class map in which a class to which each pixel in the input image belongs is classified; generating one or more texture images regarding surface textures; and synthesizing a first texture image among the one or more texture images onto a texture-required region composed of pixels classified as a first class in the input image.

Description

Electronic device for processing images and method of operation thereof

[0001] The present invention relates to an electronic device for processing images and an operating method thereof, and more particularly, to an electronic device for imparting texture by synthesizing a texture image to a classified area within an image and an operating method thereof.

[0002] In recent years, rapid advancements in communication technology have led to the expansion of mobile device functionality and the provision of more diverse user interfaces (UIs). To enhance the utility of mobile devices and satisfy the diverse needs of users, a variety of applications capable of running on these devices are being developed.

[0003] In particular, with the growing user interest in photography and video, most mobile devices now offer digital camera functionality. These devices can provide enlarged images using digital zoom. Unlike optical zoom, which physically moves the lens to enlarge an image, digital zoom crops a portion of an existing image and enlarges that portion to create the illusion of a full image. This process of enlarging a portion of an image involves artificial pixel interpolation, which can result in a loss of image resolution or detail. While digital zoom can make the subject appear closer without exceeding the camera's hardware limitations, it is more prone to quality degradation than optical zoom.

[0004] A method disclosed as a technical means for achieving a technical task may include a step of acquiring an input image. The method may include a step of acquiring a class map that classifies a class to which each pixel in the input image belongs. The method may include a step of generating one or more texture images regarding the texture of a surface. The method may include a step of synthesizing a first texture image from among the one or more texture images on a texture-requiring region composed of pixels classified into a first class in the input image.

[0005] An electronic device disclosed as a technical means for achieving a technical task may include an input / output interface, a memory, and at least one processor. The input / output interface may receive a user input requesting image processing and output a processed image according to the user input. The memory may store instructions for processing the image. By having at least one processor execute a program or at least one instruction stored in the memory, the electronic device may obtain an input image, obtain a class map classifying a class to which each pixel in the input image belongs, generate one or more texture images regarding the texture of a surface, and synthesize a first texture image from among the one or more texture images on a texture-requiring region composed of pixels classified into a first class in the input image.

[0006] A computer-readable recording medium disclosed as a technical means for achieving a technical task may have stored thereon a program for executing at least one of the embodiments of the disclosed method on a computer.

[0007] FIG. 1 is a conceptual diagram illustrating a method for processing an image according to one embodiment of the present disclosure.

[0008] FIG. 2 is a flowchart illustrating a method for processing an image according to an embodiment of the present disclosure.

[0009] FIG. 3 is a conceptual diagram illustrating a method for obtaining a class map according to an embodiment of the present disclosure.

[0010] FIG. 4 is a flowchart illustrating a method for obtaining an input image according to an embodiment of the present disclosure.

[0011] FIG. 5a is a conceptual diagram illustrating a method for generating a texture image according to an embodiment of the present disclosure.

[0012] FIG. 5b is a conceptual diagram illustrating a method for generating a texture image according to an embodiment of the present disclosure.

[0013] FIG. 5c is a conceptual diagram illustrating a method for generating a texture image according to an embodiment of the present disclosure.

[0014] FIG. 6 is a diagram illustrating texture images according to an embodiment of the present disclosure.

[0015] FIG. 7 is a flowchart illustrating a method for generating a texture image according to one embodiment of the present disclosure.

[0016] FIG. 8 is a conceptual diagram illustrating a method for synthesizing an input image and a texture image according to one embodiment of the present disclosure.

[0017] FIG. 9 is a flowchart illustrating a method for synthesizing an input image and a texture image according to one embodiment of the present disclosure.

[0018] FIG. 10 is a flowchart illustrating a method for determining synthesis weights to synthesize an input image and a texture image according to one embodiment of the present disclosure.

[0019] FIG. 11 is a block diagram illustrating a configuration of an electronic device according to an embodiment of the present disclosure.

[0020] In describing this disclosure, descriptions of technical details that are well-known in the technical field to which this disclosure pertains and are not directly related to this disclosure will be omitted. This is to avoid obscuring the gist of this disclosure by omitting unnecessary explanations and to convey it more clearly. Furthermore, the terms described below are defined based on their functions in this disclosure and may vary depending on the intent or custom of the user or operator. Therefore, their definitions should be based on the contents of this specification as a whole.

[0021] For the same reason, some components in the attached drawings are exaggerated, omitted, or schematically depicted. Furthermore, the dimensions of each component do not entirely reflect its actual size. Identical or corresponding components in each drawing are assigned the same reference numbers.

[0022] The advantages and features of the present disclosure, and methods for achieving them, will become clearer with reference to the embodiments described below in detail with the accompanying drawings. However, the present disclosure is not limited to the embodiments disclosed below and may be implemented in various different forms. The disclosed embodiments are provided to ensure that the disclosure of the present disclosure is complete and to fully inform those skilled in the art of the present disclosure of the scope of the disclosure. An embodiment of the present disclosure may be defined according to the claims. Like reference numerals denote like elements throughout the specification. In addition, when describing an embodiment of the present disclosure, if a detailed description of a related function or configuration is determined to unnecessarily obscure the gist of the present disclosure, the detailed description thereof will be omitted. In addition, the terms described below are terms defined in consideration of the functions of the present disclosure and may vary depending on the intention or custom of the user or operator. Therefore, the definitions should be made based on the contents throughout this specification.

[0023] In one embodiment, each block of the flowchart diagrams and combinations of the flowchart diagrams can be performed by computer program instructions. The computer program instructions can be installed on a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, and the instructions, when executed by the processor of the computer or other programmable data processing apparatus, can create means for performing the functions described in the flowchart block(s). The computer program instructions can also be stored in a computer-available or computer-readable memory that can direct a computer or other programmable data processing apparatus to implement the functions in a particular manner, and the instructions stored in the computer-available or computer-readable memory can also produce an article of manufacture that includes instruction means for performing the functions described in the flowchart block(s). The computer program instructions can also be installed on a computer or other programmable data processing apparatus.

[0024] Additionally, each block in the flowchart diagram may represent a module, segment, or portion of code that includes one or more executable instructions for performing a specified logical function(s). In one embodiment, the functions described in the blocks may occur out of order. For example, two blocks depicted in succession may be executed substantially simultaneously or, depending on the function, may be executed in reverse order.

[0025] The term '~ unit' used in one embodiment of the present disclosure may represent software or a hardware component such as a Field Programmable Gate Array (FPGA) or an Application Specific Integrated Circuit (ASIC), and the '~ unit' may perform a specific role. Meanwhile, the '~ unit' is not limited to software or hardware. The '~ unit' may be configured to be on an addressable storage medium and may be configured to play one or more processors. In one embodiment, the '~ unit' may include components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The functionality provided through a specific component or a specific '~ unit' may be combined to reduce the number of components or separated into additional components. In addition, in one embodiment, the '~ unit' may include one or more processors.

[0026] Below, the meanings of terms used in this disclosure are explained.

[0027] In the present disclosure, a "class" may refer to a specific category to which each pixel in an image belongs. A segmentation algorithm may be a task of classifying objects contained in an image on a pixel-by-pixel basis. Specifically, a segmentation algorithm may be a task of classifying each pixel in an image according to a specific class. Each pixel belonging to an object in an image may be classified into a specific class corresponding to the object, and a "class" may refer to a category for classifying the object.

[0028] For example, when analyzing road images, a segmentation algorithm can be performed to classify pixel-by-pixel into classes such as roads, pedestrians, vehicles, and traffic lights. The types of classes are merely examples and do not limit the technical concepts of the present disclosure.

[0029] In the present disclosure, a class map may be data that classifies the class to which each pixel in an image belongs. The class map may be data obtained by performing a segmentation algorithm on an input image. The class map may be data that classifies objects contained in the input image on a pixel-by-pixel basis. For example, the class map may be image data, but the type of data does not limit the technical concept of the present disclosure.

[0030] The type of segmentation algorithm does not limit the technical concept of the present disclosure. For example, methods such as semantic segmentation, instance segmentation algorithms, and panoptic segmentation algorithms may be utilized.

[0031] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the attached drawings so that those skilled in the art can easily practice the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In addition, in the drawings, parts that are not related to the description are omitted in order to clearly describe the present disclosure, and similar parts are designated with similar reference numerals throughout the specification. In addition, the reference numerals used in each drawing are only for the purpose of describing each drawing, and different reference numerals used in different drawings do not indicate different elements. The present disclosure will be described in detail below with reference to the attached drawings.

[0032] FIG. 1 is a conceptual diagram illustrating a method for processing an image according to one embodiment of the present disclosure.

[0033] Referring to FIG. 1, an electronic device (100) can obtain an input image (10).

[0034] In one embodiment, the input image (10) may include objects. The input image (10) may include objects (11, 12, 13) to be classified pixel by pixel through a segmentation algorithm. The objects (11, 12, 13) may include bags, books, floors, etc., which are merely examples and do not limit the technical idea of ​​the present disclosure.

[0035] In one embodiment, the electronic device (100) may include a camera and may capture an input image (10) through the camera.

[0036] In one embodiment, the electronic device (100) can acquire a photographed input image (10). The electronic device (100) can acquire the photographed input image (10) from an external device. The method of acquiring the input image (10) does not limit the technical idea of ​​the present disclosure.

[0037] In one embodiment, the electronic device (100) may include a texture generator (110) and a class classification model (120). For example, the texture generator (110) and the class classification model (120) may each include an algorithm, an artificial intelligence model, a program, software, etc. for performing the operations described below, and the electronic device (100) may perform one or more operations using the texture generator (110) and the class classification model (120).

[0038] In one embodiment, the texture generator (110) may be configured to generate a texture image (20). The texture generator (110) may be configured to generate one or more texture images (20) in which the brightness of each pixel within the image is randomly implemented. The electronic device (100) may use the texture generator (110) to generate one or more texture images (20) relating to the texture of a surface.

[0039] In one embodiment, the texture image (20) may be an image in which the brightness of each pixel is set to a random value. The texture image (20) may be composed of pixels having various brightness values ​​randomly without a uniform pattern. The texture generator (110) may randomly set the brightness value of each pixel and generate a texture image (20) composed of pixels implemented with random brightness values.

[0040] In one embodiment, the class classification model (120) may be configured to obtain a class map (30) based on an input image (10). The electronic device (100) may obtain the class map (30) using the class classification model (120).

[0041] The class classification model (120) may be a model that processes an input image (10) through a segmentation algorithm. The class classification model (120) may classify objects (11, 12, 13) in the input image (10) through the segmentation algorithm. The class classification model (120) may classify pixels in the input image (10) into a specific class through the segmentation algorithm. The class classification model (120) may classify classes for each pixel in the input image (10). The class classification model (120) may generate a class map (30) that classifies each pixel belonging to an object (11, 12, 13) in the input image (10) into a specific class corresponding to the object (11, 12, 13).

[0042] The class map (30) may be data that classifies the class to which each pixel in the image belongs. For example, the input image (10) may include objects such as a bag (11), a book (12), and a floor (13). The input image (10) may include one or more first pixels belonging to the area of ​​the bag (11), one or more second pixels belonging to the area of ​​the book (12), and one or more third pixels belonging to the area of ​​the floor (13). The class map (30) may be data that corresponds the class of 'bag' to one or more first pixels. The class map (30) may be data that corresponds the class of 'book' to one or more second pixels. The class map (30) may be data that corresponds the class of 'floor' to one or more third pixels.

[0043] In one embodiment, as illustrated in FIG. 1, the class map (30) may be an image in which the correspondence between pixels in the input image (10) or an area of ​​an object composed of pixels and a class is visually expressed. However, the data format of the class map (30) does not limit the technical idea of ​​the present disclosure, and the class map (30) may also be text data including the correspondence between pixels and classes.

[0044] In one embodiment, the electronic device (100) can apply a texture image (20) to an input image (10). The electronic device (100) can synthesize the texture image (20) to the input image (10). The electronic device (100) can synthesize the input image (10) and the texture image (20) based on a class map (30).

[0045] For example, an input image (10) may include a plurality of objects. The electronic device (100) may classify a first class corresponding to a first object among the plurality of objects included in the input image (10). The electronic device (100) may classify pixels belonging to the first object in the input image (10) into the first class.

[0046] Additionally, the electronic device (100) can obtain multiple texture images using a texture generator (110). The first class can correspond to a first texture image among the multiple texture images.

[0047] The electronic device (100) can synthesize a first texture image into an area of ​​a first object in an input image (10). The electronic device (100) can synthesize the first texture image into an area of ​​pixels classified into a first class in the input image (10).

[0048] Similarly, the electronic device (100) can classify a second class corresponding to a second object among a plurality of objects from an input image (10). The electronic device (100) can classify pixels belonging to the second object in the input image (10) into the second class. The second class can correspond to a second texture image among a plurality of texture images. The electronic device (100) can synthesize the second texture image into an area of ​​the second object in the input image (10). The electronic device (100) can synthesize the second texture image into an area of ​​pixels classified into the second class in the input image (10).

[0049] In one embodiment, the electronic device (100) can obtain a correspondence between multiple classes and multiple texture images. For example, a rule can be set such that a first texture image is applied to an area of ​​a first class within an input image (10), a second texture image is applied to an area of ​​a second class within the input image (10), and so on.

[0050] In one embodiment, the electronic device (100) may synthesize a texture image (20) to a portion of an input image (10) based on a class map (30) and a set rule. For example, the electronic device (100) may obtain a class map (30) that classifies objects or pixels belonging to objects included in the input image (10) into first to fifth classes. In addition, the electronic device (100) may obtain a first rule for applying a first texture image to an area of ​​the first class, and a second rule for applying a fourth texture image to an area of ​​the fourth class. The electronic device may synthesize the first texture image to an area of ​​the first class in the input image (10) based on the class map (30) and the first rule. The electronic device may synthesize the fourth texture image to an area of ​​the fourth class in the input image (10) based on the class map (30) and the second rule.

[0051] FIG. 2 is a flowchart illustrating a method for processing an image according to an embodiment of the present disclosure.

[0052] For convenience of explanation, parts that overlap with those described using Figure 1 are simplified or omitted.

[0053] Referring to FIG. 2, in step S210, the electronic device can obtain an input image.

[0054] In one embodiment, the electronic device may include a camera and may capture an input image through the camera. In one embodiment, the electronic device may obtain the captured input image. The electronic device may obtain the captured input image from an external device.

[0055] In step S220, the electronic device can obtain a class map that classifies the class to which each pixel in the input image belongs.

[0056] In one embodiment, the electronic device may include a class classification model. The electronic device may use the class classification model to obtain a class map from an input image. The class classification model may be a model that classifies pixels in the input image into classes corresponding to objects to which the pixels belong.

[0057] The class classification model may be a model utilizing a segmentation algorithm. The segmentation algorithm may include algorithms such as semantic segmentation and instance segmentation, but is not intended to limit the technical concepts of the present disclosure.

[0058] In step S230, the electronic device may generate one or more texture images regarding the texture of the surface.

[0059] In one embodiment, the electronic device can generate one or more texture images that randomly implement the brightness of each pixel within the image.

[0060] In one embodiment, the electronic device can determine the resolution of the texture image. For example, the electronic device can generate a texture image having a resolution of 1920*1080.

[0061] In one embodiment, the electronic device can determine a random brightness value between 0 and 255 for each pixel in the generated texture image. The electronic device can obtain a random brightness value for each pixel. Accordingly, the electronic device can obtain a texture image composed of pixels having random brightness values.

[0062] In one embodiment, an electronic device can generate a second texture image based on a first texture image. The electronic device can obtain the second texture image based on a first texture segment and a second texture segment. The electronic device can generate the first texture image. The first texture image can be a first texture segment. The electronic device can obtain a second texture segment having the same size as the first texture segment by enlarging a portion of the first texture segment. The electronic device can generate the second texture image by overlapping the first texture segment and the second texture segment.

[0063] In one embodiment, an electronic device can generate a third texture image based on a first texture image. The electronic device can obtain the third texture image based on a first texture segment, a second texture segment, and a third texture segment. The electronic device can generate the first texture image. The first texture image can be a first texture segment. The electronic device can obtain a second texture segment having the same size as the first texture segment by enlarging a portion of the first texture segment. The electronic device can obtain a third texture segment having the same size as the second texture segment by enlarging a portion of the second texture segment. The electronic device can generate the third texture image by overlapping the first texture segment, the second texture segment, and the third texture segment.

[0064] The number of texture segments used to generate a texture image does not limit the technical idea of ​​the present disclosure, and a texture image can be generated using more texture segments.

[0065] In step S240, the electronic device can synthesize a first texture image on a texture-requiring region composed of pixels classified into a first class within an input image.

[0066] In one embodiment, the electronic device may obtain a class map by classifying each pixel in an input image into one of a plurality of classes using a class classification model. The plurality of classes may include a first class. The electronic device may classify a first pixel in the input image into the first class using the class classification model. The electronic device may classify a second pixel in the input image into the first class using the class classification model. The first pixel and the second pixel may each belong to an area of ​​an object corresponding to the first class in the input image.

[0067] In one embodiment, an electronic device may obtain a texture-requiring region comprised of pixels classified into a first class within an input image. For example, the texture-requiring region may include a first pixel and a second pixel. The electronic device may synthesize one of a plurality of texture images onto the texture-requiring region. The electronic device may synthesize the first texture image onto the texture-requiring region. The electronic device may also synthesize the second texture image onto the texture-requiring region.

[0068] One of the plurality of texture images may be selected with the intention of differentiating the texture expression of the texture-requiring region. For example, if the texture-requiring region is a region of an object classified as a 'floor' class, one texture image suitable for expressing a rough texture may be selected from the plurality of texture images. As another example, if the texture-requiring region is a region of an object classified as a 'water' class, one texture image suitable for expressing a smooth texture may be selected from the plurality of texture images.

[0069] In one embodiment, multiple texture images may have different textures based on the number of overlapping texture segments. For example, a first texture image may have a fine texture or a texture composed of small particles. A second texture image, which is an overlapping of the first and second texture segments, may have a rougher texture or a texture composed of larger particles. The textures of the texture images are described again in FIG. 6 and are illustrated as images.

[0070] In one embodiment, the electronic device may obtain a class map by classifying each pixel in an input image into one of multiple classes using a class classification model. Based on the class map, the electronic device may determine that the input image does not contain a texture-requiring region composed of pixels classified into the first class. In this case, the electronic device may output the input image as is without performing an operation of synthesizing the first texture image with the input image.

[0071] FIG. 3 is a conceptual diagram illustrating a method for obtaining a class map according to an embodiment of the present disclosure. For reference, FIG. 3 illustrates a method for obtaining a class map based on various input images.

[0072] For convenience of explanation, parts that overlap with those described using Figure 1 are simplified or omitted.

[0073] Referring to FIG. 3, the electronic device (300) can obtain an input image. The electronic device (300) can generate a class map (30a, 30b) by inputting the input image into a class classification model (320).

[0074] In one embodiment, the electronic device (300) can obtain an input image. The electronic device (300) can obtain a first image (10a) as the input image. The electronic device (300) can include a camera and can capture the first image (10a) through the camera. In one embodiment, the electronic device (300) can obtain the captured first image (10a). The electronic device (300) can obtain the first image (10a) captured from an external device.

[0075] In one embodiment, the electronic device (300) can obtain a second image (10b) as an input image. After obtaining the first image (10a), the electronic device (300) can obtain the second image (10b) by enlarging a region of interest (ROI) of the first image (10a).

[0076] A region of interest (ROI) may include an area that a user wishes to enlarge. For example, the electronic device (300) may obtain a user input specifying a region of interest (ROI) to be enlarged. The electronic device (300) may obtain the region of interest (ROI) based on the user input. The user input may be obtained in various forms, such as a drag, a simple click, or a touch input, and is not intended to limit the technical concept of the present disclosure.

[0077] In one embodiment, the second image (10b) may be an image in which a region of interest (ROI) within the first image (10a) is enlarged through a digital zoom method. The resolution of the image enlarged through the digital zoom method may not change. For example, the resolution of the first image (10a) and the resolution of the second image (10b) may be the same. However, the second image (10b) may be generated by artificially interpolating pixels in the enlarged region of interest (ROI) to add new pixels. Since the resolution of the original image does not change during zooming, the image quality of the second image (10b) may be degraded compared to the first image (10a).

[0078] The operations performed in the present disclosure can provide texture to an object by synthesizing a texture image onto an object area within an input image. By providing an arbitrary texture to an object within an image, the user can visually perceive an optical illusion of improved image quality, and a natural image can be provided. In particular, the watercolor phenomenon, in which boundaries become blurred and colors spread out like a blur when the user enlarges the image, can be improved. For example, the second image (10b) may be a low-quality image compared to the first image (10a), and by providing an arbitrary texture to an object area of ​​the second image (10b), a more natural image can be provided.

[0079] In one embodiment, the electronic device (300) can acquire an input image. The electronic device (300) can acquire a first image (10a) or a second image (10b) as the input image. Hereinafter, the case where the first image (10a) is acquired as the input image (S10) and the case where the second image (10b) is acquired as the input image (S20) are separately described.

[0080] In step S10, the electronic device (300) can acquire a first image (10a) as an input image. The electronic device (300) can generate a first class map (30a) by inputting the first image (10a) into a class classification model (320). The operations performed by the class classification model (320) overlap with those described in FIG. 1 and are therefore briefly described.

[0081] In one embodiment, the class classification model (320) can classify pixels in the first image (10a) into a specific class through a segmentation algorithm. The class classification model (320) can classify classes for each pixel in the first image (10a). The class classification model (320) can generate a first class map (30a) that classifies pixels belonging to an object in the first image (10a) into a specific class corresponding to the object.

[0082] In step S20, the electronic device (300) can obtain a second image (10b) as an input image. The electronic device (300) can generate a second class map (30b) by inputting the second image (10b) into a class classification model (320).

[0083] In one embodiment, the class classification model (320) can classify pixels in the second image (10b) into a specific class through a segmentation algorithm. The class classification model (320) can classify classes for each pixel in the second image (10b). The class classification model (320) can generate a second class map (30b) that classifies pixels belonging to an object in the second image (10b) into a specific class corresponding to the object.

[0084] In one embodiment, the area included in the first image (10a) may be wider than the area included in the second image (10b). The area included in the first image (10a) may be wider than the region of interest (ROI). Accordingly, the area included in the first class map (30a) may be wider than the area included in the second class map (30b). The first class map (30a) may be a map that classifies classes for the area included in the first image (10a), and the second class map (30b) may be a map that classifies classes for the area or region of interest (ROI) included in the second image (10b). For example, as illustrated in FIG. 3, the first class map (30a) may include more pixels classified as classes of 'flower' and 'window' as classes are classified for a relatively wide area, and the second class map (30b) may not include pixels classified as classes of 'flower' and 'window' as classes are classified for a relatively narrow area.

[0085] FIG. 4 is a flowchart illustrating a method for obtaining an input image according to an embodiment of the present disclosure.

[0086] For convenience of explanation, parts that overlap with those described using Figures 2 and 3 are simplified or omitted.

[0087] Referring to FIG. 4, step S210 of FIG. 2 may include steps S410, S420, and S430.

[0088] In step S410, the electronic device can acquire a first image.

[0089] In one embodiment, the electronic device may include a camera and may capture a first image through the camera. In one embodiment, the electronic device may acquire the captured first image. The electronic device may acquire the captured first image from an external device.

[0090] In step S420, the electronic device can acquire a region of interest within the first image.

[0091] In one embodiment, the region of interest may include an area that the user wishes to enlarge. The electronic device may obtain user input specifying the region of interest. The user input may be obtained in various forms, such as drag, simple click, and touch input, and is not limited to the technical concepts of the present disclosure. In one embodiment, the electronic device may obtain the region of interest within the first image based on the user input.

[0092] In one embodiment, the electronic device can acquire a region of interest based on a preset value. The electronic device can preset the location of the region of interest when enlarging the first image and store the preset location.

[0093] For example, an electronic device may store the location of a region of interest that includes the center of a first image. When the electronic device receives a user input requesting to enlarge the first image, the electronic device may determine a region of interest that includes the center of the first image based on the desired magnification. The area of ​​the region of interest may be determined based on the desired magnification, and the location of the region of interest that includes the center of the first image may be determined.

[0094] In step S430, the electronic device can acquire a second image by enlarging the region of interest.

[0095] In one embodiment, the electronic device can magnify a region of interest within a first image. The electronic device can magnify the region of interest using a digital zoom method, but the method of magnifying the image is not limited to the technical concept of the present disclosure. In one embodiment, the electronic device can acquire a second image with the region of interest magnified.

[0096] In one embodiment, an electronic device may obtain an input image. The electronic device may obtain the input image as input data for obtaining a class map. The input image may be a first image or a second image.

[0097] For example, after step S410 is performed, step S220 of FIG. 2 may be performed based on the first image. The electronic device may obtain a class map that classifies the class to which each pixel in the first image belongs. As another example, after step S430 is performed, step S220 of FIG. 2 may be performed based on the second image. The electronic device may obtain a class map that classifies the class to which each pixel in the second image belongs.

[0098] FIG. 5a is a conceptual diagram illustrating a method for generating a texture image according to an embodiment of the present disclosure.

[0099] Referring to FIG. 5A, in one embodiment, an electronic device may generate a first texture image (510) regarding the texture of a surface. The electronic device may generate the first texture image (510) by randomly implementing the brightness of each pixel within the image.

[0100] In one embodiment, the first texture image (510) may be an image in which the brightness of each pixel is set to a random value. The first texture image (510) may be composed of pixels having various brightness values ​​randomly without a uniform pattern. The electronic device may randomly set the brightness value of each pixel and generate the first texture image (510) composed of pixels implemented with random brightness values.

[0101] For convenience of explanation, a first area (A1) including four pixels within a first texture image (510) is enlarged to specifically describe the first pixel (511), the second pixel (512), the third pixel (513), and the fourth pixel (514). Other pixels within the first texture image (510) may also be formed similarly to the first pixel (511), the second pixel (512), the third pixel (513), and the fourth pixel (514).

[0102] In one embodiment, the electronic device can randomly set brightness values ​​of pixels within a first texture image (510). The first area (A1) can include a first pixel (511), a second pixel (512), a third pixel (513), and a fourth pixel (514).

[0103] In one embodiment, the electronic device can randomly determine one brightness value from among brightness values ​​0 to 255 and display one pixel based on the determined brightness value. The electronic device can display a first pixel (511) having a brightness value of 52. The electronic device can display a second pixel (512) having a brightness value of 122. The electronic device can display a third pixel (513) having a brightness value of 28. The electronic device can display a fourth pixel (514) having a brightness value of 219. Each brightness value is an example and does not limit the technical idea of ​​the present disclosure.

[0104] As the brightness values ​​of the first to fourth pixels (511, 512, 513, 514) are randomly determined, the brightness values ​​of a plurality of pixels within the first texture image (510) can be randomly determined. Accordingly, the first texture image (510) can express a predetermined texture by a plurality of pixels having random brightness values. The electronic device can generate the first texture image (510) regarding the texture of the surface.

[0105] FIG. 5b is a conceptual diagram illustrating a method for generating a texture image according to an embodiment of the present disclosure.

[0106] For convenience of explanation, parts that overlap with those described using Fig. 5a are simplified or omitted.

[0107] Referring to FIG. 5B, in one embodiment, the electronic device may generate a second texture image (520) regarding the texture of a surface. The electronic device may generate the second texture image (520) by randomly implementing the brightness of each pixel within the image.

[0108] In one embodiment, the first texture segment (SEG1) may be a first texture image (510 of FIG. 5a).

[0109] In one embodiment, the electronic device can obtain a second texture segment (SEG2) based on the first texture image (510). The electronic device can obtain the second texture segment (SEG2) based on the first texture segment (SEG1). The electronic device can obtain the second texture segment (SEG2) by enlarging a portion of the first texture segment (SEG1). The electronic device can obtain the second texture segment (SEG2) by enlarging the second area (A2).

[0110] In one embodiment, the first texture segment (SEG1) and the second texture segment (SEG2) may have the same size. The first texture segment (SEG1) and the second texture segment (SEG2) may have the same resolution. The first texture segment (SEG1) and the second texture segment (SEG2) may be images including the same number of pixels. For example, the first texture segment (SEG1) may be an image having a size of 1000*1000 pixels, and the second texture segment (SEG2), which is an enlarged version of the second area (A2) of the first texture segment (SEG1), may be an image having a size of 1000*1000 pixels.

[0111] As the second area (A2), which is a part of the first texture segment (SEG1) having a size of 1000*1000 pixels, is enlarged, interpolation may be performed to determine a pixel value from surrounding pixels so that the second texture segment (SEG2) maintains a size of 1000*1000 pixels. For example, the electronic device may obtain the second texture segment (SEG2) using bicubic interpolation. As other examples, nearest-neighbor interpolation, bilinear interpolation, Lanczos interpolation, etc. may of course be used.

[0112] In one embodiment, the electronic device can generate a second texture image (520) by overlapping a first texture segment (SEG1) and a second texture segment (SEG2). The electronic device can generate the second texture image (520) by calculating brightness values ​​of corresponding pixels of the first texture segment (SEG1) and the second texture segment (SEG2).

[0113] For example, the electronic device may generate the second texture image (520) by adding the brightness values ​​of the corresponding pixels of the first texture segment (SEG1) and the second texture segment (SEG2). As another example, the electronic device may generate the second texture image (520) by averaging the brightness values ​​of the corresponding pixels of the first texture segment (SEG1) and the second texture segment (SEG2). The computational method for overlapping the first texture segment (SEG1) and the second texture segment (SEG2) is merely an example and does not limit the technical idea of ​​the present disclosure.

[0114] In one embodiment, an electronic device may obtain a texture image based on the sum of one or more texture segments. For example, a texture image I may be obtained based on Equation 1.

[0115]

[0116] In mathematical expression 1, I texture can be a texture image.

[0117] In mathematical expression 1, n may denote the number of overlapping texture segments. For example, when 'n=1', a texture image may be generated using one texture segment, which may correspond to an embodiment of generating a first texture image (510) illustrated in FIG. 5a. As another example, when 'n=2', a texture image may be generated using two texture segments, which may correspond to an embodiment of generating a second texture image (520) illustrated in FIG. 5b. As another example, when 'n=3', a texture image may be generated using three texture segments, which may correspond to an embodiment of generating a third texture image (530) illustrated in FIG. 5c.

[0118] In one embodiment, the larger the value of n, the more distinctly the texture of the coarse particles in the texture image can be expressed.

[0119] In mathematical expression 1, k can represent the overlap coefficient of each texture segment. Texture segments with different particle sizes can be synthesized with different proportions depending on the k value. Depending on the k value, a texture image can express various textures and textures.

[0120] In one embodiment, in order to express the texture of a texture image in various ways, the overlap coefficient k may be expressed in various formulas including n. For example, k may be expressed in the form of a logarithmic function, a linear function, a polynomial function, etc. with respect to n.

[0121] In mathematical expression 1, may refer to a texture segment having x * y size or resolution. A texture segment may be an image composed of pixels with randomly determined brightness values.

[0122] In mathematical expression 1, can be a function that means to perform interpolation on a texture segment with x * y size or resolution. For example, the interpolation can be bicubic interpolation, nearest-neighbor interpolation, bilinear interpolation, or lanczos interpolation.

[0123] In one embodiment, the electronic device can generate one or more texture images based on Equation 1. However, Equation 1 is merely an example and does not limit the technical idea of ​​the present disclosure.

[0124] FIG. 5c is a conceptual diagram illustrating a method for generating a texture image according to an embodiment of the present disclosure.

[0125] For convenience of explanation, parts that overlap with those described using FIGS. 5a and 5b are simplified or omitted.

[0126] Referring to FIG. 5c, in one embodiment, the electronic device may generate a third texture image (530) regarding the texture of a surface. The electronic device may generate the third texture image (530) by randomly implementing the brightness of each pixel within the image.

[0127] In one embodiment, the first texture segment (SEG1) may be a first texture image (510 of FIG. 5a).

[0128] In one embodiment, the electronic device can obtain a second texture segment (SEG2) based on the first texture image (510). The electronic device can obtain the second texture segment (SEG2) based on the first texture segment (SEG1). The electronic device can obtain the second texture segment (SEG2) by enlarging a portion of the first texture segment (SEG1). The electronic device can obtain the second texture segment (SEG2) by enlarging the second area (A2).

[0129] In one embodiment, the electronic device can obtain a third texture segment (SEG3) based on the second texture segment (SEG2). The electronic device can obtain the third texture segment (SEG3) by enlarging a portion of the second texture segment (SEG2). The electronic device can obtain the third texture segment (SEG3) by enlarging the third area (A3).

[0130] In one embodiment, the first texture segment (SEG1), the second texture segment (SEG2), and the third texture segment (SEG3) may have the same size. The first texture segment (SEG1), the second texture segment (SEG2), and the third texture segment (SEG3) may have the same resolution. The first texture segment (SEG1), the second texture segment (SEG2), and the third texture segment (SEG3) may be images containing the same number of pixels. For example, the first texture segment (SEG1) may be an image having a size of 1000*1000 pixels, the second texture segment (SEG2), which is an enlarged second area (A2) of the first texture segment (SEG1), may be an image having a size of 1000*1000 pixels, and the third texture segment (SEG3), which is an enlarged third area (A3) of the second texture segment (SEG2), may be an image having a size of 1000*1000 pixels.

[0131] As the second area (A2), which is part of the first texture segment (SEG1) having a size of 1000*1000 pixels, is enlarged, interpolation may be performed to determine one pixel value from surrounding pixels so that the second texture segment (SEG2) maintains a size of 1000*1000 pixels. Similarly, interpolation may be performed to determine one pixel value from surrounding pixels so that the third texture segment (SEG3) maintains a size of 1000*1000 pixels.

[0132] In one embodiment, the electronic device can generate a third texture image (530) by overlapping a first texture segment (SEG1), a second texture segment (SEG2), and a third texture segment (SEG3). The electronic device can generate the third texture image (530) by calculating brightness values ​​of corresponding pixels of the first texture segment (SEG1), the second texture segment (SEG2), and the third texture segment (SEG3).

[0133] For example, the electronic device may generate a third texture image (530) by adding the brightness values ​​of the corresponding pixels of the first texture segment (SEG1), the second texture segment (SEG2), and the third texture segment (SEG3). As another example, the electronic device may generate the third texture image (530) by averaging the brightness values ​​of the corresponding pixels of the first texture segment (SEG1), the second texture segment (SEG2), and the third texture segment (SEG3). The computational method for overlapping the first texture segment (SEG1), the second texture segment (SEG2), and the third texture segment (SEG3) is merely an example and does not limit the technical idea of ​​the present disclosure.

[0134] FIG. 6 is a diagram illustrating texture images according to an embodiment of the present disclosure.

[0135] For convenience of explanation, parts that overlap with those described using FIGS. 5a to 5c are simplified or omitted.

[0136] Referring to FIG. 6, texture images generated in one embodiment of the present disclosure are illustrated.

[0137] In one embodiment, the electronic device may acquire a first texture image (510). The first texture image (510) may be the same as the first texture image (510) described using FIG. 5A. The first texture image (510) may be an image in which pixel brightness is randomly implemented. Compared to the second texture image (520), the third texture image (530), and the fourth texture image (540) described below, the first texture image (510) may have a texture with a relatively fine particle size, and may have a texture with a relatively uniform particle distribution.

[0138] In one embodiment, the electronic device may acquire a second texture image (520). The second texture image (520) may be the same as the second texture image (520) described using FIG. 5B. The second texture image (520) may be an image generated by overlapping the first texture segment and the second texture segment. Compared to the first texture image (510), the second texture image (520) may have a texture with a relatively coarse particle size and a texture with a relatively uneven particle distribution.

[0139] In one embodiment, the electronic device may acquire a third texture image (530). The third texture image (530) may be the same as the third texture image (530) described using FIG. 5C. The third texture image (530) may be an image generated by overlapping the first texture segment, the second texture segment, and the third texture segment. Compared to the second texture image (520), the third texture image (530) may have a texture with a relatively coarse particle size and a texture with a relatively uneven particle distribution.

[0140] In one embodiment, the electronic device may acquire a fourth texture image (540). The fourth texture image (540) may be an image generated by overlapping the first texture segment, the second texture segment, the third texture segment, and the fourth texture segment. Compared to the third texture image (530), the fourth texture image (540) may have a texture having a relatively coarse particle size and may have a texture having a relatively uneven particle distribution.

[0141] Of course, by adjusting coefficients for generating texture images, such as adjusting n and k in mathematical expression 1, the electronic device can obtain texture images with various textures. The expressions for describing the textures of the first texture image (510), the second texture image (520), the third texture image (530), and the fourth texture image (540) do not limit the technical idea of ​​the present disclosure.

[0142] In one embodiment, the electronic device can provide texture to a texture-requiring region within the input image by synthesizing at least one of the respective texture images into the input image.

[0143] For example, the first texture image (510) may have a texture having a relatively fine particle size and may have a texture having a relatively uniform particle distribution. The electronic device may synthesize the first texture image (510) into a texture-requiring area within an input image, thereby imparting a texture having a fine particle size to the synthesized texture-requiring area.

[0144] As another example, the fourth texture image (540) may have a texture with a relatively coarse particle size and a texture with a relatively uneven particle distribution. The electronic device may synthesize the fourth texture image (540) into a texture-requiring area within the input image, thereby imparting a texture with a coarse particle size to the synthesized texture-requiring area.

[0145] As another example, the third texture image (530) may have a texture that is between the texture of the first texture image (510) and the texture of the fourth texture image (540). The third texture image (530) may have a texture having a grain size that is coarser than the grain size of the first texture image (510) and finer than the grain size of the fourth texture image (540). The electronic device may synthesize the third texture image (530) into a texture-requiring area within the input image, thereby imparting a texture having an intermediate grain size to the synthesized texture-requiring area.

[0146] Of course, although not shown in FIG. 6, the electronic device can obtain a fifth texture image, a sixth texture image, etc. by overlapping more texture segments. The electronic device can obtain various texture images and implement various textures by synthesizing at least one of the obtained texture images onto an input image.

[0147] FIG. 7 is a flowchart illustrating a method for generating a texture image according to one embodiment of the present disclosure.

[0148] For convenience of explanation, parts that overlap with those described using Figures 1 to 6 are simplified or omitted.

[0149] Referring to FIG. 7, step S230 of FIG. 2 may include steps S710, S720, and S730.

[0150] In step S710, the electronic device can generate a first texture segment that randomly implements the brightness of each pixel in the image.

[0151] In one embodiment, the first texture segment may be an image comprising pixels, each having a random brightness value. The brightness values ​​of the pixels may range from 0 to 255, but this is merely an example and the technical concept of the present disclosure is not limited thereto.

[0152] In step S720, the electronic device can generate a second texture segment by enlarging a portion of the first texture segment.

[0153] For example, an electronic device may set a first region of a first texture segment. The first region may be a portion of the first texture segment. The electronic device may enlarge the first region to the size or resolution of the first texture segment. The electronic device may generate a second texture segment to correspond to the size or resolution of the first texture segment.

[0154] In step S730, the electronic device can generate one or more texture images by overlapping the first texture segment and the second texture segment.

[0155] In one embodiment, the electronic device can overlap a first texture segment and a second texture segment. The electronic device can perform an operation to add the brightness values ​​of two corresponding pixels within the first texture segment and the second texture segment. The electronic device can generate a texture image by performing an operation between pairs of corresponding pixels within the first texture segment and the second texture segment.

[0156] In one embodiment, the electronic device can generate one or more texture images by overlapping the first texture segment to the third texture segment. The electronic device can perform an operation to add the brightness values ​​of three corresponding pixels within the first texture segment to the third texture segment. The electronic device can generate the texture image by performing an operation on sets of corresponding pixels within the first texture segment to the third texture segment.

[0157] FIG. 8 is a conceptual diagram illustrating a method for synthesizing an input image and a texture image according to an embodiment of the present disclosure. For reference, FIG. 8 primarily describes the effects according to an embodiment of the present disclosure through the difference between the input image and the output image, and describes in detail the operation (S830) of applying the texture image to the input image.

[0158] For convenience of explanation, parts that overlap with those described using Figures 1 to 7 are simplified or omitted.

[0159] Referring to FIG. 8, in one embodiment, an electronic device may obtain an input image. The input image may be a first image (810) or a second image (820). The electronic device may obtain the first image (810) as the input image, or may obtain a second image (820) that is an enlarged portion of the first image (810) as the input image.

[0160] In one embodiment, the electronic device can obtain a first image (810). The electronic device can obtain a region of interest (ROI) of the first image (810). The electronic device can obtain a user input for setting the region of interest (ROI).

[0161] In step S810, the electronic device can acquire a second image (820) by enlarging a region of interest (ROI). The algorithm for enlarging a portion of the image may utilize a digital zoom method, but the technical concept of the present disclosure is not limited thereto.

[0162] In one embodiment, even if the first image (810) is enlarged, the resolution of the original image, the first image (810), may remain unchanged. The first image (810) may have the same resolution as the second image (820), which is an enlarged portion of the first image (810). However, since the second image (820) represents a wider area with the same resolution, the actual resolution is reduced, which may result in a deterioration in image quality.

[0163] In step S820, the electronic device may obtain a class map (830) based on the second image (820). The electronic device may obtain a class map (830) that classifies the classes of pixels in the second image (820) using a segmentation algorithm. The electronic device may obtain a class map (830) that classifies pixels in the second image (820) into classes corresponding to objects to which the pixels belong. The method of obtaining the class map (830) overlaps with that described using FIG. 3 and is therefore omitted.

[0164] In one embodiment, the electronic device can obtain a texture image (840). Since the texture image (840) has been described using FIGS. 5A to 5C and FIG. 6, a related description thereof will be omitted. In step S830, the electronic device can apply the texture image (840) to the second image (820). The electronic device can synthesize the texture image (840) on a texture-requiring area composed of pixels classified into the first class within the second image (820).

[0165] For example, the electronic device can obtain pixels classified as a class of 'ground' in the second image (820). The electronic device can synthesize a texture image (840) on a texture-requiring area composed of the obtained pixels. The electronic device can synthesize the texture image (840) by performing an operation of adding the brightness values ​​of each pixel of the texture image (840) on the texture-requiring area in the second image (820). The electronic device can generate an output image (850) by synthesizing the texture image (840) in the second image (820).

[0166] For example, an output image (850) can be obtained based on mathematical expression 2.

[0167]

[0168] In mathematical expression 2, I final can be the output image.

[0169] In mathematical expression 2, I zoom The silver texture image may be the second image to be synthesized. I zoom may be an image that enlarges an area of ​​interest specified by user input from the first image. I zoom may be a texture-required area where a texture image will be applied.

[0170] I texture can be a texture image. I texture can be derived by mathematical formula 1.

[0171] In mathematical expression 2, w can mean the application intensity of the texture image.

[0172] In one embodiment, w may be applied differently for each class. For example, the electronic device may obtain a second image (820). The electronic device may obtain a class map (830) that classifies the pixels in the second image (820). Based on the class map (830), the electronic device may obtain a first texture-requiring area classified as a 'floor' class and a second texture-requiring area classified as a 'clothes' class.

[0173] For example, the texture of the floor may be visually rougher than that of the clothing. The electronic device may synthesize a texture image with a relatively high application intensity for the first texture-requiring area. The w value for the first texture-requiring area may be, for example, 0.8. The electronic device may synthesize a texture image with a relatively low application intensity for the second texture-requiring area. The w value for the second texture-requiring area may be, for example, 0.4.

[0174] In one embodiment, the electronic device may include a camera, and w may be applied differently based on camera metadata even within the same class. Camera metadata may be various information automatically stored by the camera when taking a photo. For example, camera metadata may include data such as the properties of the captured image, settings, image resolution, shooting time, manufacturer, ISO sensitivity, shutter speed, focal length, location information, flash information, image file format, and exposure compensation value.

[0175] For example, w can be applied differently based on ISO sensitivity even within the same class, as shown in Table 1.

[0176] ISO sensitivity (photo brightness information in camera metadata) w~1500.05 150~3200.15 320~6400.25 640~0.45

[0177] In one embodiment, the ISO sensitivity may be photo brightness information in the camera metadata. If the ISO sensitivity is 150 or less, it may mean that the second image (820) was acquired by taking a photo outdoors or in clear weather. The electronic device may determine the w value as 0.05 based on the ISO sensitivity. The electronic device may generate an output image (850) according to Equation 2 based on the determined w value. If the ISO sensitivity is 150 or more and 320 or less, it may mean that the second image (820) was acquired by taking a photo indoors or in cloudy weather. The electronic device may determine the w value as 0.15 based on the ISO sensitivity. The electronic device may generate an output image (850) according to Equation 2 based on the determined w value. If the ISO sensitivity is 320 or more and 640 or less, it may mean that the second image (820) was acquired by taking a photo in medium lighting (medium brightness) or in the evening. The electronic device can determine the w value as 0.25 based on the ISO sensitivity. The electronic device can generate an output image (850) according to Equation 2 based on the determined w value.

[0178] If the ISO sensitivity is 640 or higher, this may indicate that the second image (820) was acquired in low light or at night. The electronic device may determine the w value as 0.45 based on the ISO sensitivity. The electronic device may generate an output image (850) according to Equation 2 based on the determined w value.

[0179] In one embodiment, the second image (820) captured in low light or at night may exhibit a strong cartoon effect when zoomed in on the image. Therefore, the electronic device may determine a high w value to strongly impart texture.

[0180] However, mathematical formula 2 and table 1 are only explained as examples and do not limit the technical idea of ​​the present disclosure.

[0181] In one embodiment, compared to the second image (820), which is an input image, the output image (850) has a texture according to a texture image (840) applied to a texture-requiring area classified as a 'floor' class based on a class map (830). In the process of enlarging the first image (810) into the second image (820), the image quality may deteriorate, but by applying the texture according to the texture image (840), the user may be given an illusion that the image quality has improved to the naked eye, and a natural image may be provided.

[0182] In one embodiment, a texture-required area to which a texture image (840) is applied may be preset. For example, if the number of pixels classified as the first class is large, the area of ​​pixels classified as the first class may be wide. The texture-required area may be set based on the area comprised of pixels classified as the same class. The wider the area, the more pronounced the effect of applying the texture image may be.

[0183] As another example, an area of ​​pixels classified as a class such as 'ground', 'floor', or 'wall' may be designated as an area requiring texture. The electronic device may apply a texture image to an area of ​​pixels classified as a class such as 'ground', 'floor', or 'wall'.

[0184] FIG. 9 is a flowchart illustrating a method for synthesizing an input image and a texture image according to one embodiment of the present disclosure.

[0185] For convenience of explanation, parts that overlap with those described using Figures 1 to 8 are simplified or omitted.

[0186] Referring to FIG. 9, step S240 of FIG. 2 may include step S910 and step S920.

[0187] In step S910, the electronic device may obtain a synthesis weight regarding the degree to which the first texture image is applied to the texture-requiring area. For example, the synthesis weight may mean w in mathematical expression 2.

[0188] In one embodiment, the electronic device may determine a synthesis weight based on the class into which the pixels within the texture-requiring region are classified. For example, the electronic device may determine a first synthesis weight corresponding to a first class. The electronic device may determine a second synthesis weight corresponding to a second class. A texture-requiring region classified as the first class may require a stronger texture than a texture-requiring region classified as the second class. In this case, the first synthesis weight corresponding to the first class may be greater than the second synthesis weight corresponding to the second class. Accordingly, the electronic device may apply a texture image to the texture-requiring region classified as the first class based on the relatively higher first synthesis weight.

[0189] In one embodiment, the electronic device can obtain a relationship between a class and a synthesis weight based on user input. The electronic device can store the relationship between the class and the synthesis weight, and can utilize the relationship between the class and the synthesis weight when performing an operation of synthesizing a texture image over a texture-requiring region.

[0190] In one embodiment, the electronic device can determine synthesis weights based on camera metadata. The operation of determining synthesis weights based on camera metadata is described in detail with reference to FIG. 10.

[0191] In step S920, the electronic device can add brightness values ​​of corresponding pixels between the texture-requiring area and the first texture image based on the synthesis weight.

[0192] In one embodiment, the electronic device can acquire a portion of the first texture image to match the texture-requiring region. The electronic device can synthesize the texture-requiring region and the portion of the first texture image. The electronic device can add the brightness values ​​of the pixels of the texture-requiring region and the pixels of the portion of the first texture image, respectively, according to their positions. The electronic device can add the brightness values ​​of the corresponding pixels between the texture-requiring region and the portion of the first texture image, respectively.

[0193] In one embodiment, due to an operation of adding the brightness values ​​of pixels, if the upper limit of the brightness value is exceeded, a value corresponding to the upper limit of the brightness value can be obtained.

[0194] FIG. 10 is a flowchart illustrating a method for determining synthesis weights to synthesize an input image and a texture image according to one embodiment of the present disclosure.

[0195] For convenience of explanation, parts that overlap with those described using Figures 1 to 9 are simplified or omitted.

[0196] Referring to FIG. 10, step S910 of FIG. 9 may include step S1010 and step S1020.

[0197] In step S1010, the electronic device can obtain brightness information of an input image.

[0198] In one embodiment, the electronic device may include a camera and acquire camera metadata. Camera metadata may be various pieces of information automatically stored by the camera when taking a photo. For example, camera metadata may include data such as the properties of the captured image, settings, image resolution, time of capture, manufacturer, ISO sensitivity, shutter speed, focal length, location information, flash information, image file format, and exposure compensation values.

[0199] In one embodiment, the electronic device can obtain brightness information of an input image from camera metadata. The brightness information may be, for example, ISO sensitivity.

[0200] In step S1020, the electronic device can adjust the synthesis weights based on the brightness information.

[0201] In one embodiment, the composite weights can have values ​​between 0 and 1.

[0202] For example, the cartoon effect can be a phenomenon where the edges of an image become softly blurred and colors appear to spread. The cartoon effect can become more pronounced when enlarging an image. Dark photographs, in particular, can exhibit a strong cartoon effect. Conversely, bright photographs can exhibit a less pronounced cartoon effect.

[0203] In one embodiment, the electronic device may determine a synthesis weight based on brightness information. For example, the brightness information may be an ISO sensitivity of 100 or lower. The synthesis weight may be 0.1 based on the brightness information. That is, the electronic device may determine a lower synthesis weight for a brighter image.

[0204] As another example, the brightness information may be at an ISO sensitivity of 600 or higher. The composite weighting may be 0.6 based on the brightness information. That is, the electronic device can determine a higher composite weighting for darker images.

[0205] Hereinafter, with reference to FIG. 11, the configuration of an electronic device for performing the image processing operations described so far will be described. FIG. 11 is a block diagram illustrating the configuration of an electronic device according to an embodiment of the present disclosure.

[0206] For convenience of explanation, parts that overlap with those described using Figures 1 to 10 are simplified or omitted.

[0207] Referring to FIG. 11, an electronic device (1000) according to an embodiment may include an input / output interface (1100), a memory (1200), and a processor (1300). However, the components of the electronic device (1000) are not limited to the above-described examples, and the electronic device (1000) may include more or fewer components than the above-described components. In an embodiment, some or all of the input / output interface (1100), the memory (1200), and the processor (1300) may be implemented in the form of a single chip, and the processor (1300) may include one or more processors.

[0208] The input / output interface (1100) may include an input interface (e.g., touch screen, hard button, microphone, etc.) for receiving control commands or information from a user, and an output interface (e.g., display panel, speaker, etc.) for displaying the results of execution of an operation according to the user's control or the status of the electronic device (1000).

[0209] For example, the electronic device (1000) may acquire an image, enlarge the image, and synthesize a texture image into a texture-requiring area of ​​the enlarged image based on a user's image capture command and image enlargement command obtained through the input / output interface (1100). The processor (1300) of the electronic device (1000) may perform the image processing operations described using FIGS. 1 to 10.

[0210] The memory (1200) is a configuration for storing various programs or data, and may be configured as a storage medium such as a ROM, a RAM, a hard disk, a CD-ROM, and a DVD, or a combination of storage media. The memory (1200) may not exist separately and may be configured to be included in the processor (1300). The memory (1200) may be configured as a volatile memory, a non-volatile memory, or a combination of volatile memory and non-volatile memory. Programs or instructions for performing operations according to the embodiments described with reference to FIGS. 1 to 10 may be stored in the memory (1200). The memory (1200) may also provide stored data to the processor (1300) upon request of the processor (1300).

[0211] The processor (1300) controls a series of processes so that the electronic device (1000) operates according to the embodiments described with reference to FIGS. 1 to 10, and may be composed of one or more processors. One or more processors included in the processor (1300) may be circuitry such as a System on Chip (SoC), an Integrated Circuit (IC), etc. In this case, one or more processors may be a general-purpose processor such as a CPU, an AP, a Digital Signal Processor (DSP), a graphics-only processor such as a GPU, a Vision Processing Unit (VPU), or an artificial intelligence-only processor such as an NPU. For example, when one or more processors are artificial intelligence-only processors, the artificial intelligence-only processor may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

[0212] The processor (1300) can write data to the memory (1200) or read data stored in the memory (1200), and in particular, process data according to predefined operation rules or artificial intelligence models by executing a program or at least one instruction stored in the memory (1200). Accordingly, the processor (1300) can perform the operations described in the embodiments described above, and the operations described as being performed by the electronic device (1000) in the embodiments described above can be regarded as being performed by the processor (1300) unless otherwise specifically described.

[0213] A method according to one embodiment may include a step of obtaining an input image. The method may include a step of obtaining a class map that classifies each pixel in the input image into a class. The method may include a step of generating one or more texture images relating to the texture of a surface. The method may include a step of synthesizing a first texture image from among the one or more texture images onto a texture-requiring region composed of pixels classified into a first class in the input image.

[0214] In one embodiment, the input image may be an image acquired to include multiple objects. The class map may be a classification of classes corresponding to the multiple objects.

[0215] In one embodiment, the step of obtaining an input image may include the step of obtaining a first image. The step of obtaining the input image may include the step of obtaining a region of interest within the first image. The method may include the step of obtaining a second image by enlarging the region of interest. The input image may be the first image or the second image.

[0216] In one embodiment, the step of generating one or more texture images may be generating one or more texture images in which the brightness of each pixel within the image is randomly implemented.

[0217] In one embodiment, the step of generating one or more texture images may include the step of generating a first texture segment in which the brightness of each pixel in the image is randomly implemented. The step of generating one or more texture images may include the step of generating a second texture segment by enlarging a portion of the first texture segment. The step of generating one or more texture images may include the step of generating one or more texture images by overlapping the first texture segment and the second texture segment.

[0218] In one embodiment, the first class may correspond to the first texture image.

[0219] In one embodiment, the step of synthesizing the first texture image onto the texture-requiring area may include adding brightness values ​​of corresponding pixels between the texture-requiring area and the first texture image.

[0220] In one embodiment, the step of synthesizing the first texture image onto the texture-requiring area may include the step of obtaining a synthesis weight regarding the degree to which the first texture image is applied onto the texture-requiring area. The step of synthesizing the first texture image onto the texture-requiring area may include the step of adding brightness values ​​of corresponding pixels between the texture-requiring area and the first texture image based on the synthesis weight.

[0221] In one embodiment, the step of obtaining the synthesis weight may include the step of obtaining brightness information of the input image. The step of obtaining the synthesis weight may include the step of adjusting the synthesis weight based on the brightness information.

[0222] In one embodiment, the synthesis weights may be determined differently for each class to which a pixel within the texture-requiring region to which the first texture image is to be applied belongs.

[0223] An electronic device according to an embodiment may include an input / output interface, a memory, and at least one processor. The input / output interface may receive a user input requesting image processing, and output a processed image according to the user input. The memory may store instructions for processing the image. By having at least one processor execute a program or at least one instruction stored in the memory, the electronic device may obtain an input image, obtain a class map classifying a class to which each pixel in the input image belongs, generate one or more texture images regarding the texture of a surface, and synthesize a first texture image from among the one or more texture images on a texture-requiring area composed of pixels classified into a first class in the input image.

[0224] In one embodiment, the input image may be an image acquired to include multiple objects. The class map may be a classification of classes corresponding to the multiple objects.

[0225] In one embodiment, the electronic device is configured to acquire a first image, acquire a region of interest within the first image, and acquire a second image by enlarging the region of interest by having at least one processor execute a program or at least one instruction stored in a memory, wherein the input image can be the first image or the second image.

[0226] In one embodiment, the electronic device can generate one or more texture images in which the brightness of each pixel in the image is randomly implemented by having at least one processor execute a program or at least one instruction stored in the memory.

[0227] In one embodiment, the electronic device can generate one or more texture images by having at least one processor execute a program stored in a memory or at least one instruction to generate a first texture segment that randomly implements the brightness of each pixel in an image, generate a second texture segment by enlarging a portion of the first texture segment, and overlap the first texture segment and the second texture segment.

[0228] In one embodiment, the electronic device can add brightness values ​​of corresponding pixels between a texture-requiring area and a first texture image by having at least one processor execute a program or at least one instruction stored in a memory.

[0229] In one embodiment, the electronic device can obtain a synthesis weight regarding the degree to which the first texture image is applied to a texture-requiring area by having at least one processor execute a program stored in a memory or at least one instruction, and add brightness values ​​of corresponding pixels between the texture-requiring area and the first texture image based on the synthesis weight.

[0230] In one embodiment, the electronic device can obtain brightness information of an input image and adjust synthesis weights based on the brightness information by having at least one processor execute a program or at least one instruction stored in a memory.

[0231] In one embodiment, the synthesis weights may be determined differently for each class to which a pixel within the texture-requiring region to which the first texture image is to be applied belongs.

[0232] A non-transitory computer-readable recording medium having recorded thereon a program for performing any one of the methods according to one embodiment of the present disclosure on a computer may be provided.

[0233] Various embodiments of the present disclosure may be implemented or supported by one or more computer programs, and the computer programs may be formed from computer-readable program code and embodied in a computer-readable medium. In the present disclosure, "application" and "program" may refer to one or more computer programs, software components, instruction sets, procedures, functions, objects, classes, instances, associated data, or portions thereof suitable for implementation in computer-readable program code. "Computer-readable program code" may include various types of computer code, including source code, object code, and executable code. "Computer-readable medium" may include various types of media that can be accessed by a computer, such as read-only memory (ROM), random access memory (RAM), a hard disk drive (HDD), a compact disc (CD), a digital video disc (DVD), or various types of memory.

[0234] Additionally, a device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, a 'non-transitory storage medium' is a tangible device and may exclude wired, wireless, optical, or other communication links that transmit temporary electrical or other signals. Meanwhile, this 'non-transitory storage medium' does not distinguish between cases where data is permanently stored in the storage medium and cases where it is temporarily stored. For example, a 'non-transitory storage medium' may include a buffer where data is temporarily stored. A computer-readable medium may be any available medium that can be accessed by a computer, and may include both volatile and non-volatile media, and removable and non-removable media. A computer-readable medium includes a medium on which data can be permanently stored and a medium on which data can be stored and later overwritten, such as a rewritable optical disk or an erasable memory device.

[0235] According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0236] The above description of the present disclosure is for illustrative purposes only, and those skilled in the art will appreciate that the present disclosure can be readily modified into other specific forms without altering the technical spirit or essential characteristics of the present disclosure. For example, suitable results can be achieved even if the described techniques are performed in a different order than the described method, and / or components of the systems, structures, devices, circuits, etc. described are combined or combined in a different form than the described method, or are replaced or substituted by other components or equivalents. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. For example, each component described as being single may be implemented in a distributed manner, and similarly, components described as being distributed may be implemented in a combined form.

[0237] The scope of the present disclosure is indicated by the claims described below rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present disclosure.

Claims

1. Step of obtaining an input image; A step of obtaining a class map that classifies the class to which each pixel in the input image belongs; generating one or more texture images relating to the texture of a surface; and A method comprising the step of synthesizing a first texture image among the one or more texture images on a texture-requiring area composed of pixels classified into a first class in the input image.

2. In paragraph 1, The above input image is an image acquired to include multiple objects, A device in which the above class map classifies classes corresponding to the plurality of objects.

3. In either of paragraphs 1 and 2, The step of obtaining the above input image is: Step of acquiring a first image; A step of acquiring a region of interest within the first image; and comprising a step of acquiring a second image by enlarging the region of interest; A method wherein the input image is the first image or the second image.

4. In any one of the clauses 1 to 3, The step of generating one or more texture images comprises: A method for generating one or more texture images in which the brightness of each pixel in the image is randomly implemented.

5. In paragraph 4, The step of generating one or more texture images comprises: A step of generating a first texture segment in which the brightness of each pixel in the image is randomly implemented; A step of generating a second texture segment by enlarging a portion of the first texture segment; and A method comprising the step of generating the one or more texture images by overlapping the first texture segment and the second texture segment.

6. In any one of paragraphs 1 to 5, The step of synthesizing the first texture image onto the texture-requiring area is: A step of obtaining a synthetic weight regarding the degree to which the first texture image is applied to the texture-requiring area; and A method comprising the step of adding brightness values ​​of corresponding pixels between the texture-requiring area and the first texture image based on the synthesis weight.

7. In paragraph 6, The step of obtaining the above synthetic weight is: A step of obtaining brightness information of the input image; and A method comprising a step of determining the synthesis weight based on the brightness information.

8. An input / output interface for receiving user input requesting image processing and outputting an image processed according to the user input; Memory where commands for processing images are stored; and Contains at least one processor, The electronic device, by causing at least one processor to execute a program or at least one instruction stored in the memory, Obtain the input image, Obtain a class map that classifies the class to which each pixel in the input image belongs, Generate one or more texture images of the texture of the surface, An electronic device that synthesizes a first texture image among one or more texture images on a texture-requiring area composed of pixels classified into a first class in the input image.

9. In paragraph 8, The above input image is an image acquired to include multiple objects, An electronic device in which the above class map classifies classes corresponding to the plurality of objects.

10. In any one of paragraphs 8 and 9, The electronic device, by causing at least one processor to execute a program or at least one instruction stored in the memory, Obtain the first image Obtaining a region of interest within the first image, By enlarging the above region of interest, a second image is obtained, An electronic device wherein the input image is the first image or the second image.

11. In any one of the clauses 8 to 10, The electronic device, by causing at least one processor to execute a program or at least one instruction stored in the memory, An electronic device that generates one or more texture images in which the brightness of each pixel in the image is randomly implemented.

12. In paragraph 11, The electronic device, by causing at least one processor to execute a program or at least one instruction stored in the memory, Generate a first texture segment that randomly implements the brightness of each pixel in the image, A second texture segment is generated by enlarging a portion of the first texture segment, An electronic device that generates one or more texture images by overlapping the first texture segment and the second texture segment.

13. In any one of the clauses 8 to 12, The electronic device, by causing at least one processor to execute a program or at least one instruction stored in the memory, Obtain a synthetic weight regarding the degree to which the first texture image is applied to the texture-requiring area, An electronic device that adds the brightness values ​​of corresponding pixels between the texture-requiring area and the first texture image based on the synthesis weight.

14. In paragraph 13, The electronic device, by causing at least one processor to execute a program or at least one instruction stored in the memory, Obtain brightness information of the above input image, An electronic device that determines the synthesis weight based on the brightness information.

15. A computer-readable recording medium having recorded thereon a program for performing the method of any one of clauses 1 to 7 on a computer.

Images