Electronic device, method executed by electronic device, and computer readable medium

Abstract

The present disclosure relates to an electronic device. The electronic device includes: a memory; and a processor, which is used to determine the arrangement of multiple images for multiple areas divided according to a predetermined layout, store information about the determined arrangement in the memory, and control the electronic device to display the multiple images in the multiple areas based on the stored information about the determined arrangement, and the processor can determine the arrangement of the multiple images based on the importance level of at least one object area determined according to the category of at least one object area included in the multiple images and the degree of loss of at least one object area according to the importance level.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is based on and claims the benefit of priority under 35 U.S.C. §119(a) to Korean Patent Application No. 10-2018-0121158 filed in the Korean Intellectual Property Office on October 11, 2018, and the disclosure of the above patent application is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure relates to an electronic device for automatically editing and arranging a plurality of images and a control method thereof, and more particularly, to an electronic device for minimizing the degree of loss of object areas included in a plurality of images when arranging the plurality of images based on a predetermined layout and a control method thereof. Background Art

[0004] Due to the development and commercialization of image processing technology, ordinary people who are not professionals can easily edit images, and it is common practice for individuals to simply edit everyday images directly taken by individuals and publish the everyday images through social networking services / sites (SNS).

[0005] Specifically, a layout that enables a user to easily edit and arrange one or more images has been provided, and various applications that enable images to be arranged on the corresponding layout have been provided.

[0006] Conventionally, when arranging a plurality of images in a plurality of regions in a layout, the images are edited sequentially in each of the plurality of regions in the order in which the images are selected by a user, or the arrangement of the images is determined based only on the connection relationship between the images. Therefore, an object region which is a relatively important part of each image is often cut off. Summary of the invention

[0007] The present disclosure is to solve at least the above problems and / or disadvantages and provide at least the advantages described below. The object of the present disclosure is to provide an electronic device and a control method thereof to identify object areas included in multiple images and determine the arrangement when arranging multiple images according to a predetermined layout so that the loss of the object area is minimized.

[0008] In addition, an object of the present disclosure is to provide an electronic device and a control method thereof to determine an importance level of each object region based on a user's selection and / or a category and to arrange a plurality of images according to a layout based thereon.

[0009] According to an embodiment, an electronic device includes: a memory; and a processor, which is used to determine the arrangement of multiple images for multiple areas divided according to a predetermined layout, store information about the determined arrangement in the memory, and control the electronic device to display the multiple images in the multiple areas based on the stored information about the determined arrangement, and the processor can determine the arrangement of the multiple images based on the importance level of at least one object area corresponding to the category of at least one object area included in the multiple images and the degree of loss of at least one object area according to the importance level.

[0010] The processor may obtain an interest value reflecting the importance level for a unit area of ​​the object area, and when arranging the plurality of images in the plurality of areas, determine an arrangement of the plurality of images so that a degree of loss of the obtained interest value is minimized.

[0011] The processor can identify the arrangement of the first image when arranging the first image among multiple images in each of multiple areas so that the degree of loss of the interest value corresponding to the object area included in the first image is minimized, and identify the degree of loss of the interest value according to the arrangement of the first image, and determine the arrangement of multiple images based on the identified arrangement and loss degree of the first image.

[0012] The processor can identify the arrangement of the second images when arranging the second image among the multiple images in each of the multiple regions so that the degree of loss of the interest value corresponding to the object area included in the second image is minimized, and identify the loss of the interest value according to the arrangement of the second images, and determine the arrangement of the multiple images based on the identified arrangement and loss degree of the second images.

[0013] The processor may determine the importance level of each object region based on the number of object regions corresponding to each category among the object regions included in the plurality of images.

[0014] The processor may obtain a ratio of object regions corresponding to each category among object regions included in a plurality of images, and determine the importance level of an object region corresponding to a category obtaining the highest ratio among the categories from among the object regions included in the plurality of images to be the highest.

[0015] The processor may determine the importance level of the object region based on a user input selecting at least one of the categories of the object regions included in the plurality of images.

[0016] The processor of the electronic device can store information about user input for selecting at least one of the categories of object areas included in multiple images in a memory, and when determining the arrangement of multiple other images of multiple areas divided according to a predetermined layout, determine the importance level of the object areas included in the multiple other images based on the category of at least one object area included in the multiple other images and the stored information about the user input.

[0017] The processor can reduce or enlarge the first image based on the size of each of the multiple regions while maintaining the aspect ratio of the first image, and in response to the aspect ratio of each of the multiple regions being different from the aspect ratio of the first image, identify the arrangement of the first image when only a portion of the reduced or enlarged first image corresponding to the size of each of the multiple regions is arranged in each of the multiple regions so that the degree of loss of interest values ​​corresponding to the object region included in the first image is minimized.

[0018] According to an embodiment, a control method of an electronic device includes: determining an arrangement of multiple images for multiple areas divided according to a predetermined layout, and controlling the electronic device to display the multiple images in the multiple areas based on the determined arrangement, and determining the arrangement of the multiple images may include determining the arrangement of the multiple images based on an importance level of at least one object area corresponding to a category of at least one object area included in the multiple images and a degree of loss of at least one object area according to the importance level.

[0019] Determining the arrangement of the plurality of images may include obtaining an interest value reflecting the importance level for a unit area of ​​the object area, and determining the arrangement of the plurality of images so that a degree of loss of the obtained interest value is minimized when the plurality of images are arranged in the plurality of areas.

[0020] Determining the arrangement of multiple images may include identifying the arrangement of the first image so that the degree of loss of the interest value corresponding to the object area included in the first image is minimized when arranging the first image among the multiple images in each of the multiple areas, and identifying the degree of loss of the interest value according to the arrangement of the first image, and determining the arrangement of the multiple images based on the identified arrangement of the first image and the degree of loss.

[0021] Determining the arrangement of multiple images may include identifying the arrangement of the second images when arranging a second image among the multiple images in each of a plurality of regions so that the degree of loss of an interest value corresponding to an object region included in the second image is minimized, identifying the loss of the interest value based on the arrangement of the second images, and determining the arrangement of the multiple images based on the identified arrangement of the second images and the degree of loss.

[0022] Determining the arrangement of the plurality of images may include determining an importance level of each object region based on the number of object regions corresponding to each category among the object regions included in the plurality of images.

[0023] Determining the arrangement of multiple images may include obtaining the proportion of object areas corresponding to each category among the object areas included in the multiple images, and determining the importance level of the object areas corresponding to the category with the highest proportion among the categories from the object areas included in the multiple images.

[0024] Determining the arrangement of the plurality of images may include determining an importance level of the object region based on a user input selecting at least one of categories of the object region included in the plurality of images.

[0025] Determining the arrangement of multiple images may include storing information about user input of at least one of the categories of object areas included in the multiple images in a memory, and when determining the arrangement of multiple other images of multiple areas divided according to a predetermined layout, determining the importance level of the object areas included in the multiple other images based on the category of at least one object area included in the multiple other images and the stored information about the user input.

[0026] Determining the arrangement of multiple images may include reducing or enlarging the first image based on the size of each of the multiple regions while maintaining the aspect ratio of the first image, and identifying the arrangement of the first image such that the degree of loss of interest values ​​corresponding to the object area included in the first image is minimized when only a portion of the reduced or enlarged first image corresponding to the size of each of the multiple regions is arranged in each of the multiple regions in response to the aspect ratio of each of the multiple regions being different from the aspect ratio of the first image.

[0027] According to an embodiment, a non-temporary computer-readable medium stores computer instructions, which, when executed by a processor of an electronic device, cause the electronic device to perform operations, the operations including: determining an arrangement of multiple images for multiple areas divided according to a predetermined layout; and controlling the electronic device to display the multiple images in multiple areas based on the determined arrangement, and determining the arrangement of the multiple images may include determining the arrangement of the multiple images based on an importance level of at least one object area corresponding to a category of at least one object area included in the multiple images and a degree of loss of at least one object area according to the importance level.

[0028] According to an embodiment, an electronic device and a control method thereof can automatically determine an arrangement in which loss of an object area is minimized when arranging a plurality of images according to a predetermined layout, thereby increasing user convenience.

[0029] Specifically, since the importance level of each object area is determined based on the user's selection and / or category, and the degree of loss of the object area reflecting the importance level is considered, the optimal image arrangement reflecting the user's desired elements when arranging multiple images according to the layout can be determined. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The above and other aspects and advantages of certain examples of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings, in which:

[0031] Figure 1 shows a layout and multiple images for describing an electronic device according to various embodiments;

[0032] Figure 2 is a block diagram for describing a configuration of an electronic device according to an embodiment;

[0033] Figure 3 An example of identifying the importance level of each of the object regions when the object regions are obtained from a plurality of images is shown;

[0034] Figure 4 An example of identifying the importance level of each of the object regions when the object regions are obtained from a plurality of images is shown;

[0035] Figure 5 An example of obtaining a minimum loss value of an interest value when arranging each image in each of a plurality of regions is described;

[0036] Fig. 6A and Figure 6B An example of obtaining a minimum loss value of an interest value when arranging each image in each of a plurality of regions is described;

[0037] Figure 7 An example of obtaining a minimum loss value of an interest value when arranging each image in each of a plurality of regions is described;

[0038] Figure 8 is a table describing minimum loss values ​​of interest values ​​when each image is arranged in each of a plurality of regions, to describe a case where loss of interest values ​​is minimized when a plurality of images are arranged in a plurality of regions;

[0039] Fig. 9 The result of arranging multiple images in multiple regions so that the loss of interest values ​​is minimized is shown;

[0040] Fig.10 is a block diagram for describing a configuration of an electronic device according to various embodiments;

[0041] Fig.11A , Fig. 11B and Fig. 11C is a diagram for describing a configuration of an electronic device according to various embodiments;

[0042] Fig.12 is a flow chart of a control method of another electronic device according to an embodiment;

[0043] Fig.13 Yes Fig.12 A flowchart of an example of an arrangement of multiple images;

[0044] Fig.14 is a flowchart of an example of identifying the importance level of an object area; and

[0045] Fig.15 An algorithm that describes an example of obtaining an interest value reflecting the importance level of an object region and determining an arrangement such that loss of the interest value is minimized when arranging a plurality of images in a plurality of regions. DETAILED DESCRIPTION

[0046] Before describing the present disclosure in detail, a method for presenting the present specification and the accompanying drawings will be described.

[0047] The terms used in this specification and claims are general terms identified in consideration of the functions of the various embodiments of the present disclosure. However, these terms may vary according to the intentions of technicians in the relevant fields, legal or technical interpretations, the emergence of new technologies, etc. In addition, there may be some terms arbitrarily identified by the applicant. Unless there is a specific definition of a term, the term can be interpreted based on the overall content and the technical common sense of technicians in the relevant fields.

[0048] In addition, similar reference numerals indicate similar components that perform substantially the same function throughout the specification. For ease of description and understanding, the same reference numerals or symbols are used and described in different exemplary embodiments. In other words, although the elements with the same reference numerals are all shown in multiple figures, multiple figures do not mean an exemplary embodiment.

[0049] Terms such as "first", "second", etc. may be used to describe various elements, but the elements should not be limited by these terms. The terms are only used for the purpose of distinguishing different elements. For example, elements associated with ordinal numbers should not be limited by numerical order or order of use. If necessary, ordinal numbers can be substituted for each other.

[0050] Unless otherwise stated, singular terms include plural terms. It should be understood that terms such as "comprising" or "consisting of" are used herein to specify the presence of characteristics, numbers, steps, operations, elements, components, or combinations thereof, and do not exclude the presence of one or more other characteristics, numbers, steps, operations, elements, components, or combinations thereof or the possibility of adding one or more characteristics, numbers, steps, operations, elements, components, or combinations thereof.

[0051] Terms such as "module", "unit", "part", etc. are used to refer to elements that perform at least one function or operation, and such elements can be implemented as hardware or software or a combination of hardware and software. In addition, except for the case where each of multiple "modules", "units", "parts", etc. needs to be implemented in separate hardware, the components can be integrated in at least one module or chip and can be implemented in at least one processor.

[0052] In addition, when any part is connected to another part, this includes direct connection and indirect connection through another medium. In addition, when a certain part includes a certain element, unless otherwise specified, it means that another element can be additionally included, rather than excluding another element.

[0053] Figure 1 Layouts and a plurality of images for describing an electronic device and a control method of the electronic device according to various embodiments are shown.

[0054] refer to Figure 1 , seven images 10-1, 10-2, ..., 10-7 and a layout 20 divided into seven regions can be identified. For each of the seven regions in the layout 20, an order 1 to 7 is defined, and a corresponding aspect ratio (eg, 1.17:1) is indicated.

[0055] The present disclosure relates to an electronic device for determining (or recognizing) the arrangement of multiple images for multiple areas divided according to a layout and a control method thereof. Figure 1 An example of a detailed process in which seven images 10 - 1 , 10 - 2 , . . . , 10 - 7 are arranged in seven areas divided according to the layout 20 is shown.

[0056] Figure 2 is a block diagram for describing a configuration of an electronic device 100 according to an embodiment.

[0057] The electronic device 100 may be a display device that is implemented as a terminal device such as a mobile phone, a smart phone, etc., or may display one or more images such as a TV, a smart TV, a notebook PC, a desktop PC, a tablet PC, a self-service terminal, and a monitor. In addition, the electronic device 100 may be implemented as a set-top box, an over-the-top (OTT), or a server, and may transmit data about an image to an external device wirelessly or by wire, so that the external device including a display may display one or more images.

[0058] refer to Figure 2 , the electronic device 100 may include a memory 110 and a processor 120 .

[0059] The memory 110 is a configuration for storing instructions or data related to an operating system (OS) and elements of the electronic device 100 for controlling overall operations of the elements of the electronic device 100 .

[0060] To this end, the memory 110 may be implemented as a nonvolatile memory (eg, a hard disk, a solid state drive (SSD), and a flash memory), a volatile memory, or the like.

[0061] The memory 110 may store information about predetermined one or more layouts or information about one or more images.

[0062] Layout may mean a method for editing and arranging one or more images. When arranging one or more images on a specific screen, the corresponding screen may be divided into one or more areas according to the layout. In the divided areas, images may be arranged in each divided area accordingly, and the same number of images as the number of areas divided according to the layout may be arranged in the area inside the layout. In addition, the arranged images may be displayed by the electronic device 100 or an external device (not shown).

[0063] The information about the layout may include the number of regions divided according to the layout, a horizontal / vertical size or ratio of each region, a position of each region, a number or order of each region, and the like.

[0064] The image stored in the memory 110 may be obtained based on image data received from the outside through a communicator (not shown), an input / output port (not shown), a broadcast receiver (not shown), etc. of the electronic device 100. The image may be an image drawn based on a user's manipulation of at least one button, keyboard, mouse, etc. included in a user input device (not shown) of the electronic device 100 and / or a user input to a touch pad / panel included in the user input device (not shown). Alternatively, the image may be an image captured and obtained by at least one camera (not shown) included in the user input device (not shown) of the electronic device 100.

[0065] The memory 110 may store information about the arrangement of the plurality of images divided according to the predetermined layout. Specifically, information about which image of the plurality of images is arranged in each of the plurality of divided regions, the ratio of enlargement / reduction of the image when each image is arranged in each region, and information about the region in the image that is cut out when each image is arranged in each region may be stored. At this time, the arrangement of the plurality of images may be determined by the processor 120.

[0066] In addition, the memory 110 may store information about objects included in each image, information about the category of each object region, information about the category selected by the user, etc. This will be combined with the information obtained by Figure 3 and Figure 4 The operation of the electronic device 100 is shown and described together.

[0067] The processor 120 may control the overall operation of the electronic device 100 .

[0068] For this purpose, the processor 120 may include a random access memory (RAM) (not shown), a read-only memory (ROM) (not shown), a central processing unit (CPU) (not shown), a graphics processing unit (GPU) (not shown), a system bus (not shown), etc., and may perform operations or data processing related to control of other elements included in the electronic device 100.

[0069] The processor 120 may determine (or recognize) the arrangement of the plurality of images for the plurality of regions divided according to the predetermined layout. At this time, the processor 120 may determine the arrangement of the plurality of images temporarily or non-temporarily stored in the memory 110, or may determine the arrangement of the plurality of images stored in the external device (not shown) using information about at least one of identification numbers, sizes, objects, and categories of the plurality of images stored in the external device (not shown). The processor 120 may store the information about the plurality of determined images on the memory 110.

[0070] At this time, the processor 120 may determine the arrangement of the plurality of images based on an importance level of an object region according to a category of at least one object region included in the plurality of images and a loss degree of at least one object region according to the importance level.

[0071] In this case, the processor 120 may first determine (or identify) the importance level of the object region based on the category of at least one object region included in the plurality of images. When arranging the plurality of images in the plurality of regions, the arrangement of the plurality of images relative to the plurality of regions may be determined based on the degree of loss of at least one object region according to the importance level.

[0072] The object region may be a region in the image including an object with a high weight or a region including a predetermined object. The object region is not necessarily one per image, and there may be no object region or at least two object regions for a specific image.

[0073] In general, an object region is regarded as a relatively important region in an image, and therefore, when editing and arranging a plurality of images to correspond to a plurality of regions divided according to a layout, it is necessary to reduce the degree of loss of the object region.

[0074] The processor 120 may extract the object region using information on the color and the edge of each of the plurality of images.

[0075] For example, the processor 120 may detect edges in an image based on a Canny algorithm, a Sobel algorithm, etc. The detected edge is a high-frequency component that sensitively responds to human visual perception ability and is therefore suitable as an element for detecting an object region.

[0076] The processor 120 may identify the color distribution of the vicinity surrounding the detected edge. Specifically, the processor 120 may obtain the average value of the color values ​​(at least one of red, green, and blue) of the pixels included in a block of a specific size centered on the edge region in the image, and then calculate the color distribution of the region smaller than the average value and the color distribution of the region larger than the average value, respectively. Thereafter, the processor 120 may calculate the value of each pixel by combining the degree of high frequency surrounding the edge region with the degree of color distribution, and then identify the region including the pixel with the highest value as the object region.

[0077] The processor 120 may recognize an object included in an image using an artificial intelligence (AI) model that learns to recognize an object included in an image by analyzing the image and determine a region including the object as the object region.

[0078] For this purpose, the memory 110 may store an AI model that learns to analyze an image and recognize one or more objects included in the image. At this time, the AI ​​model may be stored in the memory 110 after learning through an external device (not shown) or the like, or may be learned through the processor 120, which includes a dedicated AI chip such as a deep neural network (DNN) for machine learning.

[0079] Alternatively, the processor 120 may receive a determination result of an AI model stored in an external device (not shown) in the form of data, and recognize an object included in an image based on the determination result by communicating with the external device (not shown).

[0080] The object may be an animal, the sky, the ground, the sea, the sun, the stars, the moon, a human being, various foods, a human face, other things, or may be an object corresponding to a natural element such as a flower, grass, a tree, etc. The object recognized by the processor 120 becomes different according to the data used for learning of the AI ​​model recognized by the processor 120, and therefore, the object is not limited to the above examples.

[0081] Figure 3 An example of identifying an object region from a plurality of images by the processor 120 is shown.

[0082] refer to Figure 3 , the processor 120 can identify object regions corresponding to natural scenery and pizza from the first image 30-1, and identify object regions corresponding to faces and food from the second image 30-2. In this way, a total of 14 various object regions are identified from the seven images 30-1, 30-2, ..., 30-7.

[0083] The processor 120 may identify the category corresponding to each identified object region based on pre-stored information about the category. In this case, the category corresponding to the pre-stored information may be variously predetermined, such as face, food, nature, animal, plant, etc.

[0084] The processor 120 may determine the importance level of each object region based on the category of the object included in each object region. For example, an object region including objects such as pizza, hamburger, etc. may be identified as corresponding to the "food" category.

[0085] For a predetermined category, the processor 120 may identify its priority as higher than another category. Specifically, for an object area corresponding to a predetermined category, the importance level of an object area corresponding to another category may be identified as higher. When determining to edit and arrange a plurality of images so as to insert the plurality of images into a plurality of areas divided according to a layout, the processor 120 determines the editing and arranging so as to minimize the degree of loss of the object areas included in the plurality of images. At this time, even if each object area loses the same area, it may be determined that the degree of loss of the object area with a higher importance level is greater.

[0086] The processor 120 may determine (or identify) the importance level of each object region based on the number of object regions corresponding to each category among object regions included in the plurality of images.

[0087] Specifically, the processor 120 can obtain the proportion of object areas corresponding to each category among the object areas included in multiple images, and determine the importance level of the object area corresponding to the category with the highest obtained proportion among the object areas included in the multiple images as the highest.

[0088] Figure 3 The object regions identified in the seven images 30-1, 30-2, ..., 30-7 correspond to one of food, nature, and face, respectively. At this time, the processor 120 can recognize that, among the 14 object regions included in the seven images 30-1, 30-2, ..., 30-7, six correspond to the "food" category, five correspond to the "nature" category, and three correspond to the "face" category. Among the 14 object regions, the processor 120 can recognize the importance of the object region corresponding to the "food" category with the highest proportion as the highest, and recognize the importance of the object region corresponding to the "nature" category with the second highest proportion as the second highest.

[0089] The above example in which the processor 120 recognizes importance based on the category ratio of the object area is based on the point that the category corresponding to the largest object area among the object areas included in the plurality of images is highly likely to be the category matching the aesthetics or intention of the user who photographed or obtained the plurality of images.

[0090] The processor 120 may identify the importance of the category and the object region based on direct user input. Specifically, for a category selected based on user input, it may be identified as having a higher importance than another category.

[0091] At this time, the processor 120 may set the ranking of the importance level of each category stored based on a user input for selecting at least one of the plurality of categories stored in the memory 110. Therefore, the processor 120 may identify the importance level of the object area corresponding to the category with high importance as higher than the importance level of the object area corresponding to the category with low importance set based on the user input.

[0092] The processor 120 may obtain object regions included in a plurality of images, and then determine the importance of each object region based on a user input for selecting at least one of the categories of the obtained object regions.

[0093] For example, when Figure 3 When object areas corresponding to the categories of "food", "nature", and "face" are obtained in the seven images 30-1, 30-2, ..., 30-7 shown, if a user input for setting the ranking of importance levels in the order of "face", "nature", and "food" is received, the processor 120 may identify the importance of the object area corresponding to the category of the "face" category as the highest level, and identify the importance of the object area corresponding to the category of the "nature" category as the second highest level.

[0094] The processor 120 may obtain object regions included in a plurality of images, and then determine the importance of each object region based on a user input directly selecting at least one of the obtained object regions.

[0095] Meanwhile, the processor 120 may determine importance levels of object regions included in other multiple images based on a user input history for selecting at least one of the object regions included in a plurality of specific images or at least one of the categories of the object regions.

[0096] Specifically, the processor 120 may store information about user input for selecting at least one of the categories of object areas included in multiple images in the memory 110, and when determining the arrangement of multiple other images for multiple areas divided according to the above-mentioned layout or another predetermined layout, the processor 120 may determine the importance level of the object areas included in the multiple other images based on the category of at least one object area included in the multiple other images and the stored information about the user input.

[0097] For example, when there is a history of receiving user input for selecting a "face" category from among categories of object areas included in multiple images, the processor 120 may store the history in the memory 110, and reflect the history when determining the importance levels of object areas included in multiple other images, and determine the importance of the object area corresponding to the "face" category among the object areas included in the above-mentioned multiple other images as the highest.

[0098] Meanwhile, if the importance level is identified through the above embodiment, the processor 120 may reflect the identified importance to each object region and arrange multiple images in multiple regions in a layout so that the loss degree of the object region reflecting the importance can be minimized.

[0099] Specifically, the processor 120 can obtain an interest value reflecting the importance of identifying each object area for a unit area of ​​the object area, and when arranging multiple images in multiple areas, can determine the arrangement of the multiple images so that the degree of loss of the obtained interest value can be minimized.

[0100] Figure 4 40 - 1 and 40 - 2 , . . . , and 40 - 7 , the brightness of each object region is displayed differently according to the importance (interest value). Figure 4 The darker the area with higher interest value is, the darker (darker) it appears.

[0101] like Figure 4 As shown, when the importance of categories is identified in the order of object areas corresponding to “food”, object areas corresponding to “nature”, and object areas corresponding to “face”, the processor 120 may, for example, assign an interest value of “3” to each unit area of ​​the object area corresponding to “food”, assign an interest value of “2” to each unit area of ​​the object area corresponding to “nature”, and assign an interest value of “1” to each unit area of ​​the object area corresponding to “face”.

[0102] At this time, if a predetermined area has to drop one of the object area corresponding to "food" and the object area corresponding to "nature" when arranging multiple images in multiple areas, the processor 120 can determine the arrangement of the multiple images so that the object area corresponding to "nature" is dropped by the corresponding area in consideration of the interest value.

[0103] Meanwhile, in the above-mentioned embodiment, regarding the internal area of ​​each object, the description has been made on the premise that the processor 120 obtains the same interest value for each unit area, but the processor 120 may identify the importance level of the area corresponding to the recognized object among the object areas as higher than the area not corresponding to the recognized object. Figure 3 In the first image 30-1, since the pizza does not fill the quadrilateral object area in the object area including the pizza, the processor may identify the interest value of the area corresponding to the pizza itself among the object area as being higher than the interest value of the remaining area.

[0104] When determining the arrangement of the plurality of images for the plurality of regions divided according to the layout, in order to determine the arrangement that enables minimizing the degree of loss of the interest value, the processor 120 may determine the optimal arrangement in which the degree of loss of the interest value corresponding to each image is minimized when arranging each of the plurality of images in each of the plurality of regions, and may identify the degree of loss of the interest value. In addition, the processor 120 may determine the arrangement of each of the plurality of images for each of the plurality of regions in consideration of the determined optimal arrangement and the degree of loss of the interest value, that is, the arrangement in which the degree of loss of the interest value of the object region included in the plurality of images is minimized.

[0105] Specifically, when a first image among multiple images is arranged in each of multiple regions respectively, the processor 120 can identify the arrangement of the first image so that the degree of loss of the interest value corresponding to the object region included in the first image is minimized, and identify the degree of loss of the interest value.

[0106] At this time, the processor 120 may reduce or enlarge the first image while maintaining the aspect ratio based on the size of each of the plurality of regions. If the aspect ratio of each of the plurality of regions is different from the aspect ratio of the first image, when only a portion corresponding to the size of each of the plurality of regions in the reduced or enlarged first image is arranged in each of the plurality of regions, the arrangement of the first image may be identified so that the degree of loss of the interest value corresponding to the object region included in the first image is minimized. In addition, in this case, the degree of loss of the interest value may be identified.

[0107] For example, if the aspect ratio of the first image is 2:1, when the aspect ratio of the specific area where the first image is to be arranged is 1:1, the corresponding area should be filled with the first image, and the processor 120 can enlarge or reduce the first image while maintaining the aspect ratio of the first image, so that the vertical size of the first image is equal to or greater than the vertical size of the corresponding area. At this time, if the processor 120 enlarges or reduces the first image so that the vertical size of the first image is equal to the vertical size of the corresponding area, the horizontal size of the corresponding area is twice the horizontal size of the first image. Therefore, the processor 120 can remove the area corresponding to half the size of the horizontal size of the enlarged or reduced first image, so that the degree of loss of interest value is minimized, and the first image is arranged in the corresponding area.

[0108] Meanwhile, if the information about the size of the layout has not been determined, in other words, if only the aspect ratio of each of the plurality of regions divided according to the layout has been determined, when arranging the first image in each of the plurality of regions, the processor 120 may arrange only a portion of the first image corresponding to the aspect ratio of each of the plurality of regions in each of the plurality of regions. In addition, when the information about the size of the layout is determined later, the processor 120 may enlarge or reduce a portion of the first image corresponding to the aspect ratio of each of the plurality of regions to correspond to each of the plurality of regions and be arranged in each of the plurality of regions.

[0109] For example, when the aspect ratio of the first image is 2:1 and the aspect ratio of the specific area in which the first image is to be arranged is 1:1, if the size of the corresponding area has not yet been determined, when the first image is arranged in the corresponding area, the first image must fill the corresponding area. Therefore, the processor 120 can obtain an image of a portion corresponding to an aspect ratio of 1:1 from the first image, and arrange it in the corresponding area. At this time, the processor 120 can obtain a partial image including as many interest values ​​of the first image as possible. In addition, when the information about the size of the layout is determined and the size of the corresponding area is determined, the aspect ratio of the partial image and the corresponding area is already the same as 1:1, and therefore the processor 120 can enlarge or reduce the partial image according to the size of the corresponding area, and arrange the partial image in the corresponding area.

[0110] Figure 5 It is shown that when a specific image 10-7 among seven images 10-1, 10-2, ..., 10-7 is arranged in each of a plurality of regions divided according to the layout 20, the image 10-7 is arranged in each of the plurality of regions according to an optimal arrangement in which the degree of loss of interest values ​​corresponding to object regions included in the image 10-7 is minimized.

[0111] exist Figure 5 , with reference to the upper left region 510 of the seven regions divided according to the layout 20, the result of matching the horizontal sizes of the upper left region 510 and the image 10-7 is that the vertical length of the image 10-7 having an aspect ratio different from that of the upper left region 510 becomes greater than the vertical length of the upper left region. Therefore, a portion of at least one of the two object regions 520 and 530 included in the image 10-7 must be cut off. Therefore, when the image 10-7 is arranged on the upper left region 510, it can be recognized that the lost area of ​​the object region 520 corresponding to the "food" category having a relatively high importance level (interest value per unit area) is arranged to be smaller than the lost area of ​​the object region 530 corresponding to the "nature" category.

[0112] Meanwhile, in determining the degree of minimizing the loss of the interest value, the processor 120 may use an integral image in which the interest value is reflected as a number when arranging the enlarged or reduced image on a specific area.

[0113] FIG. 6A to FIG. 6B is a view of an example of using an integral image reflecting an interest value of a specific image when the processor 120 determines an arrangement of a specific image for a specific region.

[0114] refer to Fig. 6A, when inserting the specific image 610 into the specific area 650, the horizontal size of the image 610 is larger than the horizontal size of the corresponding area 650, and thus in order to arrange the image 610 in the corresponding area 650, a portion of the image 610 needs to be removed.

[0115] Fig. 6A Assume the following case: when the processor 120 arranges the image 610 in the corresponding area 650, the processor 120 needs to determine the arrangement so that the degree of loss of the interest value of the object area included in the image 610 becomes minimum.

[0116] refer to Figure 6B , the processor 120 can obtain an image 612 that reflects the interest value and is digitized for each unit area of ​​the image. For example, the upper left area 612-1 of the digitized image 612 does not include an object area and therefore has a relatively small interest value of "7", while in a relatively wide area 612-2 at the lower left, an object area with an importance level is located and the interest value is "8".

[0117] refer to Figure 6B , the processor 120 can obtain the integral image 614 from the digitized image 612. The processor 120 can obtain the integral image 614 by setting the value obtained by adding all the values ​​of the coordinate, the left area of ​​the coordinate, the upper area of ​​the coordinate, and the upper left area of ​​the coordinate as the numerical value of the corresponding coordinate based on the specific coordinate of the digitized image 612 with respect to the specific coordinate of the digitized image 612. For example, "48" (48 = 7 + 9 + 8 + 8 + 8 + 8) located at (3 (row), 2 (column)) on the integral image 614 indicates that the numbers on the left side, the upper side, and the upper left side are added up, including the "8" located at (3, 2) on the digitized image 612.

[0118] refer to Figure 6B , the case of arranging the image 610 in the corresponding area 650 includes the case of arranging 650-1 the left area of ​​the image 610, the case of arranging 650-2 the central area of ​​the image 610, and the case of arranging 650-3 the right area of ​​the image 610. The processor 120 can determine one arrangement from the above three arrangements as the best arrangement, in which the loss degree of the interest value is the smallest, that is, the loss degree of the value on the digitized image 612 is the smallest.

[0119] refer to Figure 6B , the processor 120 may determine the third arrangement 650-3 having the largest sum ("96") of the numerical values ​​corresponding to the parts arranged in the corresponding area 650 as the best arrangement having the least degree of loss of interest values.

[0120] When arranging a first image among multiple images in each of multiple regions, the processor 120 can identify the arrangement of the first image so that the degree of loss of interest values ​​is minimized, and identify the degree of loss of interest values ​​corresponding to the object region included in the first image.

[0121] Figure 7 The optimal arrangement is shown in which the degree of loss of the interest value corresponding to the image 10-7 is minimized when a specific image 10-7 among the seven images 10-1, 10-2, ..., 10-7 is arranged in each area in the layout 20. Figure 7 , in each region in the layout 20, the degree of loss of the interest value according to the best arrangement for each region according to the image 10-7 is expressed as a ratio.

[0122] The processor 120 may determine the arrangement of the plurality of images based on the optimal arrangement of the first image for each region and the degree of loss of the interest value at that time. Specifically, when arranging the plurality of images in the plurality of regions, the processor 120 may determine the arrangement of the plurality of images so that the degree of loss of the interest value corresponding to the object region included in the plurality of images is minimized.

[0123] For each other image among the plurality of images and the first image, the processor 120 may identify an optimal arrangement for each of the plurality of regions and a degree of loss of interest values ​​according to the optimal arrangement.

[0124] Specifically, when arranging a second image that is not a first image among a plurality of images in each of a plurality of regions, the processor 120 may identify the (optimal) arrangement of the second images so that the degree of loss of the interest value according to the object region included in the second image is minimized, and identify the degree of loss of the interest value at this time. In addition, the processor 120 may determine the arrangement of the plurality of images based on the identified optimal arrangement of the second images and the degree of loss of the interest value at that time.

[0125] In other words, when multiple images are arranged in multiple areas, the arrangement of the multiple images can be determined based on all of the following so that the degree of loss of interest values ​​corresponding to the object areas included in the multiple images is minimized: the previously identified optimal arrangement of the first images, the degree of loss of interest values ​​based thereon, the identified optimal arrangement of the second images, and the degree of loss of interest values.

[0126] When the number of the plurality of images is three or more and includes one or more other images in addition to the first image and the second image, the processor 120 may identify an optimal arrangement of each of the plurality of regions and a degree of loss of interest values ​​of the other images.

[0127] As described above, since the optimal arrangement of each of the multiple images for each of the multiple regions and the degree of loss of interest values ​​corresponding thereto are respectively identified, the processor 120 can use the identification results to determine the optimal arrangement, that is, the arrangement in which the degree of loss of interest values ​​corresponding to the multiple images is minimized.

[0128] Figure 8 is a table describing minimum loss values ​​of interest values ​​when each image is arranged in each of a plurality of regions, to describe a case where loss of interest values ​​is minimized when a plurality of images are arranged in a plurality of regions.

[0129] exist Figure 8 In FIG. 1 , a row corresponds to each of a plurality of images 10-1, 10-2, ..., 10-7, and an inner column corresponds to each of a plurality of regions (see Figure 1 ).

[0130] refer to Figure 8 , in the case of the first image 10-1, if the first image 10-1 is arranged in the sixth region among the plurality of regions, the loss of the interest value can be minimized (4.7), and in the case of the second image, if the second image is arranged in the sixth region, the loss of the interest value can be minimized (1.5). However, the problem is that both the first image and the second image may not be arranged in the sixth region.

[0131] Therefore, the processor 120 may use the Hungarian algorithm of the N×M optimization matching algorithm so that the loss of interest values ​​corresponding to all of the multiple images is minimized.

[0132] Therefore, reference Figure 8 , the processor 120 can determine that when the first image 10-1 is arranged in the first area, the second image 10-2 is arranged in the sixth area, the third image 10-3 is arranged in the seventh area, the fourth image 10-4 is arranged in the third area, the fifth image 10-5 is arranged in the fourth area, the sixth image 10-6 is arranged in the second area and the seventh image 10-7 is arranged in the fifth area, the loss of interest values ​​of the object areas included in the multiple images 10-1, 10-2, ..., 10-7 can be minimized (at this time, when arranging each image in each area, the arrangement naturally follows the optimal arrangement of each image for each identified area).

[0133] Finally, the processor 120 may finally determine the arrangement of the multiple images for the multiple regions divided according to the layout. The processor 120 may then store information about the determined arrangement on the memory 110. Thereafter, the processor 120 may control the electronic device 100 to display multiple images on multiple regions based on the information about the determined arrangement stored in the memory 110. At this point, the processor 120 may control a display (not shown) included in the electronic device 100 to display the multiple images. Alternatively, in order for an external display device (not shown) to display multiple images, data about the multiple images and / or data about the determined arrangement may be transmitted to the external display device (not shown) wirelessly or by wire through a communicator (not shown) or an input / output port (not shown) of the electronic device 100.

[0134] Fig. 9 The following example is shown: when seven of the images 10-1, 10-2, ..., 10-7 are arranged in a plurality of areas divided according to the layout 20, an arrangement in which the degree of loss of interest value is minimized is finally determined, and the seven images 10-1, 10-2, ..., 10-7 are displayed accordingly.

[0135] Meanwhile, the above-mentioned embodiment mainly describes the case where a plurality of images are arranged in a plurality of regions. However, when one image is arranged in one region by the processor 120, the interest value of each object region can be obtained based on the importance of each of at least one object region included in the corresponding image, and the arrangement of the corresponding images in which the loss of the interest value is minimized can be determined.

[0136] Fig.10 and FIG. 11A to FIG. 11C are block diagrams and views for describing the configuration of the electronic device 100 according to various embodiments.

[0137] refer to Fig.10 In addition to the memory 110 and the processor 120 , the electronic device 100 may further include at least one of a communicator 130 , a broadcast receiver 140 , an input / output port 150 , a user input 160 , a display 170 , and an audio outputter 180 .

[0138] The communicator 130 is used to perform data communication (not shown) with an external device (not shown) by wire or wirelessly. The processor 120 can communicate with various external devices using the communicator 130.

[0139] When performing data communication with an external device using a wireless communication method, the communicator 130 may include at least one of the following: a WIFI direct communication module, a Bluetooth module, an infrared data association (IrDA) module, a near field communication (NFC) module, a Zigbee module, a cellular communication module, a third generation (3G) mobile communication module, a fourth generation (4G) mobile communication module, and a 4G long term evolution (LTE) communication module.

[0140] When performing data communication with an external device using a wired communication method, the communicator 130 may be connected to a coaxial cable, an optical fiber cable, etc. to perform a local area network (LAN) communication to transmit / receive various data.

[0141] The processor 120 may receive information about one or more images and / or layouts from the outside through the communicator 130 .

[0142] For example, when the electronic device 100 is implemented as a mobile phone or a smart phone (such as Fig. 11C ) and when communication with an external device 200 (which is a server) is available through the communicator 130, the electronic device 100 can receive data of one or more images and / or layouts from the external device 200 (which is a server), and can determine the arrangement of images for one or more areas distinguished according to the layout.

[0143] The communicator 130 may separately include a first communicator (not shown) for transmitting and receiving data with a remote control device (not shown) to control the electronic device 100, and a second communicator (not shown) for transmitting and receiving data with an external server (not shown). In this case, the first communicator (not shown) may be implemented as an Ethernet modem, a Wi-Fi module, a Bluetooth module, etc., and the second communicator may be implemented as a separate Wi-Fi module, an Ethernet module, etc.

[0144] The broadcast receiver 140 may receive a signal regarding broadcast content. The broadcast content may include images, audio, and additional data (e.g., an electronic program guide (EPG)), and the broadcast receiver 140 may receive broadcast content signals from various sources such as terrestrial broadcasting, cable broadcasting, satellite broadcasting, Internet broadcasting, and the like.

[0145] The broadcast receiver 140 may be implemented to include a configuration for receiving broadcast contents transmitted from a broadcaster, such as a tuner (not shown), a demodulator (not shown), and an equalizer (not shown).

[0146] Through the input / output port 150 , the electronic device 100 may receive image and / or audio signals from the outside or transmit image and / or audio signals to the outside.

[0147] For this purpose, the input / output port 150 may be implemented as a wired port such as a High-Definition Multimedia Interface (HDMI) port, a display port, a Red, Green, and Blue (RGB) port, a Digital Video Interface (DVI) port, a Thunderbolt port, a component port, etc. Alternatively, the input / output port 150 may be implemented as a port for wireless communication such as Wi-Fi and Bluetooth communication.

[0148] The input / output port 150 may be implemented as an HDMI port or a Thunderbolt port so as to also transmit image and audio signals, but a first port transmitting an image signal and a second port transmitting an audio signal may be separately implemented.

[0149] The input / output port 150 may include an interface module such as a USB, and transceive audio or image data or transceiver firmware data to perform firmware upgrade through physical connection with an external terminal device such as a PC via the interface module.

[0150] For example, Fig.11A As shown, when the electronic device 100 is a display device including a display such as a TV, the electronic device 100 may receive one or more images from an external device 200 such as a set-top box. In addition, the processor 120 may determine an arrangement for inserting one or more input images into a plurality of regions divided according to a predetermined layout.

[0151] As another example, when the electronic device 100 is implemented as Fig. 11B When the electronic device 100 is connected to a set-top box as shown in the figure, the electronic device 100 can determine the arrangement of the plurality of images for the plurality of areas divided according to the layout, and then transmit the data of the plurality of images arranged in the plurality of areas to the external device 200 with a display such as a TV through the input / output port 150 according to the determined layout. Therefore, the external device 200 can display the plurality of images arranged in the plurality of areas according to the arrangement determined by the electronic device 100.

[0152] The user inputter 160 is a configuration that enables the electronic device 100 to receive a user input including a command or information from a user.

[0153] The processor 120 may receive a user input for selecting at least one category corresponding to at least one object region or at least one object region included in one or more images through the user inputter 160 .

[0154] The user inputter 160 may include at least one button (not shown), a keyboard (not shown), and a mouse (not shown) for receiving a user input of selecting at least one object region or category.

[0155] In order to receive a user input for selecting at least one object region or category in a touch format, the user inputter 160 may include a touch panel (not shown) implemented together with the display 170 or a separate touch pad (not shown).

[0156] The user input 160 may obtain at least one image based on user manipulation of a button, keyboard, mouse provided in the user input 160 and / or user input to a touch panel / touch pad provided in the user input 160. That is, the user may draw one or more images through the user input 160.

[0157] The user inputter 160 may include one or more cameras (not shown) and obtain an image photographed by the camera (not shown). At this time, the one or more cameras (not shown) may be digital cameras including an optical sensor.

[0158] The user inputter 160 may include a microphone (not shown) to receive a user input in voice to select at least one object region or category.

[0159] At the same time, if Fig.11A or Fig. 11B As shown, if there is a remote controller 300 for controlling the electronic device 100, the electronic device 100 can receive data on the user's voice received by a microphone (not shown) provided in the remote controller 300 from the remote controller, and can select at least one object area or category based on the received user input. At this time, when an analog audio signal is received by a microphone (not shown) provided in the remote controller 300, the remote controller 300 can digitize the signal and transmit the signal to the electronic device 100 via Bluetooth or Wi-Fi communication.

[0160] Specifically, the processor 120 can recognize data on the user's voice received through sound from a microphone (not shown) set in the user input 160 or a microphone (not shown) set in the remote controller 300 and convert the data into text, and recognize the content of the user input to select at least one object area or category based on the converted text.

[0161] Meanwhile, the remote controller 300 can be implemented as a controller for controlling the electronic device 100 (such as Fig.11A or Fig. 11BThe remote controller 300 may be a remote controller of an electronic device 100 (which is a TV or a set-top box), or may be implemented to control the electronic device 100 (which is a TV or a set-top box) because the remote control application is installed and stored in a terminal device such as a smart phone. At this time, the remote controller 300 may transmit a control signal to the electronic device 100 based on Wi-Fi, Bluetooth, infrared communication, etc., and the electronic device 100 may include a first communication unit (not shown) implemented as a Wi-Fi module, a Bluetooth module, an infrared communication module, etc., which is capable of receiving a control signal of the remote controller 300 according to the aforementioned communication method, and a second communication unit implemented as a Wi-Fi module, an Ethernet module, etc., which is capable of communicating with an external device (not shown) such as a server. The first communication unit and the second communication unit may be implemented as one or separately.

[0162] The display 170 is a configuration that displays one or more images according to the control of the processor 120. For this purpose, the display 170 may be implemented as a liquid crystal display (LCD), a plasma display panel (PDP), an organic light emitting diode (OLED), a transparent OLED (TOLED), etc. In the case of an LCD, the display 170 may include a driving circuit (not shown) and a backlight unit (not shown), which may be implemented in the form of an amorphous silicon TFT, a low temperature polycrystalline silicon (LTPS) TFT, an organic TFT (OTFT), etc.

[0163] Meanwhile, the display 170 may be implemented as a touch screen including a touch panel capable of sensing a user's touch manipulation. In this case, the display 170 may be included in the user input 160 .

[0164] The display 170 can display images included in the broadcast signal received through the broadcast receiver 140, images included in the data received through the communicator 130, images obtained through the user input 160, images included in the image file stored in the memory 110, images included in the signal received from the outside through the input / output port 150, and the like.

[0165] The audio outputter 180 is a configuration for outputting specific audio under the control of the processor 120. The audio outputter 180 may output audio included in a broadcast signal received through the broadcast receiver 140, audio included in a signal received through the communicator 130, a microphone (not shown) provided in the user inputter 160, audio input through an external microphone (not shown) and received by the electronic device 100, audio included in an audio file stored in the memory 110, audio included in a signal received from the outside through the input / output port 150, and the like.

[0166] For this purpose, the audio output 180 may be implemented as a speaker 190 and / or a headphone / earphone output terminal (not shown).

[0167] pass Figures 12 to 15 , a control method of an electronic device will be described.

[0168] Fig.12 is a flow chart of another control method of an electronic device according to an embodiment. Fig.12 In step S1210, the control method may determine (or identify) the arrangement of the plurality of images for the plurality of regions divided according to the predetermined layout, and in step S1220, control the electronic device to display the plurality of images in the plurality of regions based on the determined arrangement. At this time, the plurality of images arranged according to the determined arrangement may be displayed on the display of the electronic device, or on the display of an external device that receives data of the plurality of images from the electronic device.

[0169] When determining the arrangement of multiple images in step S1210, the control method can determine the arrangement of multiple images based on the importance of at least one object area corresponding to the category of at least one object area included in the multiple images and the loss degree of at least one object area according to the importance. Fig.13 is a flowchart for describing this operation according to a more specific operation sequence.

[0170] refer to Fig.13 , when determining the arrangement of the plurality of images in step S1310, the control method may determine the importance of the object region based on a category of at least one object region included in the plurality of images.

[0171] The object region may be a region including an object with high importance or a region including a predetermined object in an image. The object region need not be one per image, but a specific image may have no object region or may have two or more object regions.

[0172] The categories may be variously pre-stored, such as food, nature, animals, plants, buildings, faces, etc. For example, if an object included in a specific object region is a hamburger, the object region may be recognized as corresponding to the food category.

[0173] Thereafter, an importance ranking in each object region may be determined according to the identified categories.

[0174] At this time, among the object areas included in the multiple images, the importance of each object area can be determined based on the number of object areas corresponding to each category. Specifically, among the object areas included in the images, the proportion of object areas corresponding to each category among the object areas included in the multiple images can be obtained, and the importance of the object area corresponding to the category with the highest proportion can be determined to be the highest.

[0175] Meanwhile, the importance of the object area may be determined based on a user input selecting at least one of the categories of the object area included in the plurality of images. In addition, information on the user input selecting at least one of the categories of the object area included in the plurality of images may be stored.

[0176] Thereafter, when determining the arrangement of multiple different images relative to multiple areas divided according to a predetermined layout, the importance level of the object area included in the multiple other images can be determined based on the category of at least one object area included in the other multiple images and the stored information about user input.

[0177] When determining the importance level, when arranging the plurality of images in the plurality of regions, the arrangement of the plurality of images for the plurality of regions may be determined according to a loss degree of at least one object region according to the importance level in step S1320.

[0178] At this time, the interest value reflecting the importance recognized for each unit region of the object region can be obtained. When arranging the plurality of images in the plurality of regions, the arrangement of the plurality of images can be determined so that the degree of loss of the obtained interest value can be minimized.

[0179] Specifically, when a first image among multiple images is arranged in each of multiple regions, when the first image is arranged so that the loss degree of the interest value corresponding to the object region included in the first image is minimized, the loss degree of the interest value can be identified.

[0180] At this time, the first image can be reduced or enlarged while maintaining the aspect ratio based on the size of each of the multiple regions. In this case, if the aspect ratio of each of the multiple regions is different from the aspect ratio of the first image, only a portion corresponding to the size of each of the multiple regions in the reduced or enlarged first image is set in each of the multiple regions, and the arrangement of the first image can be identified so that the degree of loss of the interest value corresponding to the object area included in the first image is minimized. In addition, the degree of loss of the interest value according to the (optimal) arrangement of the first image can be identified, so that the degree of loss of the interest value can be minimized.

[0181] The arrangement of the multiple images can be determined based on the arrangement of the identified first image and the degree of loss of the interest value therefrom. Specifically, when the multiple images are arranged in multiple areas, the arrangement of the multiple images can be determined so that the degree of loss of the interest value corresponding to the object area included in the multiple images is minimized.

[0182] When identifying the (optimal) arrangement of the first image for each of the plurality of regions, the optimal arrangement of a second image different from the first image among the plurality of images and the degree of loss of interest value may also be identified.

[0183] Specifically, when arranging the second image among the plurality of images in the plurality of regions, the (optimal) arrangement of the second image can be identified so that the degree of loss of the interest value according to the object region included in the second image is minimized. At this time, the degree of loss of the interest value according to the arrangement of the second image can also be identified.

[0184] Based on the (optimal) arrangement of the second image for each of the plurality of regions and the degree of loss of the interest value according thereto, the arrangement of the plurality of images may be determined.

[0185] That is, when multiple images are arranged in multiple regions based on the (optimal) arrangement of the first image and the loss of interest value in each of the multiple regions, the (optimal) arrangement of the second image and the loss of interest value in each of the multiple regions, and so on, the arrangement of the multiple images can be determined so that the degree of loss of interest values ​​corresponding to the object regions included in the multiple images is minimized.

[0186] If the number of the plurality of images is greater than or equal to three, and other images are included in addition to the first image and the second image, the optimal arrangement of each of the plurality of regions and the degree of loss of the interest value may also be identified for the other images. Based on the recognition result, the arrangement of the plurality of images for the plurality of regions may be identified so that the degree of loss of the interest value corresponding to the object region included in the plurality of images can be minimized.

[0187] Fig.14 yes Fig.13 Flowchart of an example of identifying the importance level of the object area in step S1310.

[0188] refer to Fig.14 , first, in step S1410, an object region may be obtained from a plurality of images. At this time, the object region may be obtained based on color information and edge information of each of the plurality of images. Alternatively, the object included in the image may be identified by using an AI model that learns to identify the object included in the image, and the region including the object may be identified as the object region.

[0189] When the category of the object included in each object region is identified in step S1420 , the ratio of each of the identified categories may be obtained in step S1430 .

[0190] For example, if five object areas among eight object areas included in five images correspond to the "nature" category and three object areas correspond to the "food" category, the ratio of the "nature" category is 5 / 8 and the ratio of the "food" category can be identified as 3 / 8.

[0191] In addition, the larger the ratio of the category corresponding to the object area, the higher the importance of the object area in step S1440. For example, the ratio of the "nature" category among the categories of the object area is 5 / 8 and the ratio of the "food" category is 3 / 8, and the importance of the object area corresponding to the "nature" category can be identified as higher than the importance of the object area corresponding to the "food" category.

[0192] Fig.15 yes Fig.13 A specific embodiment of step S1320 of the embodiment of the present invention describes an algorithm for describing an example of obtaining an interest value reflecting the importance of an object area, and determining an arrangement so that the loss of the interest value can be minimized when arranging multiple images in multiple areas.

[0193] refer to Fig.15 ,like Fig.14 As shown, object regions are obtained from a plurality of (n) images in step S1510, and the importance of each identified object region may be identified in step S1520. Here, the importance of each object region may be identified based on a category of each object region and / or a user input.

[0194] In step S1530, an interest value reflecting the importance of the object region may be obtained. Specifically, a digitized interest value reflecting the corresponding importance to each unit region of the object region may be obtained.

[0195] In step S1540, when each of a plurality of images is arranged in each of a plurality of (n) regions, an optimal arrangement can be identified so that the degree of loss of interest values ​​of the object region included in each image is minimized, and a (minimum) loss value of the interest value according to the optimal arrangement can be identified.

[0196] In this case, in step S1540', for example, when a specific image (the i-th image among the plurality of images) is arranged in a specific region (the j-th region among the plurality of regions), an optimal arrangement of the image (the i-th image) can be identified so that the degree of loss of the interest value corresponding to the object region included in the image (the i-th image) is minimized, and a (minimum) loss value of the interest value according to the optimal arrangement can be identified. This step can be repeated for all cases in which each of the 1st to nth images is arranged in each of the 1st to nth regions.

[0197] At this time, in step S1550, considering the identified optimal arrangement and the identified (minimum) loss value, the arrangement of multiple images of the multiple regions can be finally determined, wherein the degree of loss of the interest value corresponding to the entire object region included in the multiple images is minimized. At this time, the Hungarian method (which is an N×N optimization matching method) can be used so that the loss of the interest value corresponding to the entire object region included in the multiple images is minimized.

[0198] As described above, the electronic device 100 and the control method thereof can obtain the object areas included in the multiple images when arranging the multiple images in the multiple areas divided according to the layout, and recognize the importance level of each of the object areas based on the user's intention and the characteristics of the multiple images and automatically reflect the importance level. Therefore, the effect is that the inconvenience of the user editing the multiple images separately or arbitrarily specifying their arrangement can be solved.

[0199] via Figures 12 to 15 The control method of the electronic device described can be performed by Figure 2 and Fig.10 The electronic device 100 shown and described is implemented.

[0200] Alternatively, the control method may be implemented by a system including the electronic device 100 and at least one external device (not shown). In this regard, it may be assumed that the control method of the electronic device is performed by the electronic device 100 and the server (not shown). At this time, the electronic device 100 and the server (not shown) may share and execute the reference in various forms. Figures 12 to 15 Describe the steps of the control method.

[0201] For example, the electronic device 100 may receive one or more images in the form of data or signals through the communicator 130, the broadcast receiver 140, the input / output port 150, etc., or obtain one or more images captured by a camera included in the user input device 160, and transmit the received or obtained images to the server. When arranging one or more images according to a layout pre-stored in the server, the server may determine the arrangement of the one or more images so as to minimize the degree of loss of horizontal importance according to the object area in at least one image, and transmit information about the determined arrangement to the electronic device 100. At this time, the electronic device 100 may directly display the one or more images arranged according to the determined arrangement, or transmit information about the one or more images arranged according to the determined arrangement to a connected external device (not shown) through the input / output port 150 so that the corresponding external device displays the information.

[0202] Alternatively, when the electronic device 100 transmits one or more images to the server, the server may identify the object region from the one or more images and transmit the information to the electronic device 100, and the electronic device 100 may identify the importance level of each object region based on a user input for selecting at least one object among the identified object regions. When arranging the one or more images in the one or more regions divided according to the layout pre-stored in the memory 110, the electronic device 100 may determine the arrangement according to the importance level so as to minimize the degree of loss of the object region.

[0203] The above-described various embodiments may be implemented in a recordable medium readable by a computer or a computer-like device using software, hardware, or a combination of software and hardware.

[0204] In terms of hardware implementation, the embodiments of the present disclosure may be implemented using at least one of the following: an application specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a processor, a controller, a microcontroller, a microprocessor, or an electronic unit for performing other functions.

[0205] In some cases, the embodiments described herein may be implemented by the processor 120 itself. According to software implementation, embodiments such as the procedures and functions described herein may be implemented with separate software modules. Each of the above software modules may perform one or more of the functions and operations described herein.

[0206] Meanwhile, computer instructions for executing processing operations in the electronic device 100 according to the above-mentioned various embodiments may be stored in a non-transitory computer-readable medium. The computer instructions stored in this non-transitory computer-readable medium, when executed by a processor of a specific device, cause the above-mentioned specific device to execute processing operations in the electronic device 100 according to the above-mentioned various embodiments.

[0207] A non-transitory computer-readable medium refers to a medium that stores data semi-permanently rather than for a short period of time, such as a register, a cache, a memory, etc., and can be read by a device. In detail, the above-mentioned various applications or programs can be stored in a non-transitory computer-readable medium, such as a compact disc (CD), a digital versatile disc (DVD), a hard disk, a Blu-ray disc, a universal serial bus, a memory card, a read-only memory (ROM), etc., and the application or program can be provided.

[0208] The foregoing embodiments and advantages are merely exemplary and should not be construed as limiting the present disclosure. The present teachings may be easily applied to other types of devices. In addition, the description of the embodiments of the present disclosure is intended to be illustrative rather than limiting the scope of the claims, and many substitutions, modifications, and variations will be apparent to those skilled in the art.

Claims

1. An electronic device, comprising: Memory; as well as Processor, configured as: identifying an arrangement of a plurality of images for a plurality of areas divided according to a predetermined layout, storing information about the identified arrangement in the memory, and controlling the electronic device to display the plurality of images in the plurality of areas based on the stored information about the identified arrangement and / or transmitting data for an external device to display the plurality of images in the plurality of areas based on the stored information about the identified arrangement, wherein the plurality of images include an object area, and Wherein, identifying the arrangement of multiple images includes identifying the arrangement of each of the multiple images in each of the multiple areas based on the degree of loss of the object area related to the importance of the object area, so that in each area, the degree of loss of the at least one object area included in each image is minimized, and identifying the arrangement of the multiple images based on the identified arrangement of each image, wherein the importance of the object area is obtained based on the category of the object included in the object area.

2. The electronic device of claim 1, wherein the processor is configured to: obtaining an interest value reflecting the importance for a unit area of ​​the object area, and The arrangement of each of the plurality of images in each of the plurality of regions is identified according to the obtained interest value, so that the loss degree of the object region is minimized.

3. The electronic device according to claim 2, wherein: A first image of the plurality of images includes at least one object region, and To identify the arrangement of the plurality of images, the processor is configured to: identifying an arrangement of the first image in each of the plurality of regions according to the obtained interest value, so that in each region, a degree of loss of the at least one object region included in the first image is minimized, and The arrangement of the plurality of images is identified based on the identified arrangement of the first image.

4. The electronic device as claimed in claim 3, wherein A second image of the plurality of images includes at least one object region, and To identify the arrangement of the plurality of images, the processor is configured to: identifying an arrangement of the second image in each of the plurality of regions according to the obtained interest value, so that in each region, a degree of loss of the at least one object region included in the second image is minimized, and The arrangement of the plurality of images is identified based on the identified arrangement of the second images. 5 . The electronic device of claim 1 , wherein the processor is configured to identify the importance of the object region based on the number of object regions corresponding to each of a plurality of categories.

6. The electronic device of claim 5, wherein the processor is configured to: For each of the plurality of categories, obtaining a ratio of the number of object regions in each category to the total number of object regions in the plurality of categories, and The importance is identified such that the highest importance is identified for the object region in the category having the highest ratio among the plurality of categories. 7 . The electronic device of claim 1 , wherein the processor is configured to identify importance based on user input selecting at least one of the plurality of categories.

8. The electronic device of claim 7, wherein the processor is configured to: The at least one category among the plurality of categories is identified as having a higher importance than the other categories.

9. The electronic device of claim 3, wherein to identify the arrangement of the first image, the processor is configured to: reducing or enlarging the first image based on a size of each of the plurality of regions while maintaining an aspect ratio of the first image, and In response to the aspect ratio of each of the multiple areas being different from the aspect ratio of the first image, when a portion of the first image in a reduced or enlarged first image corresponding to the size of each of the multiple areas is arranged in each of the multiple areas, the arrangement of the first image is identified based on the obtained interest value so that the degree of loss of the at least one object area included in the first image is minimized.

10. A method performed by an electronic device, comprising: identifying an arrangement of a plurality of images for a plurality of regions divided according to a predetermined layout; as well as displaying the plurality of images in the plurality of regions based on the identified arrangement and / or transmitting data so that the plurality of images are displayed in the plurality of regions by an external device based on the identified arrangement, wherein the plurality of images include an object region, and Identifying the arrangement of multiple images includes identifying the arrangement of each of the multiple images in each of the multiple areas based on the degree of loss of the object area related to the importance of the object area, so that in each area, the degree of loss of the at least one object area included in each image is minimized, and identifying the arrangement of the multiple images based on the identified arrangement of each image, wherein the importance of the object area is obtained based on the category of the object included in the object area.

11. The method of claim 10, wherein identifying the arrangement of the plurality of images comprises: obtaining an interest value reflecting the importance for a unit area of ​​the object area, and The arrangement of each of the plurality of images in each of the plurality of regions is identified according to the obtained interest value, so that the loss degree of the object region is minimized.

12. The method of claim 11, wherein A first image of the plurality of images includes at least one object region, and The identifying the arrangement of the plurality of images comprises: identifying an arrangement of the first image in each of the plurality of regions according to the obtained interest value, so that in each region, a degree of loss of the at least one object region included in the first image is minimized, and The arrangement of the plurality of images is identified based on the identified arrangement of the first image.

13. The method of claim 12, wherein A second image of the plurality of images includes at least one object region, and The identifying the arrangement of the plurality of images comprises: identifying an arrangement of the second image in each of the plurality of regions according to the obtained interest value, so that in each region, a degree of loss of the at least one object region included in the second image is minimized, and The arrangement of the plurality of images is identified based on the identified arrangement of the second images.

14. The method of claim 10, wherein the identifying the arrangement of the plurality of images comprises identifying importance of object regions based on a number of object regions corresponding to each of a plurality of categories.

15. The method of claim 14, wherein said identifying said arrangement of said plurality of images comprises: For each of the plurality of categories, obtaining a ratio of the number of object regions in each category to the total number of object regions in the plurality of categories, and The importance is identified such that the highest importance is identified for the object region in the category having the highest ratio among the plurality of categories.

Images