Method and system for context-aware clipping in camera previews

Abstract

A method performed by an object cropping system of an electronic device includes: receiving a preview image including a plurality of objects from a camera; determining contour boundaries of the plurality of objects; determining a correlation score between the plurality of objects based on a context of the plurality of objects and a relationship between the plurality of objects; generating a plurality of boundary suggestions including a plurality of objects based on the correlation scores; receiving a user input indicating a selection of one of the plurality of boundary suggestions or a selection of at least one object from a plurality of objects present in the plurality of boundary suggestions, where the relevance score is updated for the at least one object based on the user input; and cropping the at least one object from the preview image based on the user input and the updated relevance score using the contour boundary determined for the plurality of objects.

Description

Technical Field

[0001] The present disclosure relates generally to image processing using artificial intelligence (AI) techniques, and more particularly, but not exclusively, to methods and systems for context-aware cropping in camera preview. Background Art

[0002] Typically, the cropping boundaries that appear in the capture preview may be rectangular and can only be adjusted in a rectangular manner. In many cases, users may prefer non-regular and non-polygonal boundaries for some objects present in the scene. In one embodiment, a food blogger may want a triangular / elliptical / circular boundary for cropping pizza slices, food plates, etc. Such non-regular and non-polygonal cropping boundaries may exist in some arbitrary third-party applications, and such third-party applications will only be helpful after the image has been captured.

[0003] For example, in Adobe ® In scanning applications, there are options for cropping with non-rectangular boundaries. However, cropping the non-rectangular boundaries only after the image is captured may not be sufficient for professional and recreational purposes. Moreover, such third-party applications require manual input for understanding the area to be cropped, which is another limitation. Most of the time, users want to see different options based on various aspects of the scene captured in the image, which allows the user to select the most appropriate option for cropping.

[0004] However, third-party applications do not provide such options, instead they rely entirely on manual input to crop the image after image capture. Currently, there is no existing technology that allows for boundary detection and cropping during the preview stage of imaging. This reduces the flexibility of the user to dynamically decide the boundaries of the image at the time of capture.

[0005] The information disclosed in this Background section of the present disclosure is only for enhancement of understanding of the general background of the present disclosure and should not be taken as an acknowledgement or any form of suggestion that this information forms the prior art already known to a person skilled in the art. Summary of the invention

[0006] Solution to the problem

[0007] According to an aspect of the present disclosure, a method performed by an object cropping system of an electronic device includes: receiving a preview image including multiple objects from a camera; determining contour boundaries of the multiple objects; determining relevance scores between the multiple objects based on contexts of the multiple objects and relationships between the multiple objects; generating multiple boundary suggestions including the multiple objects based on the relevance scores; receiving user input indicating a selection of one of the multiple boundary suggestions or selecting at least one object from multiple objects present in the multiple boundary suggestions, wherein the relevance score is updated for at least one object based on the user input; and cropping at least one object from the preview image based on the user input and the updated relevance score using the contour boundaries determined for the multiple objects.

[0008] The contour boundaries of the plurality of objects may be determined by using at least one first type of artificial intelligence (AI) based model.

[0009] At least one first type of AI-based model may include an enhanced Region Proposal Network (RPN) for contour boundary determination, and the enhanced RPN is integrated with an instance segmentation deep neural network (DNN).

[0010] The plurality of objects may have at least one of a polygonal shape and a non-polygonal shape.

[0011] A relevance score between the plurality of objects based on the contexts of the plurality of objects and the relationships between the plurality of objects may be determined by using at least one second type of AI-based model.

[0012] Contour boundaries determined for a plurality of objects may be visualized using boundaries drawn by a boundary line construction model associated with an object clipping system.

[0013] The at least one second type of AI-based model may include a Neuro-symbolic model for determining a context of the plurality of objects and a relationship between the plurality of objects.

[0014] The relationship between multiple objects is determined based on the scene description and the context of the multiple objects.

[0015] The method may also include pre-processing the preview image to eliminate at least one of redundant image frames and irrelevant image frames.

[0016] The method may further include storing the cropped at least one object in the single image or in the multiple images.

[0017] The method may also include, based on the preview image including a single object having one of a polygonal shape and a non-polygonal shape: determining a contour boundary of the single object using at least one first type AI based model; generating at least one boundary suggestion including the single object; receiving user input indicating a selection of one of the at least one boundary suggestion; and cropping the single object from the preview image based on the user input using the contour boundary determined for the single object.

[0018] According to an aspect of the present disclosure, an electronic device includes: a memory storing instructions; and a processor configured to execute the instructions to: receive a preview image including multiple objects; determine contour boundaries of the multiple objects; determine correlation scores between the multiple objects based on contexts of the multiple objects and relationships between the multiple objects; generate multiple boundary suggestions including the multiple objects based on the correlation scores; receive user input indicating selection of one of the multiple boundary suggestions or selection of at least one object from multiple objects present in the multiple boundary suggestions, wherein the correlation score is updated for at least one object based on the user input; and use the contour boundaries determined for the multiple objects to crop at least one object from the preview image based on the user input and the updated correlation score.

[0019] The processor may be further configured to execute the instructions to determine contour boundaries of the plurality of objects by using at least one first type of artificial intelligence (AI) based model.

[0020] At least one first type of AI-based model may include an enhanced region proposal network (RPN) for contour boundary determination, wherein the enhanced RPN is integrated with an instance segmentation deep neural network (DNN).

[0021] The plurality of objects may have at least one of a polygonal shape and a non-polygonal shape.

[0022] The foregoing summary is illustrative only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments, and features described above, further aspects, embodiments, and features will become apparent by reference to the drawings and the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent through the following description in conjunction with the accompanying drawings, in which:

[0024] Figure 1A An exemplary architecture for context-aware cropping in camera preview according to some embodiments of the present disclosure is shown;

[0025] Figure 1Bshows an exemplary preview image including a plurality of objects according to some embodiments of the present disclosure;

[0026] Figure 1C shows an exemplary preview image including a single object according to some embodiments of the present disclosure;

[0027] Figure 1D shows an exemplary outline boundary of a single object (pizza slice) according to some embodiments of the present disclosure;

[0028] Figure 2A shows a detailed block diagram of an object cropping system for context-aware cropping in camera preview according to some embodiments of the present disclosure;

[0029] Figure 2B An exemplary neural symbolic model according to some embodiments of the present disclosure is shown;

[0030] Figure 2C shows an exemplary image in which regions of interest and relevance have been marked according to some embodiments of the present disclosure;

[0031] Figure 2D shows exemplary relationship information between multiple objects according to some embodiments of the present disclosure;

[0032] Figure 2E shows an exemplary image including relevance scores for a plurality of objects according to some embodiments of the present disclosure;

[0033] Figure 2F shows an exemplary image indicating user input for a suggested boundary according to some embodiments of the present disclosure;

[0034] Figure 2G shows an exemplary image indicating a fitted boundary for a user-selected object according to some embodiments of the present disclosure;

[0035] Figure 2H and Fig.2I showing objects individually cropped from a preview image according to some embodiments of the present disclosure;

[0036] Figure 3A A flow chart illustrating a method for context-aware cropping in camera preview when multiple objects are present in the preview image according to some embodiments of the present disclosure is shown;

[0037] Figure 3B A flow chart illustrating a method for context-aware cropping in camera preview when a single object is present in the preview image according to some embodiments of the present disclosure is shown; and

[0038] Figure 4is a block diagram of an exemplary computer system for implementing embodiments consistent with the present disclosure. DETAILED DESCRIPTION

[0039] Before the following detailed description, it may be advantageous to set forth the definitions of certain words and phrases used in the present disclosure. The term "coupling" and its derivatives refer to any direct or indirect communication between two or more elements, whether or not these elements are in physical contact with each other. The terms "send", "receive" and "communication" and their derivatives include direct and indirect communication. The terms "include" and "comprise" and their derivatives refer to unlimited inclusion. The term "or" is an inclusive term meaning "and / or". The phrase "associated with..." and its derivatives refer to include, be included in, interconnected with, be included in, be connected to or connected with, be coupled to or coupled with, communicateable with, cooperate with, interweave, juxtapose, approach, be combined with or combined with, have, have...attributes, have a relationship with or have a relationship with, etc. The term "controller" refers to any device, system or part thereof that controls at least one operation. Such a controller can be implemented with hardware or a combination of hardware and software and / or firmware. The function associated with any particular controller can be centralized or distributed, whether local or remote. When used with a list of items, the phrase "at least one of" means that different combinations of one or more of the listed items may be used, and only one of the items in the list may be needed. For example, "at least one of A, B, and C" includes any combination of: A, B, C, A and B, A and C, B and C, and A and B and C. Similarly, the term "set" means one or more. Thus, the set of items can be a single item or a set of two or more items.

[0040] In addition, the various functions described below can be implemented or supported by one or more computer programs, each of which is formed by a computer-readable program code and embodied in a computer-readable medium. The terms "application" and "program" refer to one or more computer programs, software components, instruction sets, processes, functions, objects, classes, instances, related data or parts thereof suitable for implementation in a suitable computer-readable program code. The phrase "computer-readable program code" includes any type of computer code, including source code, object code, and executable code. The phrase "computer-readable medium" includes any type of medium that can be accessed by a computer, such as a read-only memory (ROM), a random access memory (RAM), a hard drive, a compact disk (CD), a digital video disk (DVD), or any other type of memory. "Non-transitory" computer-readable media do not include wired, wireless, optical or other communication links that transmit temporary electrical or other signals. Non-transitory computer-readable media include media that can permanently store data and media that can store data and overwrite it later, such as rewritable optical disks or erasable memory devices.

[0041] Definitions for certain other words and phrases are provided throughout this disclosure. Those of ordinary skill in the art should understand that in many, if not most instances, such definitions apply to prior, as well as future uses of such defined words and phrases.

[0042] Those skilled in the art will appreciate that any block diagram herein represents a conceptual view of an illustrative system that embodies the principles of the present disclosure. Similarly, those skilled in the art will appreciate that any flow chart, flow diagram, state transition diagram, pseudo code, etc., represent various processes that can be substantially represented in a computer-readable medium and executed by a computer or processor, whether or not such a computer or processor is explicitly shown.

[0043] In one embodiment

[0044] A method and system for context-aware cropping in camera preview is disclosed herein. The present disclosure provides a context-aware mechanism, i.e., holistic reasoning of more than one object / entity and the relationship between them in a single scene of a preview image or across preview images / frames to identify one or more objects in an image that may be of most interest based on scene context or inter-frame context, which is effective for execution on preview frames.

[0045] The present disclosure also provides multiple cropping boundary suggestions via an artificial intelligence (AI) framework that will use neural symbolic reasoning and region proposal models together. In addition, the present disclosure suggests cropping boundaries that can be irregular, non-polygonal, fitted to an object / region or a collection of objects / regions (overlapping or non-overlapping). The suggested cropping boundaries are provided in the preview stage so that the user can understand how to best point the camera or select a focus to achieve optimal cropping.

[0046] The present disclosure also provides a feature of "early exits" that allows the object cropping system to eliminate irrelevant areas, empty spaces, and irrelevant frames during the preview stage itself, which reduces the computation of irrelevant frames.

[0047] The description of an embodiment with several components in communication with each other does not imply that all such components are required. Rather, various optional components are described to illustrate various possible embodiments of the present disclosure.

[0048] In the following detailed description of the embodiments of the present disclosure, reference is made to the accompanying drawings which form a part thereof, and in which specific embodiments in which the present disclosure may be practiced are shown by way of illustration. These embodiments are described in sufficient detail to enable those skilled in the art to practice the present disclosure, and it should be understood that other embodiments may be utilized and changes may be made without departing from the scope of the present disclosure. Therefore, the following description should not be considered to have a limiting meaning.

[0049] Figure 1A An exemplary architecture for context-aware cropping in camera preview according to some embodiments of the present disclosure is shown.

[0050] The architecture 100 includes a user device 101 (e.g., an electronic device), an image capture unit 103, and an object cropping system 105. In some embodiments, the image capture unit 103 and the object cropping system 105 may be configured in the user device 101. In some other embodiments, the image capture unit 103 and the object cropping system 105 may be communicatively connected to the user device 101 via a communication network. The communication network may be one of a wired communication network, a wireless communication network, or a combination of both.

[0051] In one embodiment, the user device 101 may include, but is not limited to, a mobile phone, a tablet phone, a laptop computer, a desktop computer, etc. In one embodiment, the image capture unit 103 may be a camera or a camera device capable of communicatively interfacing with the user device 101. In the context of the present disclosure, the description is explained considering the user device 101 as a mobile phone or a tablet phone with an integrated camera as the image capture unit 103. However, this should not be interpreted as a limitation of the present disclosure, as the user device 101 and the image capture unit 103 may be associated in any manner, as described above in this paragraph.

[0052] Object clipping system 105 may include processor 107, input / output (I / O) interface 109, and memory 111. I / O interface 109 may receive a preview image from image capture unit 103 during a preview phase. In some embodiments, the preview image may include a plurality of objects having at least one of a polygonal shape and a non-polygonal shape.

[0053] In some embodiments, the non-polygonal shaped object may include an object having an irregular shape. In one embodiment, the irregular shape may be any shape that does not belong to a standard geometric shape (such as a circle, triangle, rectangle, trapezoid, square, etc.).

[0054] In one embodiment, consider Figure 1B The preview image is shown. Figure 1B The multiple objects in the preview image shown may be a chessboard, a plate, food on a plate, a bottle, a salt shaker, etc. In this example, the circular-shaped plate is an object of a non-polygonal shape, the rectangular-shaped chessboard in the image may be an object of a polygonal shape, and the food on the plate (such as vegetables and French fries) may be objects of a non-polygonal and irregular shape.

[0055] In some embodiments, upon receiving the preview image, the processor 107 may pre-process the preview image to eliminate at least one of redundant image frames and irrelevant image frames. When pre-processing the preview image, the processor 107 may use one or more first-type artificial intelligence (AI)-based models to determine contour boundaries of multiple objects present in the preview image. In one embodiment, the one or more first-type AI-based models include an enhanced region prototyping network (RPN) for contour boundary determination.

[0056] In some embodiments, the enhanced RPN can be integrated with an instance segmentation deep neural network (DNN). The processor 107 can visualize the contour boundaries determined for multiple objects using boundaries drawn by a boundary line construction model associated with the object clipping system 105. When determining the boundaries, the processor 107 can use one or more second types of AI-based models to determine the correlation scores between multiple objects based on the context of the multiple objects and the relationship between the multiple objects. In the context of the present disclosure, the correlation score can be a score that establishes or quantifies the correlation between multiple objects in the preview image. The relationship between multiple objects (one of the parameters that helps determine the correlation score) can be determined based on the scene description and the context of the multiple objects. In some embodiments, one or more second types of AI models may include, but are not limited to, a neural symbolic model for determining the context of multiple objects and the relationship between multiple objects.

[0057] In addition, the processor 107 may generate a plurality of boundary suggestions including a plurality of objects based on the correlation score. A user of the user device may provide a user input indicating a selection of one of the plurality of boundary suggestions or a selection of one or more objects based on the plurality of objects present in the plurality of boundary suggestions. The processor 107 may receive the user's input, and thereafter, the processor 107 may update the two or more objects based on the user's input.

[0058] Subsequently, the processor 107 may crop one or more objects from the preview image based on the user input and the updated relevance scores using the contour boundaries determined for the multiple objects. In some embodiments, the processor 107 may store the cropped one or more objects in a single image or multiple images in the memory 111. For example, if considering Figure 1B For cropping, the cropped objects (such as a pizza, a salt shaker, and a food plate) can all be stored in a single image. Alternatively, the cropped pizza can be stored as a separate image, the cropped salt shaker can be stored as a separate image, and the food plate can be stored as a separate image.

[0059] In some embodiments, when there are multiple objects in the scene captured in the preview image, the above method of determining the relationship between the objects in the preview image and the correlation scores to determine the boundary to be cropped may be applicable. However, when the preview image includes a single object having a polygonal or non-polygonal shape (e.g., Figure 1C When a pizza slice is placed on a plate as shown, processor 107 may use one or more first type of AI-based models to initially determine the contour boundaries of a single object.

[0060] Thereafter, the processor 107 may generate one or more boundary suggestions including a single object. For example, in one suggestion, a contour boundary may be generated for a pizza slice, and in another suggestion, a contour boundary may be generated for a pizza slice and a plate on which the pizza slice is placed.

[0061] In addition, processor 107 may receive user input indicating a selection of one of the one or more boundary suggestions. Based on the user input, processor 107 may crop a single object from the preview image using the outline boundary determined for the single object. For example, if the user has selected a boundary suggestion that includes a pizza slice, processor 107 may crop an object such as Figure 1D The outline boundary of the pizza slice shown is captured and the image is stored in memory 111.

[0062] Figure 2A A detailed block diagram of an object cropping system for context-aware cropping in camera preview according to some embodiments of the present disclosure is shown.

[0063] In some implementations, the object clipping system 105 may include data 203 and module 205. In one embodiment, the data 203 is stored in a Figure 2A The object clipping system 105 is shown in the memory 111 configured in the embodiment. In one embodiment, the data 203 may include boundary data 207, AI model data 209, score data 211, clipped image data 213 and other data 215.

[0064] In an embodiment, data 203 may be stored in memory 111 in the form of various data structures. In addition, data models (such as relational or hierarchical data models) may be used to organize data 203. Other data 215 may store data generated by module 205 for performing various functions of object clipping system 105, including temporary data and temporary files.

[0065] In an embodiment, the data 203 stored in the memory 111 may be processed by a module 205 of the object clipping system 105. The module 205 may be stored in the memory 111. In an example, the module 205 communicatively coupled to the processor 107 configured in the object clipping system 105 may also be present in the memory 111. Figure 2A The memory 111 is shown as external and implemented as hardware. As used herein, the term module refers to an application specific integrated circuit (ASIC), an electronic circuit, a processor (shared, dedicated or group) and memory that executes one or more software or firmware programs, a combinational logic circuit and / or other suitable components that provide the described functionality.

[0066] In an embodiment, the modules 205 may include, for example, a receiving module 221, a preprocessing module 223, a boundary determination module 225, a score determination module 227, a boundary suggestion module 229, a clipping module 231, and other modules 233. The other modules 233 may be used to perform various miscellaneous functions of the object clipping system 105. Such the aforementioned modules 205 may be represented as a single module or a combination of different modules.

[0067] In some embodiments, the receiving module 221 may receive a preview image from the image capture unit 103 associated with the object cropping system 105 during the preview phase. In some embodiments, the preview image may include, but is not limited to, a plurality of objects having at least one of a polygonal shape and a non-polygonal shape. In some embodiments, the non-polygonal shaped object may include an object having an irregular shape.

[0068] In some embodiments, the preprocessing module 223 can preprocess the preview image to eliminate at least one of redundant image frames and irrelevant image frames. In the context of the present disclosure, this process of preprocessing and removing redundant and irrelevant image frames in the preview stage itself can also be referred to as "early exit".

[0069] In some embodiments, in a given preview image, the pre-processing module 223 can remove areas of no interest, blank areas, out-of-focus areas, etc. This in turn reduces processing burden and symbol computation because unwanted and irrelevant portions of the preview image are eliminated even before further processing occurs.

[0070] In some embodiments, the boundary determination module 225 may use one or more first-type AI-based models to determine the contour boundaries of multiple objects. The one or more first-type AI-based models may include, but are not limited to, an enhanced region generation network (RPN) for contour boundary determination. The enhanced RPN may be integrated with an instance segmentation deep neural network (DNN).

[0071] In one embodiment, the instance segmentation DNN may be a Feature Pyramid Network (FPN). One or more first type AI-based models may be stored as AI data 209. In some embodiments, the boundary determination module 225 may use an enhanced RPN to initially determine a plurality of relevant and interesting regions based on the context of a plurality of objects, or in other words, objects in the preview image. Thereafter, the boundary determination module 225 may use an instance segmentation DNN such as an FPN to determine initial contour boundaries of a plurality of objects in the preview image.

[0072] In some embodiments, the contour boundaries determined for the plurality of objects may be visualized using boundaries drawn by a boundary line construction model associated with the object clipping system 105. The contour boundaries may be visualized in one or more colors. The initially determined contour boundaries may be stored in the memory 111 as boundary data 207.

[0073] In some embodiments, the score determination module 227 may use one or more second types of AI-based models to determine the relevance scores between the multiple objects based on the context of the multiple objects and the relationships between the multiple objects. In some embodiments, the one or more second types of AI-based models may include a neural symbolic model for determining the context of the multiple objects and the relationships between the multiple objects.

[0074] In one embodiment, the neural symbolic model may be a neural logic machine or a logic tensor network. Figure 2B An exemplary neural symbolic model is shown in . The exemplary neural symbolic model includes an input layer, a hidden layer, and an output layer. One or more second type of AI-based models can be stored as AI data 209.

[0075] In some embodiments, the neural symbolic model may determine contexts of the plurality of objects based on information learned from at least one of an inter-frame and an intra-frame. In the present disclosure, the contexts of the plurality of objects may be defined as holistic reasoning about the plurality of objects and the relationships between them in a single scene of a preview image or across a plurality of preview image frames in order to identify higher-level information beyond pixels.

[0076] Thus, the neural symbolic model can be trained to classify multiple objects based on the contexts of the multiple objects using symbolic reasoning. Thereafter, the neural symbolic model can also be trained to learn the relationships between the multiple objects classified based on the contexts of the multiple objects. In some embodiments, the score determination module 227 can thereafter determine the relevance scores between the multiple objects based on the scene description and the contexts of the multiple objects and the relationships between the multiple objects.

[0077] In one embodiment, if the scene in the preview is a scene of food placed on a table, multiple objects in the preview image that are present on the table but are related to the food may be determined as related objects, and thus, a correlation score may be established between such objects determined to be related. In the context of the present disclosure, a correlation score may be a score that establishes or quantifies the correlation between multiple objects in the preview image. The correlation score thus determined may be stored in the memory 111 as score data 211.

[0078] In some embodiments, the boundary suggestion module 229 can generate multiple boundary suggestions including multiple objects based on the relevance score. In some embodiments, the generated boundary suggestions can be suggested to the user starting from multiple objects determined to have the highest relevance scores. In other words, the generated boundary suggestions can include multiple objects determined to be contextually relevant and related to each other.

[0079] In some embodiments, the generated boundary can be a fitted boundary for non-polygonal and non-regular objects in the preview image. The boundary generation loss can include context divergence over multiple relevant regions, and the loss will be used to train an "instance segmentation DNN" such as FPN to generate a fitted boundary.

[0080] In some embodiments, the receiving module 221 may also receive a user input indicating the selection of one of the multiple boundary suggestions or the selection of one or more objects according to the multiple objects present in the multiple boundary suggestions. This means that if the multiple boundary suggestions have successfully captured the object that the user wants to keep and crop in the preview image, the user can select one of the multiple boundary suggestions as the user input.

[0081] However, if the multiple boundary suggestions have not successfully captured all objects that the user desires to retain and crop in the preview image, the user can provide a selection of one or more objects that need to be retained and cropped in the preview image according to the multiple objects as user input. In some embodiments, based on the user input, that is, the score determination module 227 can update the relevance score of two or more objects based on the user input.

[0082] In one embodiment, among the four boundary suggestions provided to the user in descending order of the correlation scores between the multiple objects, the user selects the third boundary suggestion. In that case, the score determination module 227 can update the correlation scores between the two or more objects present in the third boundary suggestion so that the updated correlation score will become the highest among the other correlation scores.

[0083] In another scenario, the user may select one or more objects to be added to or removed from the boundary suggestions that have been provided. Even in such a scenario, based on the user input, the score determination module 227 may update the correlation scores between the objects that the user wishes to retain and crop in the preview image so that the updated correlation scores are the highest among the other correlation scores. The updated correlation scores may also be stored in the memory 111 as score data 211.

[0084] In some embodiments, the cropping module 231 can crop one or more objects from the preview image based on user input and updated relevance scores using the contour boundaries determined for the multiple objects. In some embodiments, the cropping can be performed when the non-polygonal and / or non-regular cropping boundaries that best fit to the multiple objects are used.

[0085] In some embodiments, the cropping module 231 can store the cropped one or more objects in a single image or multiple images as cropped image data 213 in the memory 111. For example, if considering Figure 1B If one or more objects are shown for cropping, the cropped objects (such as a pizza, a salt shaker, and a food plate) can all be stored in a single image. Alternatively, the cropped pizza can be stored as a separate image, the cropped salt shaker can be stored as a separate image, and the food plate can be stored as a separate image.

[0086] In some embodiments, when there are multiple objects in the scene captured in the preview image, the above method of determining the relationship between the objects in the preview image and the relevance score in order to determine the boundary to be cropped may be applicable. However, when the preview image includes a single object having a polygonal or non-polygonal shape (e.g., a pizza slice placed on a plate), the boundary determination module 225 may use one or more first type of AI-based models to initially determine the contour boundary of the single object. Thereafter, the boundary determination module 225 may generate one or more boundary suggestions including the single object.

[0087] In addition, receiving module 221 may receive user input indicating selection of one of the one or more boundary suggestions. Based on the user input, cropping module 231 may crop a single object from the preview image using the contour boundary determined for the single object. For example, if the user has selected a boundary suggestion including a pizza slice, cropping module 231 may crop the contour boundary of the pizza slice and store the image in memory 111.

[0088] The above operations are explained below with the help of an exemplary scenario. However, this should not be interpreted as a limitation of the present disclosure, as this is only provided for the purpose of understanding.

[0089] Consider Figure 1B. The preview frame can be provided as input to one or more first AI models including an enhanced RPN model. The enhanced RPN model can determine that multiple objects in the preview image as identified in the preview image can be a food plate, a chess board, a bottle, a table, a salt shaker, food (vegetables and chips), and a place mat, and initially fit a boundary around each of the multiple objects. Thereafter, the object cropping system 105 can use the one or more first AI models to determine that the most relevant and interesting regions of context sensitivity are the food plate, the place mat, and the salt shaker, as shown in FIG. Figure 2C shown.

[0090] In some embodiments, one or more second AI models including a neural symbolic model may be used to determine context information and relationships between multiple objects. In this example, the context information (i.e., the context between multiple objects) may be food-related. The one or more second AI models may learn relationship information between multiple objects, such as Figure 2D shown.

[0091] For example, in Figure 2D In the example of the present invention, it is learned that the food plate contains vegetables, salad, and French fries. Vegetables and salad require salt, and salt is contained in the salt shaker. Based on this understanding of the relationship between multiple objects and the context of multiple objects, the object clipping system 105 can determine the relevance scores between multiple objects.

[0092] In this case, the correlation is shown in terms of high, low, and medium rather than scores. However, this should not be construed as a limitation of the present disclosure, as correlation may also be indicated in terms of scores. Example correlation scores indicated for multiple objects (i.e., food plates, salt shakers, and placemats) are shown in FIG. Figure 2E shown.

[0093] According to the relevance scores, since the food plate and the salt shaker have a relevance score of "high", and the placemat has a relevance score of "medium", the object clipping system 105 can generate a boundary suggestion including the food plate, the salt shaker, and the placemat. Figure 2F The form of user input shown provides his or her selections, i.e., the user selects only the food plate and the salt shaker. Thus, the object clipping system 105 adjusts and fits the selected objects (i.e., as Figure 2G The food plate and salt shaker shown in the figure are drawn, and the objects are cropped from the preview image, such as Figure 2H and Fig.2I shown.

[0094] Figure 3A A flow chart illustrating a method 300a for context-aware cropping in camera preview when multiple objects are present in the preview image is shown according to some embodiments of the present disclosure.

[0095] like Figure 3A As shown, method 300a includes one or more operations illustrating a method for context-aware cropping in camera preview when multiple objects are present in the preview image. Method 300a may be described in the general context of computer-executable instructions. In general, computer-executable instructions may include routines, programs, objects, components, data structures, processes, modules, and functions that perform functions or implement abstract data types.

[0096] The order in which method 300a is described is not intended to be construed as limiting, and any number of the described method operations may be combined in any order to implement method 300a. Additionally, individual operations may be deleted from the method without departing from the spirit and scope of the subject matter described herein. Furthermore, method 300a may be implemented in any suitable hardware, software, firmware, or combination thereof.

[0097] At operation 301, method 300a may include receiving, by a processor 107 configured in an object cropping system 105, a preview image including a plurality of objects from an image capture unit 103, the object cropping system 105 being configured in turn in a user device 101. In some embodiments, the plurality of objects may have at least one of a polygonal shape and a non-polygonal shape. In some embodiments, the processor 107 may pre-process the preview image to eliminate at least one of redundant image frames and irrelevant image frames.

[0098] At operation 303, method 300a may include determining, by processor 107, contour boundaries of a plurality of objects using one or more first type AI-based models. In some embodiments, the one or more first type AI-based models include an enhanced region generation network (RPN) for contour boundary determination. The enhanced RPN is integrated with an instance segmentation deep neural network (DNN).

[0099] At operation 305, method 300a may include determining, by processor 107, a relevance score between the plurality of objects based on a context of the plurality of objects and a relationship between the plurality of objects using one or more second-type AI-based models. In some embodiments, the relationship between the plurality of objects may be determined based on a scene description and the context of the plurality of objects. In some embodiments, the one or more second-type AI-based models include a neural symbolic model for determining the context of the plurality of objects and the relationship between the plurality of objects.

[0100] At operation 307 , the method 300 a may include generating, by the processor 107 , a plurality of boundary proposals including the plurality of objects based on the relevance scores.

[0101] At operation 309, method 300a may include receiving, by processor 107, a user input indicating selection of one of the plurality of boundary suggestions or selection of one or more objects from the plurality of objects present in the plurality of boundary suggestions. In some embodiments, the relevance score is updated for two or more objects based on the user input.

[0102] At operation 311, method 300a may include cropping, by processor 107, one or more objects from the preview image based on user input and updated relevance scores using the contour boundaries determined for the plurality of objects. In some embodiments, the cropped one or more objects may be stored in a single image or in multiple images.

[0103] Figure 3B A flow chart illustrating a method for context-aware cropping in camera preview when a single object is present in the preview image according to some embodiments of the present disclosure is shown.

[0104] like Figure 3B As shown, method 300b includes one or more operations illustrating a method for context-aware cropping in camera preview when a single object is present in the preview image. Method 300b may be described in the general context of computer-executable instructions. In general, computer-executable instructions may include routines, programs, objects, components, data structures, procedures, modules, and functions that perform functions or implement abstract data types.

[0105] The order in which method 300b is described is not intended to be construed as limiting, and any number of the described method operations may be combined in any order to implement method 300b. In addition, individual operations may be deleted from the method without departing from the spirit and scope of the subject matter described herein. In addition, method 300b may be implemented with any suitable hardware, software, firmware, or a combination thereof.

[0106] At operation 321, method 300b may include determining, by a processor 107 configured in an object clipping system 105, one or more first type AI-based models, a contour boundary of a single object, the object clipping system 105 being configured in turn in the user device 101. In some embodiments, the single object may be one of a polygonal shape and a non-polygonal shape.

[0107] At operation 323 , the method 300 b may include generating, by the processor 107 , one or more boundary proposals including a single object.

[0108] At operation 325 , the method 300 b may include receiving, by the processor 107 , a user input indicating a selection of one of the one or more boundary suggestions.

[0109] At operation 327 , the method 300 b may include cropping, by the processor 107 , the single object from the preview image based on the user input using the contour boundary determined for the single object.

[0110] Figure 4 is a block diagram of an exemplary computer system for implementing embodiments consistent with the present disclosure.

[0111] In some embodiments, Figure 4 A block diagram of an exemplary computer system 400 for implementing embodiments consistent with the present disclosure is shown. In some embodiments, computer system 400 may be object cropping system 105 for context-aware cropping in camera preview. Computer system 400 may include a central processing unit ("CPU" or "processor") 402.

[0112] Processor 402 may include at least one data processor for executing program components for executing user or system generated business processes. A user may include a person, a person using a device such as those included in the present disclosure, or such a device itself. Processor 402 may include a special purpose processing unit such as an integrated system (bus) controller, a memory management control unit, a floating point unit, a graphics processing unit, a digital signal processing unit, etc.

[0113] The processor 402 may be configured to communicate with an input device 411 and an output device 412 via an I / O interface 401. The I / O interface 401 may employ communication protocols / methods such as, but not limited to, audio, analog, digital, stereo, IEEE-1394, serial bus, universal serial bus (USB), infrared, PS / 2, BNC, coaxial, component, composite, digital video interface (DVI), high-definition multimedia interface (HDMI), radio frequency (RF) antenna, S-video, video graphics array (VGA), IEEE 802.n / b / g / n / x, Bluetooth, cellular (e.g., code division multiple access (CDMA), high-speed packet access (HSPA+), global system for mobile communications (GSM), long term evolution (LTE), WiMax, etc.), etc. Using the I / O interface 401, the computer system 400 may communicate with the input device 411 and the output device 412.

[0114] In some embodiments, the processor 402 may be configured to communicate with a communication network 409 via a network interface 403. The network interface 403 may communicate with the communication network 409. The network interface 403 may employ connection protocols including, but not limited to, direct connection, Ethernet (e.g., twisted pair 10 / 100 / 1000 Base T), Transmission Control Protocol / Internet Protocol (TCP / IP), Token Ring, IEEE 802.11a / b / g / n / x, etc. Using the network interface 403 and the communication network 409, the computer system 400 may communicate with the user device 101, and the user device 101 may be communicatively associated with the image capture unit 103. The user device 101 may include, but not limited to, a mobile phone, a laptop computer, a tablet phone, a desktop computer, etc.

[0115] The communication network 409 may be implemented as one of different types of networks, such as an intranet or local area network (LAN) within an organization, etc. The communication network 409 may be a dedicated network or a shared network, which represents an association of different types of networks that communicate with each other using various protocols (e.g., Hypertext Transfer Protocol (HTTP), Transmission Control Protocol / Internet Protocol (TCP / IP), Wireless Application Protocol (WAP), etc.). In addition, the communication network 409 may include various network devices, including routers, bridges, servers, computing devices, storage devices, etc. In some embodiments, the processor 402 may be configured to communicate with a memory 405 (e.g., RAM, ROM, etc.) via a storage interface 404. The storage interface 404 may be connected to the memory 405, which includes but is not limited to a memory drive, a removable disk drive, etc., which uses connection protocols such as Serial Advanced Technology Attachment (SATA), Integrated Drive Electronics (IDE), IEEE-1394, Universal Serial Bus (USB), Fiber Channel, Small Computer Systems Interface (SCSI), etc. The storage drives may also include drums, magnetic disk drives, magneto-optical drives, optical drives, redundant arrays of independent disks (RAID), solid-state memory devices, solid-state drives, and the like.

[0116] The memory 405 may store a collection of program or database components, including but not limited to a user interface 406, an operating system 407, a web browser 408, etc. In some embodiments, the computer system 400 may store user / application data, such as data, variables, records, etc. described in the present disclosure. Such a database may be implemented as a fault-tolerant, relational, scalable, secure database, such as Oracle or Sybase.

[0117] Operating system 407 can facilitate resource management and operation of computer system 400. Examples of operating systems include, but are not limited to, APPLE ® MACINTOSH ® OS X ® ,UNIX ® , UNIX-like ® System distribution (for example, BERKELEYSOFTWARE DISTRIBUTION ® (BSD), FREEBSD ® , NETBSD ® , OPENBSD, etc.), LINUX ® Distributions (for example, RED HAT ® 、UBUNTU ® 、KUBUNTU ® etc.), IBM ® OS / 2 ® 、MICROSOFT ® WINDOWS ® (XP ® VISTA ® / 7 / 8, 10, etc.), APPLE ® iOS ® 、GOOGLE TM ANDROID TM , BLACKBERRY ® OS, etc. User interface 406 can facilitate the display, execution, interaction, manipulation or operation of program components through text or graphical facilities. For example, the user interface can provide computer interactive interface elements such as cursors, icons, check boxes, menus, scrollers, windows, widgets, etc. on a display system operably connected to computer system 400. Graphical user interfaces can be used, including but not limited to Apple ® Macintosh ® Operating system, Aqua ® 、IBM ® OS / 2® , Microsoft ® Windows ® (e.g., Aero, Metro, etc.), web interface libraries (e.g., ActiveX ® , Java ® 、Javascript ® , AJAX, HTML, Adobe ® Flash ® etc.

[0118] Computer system 400 may implement a program component stored in web browser 408. Web browser 408 may be a hypertext viewing application such as MICROSOFT ® INTERNET EXPLORER ® 、GOOGLE TM CHROME TM 、MOZILLA ® FIREFOX ® 、APPLE ® SAFARI ® The web browser 408 can use secure hypertext transfer protocol (HTTPS), secure sockets layer (SSL), transport layer security (TLS), etc. to provide secure web browsing. The web browser 408 can use AJAX, DHTML, ADOBE ® FLASH ® 、JAVASCRIPT ® , JAVA ® , application programming interface (API), etc. The computer system 400 can implement a program component stored in a mail server. The mail server can be an Internet mail server, such as Microsoft Exchange, etc. The mail server can utilize a mail server such as ASP, ACTIVEX, etc. ® , ANSI ® C++ / C#、MICROSOFT ® NET、CGI SCRIPTS、JAVA ® 、JAVASCRIPT ® , PERL ® , PHP, PYTHON ® 、WEBOBJECTS ® Mail servers can use communication protocols such as Internet Message Access Protocol (IMAP), Messaging Application Programming Interface (MAPI), MICROSOFT ®exchange, Post Office Protocol (POP), Simple Mail Transfer Protocol (SMTP), etc. In some embodiments, the computer system 400 may implement a program component stored in a mail client. The mail client may be a mail viewing application, such as APPLE ® MAIL, MICROSOFT ® ENTOURAGE ® MICROSOFT ® OUTLOOK ® 、MOZILLA ® THUNDERBIRD ® wait.

[0119] In addition, one or more computer-readable storage media may be used to implement embodiments consistent with the present disclosure. A computer-readable storage medium refers to any type of physical memory on which information or data readable by a processor can be stored. Therefore, a computer-readable storage medium may store instructions for execution by one or more processors, including instructions for causing a processor to execute steps or stages consistent with the embodiments described herein. The term "computer-readable medium" should be understood to include tangible items and exclude carrier waves and transient signals, i.e., non-transitory. Examples include random access memory (RAM), read-only memory (ROM), volatile memory, non-volatile memory, hard drives, compact disk (CD) ROMs, digital video disks (DVDs), flash drives, disks, and any other known physical storage media.

[0120] The present disclosure provides an overall multi-region of interest "fitting" boundary generation neural model that is effectively performed on preview frames. The cropped boundary can be irregular, non-polygonal, and fit to an object / region or a collection of objects / regions (overlapping or non-overlapping). Therefore, unlike the prior art that allows boundary cropping that is always polygonal in shape, or boundary cropping when the user draws a boundary around the image, the present disclosure uses AI technology to determine the various objects in the image, the relationship between objects based on context and scene description, and then the correlation between objects, and provide boundary suggestions to the user. Therefore, there is no manual interference, and the method also involves implementing AI-based technology in the preview stage itself. Implementing the method in the preview stage enables amateur or professional photographers to select / focus on multiple most relevant areas in the image during the preview stage, and provide options for cropping one or more parts of the image. With respect to video frames during the preview stage, multiple boundaries will also be generated on the video frames similarly, from which the user chooses to focus on one or more objects in the video and then crop.

[0121] The present disclosure also provides an "early exit" feature that allows the object cropping system to eliminate irrelevant areas, empty spaces, and irrelevant frames during the preview phase itself, which reduces the computation of irrelevant frames.

[0122] The description of an embodiment with several components in communication with each other does not imply that all such components are required. Rather, various optional components are described to illustrate various possible embodiments of the present disclosure.

[0123] When a single device or article is described herein, it is apparent that more than one device / article (whether or not they cooperate) may be used in place of a single device / article. Similarly, where more than one device or article is described herein (whether or not they cooperate), it is apparent that a single device / article may be used in place of more than one device or article, or a different number of devices / articles may be used in place of the number of devices or procedures shown.

[0124] The functions and / or features of a device may be embodied alternatively by one or more other devices that are not explicitly described as having such functions / features. Therefore, other embodiments of the present disclosure need not include the device itself.

[0125] This specification has described methods and systems for context-aware cropping in camera preview. The steps described are set forth to explain the exemplary embodiments shown, and it is expected that ongoing technical developments will change the manner in which specific functions are performed. These examples are presented herein for purposes of illustration and not limitation. In addition, for ease of description, the boundaries of functional building blocks are arbitrarily defined herein. Alternative boundaries may be defined as long as the specified functions and their relationships are appropriately performed.

[0126] Based on the teachings contained herein, alternatives (including equivalents, extensions, variations, deviations, etc. of those described herein) will be apparent to those skilled in the relevant art. Such alternatives fall within the scope and spirit of the disclosed embodiments.

[0127] Furthermore, the words "comprising," "having," "containing," and "including," and other similar forms are intended to be equivalent in meaning and to be open ended, in that one or more items followed by any of these words does not imply an exhaustive list of such one or more items or means limited to the listed one or more items. It must also be noted that as used herein and in the appended claims, the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise.

[0128] Finally, the language used in this disclosure is selected primarily for readability and instructional purposes, and may not be selected to describe or limit the subject matter of the present invention. Therefore, it is intended that the scope of the present disclosure is not limited by this detailed description, but rather by any claims filed on an application based thereon. Therefore, the embodiments of the present disclosure are intended to illustrate, but not to limit, the scope of the present disclosure, which is set forth in the appended claims.

Claims

1. A method performed by an object clipping system of an electronic device, the method comprising: receiving a preview image including a plurality of objects from a camera; determining contour boundaries of the plurality of objects; determining a relevance score between the plurality of objects based on contexts of the plurality of objects and relationships between the plurality of objects; generating a plurality of boundary proposals including the plurality of objects based on the relevance scores; receiving a user input indicating a selection of one of the plurality of boundary suggestions or a selection of at least one object from the plurality of objects present in the plurality of boundary suggestions, wherein the relevance score is updated for the at least one object based on the user input; as well as The at least one object is cropped from the preview image based on user input and the updated relevance score using the contour boundaries determined for the plurality of objects.

2. The method according to claim 1, wherein: The contour boundaries of the plurality of objects are determined by using at least one first type of artificial intelligence (AI) based model.

3. The method according to claim 2, wherein: The at least one first type of AI-based model includes an enhanced region proposing network (RPN) for contour boundary determination, and Among them, the enhanced RPN is integrated with the instance segmentation deep neural network (DNN).

4. The method according to claim 1, wherein: The plurality of objects have at least one of a polygonal shape and a non-polygonal shape.

5. The method according to claim 3, wherein: A relevance score between the plurality of objects based on contexts of the plurality of objects and relationships between the plurality of objects is determined by using at least one second type of AI-based model.

6. The method according to claim 1, wherein: The outline boundaries determined for the plurality of objects are visualized using boundaries drawn by a boundary line construction model associated with an object clipping system. 7 . The method of claim 5 , wherein the at least one second type of AI-based model comprises a neural symbolic model for determining a context of the plurality of objects and a relationship between the plurality of objects.

8. The method according to claim 1, wherein: The relationship between the plurality of objects is determined based on the scene description and the context of the plurality of objects.

9. The method of claim 1, further comprising pre-processing the preview image to eliminate at least one of redundant image frames and irrelevant image frames.

10. The method of claim 1, further comprising storing the cropped at least one object in a single image or in multiple images.

11. The method of claim 2, further comprising: based on the preview image including a single object having one of a polygonal shape and a non-polygonal shape: determining a contour boundary of the single object using the at least one first type of AI-based model; generating at least one boundary proposal including the single object; receiving user input indicating a selection of one of the at least one boundary suggestion; and The single object is cropped from the preview image based on user input using the contour boundary determined for the single object.

12. An electronic device comprising: Memory, which stores instructions; as well as A processor configured to execute instructions to: receiving a preview image including a plurality of objects; determining contour boundaries of the plurality of objects; determining a relevance score between the plurality of objects based on contexts of the plurality of objects and relationships between the plurality of objects; generating a plurality of boundary proposals including the plurality of objects based on the relevance scores; receiving a user input indicating a selection of one of the plurality of boundary suggestions or a selection of at least one object from the plurality of objects present in the plurality of boundary suggestions, wherein the relevance score is updated for the at least one object based on the user input; as well as The at least one object is cropped from the preview image based on user input and the updated relevance score using the contour boundaries determined for the plurality of objects.

13. The electronic device according to claim 12, wherein: The processor is also configured to execute instructions to determine contour boundaries of the plurality of objects by using at least one first type of artificial intelligence (AI) based model.

14. The electronic device according to claim 13, wherein: The at least one first type of AI-based model includes an enhanced region proposal network (RPN) for contour boundary determination, wherein the enhanced RPN is integrated with an instance segmentation deep neural network (DNN).

15. The electronic device according to claim 12, wherein: The plurality of objects have at least one of a polygonal shape and a non-polygonal shape.

Images