Electronic apparatus for performing photographing by predicting photographing composition and motion path, and control method thereof

Abstract

An electronic device includes: a camera; a driver; at least one memory storing one or more instructions; and one or more processors operably connected to the camera, the driver, and the at least one memory, where the one or more instructions, when executed by the one or more processors, may cause the electronic device to perform the following operations: acquiring an image captured by the camera; and controlling the driver to change a capturing method of the camera and an arrangement state of the electronic device based on an object included in the image.

Description

TECHNICAL FIELD

[0001] The disclosure relates to an electronic device and a control method thereof, and more particularly, to an electronic device that performs photographing by predicting a photographing composition and a motion and a control method thereof. BACKGROUND

[0002] With the development of electronic technology, electronic devices providing various functions are being developed. In particular, the increase in the number of video platforms has led to a surge in demand for video recording using various recording equipment.

[0003] However, when an individual photographs a video, the individual's ability to independently perform photographing and editing is limited. In addition, individuals often lack professional photographing knowledge, and thus individuals often photograph a video using fixed photographing equipment. As a result, the quality of their videos is often low.

[0004] On the other hand, although most robots capable of automatic driving are equipped with a camera, there is no robot that automatically photographs a video.

[0005] Therefore, there is a need for a method that allows high-quality photographing while improving user convenience in video photographing. SUMMARY

[0006] TECHNICAL SOLUTION An electronic device and a control method thereof are provided, in which the electronic device automatically identifies an optimal photographing composition and a motion path, and performs photographing based on the identified photographing composition and motion path, thereby improving user convenience.

[0007] According to an aspect of the disclosure, an electronic device includes a camera, a driver, at least one memory storing one or more instructions, and one or more processors operatively connected to the camera, the driver, and the at least one memory, wherein the one or more instructions, when executed by the one or more processors, cause the electronic device to: obtain an image captured through the camera; and control the driver to change a photographing method of the camera and a placement state of the electronic device based on an object included in the image.

[0008] The one or more instructions, when executed by the one or more processors, can cause the electronic device to, based on the object included in the image: control the camera to zoom in or zoom out, and control the driver to perform at least one of changing a position of the electronic device or changing a direction of the camera.

[0009] The one or more instructions, when executed by the one or more processors, can further cause the electronic device to: obtain a recaptured image through the camera when a location of the electronic device is changed; and control the driver to adjust a photographing method of the camera and to adjust a placement state of the electronic device based on an object included in the recaptured image.

[0010] The at least one memory stores a first neural network model and a second neural network model, and the one or more instructions, when executed by the one or more processors, can cause the electronic device to: obtain a target image corresponding to the object included in the image by inputting the image into the first neural network model, obtain placement information of the electronic device corresponding to the target image by inputting the image and the target image into the second neural network model, obtain a plurality of candidate images corresponding to each of a plurality of pieces of preliminary placement information based on the placement information, and control the driver to change a placement state of the electronic device based on a candidate image corresponding to the target image among the plurality of candidate images.

[0011] The placement information can include information about a location of the electronic device and a direction of the camera, and the one or more instructions, when executed by the one or more processors, can cause the electronic device to: obtain the plurality of pieces of preliminary placement information by applying a plurality of predetermined values to each of the location of the electronic device and the direction of the camera; and obtain the plurality of candidate images by remapping the image based on the plurality of pieces of preliminary placement information.

[0012] The at least one memory can store three-dimensional (3D) model information about the object, and the one or more instructions, when executed by the one or more processors, can cause the electronic device to: obtain the plurality of candidate images by remapping the image based on the 3D model information and the plurality of pieces of preliminary placement information.

[0013] The one or more instructions, when executed by the one or more processors, can further cause the electronic device to obtain a recaptured image through the camera when a position of the electronic device is changed, obtain an updated target image corresponding to an object included in the recaptured image by inputting the recaptured image into the first neural network model, obtain updated placement information of the electronic device corresponding to the updated target image by inputting the recaptured image and the updated target image into the second neural network model, and stop an operation of the driver based on the placement state of the electronic device corresponding to the updated placement information.

[0014] The one or more instructions, when executed by the one or more processors, can further cause the electronic device to obtain a plurality of images through the camera at predetermined time intervals after the operation of the driver is stopped, and input an identified image to the first neural network model based on identifying an image in which a changed state of the object can be included among the plurality of images.

[0015] The at least one memory stores information about a movement range of the electronic device, and the one or more instructions, when executed by the one or more processors, can cause the electronic device to control the driver to change the placement state of the electronic device within the movement range.

[0016] The electronic device can further include a sensor, the movement range can include information about an area in which the electronic device can move, and the information about the area in which the electronic device can move is based on an output of the sensor when the electronic device moves to a position.

[0017] The electronic device can further include a user interface, and the one or more instructions, when executed by the one or more processors, can further cause the electronic device to control the driver to change the placement state of the electronic device based on a user command received through the user interface.

[0018] According to an aspect of the disclosure, a method of controlling an electronic device includes obtaining an image captured through a camera included in the electronic device, and changing a photographing method of the camera and a placement state of the electronic device based on an object included in the image.

[0019] The changing can include, based on the object included in the image, zooming in or out the camera, and performing at least one of changing a position of the electronic device or changing a direction of the camera.

[0020] The method can further include obtaining a recaptured image through the camera based on a change in a location of the electronic device, and adjusting a photographing method of the camera and adjusting a placement state of the electronic device based on an object included in the recaptured image.

[0021] The changing can include obtaining a target image corresponding to an object included in the image by inputting the image into a first neural network model, obtaining placement information of the electronic device corresponding to the target image by inputting the image and the target image into a second neural network model, obtaining a plurality of candidate images corresponding to each of a plurality of pieces of preliminary placement information based on the placement information, and changing the placement state of the electronic device based on a candidate image corresponding to the target image among the plurality of candidate images. BRIEF DESCRIPTION OF DRAWINGS

[0022] The above and other aspects, features and advantages of certain embodiments of the present disclosure will be more apparent from the following description taken in conjunction with the accompanying drawings, in which: Figure 1 is a block diagram provided to explain a hardware configuration of an electronic device according to an embodiment; Figure 2 is a block diagram provided to explain a software configuration of an electronic device according to an embodiment; Figure 3 is a block diagram illustrating a detailed configuration of an electronic device according to an embodiment; Figure 4 is a diagram provided to explain a method of performing photographing while changing a composition according to an embodiment; Figure 5 is a diagram provided to explain a movement range of an electronic device according to an embodiment; Figure 6 is a diagram provided to explain 3D model information according to an embodiment; Figure 7 and Figure 8 are a flowchart and a block diagram provided to explain a photographing method according to an embodiment, respectively; Figure 9 and Figure 10 are diagrams provided to explain use examples according to various embodiments; and Figure 11 is a flowchart provided to explain a method of controlling an electronic device according to an embodiment. DETAILED DESCRIPTION

[0023] Various modifications can be made to the embodiments of the disclosure. Therefore, specific example embodiments are shown in the drawings and are described in detail in the detailed description. However, it will be understood that the disclosure is not limited to the specific example embodiments, but includes all modifications, equivalents, and substitutions without departing from the scope and spirit of the disclosure. Furthermore, since well-known functions or constructions can obscure the disclosure with unnecessary detail, well-known functions or constructions are not described in detail.

[0024] Hereinafter, example embodiments of the disclosure will be described in greater detail with reference to the accompanying drawings.

[0025] In consideration of their functions in the disclosure, general terms widely used at present are selected as terms used in the embodiments of the disclosure. However, these terms can be changed according to intentions of those skilled in the art or judicial precedents, appearance of new technologies, and the like. Also, in certain cases, there can be terms arbitrarily selected by the applicant, and in this case, the meanings of the terms are clearly mentioned in the corresponding description in the disclosure. Therefore, the terms used in the disclosure need to be defined based on the meanings of the terms and the content throughout the disclosure, rather than based on simple names of the terms.

[0026] In the specification, the expressions "have," "may have," "include," "may include," etc. indicate the presence of the corresponding feature (for example, numerical value, function, operation, or component such as part) and do not exclude the presence of additional features.

[0027] The expressions "at least one of A and B" and "at least one of A or B" shall be construed to indicate one of "A," "B," or "both A and B."

[0028] The expressions "first," "second," etc. used in the specification can define various components regardless of the order or importance of the components. These expressions are used only to distinguish one component from another component, and do not limit the corresponding components.

[0029] Unless explicitly stated otherwise, the singular terms used herein are intended to include the plural forms. It will be understood that the terms "include," "formed by," and the like used herein specify the presence of stated features, numbers, steps, operations, components, parts, or combinations thereof in the specification, but do not preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0030] In the disclosure, the term "user" can refer to a person using an electronic device or a device (for example, an artificial intelligence electronic device) using an electronic device.

[0031] Hereinafter, various embodiments of the disclosure will be described in greater detail with reference to the accompanying drawings.

[0032] Figure 1 is provided to explain a block diagram of a hardware configuration of the electronic device 100 according to an embodiment.

[0033] The electronic device 100 can be a device for capturing an image. For example, the electronic device 100 can include a mobile device and a photographing device, and can be a device that controls the mobile device to move the position of the electronic device 100 and photographs an image through the photographing device. Here, the mobile device and the photographing device can be implemented as separate apparatuses, and the electronic device 100 can be implemented as a combination of the mobile device and the photographing device. However, the electronic device 100 can be any device including the mobile device and the photographing device.

[0034] Referring to Figure 1 , the electronic device 100 can include a camera 110, a driver 120, and a processor 130. However, the electronic device 100 is not limited thereto, and the electronic device 100 can be implemented in a form excluding certain configurations. For example, the electronic device 100 can include only the camera 110 and the processor 130, be connected to a separately implemented driver, and be used in a mobile form. Alternatively, the electronic device 100 can include only the driver 120 and the processor 130, and can be connected to a separately implemented photographing apparatus to perform photographing.

[0035] The camera 110 can be configured to capture a still image or a moving image. The camera 110 can capture a still image at a certain point in time, but can also continuously capture a still image.

[0036] The camera 110 can include a lens, a shutter, an aperture, a solid-state acquisition element, an analog front end (AFE), and a timing generator (TG). The shutter controls the time when light reflected from a subject enters the camera 110, and the aperture controls the amount of light entering the lens by mechanically increasing or decreasing the size of an opening through which light enters. The solid-state acquisition element outputs photoelectric charges as an electrical signal when light reflected from a subject is accumulated. The TG outputs a timing signal to read out pixel data of the solid-state acquisition element, and the AFE samples and digitizes the electrical signal output from the solid-state acquisition element.

[0037] The driver 120 can be configured to move the position of the electronic device 100 under the control of the processor 130, and can include a wheel, a motor, a transmission gear, etc. However, the driver 120 is not limited thereto, and the driver 120 can be any configuration capable of moving the position of the electronic device 100.

[0038] The processor 130 can be configured to control overall operations of the electronic device 100. In particular, the processor 130 can be connected to each configuration of the electronic device 100 to control operations of the electronic device 100. For example, the processor 130 can be connected to configurations such as the camera 110, the driver 120, the memory, etc., to control operations of the electronic device 100.

[0039] The at least one processor 130 can include one or more of a CPU, a graphics processing unit (GPU), an accelerated processing unit (APU), an integrated many-core (MIC), a neural processing unit (NPU), a hardware accelerator, or a machine learning accelerator. The at least one processor 130 can control one or any combination of other components of the electronic device 100 and can perform operations or data processing related to communication. The at least one processor 130 can execute one or more programs or instructions stored in the memory. For example, the at least one processor 130 can perform a method according to an embodiment by executing one or more instructions stored in the memory.

[0040] When a method according to an embodiment includes a plurality of operations, the plurality of operations can be performed by one processor or a plurality of processors. For example, when a first operation, a second operation, and a third operation are performed by a method according to an embodiment, all of the first operation, the second operation, and the third operation can be performed by a first processor, or the first operation and the second operation can be performed by a first processor (e.g., a general-purpose processor) and the third operation can be performed by a second processor (e.g., an AI-only processor).

[0041] The at least one processor 130 can be implemented as a single core processor including one core or one or more multi-core processors including a plurality of cores (e.g., homogeneous cores or heterogeneous cores). When the at least one processor 130 is implemented as a multi-core processor, each core included in the multi-core processor can include an internal memory such as a cache and an on-chip memory, and a common cache shared by the plurality of cores can be included in the multi-core processor. Furthermore, each core included in the plurality of cores of the multi-core processor (or some of the plurality of cores) can independently read and execute program instructions for implementing a method according to an embodiment, and all of the plurality of cores (or some of the plurality of cores) can be linked to read and execute program instructions for implementing a method according to an embodiment.

[0042] When the method according to the embodiment includes a plurality of operations, the plurality of operations can be performed by one core included in the plurality of cores of the multi-core processor, or can be performed by the plurality of cores. For example, when a first operation, a second operation, and a third operation are performed by the method according to the embodiment, all of the first operation, the second operation, and the third operation can be performed by a first core included in the multi-core processor, or the first operation and the second operation can be performed by the first core, and the third operation can be performed by a second core included in the multi-core processor.

[0043] In one or more embodiments of the disclosure, the at least one processor 130 can be a system on chip (SoC) integrated with one or more processors and other electronic components, a single core processor, a multi-core processor, or a core included in the single core processor or the multi-core processor. Here, the core can be implemented as a CPU, a GPU, an APU, a MIC, an NPU, a hardware accelerator, a machine learning accelerator, etc., but the disclosure is not limited thereto. However, hereinafter, for convenience of explanation, the term "processor 130" will be used to describe the operation of the electronic device 100.

[0044] The processor 130 can obtain an image captured by the camera 110, and can control the camera 110 and the driver 120 to change a photographing method of the camera 110 and change a placement state of the electronic device 100 (i.e., reposition the electronic device 100) based on an object included in the image.

[0045] For example, the processor 130 can control the driver 120 to perform zooming in or out of the camera 110 based on the object included in the image, and perform one of moving a position of the electronic device 100 or changing a direction of the camera 110. Alternatively, the processor 130 can control the camera 110 or the driver 120 to change a photographing method of the camera 110 or change a placement state of the electronic device 100.

[0046] However, the disclosure is not limited thereto, and the processor 130 can change the photographing method of the camera 110 in various ways. For example, the processor 130 can change the photographing method of the camera 110 by changing at least one of an aperture value, a focus, an angle of view, a white balance, an image size, a flash, a filter, or an image stabilization.

[0047] Through the above-described operation, the processor 130 can photograph the object in a photographing composition optimized for the object.

[0048] The processor 130 can obtain a recaptured image by recapturing an image through the camera 110 as the location of the mobile electronic device 100 changes, and can control the driver 120 to adjust the photographing method of the camera 110 and adjust the placement state of the electronic device 100 based on an object included in the recaptured image. In other words, the processor 130 can change the photographing angle of the object, the distance of the electronic device 100 from the object, etc. as at least one of the photographing method or the placement state changes, and can control the driver 120 to reposition the camera 110 to obtain a new photographing composition optimized for the changed object to adjust the photographing method of the camera 110 and adjust the placement state of the electronic device 100. Alternatively, even if the photographing method and the placement state do not change, the processor 130 can change the photographing angle of the object, the distance of the electronic device 100 from the object, etc. due to the movement of the object, and can control the driver 120 to reposition the camera to obtain a new photographing composition optimized for the changed object to adjust the photographing method of the camera 110 and adjust the placement state of the electronic device 100.

[0049] As described above, the processor 130 can change the photographing method and change the placement state at a predetermined time interval, and can reduce the predetermined time interval to further improve the photographing quality.

[0050] Further, the processor 130 can change the predetermined time interval based on the degree of change of the object photographed at the predetermined time interval. For example, when the location of the object changes by more than a predetermined distance or the size of the object changes by more than a threshold percentage in a first image and a second image photographed one second after the first image is photographed, the processor 130 captures a third image 0.3 seconds after the second image is photographed.

[0051] The above-described operations of the processor 130 will be described in more detail with reference to various modules of Figure 2 .

[0052] Figure 2 is a block diagram provided to explain the software configuration of the electronic device 100 according to an embodiment. In Figure 2 , the locations of the plurality of modules within the processor 130 are intended to indicate the state in which the plurality of modules are loaded (or executed) by the processor 130 and operate on the processor 130, and the plurality of modules can be pre-stored in the memory 140.

[0053] The memory 140 can refer to hardware that stores information (such as data) in an electrical form or a magnetic form so as to be accessed by the processor 130 or the like. To this end, the memory 140 can be implemented as at least one of a non-volatile memory, a volatile memory, a flash memory, a hard disk drive (HDD), or a solid state drive (SSD), a RAM, a ROM, or the like.

[0054] The memory 140 can store at least one instruction required for the operation of the electronic device 100 or the processor 130. Here, the instruction is a code unit that directs the operation of the electronic device 100 or the processor 130, and can be written in a machine language, which is a language that a computer can understand. Alternatively, the memory 140 can store a plurality of instructions that perform a specific task of the electronic device 100 or the processor 130 as an instruction set.

[0055] The memory 140 can store data, in which the data is bit or byte information that can represent characters, numbers, images, etc. For example, the memory 140 can store the target image acquisition module 130-1, the placement information acquisition module 130-2, the candidate image acquisition module 130-3, and the driver control module 130-4, as well as other modules such as a neural network module, a movement range module, 3D model information, etc.

[0056] The memory 140 can be accessed by the processor 130, and the instruction, the instruction set, or the data can be read / written / modified / deleted / updated by the processor 130.

[0057] The memory 140 can store a first neural network model and a second neural network model. Here, the first neural network model can be a model that has learned features of data through deep reinforcement learning (DRL). For example, the first neural network model can be a model that has learned a plurality of sample videos to learn objects and a shooting composition in each video, and when an obtained image is input, the first neural network model can output a target image according to the best shooting composition.

[0058] However, the present disclosure is not limited thereto, and the first neural network model can also be a model that has learned a plurality of sample videos to learn a plurality of objects and a shooting composition in each video.

[0059] The second neural network model can be implemented as a recurrent neural network (RNN) that can output a pose for a shooting composition corresponding to a target image when an obtained image and the target image are input.

[0060] The memory 140 can store 3D model information about an object. For example, 3D model information of an object that is a subject of a photograph can be stored in the memory 140 by a user. Alternatively, the processor 130 can obtain 3D model information about an object from a plurality of images photographed during movement of the electronic device 100, and store the obtained 3D model information in the memory 140.

[0061] The memory 140 can store information about a movement range in which the electronic device 100 is capable of moving. For example, the information about the movement range in which the electronic device 100 is capable of moving can be stored in the memory 140 by a user. Alternatively, the processor 130 can obtain the information about the movement range in which the electronic device 100 is capable of moving in the process of moving the electronic device 100, and store the information about the movement range in which the electronic device 100 is capable of moving in the memory 140.

[0062] The processor 130 can control the overall operation of the electronic device 100 by executing the modules or instructions stored in the memory 140. Specifically, the processor 130 can read and interpret the modules or instructions and determine timing for data processing, and can control the operation of other configurations such as the memory 140 by transmitting control signals to control the operation of the other configurations accordingly.

[0063] The processor 130 can obtain a target image of a composition corresponding to an object included in the obtained image by inputting the obtained image to a first neural network model, via execution of a target image acquisition module 130-1. The target image can be an image that is expected to be captured in the best composition based on the object included in the obtained image.

[0064] The processor 130 can obtain placement information of the electronic device 100 corresponding to the target image by inputting the obtained image and the target image to a second neural network model, via execution of a placement information acquisition module 130-2. For example, the placement information can include information about at least one of a location of the electronic device 100 or a photographing direction of the camera 110. For example, the placement information can include information about a location of the electronic device 100 capable of capturing an image such as the target image and a photographing direction of the camera 110. As another example, the placement information can include information about a change in a location of the electronic device 100 capable of capturing an image such as the target image and a change in a photographing direction of the camera 110 based on a comparison of the captured image and the target image.

[0065] The processor 130 can obtain a plurality of candidate images based on the placement information by executing the candidate image obtaining module 130-3, wherein each of the candidate images corresponds to a plurality of preliminary placement information. For example, the processor 130 can obtain a plurality of preliminary placement information by respectively applying a plurality of predetermined values to the position of the electronic device 100 and the photographing direction of the camera 110, and can obtain a plurality of candidate images by remapping the obtained image based on the plurality of preliminary placement information. For example, by respectively applying a plurality of predetermined values to the position A of the electronic device 100 and the photographing direction B of the camera 110, a plurality of preliminary placement information such as (A, B), (A, B+1), (A+1, B), (A+1, B+1), (A, B-1), (A-1, B), (A-1, B-1), (A-1, B-1), (A-1, B+1), and (A-1, B+1) can be obtained, and nine candidate images can be obtained by remapping the obtained image based on the plurality of preliminary placement information.

[0066] The processor 130 can remap the obtained image based on the 3D model information about the object stored in the storage 140 and the plurality of preliminary placement information to obtain a plurality of candidate images. In this case, more complex candidate images can be obtained.

[0067] The processor 130 can control the driver 120 to change the placement state of the electronic device 100 based on the candidate image corresponding to the target image among the plurality of candidate images by executing the driver control module 130-4. In the above example, the processor 130 can change the placement state of the electronic device 100 based on the candidate image most similar to the target image among the nine candidate images. For example, if the candidate image based on (A+1, B+1) among the nine candidate images is most similar to the target image, the processor 130 can change the position of the electronic device 100 based on A+1 and the photographing direction of the camera 110 based on B+1.

[0068] The processor 130 can obtain a recaptured image rephotographed through the camera 110 while the mobile electronic device 100 is positioned, input the recaptured image to the first neural network model to obtain an updated target image of the optimal composition corresponding to the object included in the recaptured image, input the recaptured image and the updated target image to the second neural network model to obtain updated placement information of the electronic device 100 corresponding to the target image, and stop the operation of the driver 120 when the placement state of the electronic device 100 corresponds to the updated placement information. In other words, even while the mobile electronic device 100 is positioned, the processor 130 can repeatedly the operations of obtaining the obtained image, obtaining the target image, obtaining the placement information, obtaining a plurality of candidate images, and changing the placement state based on the candidate images, and these operations enable the optimal composition to be changed in real time. Subsequently, when the placement state and the updated placement information correspond to each other, the processor 130 can recognize the photographing of the optimal composition and stop the operation of the driver 120.

[0069] After the operation of the driver 120 is stopped, the processor 130 can obtain an image through the camera 110 at a predetermined time interval, and when an image in which the state of the object has changed is recognized in the image obtained at the predetermined time interval, the processor 130 can input the recognized image into the first neural network model. In other words, even when photographing is performed in the optimal composition, the optimal composition can change due to a change in the photographing angle of the object, the distance between the electronic device 100 and the object, etc., according to the movement of the object, and thus even after the operation of the driver 120 is stopped, the processor 130 can maintain the operations of obtaining a photographed image, obtaining a target image, obtaining placement information, obtaining a plurality of candidate images, and comparing the placement state of the electronic device 100 with placement information corresponding to the candidate images.

[0070] The processor 130 can control the driver 120 to change the placement state of the electronic device 100 within a movement range based on information about the movement range in which the electronic device 100 is capable of moving, which is stored in the memory 140.

[0071] Here, the electronic device 100 further includes a sensor, and the movement range can include information about an area in which the electronic device 100 is movable, which is detected by the sensor when the electronic device 100 is moved to a random position.

[0072] The electronic device 100 further includes a user interface, and the processor 130 can control the driver 120 to change the placement of the electronic device 100 based on a user command received through the user interface.

[0073] The functions related to artificial intelligence according to the embodiments can be operated through the processor 130 and the memory 140.

[0074] The processor 130 can consist of one processor or a plurality of processors. Here, the one processor or the plurality of processors can include a general-purpose processor such as a CPU, an application processor (AP), a digital signal processor (DSP), etc., a graphics-specialized processor such as a GPU, a visual processing unit (VPU), etc., or an AI-specialized processor such as a neural processing unit (NPU).

[0075] The one processor or the plurality of processors can process input data according to a pre-defined operation rule or an AI model stored in the memory 140. Alternatively, when the one processor or the plurality of processors correspond to an AI-specialized processor, the AI-specialized processor can be designed to have a hardware structure specialized for processing a specific AI model. The pre-defined operation rule or the AI model can be formulated through training.

[0076] Here, when the AI model is formulated through training, this can mean that a base AI model is trained based on a learning algorithm by using a plurality of training data, so that a pre-defined operation rule or an AI model set to perform a desired feature (or a desired purpose) is formulated. Such learning can be performed by a device on which an AI according to the disclosure is implemented, or by a separate server and / or system. Examples of the learning algorithm can include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0077] The artificial intelligence model can consist of a plurality of neural network layers. Each of the plurality of neural network layers has a plurality of weight values, and performs a neural network calculation through a calculation between results of the plurality of weights. The plurality of weights of the plurality of neural network layers can be optimized by a learning result of the artificial intelligence model. For example, the plurality of weights can be updated to reduce or minimize a loss value or a generation value obtained by the artificial intelligence model during a learning process.

[0078] The artificial neural network can include a deep neural network (DNN) such as, but not limited to, a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a generative adversarial network (GAN), or a deep Q-network.

[0079] Figure 3 is a block diagram illustrating a detailed configuration of the electronic device 100 according to an embodiment.

[0080] The electronic device 100 can include a camera 110, a driver 120, a processor 130, and a memory 140. Referring to FIG. 1, the electronic device 100 can include the camera 110, the driver 120, the processor 130, and the memory 140. Figure 3The electronic device 100 can further include a sensor 150, a user interface 160, a microphone 170, a communication interface 180, a display 190, and a speaker 195. Figure 3 Some of the components shown in FIG. 1 can overlap with the components shown in Figure 1 and Figure 2 and thus a further description will be omitted.

[0081] The sensor 150 can include a cliff sensor, a time-of-flight (ToF) sensor, etc. The cliff sensor can be a sensor that outputs light toward the ground and receives light reflected from the ground. The ToF sensor can be a three-dimensional sensor that recognizes the three-dimensionality, spatial information, and movement of an object by calculating the time distance for light projected to the object by an infrared wave to bounce back. The processor 130 can measure a distance from the ground based on information received from the sensor 150. Furthermore, when no reflected light is received, the processor 130 can recognize a fall, and can recognize a boundary of a movable area of the electronic device 100.

[0082] The user interface 160 can be implemented as a button, a touchpad, a mouse, a keyboard, etc., or can be implemented as a touch screen that can also perform a display function and a manipulation input function. Here, the button can be various types of buttons such as a mechanical button, a touchpad, a scroll wheel, etc. formed on any area such as a front surface, a side surface, or a rear surface of the main body of the electronic device 100.

[0083] The microphone 170 is configured to receive sound and convert it into an audio signal. The microphone 170 is electrically connected to the processor 130 and can receive sound under the control of the processor 130.

[0084] For example, the microphone 170 can be formed as an integral part of the top, the front, the side, etc. of the electronic device 100. Alternatively, the microphone 170 can be provided on a remote controller or the like that is separate from the electronic device 100. In this case, the remote controller can receive sound through the microphone 170 and provide the received sound to the electronic device 100.

[0085] The microphone 170 can include various configurations such as a microphone for collecting sound in an analog form, an amplifier circuit for amplifying the collected sound, an A / D conversion circuit for sampling the amplified sound and converting the amplified sound into a digital signal, a filter circuit for removing a noise component from the converted digital signal, etc.

[0086] The microphone 170 can be implemented in the form of a sound sensor and can be configured in any manner capable of collecting sound.

[0087] The communication interface 180 is configured to perform communication with various types of external devices according to various types of communication methods. For example, the electronic device 100 can perform communication with a content server or a user terminal device through the communication interface 180.

[0088] The communication interface 180 can include a Wi-Fi module, a Bluetooth module, an infrared communication module, a wireless communication module, etc. Here, each communication module can be implemented in the form of at least one hardware chip.

[0089] The Wi-Fi module and the Bluetooth module perform communication using a Wi-Fi method and a Bluetooth method, respectively. When using the Wi-Fi module or the Bluetooth module, various connection information such as an SSID and a session key is first transmitted and received, and various information can be transmitted and received after a communication connection is established using the various connection information. The infrared communication module performs communication according to an Infrared Data Association (IrDA) communication technology that wirelessly transmits data over a short distance using infrared rays between visible light and millimeter waves.

[0090] In addition to the above-described communication methods, the wireless communication module further includes at least one communication chip that performs communication according to various wireless communication standards such as zigbee, third generation (3G), third generation partnership project (3GPP), long term evolution (LTE), LTE-advanced (LTE-A), fourth generation (4G), fifth generation (5G), etc.

[0091] Optionally, the communication interface 180 can include a wired communication interface such as HDMI, DP, Thunderbolt, USB, RGB, D-SUB, DVI, etc.

[0092] In addition, the communication interface 180 can include at least one of a local area network (LAN) module, an Ethernet module, or a wired communication module that performs communication using a twisted pair, a coaxial cable, or an optical fiber cable.

[0093] The display 190 is configured to display an image, and can be implemented as various types of displays such as a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display panel (PDP), etc. The display 190 can further include a driving circuit, a backlight unit, etc. that can be implemented in the form of a-si TFT, low temperature poly-silicon (LTPS) TFT, organic TFT (OTFT), etc. The display 190 can be implemented as a touch screen combined with a touch sensor, a flexible display, a 3D display, etc.

[0094] The speaker 195 is configured to output not only various audio data processed by the processor 130 but also various notification sounds, voice messages, etc.

[0095] As described above, the electronic device 100 can perform automatic image capture with optimal composition based on the object, and can improve user convenience.

[0096] In the following text, reference will be made to Figures 4 to 10 The operation of electronic device 100 will be described in more detail. For ease of explanation, [details to be added]. Figures 4 to 10 Various embodiments are described herein. However, Figures 4 to 10 The various embodiments can be implemented in any combination.

[0097] Figure 4 These are illustrations provided to explain a method of taking a picture when the composition is changed according to an embodiment.

[0098] Processor 130 can acquire an image by performing a shot from the left side of the object at a viewpoint 410, such as Figure 4 As shown on the left.

[0099] Processor 130 can obtain a target image corresponding to a composition of objects included in the obtained image by inputting the acquired image into a first neural network model. For example, processor 130 can obtain the target image when the object is photographed in a right-to-left orientation, such as... Figure 4 As shown in the upper center. Here, the target image can be an image that is expected to be captured with the best shooting composition based on the objects included in the obtained image.

[0100] The processor 130 can obtain placement information of the electronic device 100 corresponding to the target image by inputting the acquired image and the target image into a second neural network model, and can obtain multiple candidate images 420-1, 420-2, 420-3, and 420-4 based on the placement information, wherein each candidate image corresponds to multiple preliminary placement information, such as... Figure 4 As shown at the bottom center.

[0101] The processor 130 can control the camera 110 and the driver 120 to change the shooting method of the camera 110 and the placement state of the electronic device 100 based on the candidate image most similar to the target image among multiple candidate images 420-1, 420-2, 420-3, and 420-4. For example, the processor 130 can control the camera 110 and the driver 120 to change the shooting method of the camera 110 and the placement state of the electronic device 100 based on the candidate image 420-4 most similar to the target image among multiple candidate images 420-1, 420-2, 420-3, and 420-4.

[0102] Through the above operations, the processor 130 can control the camera 110 and the driver 120 to change the shooting method of the camera 110 and change the placement state of the electronic device 100, such as... Figure 4 The 430 is shown.

[0103] Subsequently, the processor 130 can acquire the updated image, the updated target image, the updated placement information, a plurality of updated candidate images, and adjust the placement state based on the plurality of updated candidate images, as shown in 440, thereby performing the operation as shown in 430. Figure 4 Figure 4

[0104] The processor 130 can repeat the operation of 440 and the operation of 430 at a predetermined time interval, thereby capturing an image in an optimal composition in real time.

[0105] Although the operation of 440 is described with reference to a single object, Figure 4 the processor 130 can operate considering a plurality of objects. For example, the processor 130 can not only consider the elephant object but also consider a user's hand in Figure 4 to control the camera 110 and the driver 120 to change the photographing method of the camera 110 and change the placement state of the electronic device 100.

[0106] Figure 5 is a diagram provided to explain a movement range of the electronic device 100 according to an embodiment.

[0107] As shown in Figure 5 , the processor 130 can obtain information about an area in which the electronic device 100 can move, which is detected by the sensor 150 when the electronic device 100 moves to a random location, and store information about a final movable area as a movement range in the storage 140.

[0108] Subsequently, the processor 130 can move only within the movement range while performing the operation as shown in Figure 4 In the example of Figure 4 , if the processor 130 identifies that movement according to the candidate image 420-4, which is most similar to the target image, among the plurality of candidate images 420-1, 420-2, 420-3, 420-4, exceeds the movement range, the processor 130 can control the camera 110 and the driver 120 to change the photographing method of the camera 110 and change the placement state of the electronic device 100 based on a candidate image, which is second most similar to the target image, among the plurality of candidate images 420-1, 420-2, 420-3, 420-4.

[0109] However, the present disclosure is not limited thereto, and the storage 140 can not store any information about the movement range. In this case, the processor 130 can operate in such a way that, while performing the operation as shown in Figure 4 , the processor 130 identifies whether it is movable through the sensor 150, and if it is identified that it is not movable, changes the above-described candidate image. ​​

[0110] Figure 6 is provided to explain 3D model information according to an embodiment.

[0111] For a more complex operation, the user can also utilize the electronic device 100 to obtain 3D model information about an object. For example, when receiving a user command for obtaining 3D model information about an object, as Figure 6 indicated, the processor 130 can move around the object 610 to capture a plurality of images 620-1, 620-2, and 620-3, and obtain 3D model information about the object based on the plurality of images 620-1, 620-2, and 620-3. However, the disclosure is not limited thereto, and the user can directly provide 3D model information to the electronic device 100.

[0112] The processor 130 can identify an object included in the obtained image, and can identify whether the identified object corresponds to 3D model information stored in the memory 140. If the identified object corresponds to the 3D model information, the processor 130 can obtain a plurality of candidate images by remapping the obtained image based on the 3D model information and a plurality of preliminary placement information.

[0113] The above-described operation allows photographing in a more complex composition.

[0114] Figure 7 and Figure 8 is provided to explain a photographing method according to an embodiment.

[0115] First, the processor 130 can obtain an image (S710).

[0116] The processor 130 can obtain a target image based on the obtained image (S720). For example, as Figure 8 indicated, the processor 130 can obtain a target image (composition data 830) by inputting the obtained image (current scene 810) into a first neural network model (DRL 820-1). Subsequently, the processor 130 can input the obtained image and the target image into a second neural network model (RNN 820-2) to obtain placement information (predicted area 840) of the electronic device 100 corresponding to the target image.

[0117] Based on the placement information, the processor 130 can obtain a plurality of candidate images, each of which corresponds to a plurality of preliminary placement information (S730).

[0118] The processor 130 can obtain a target placement state of the electronic device 100 based on a candidate image most similar to the target image among the plurality of candidate images (S740), and can control the driver 120 to change the placement state of the electronic device 100 based on the target placement state (S750).

[0119] Figure 9 and Figure 10 are diagrams provided to explain use examples according to various embodiments.

[0120] The electronic device 100 can further include a projection unit, and the processor 130 can control the projection unit to project an image, as Figure 9 indicated on the upper side. In this case, the processor 130 can further identify a user as an object in the same manner as in Figure 5 and control the driver 120 to change a projection area when the position of the user is changed, as Figure 9 indicated on the lower side.

[0121] Alternatively, the electronic device 100 can be implemented as a service robot. For example, when a user is eating, the processor 130 can identify a first user among the users as an object in the same manner as in Figure 5 and perform a service based on the first user, as Figure 10 indicated on the upper side. Subsequently, the processor 130 can identify a second user among the users as an object in the same manner as in Figure 5 and perform a service based on the second user, as Figure 10 indicated on the lower side.

[0122] Figure 11 is a flowchart provided to explain a control method of an electronic device according to an embodiment.

[0123] First, an image captured through a camera included in the electronic device is obtained (S1110), and based on an object included in the obtained image, a photographing method of the camera is changed and a placement state of the electronic device is changed (S1120).

[0124] Further, the operation of changing (S1120) can include zooming in or out the camera, moving a position of the electronic device, or changing a photographing direction of the camera based on the object included in the obtained image.

[0125] The operation can further include obtaining a recaptured image of the electronic device through the camera while the electronic device is being repositioned, and adjusting a photographing method of the camera and adjusting a placement state of the electronic device based on an object included in the recaptured image.

[0126] Further, the changing operation (S1120) can include obtaining a target image of a composition corresponding to the object included in the obtained image by inputting the obtained image into the first neural network model, obtaining placement information of the electronic device corresponding to the target image by inputting the obtained image and the target image into the second neural network model, obtaining a plurality of candidate images each corresponding to a plurality of preliminary placement information based on the placement information, and changing the placement state of the electronic device based on a candidate image corresponding to the target image among the plurality of candidate images.

[0127] The placement information can include information about a position of the electronic device and a photographing direction of the camera, and the operation of obtaining the plurality of candidate images can include applying a plurality of predetermined values to the position of the electronic device and the photographing direction of the camera, respectively, to obtain a plurality of preliminary placement information, and remapping the obtained image based on the plurality of preliminary placement information to obtain the plurality of candidate images.

[0128] Further, the operation of obtaining the plurality of candidate images can include remapping the obtained image based on 3D model information of the object and the plurality of preliminary placement information to obtain the plurality of candidate images.

[0129] The operation can further include obtaining a recaptured image of the electronic device by the camera while the electronic device is being repositioned, obtaining an updated target image of a composition corresponding to the object included in the recaptured image by inputting the recaptured image into the first neural network model, obtaining updated placement information of the electronic device corresponding to the updated target image by inputting the recaptured image and the updated target image into the second neural network model, and stopping the position movement of the electronic device if the placement state of the electronic device and the updated placement information correspond to each other.

[0130] The operation can further include obtaining an image by the camera at a predetermined time interval after stopping the position movement of the electronic device, and inputting an identified image into the first neural network model if an image in which a state of the object has been changed is recognized in the image obtained at the predetermined time interval.

[0131] Further, the changing operation (S1120) can include changing the placement state of the electronic device within a movement range in which the electronic device is movable.

[0132] Further, the movement range can include information about a region in which the electronic device is movable, wherein the information about the region in which the electronic device is movable is detected by a sensor included in the electronic device when the electronic device is moved to a random position.

[0133] The operation can further include receiving a user command and changing the placement of the electronic device based on the user command.

[0134] According to one or more embodiments of the disclosure, an electronic device can perform automatic image photographing in an optimal composition according to an object, and can improve user convenience.

[0135] According to an embodiment, the various embodiments described above can be implemented as software including instructions stored in a machine-readable storage medium which is readable by a machine (e.g., a computer). A machine is a device that invokes the instructions stored in the storage medium and operates according to the invoked instructions, and the device can include an electronic device (e.g., electronic device (A)) according to the aforementioned embodiments. In the case where the instructions are executed by a processor, the processor can perform functions corresponding to the instructions by itself or by using other components under the control of the processor. The instructions can include a code generated by a compiler or a code executed by an interpreter. The machine-readable storage medium can be provided in the form of a non-transitory storage medium. Here, the term "non-transitory" simply means that the storage medium is tangible, and does not include a signal per se, and does not distinguish whether data is semi-permanently stored in the storage medium or temporarily stored in the storage medium.

[0136] In addition, according to an embodiment, the above-described method according to various embodiments can be included and provided in a computer program product. The computer program product can be traded between a seller and a buyer as a product. The computer program product can be distributed in the form of a machine-readable storage medium (e.g., a compact disc read only memory (CD-ROM)) or be distributed online through an application store (e.g., PlayStore TM ). In the case of online distribution, at least a portion of the computer program product can be temporarily stored in a storage medium such as a manufacturer's server, an application store's server, or a relay server, or be temporarily generated. In addition, according to an embodiment, a program instruction can be provided to a machine in a transmittable data format (e.g., a machine-readable storage medium). Various embodiments of the disclosure can be embodied in such transmittable data format.

[0137] In addition, according to an embodiment, the various embodiments described above can be implemented using software, hardware, or a combination thereof in a recording medium readable by a computer or a similar device. In some cases, the embodiments described herein can be implemented by a processor itself. According to software implementation, embodiments such as the processes and functions described in the specification can be implemented as separate software. Each software can perform one or more functions and operations described herein.

[0138] Computer instructions for performing the processing operations of the apparatus according to the various embodiments described above can be stored in a non-transitory computer readable medium. When executed by the processor of a specific apparatus, the computer instructions stored in such a non-transitory computer readable medium allow the specific apparatus to perform the processing operations in the apparatus according to the various embodiments described above. The non-transitory computer readable medium refers to a medium that stores data semi-permanently and is readable by an apparatus, rather than a medium that stores data for a short time, such as a register, a cache, and a memory. Specific examples of the non-transitory computer readable medium can include a CD, a DVD, a hard disk, a Blu-ray disk, a USB, a memory card, a ROM, and the like.

[0139] Components (e.g., modules or programs) according to the various embodiments described above can include single entities or multiple entities, and some of the above-described respective sub-components can be omitted, or other sub-components can be further included in the various embodiments. Alternatively or additionally, some components (e.g., modules or programs) can be integrated into one entity, and perform the same or similar functions performed by each corresponding component before the integration. The operations performed by the modules, programs, or other components according to the various embodiments can be executed in a sequential, parallel, iterative or heuristic way, at least some of the operations can be executed in a different order or omitted, or other operations can be added.

[0140] Example embodiments have been shown and described, but the present disclosure is not limited to the above-described specific embodiments, and can be implemented by those of ordinary skill in the art related to the present disclosure without departing from the spirit of the claims. Of course, various modifications can be made from the present disclosure, and these modifications should not be individually understood from the technical spirit or perspective of the present disclosure.

Claims

1. An electronic device, comprising: camera; drive; At least one memory, storing one or more instructions; as well as One or more processors are operatively connected to the camera, the driver, and the at least one memory. Wherein, when executed by the one or more processors, the one or more instructions cause the electronic device to perform the following operations: Obtain the image captured by the camera; and Based on the objects included in the image, the driver is controlled to change the shooting method of the camera and the placement state of the electronic device.

2. The electronic device as claimed in claim 1, wherein, When executed by the one or more processors, the one or more instructions cause the electronic device to perform the following operations based on the object included in the image: Control the camera to zoom in or out, and The driver is controlled to perform at least one of changing the position of the electronic device or changing the orientation of the camera.

3. The electronic device as claimed in claim 1, wherein, When the one or more instructions are executed by the one or more processors, the electronic device also performs the following operations: When the position of the electronic device is changed, the camera acquires a recaptured image; and Based on the objects included in the recaptured image, the driver is controlled to adjust the camera's shooting method and the placement of the electronic device.

4. The electronic device as claimed in claim 1, wherein, The at least one memory stores the first neural network model and the second neural network model, and Wherein, when executed by the one or more processors, the one or more instructions cause the electronic device to perform the following operations: By inputting the image into the first neural network model, a target image corresponding to the object included in the image is obtained. By inputting the image and the target image into the second neural network model, placement information of the electronic device corresponding to the target image is obtained; Based on the placement information, multiple candidate images are obtained corresponding to each of the multiple preliminary placement information. Based on the candidate image corresponding to the target image among the plurality of candidate images, the driver is controlled to change the placement state of the electronic device.

5. The electronic device as claimed in claim 4, wherein, The placement information includes information about the location of the electronic device and the orientation of the camera. Wherein, when executed by the one or more processors, the one or more instructions cause the electronic device to perform the following operations: By applying multiple predetermined values ​​to each of the position of the electronic device and the orientation of the camera, the multiple preliminary placement information are obtained, and The multiple candidate images are obtained by remapping the image based on the multiple initial placement information.

6. The electronic device as claimed in claim 5, wherein, The at least one memory stores three-dimensional (3D) model information about the object; and When executed by the one or more processors, the electronic device performs the following operations: remapping the image based on the 3D model information and the multiple preliminary placement information to obtain the multiple candidate images.

7. The electronic device as claimed in claim 4, wherein, When the one or more instructions are executed by the one or more processors, the electronic device also performs the following operations: When the position of the electronic device is changed, the camera acquires a recaptured image. By inputting the recaptured image into the first neural network model, an updated target image corresponding to the objects included in the recaptured image is obtained. By inputting the recaptured image and the updated target image into the second neural network model, updated placement information of the electronic device corresponding to the updated target image is obtained; as well as Based on the placement status of the electronic device and the updated placement information, the operation of the driver is stopped.

8. The electronic device as claimed in claim 7, wherein, When the one or more instructions are executed by the one or more processors, the electronic device also performs the following operations: After the operation of the drive stops, multiple images are acquired through the camera at predetermined time intervals, and Based on the images identified from the plurality of images, including the image showing the changed state of the object, the identified images are input into the first neural network model.

9. The electronic device as claimed in claim 1, wherein, The at least one memory stores information about the range of motion of the electronic device, and When executed by the one or more processors, the one or more instructions cause the electronic device to perform the following operations: control the driver to change the placement state of the electronic device within the range of motion.

10. The electronic device of claim 9, further comprising a sensor. in, The range of movement includes information about the area where the electronic device can move, and The information regarding the area that the electronic device can move is based on the output of the sensor when the electronic device moves to the desired location.

11. The electronic device of claim 1, further comprising a user interface. in, When executed by the one or more processors, the one or more instructions also cause the electronic device to perform the following operations: control the driver to change the placement state of the electronic device based on user commands received through the user interface.

12. A method for controlling an electronic device, the method comprising: To obtain images captured by a camera included in the electronic device; as well as Based on the objects included in the image, the shooting method of the camera and the placement of the electronic device are changed.

13. The method of claim 12, wherein, The change includes performing the following operations based on the object included in the image: To zoom in or out of the camera; and Perform at least one of changing the position of the electronic device or changing the orientation of the camera.

14. The method of claim 12, further comprising: Based on the change in the position of the electronic device, the camera obtains a recaptured image; as well as Based on the objects included in the recaptured image, the camera's shooting method and the placement of the electronic device are adjusted.

15. The method of claim 12, wherein, The changes include: By inputting the image into a first neural network model, a target image corresponding to the object included in the image is obtained; By inputting the image and the target image into a second neural network model, placement information of the electronic device corresponding to the target image is obtained; Based on the placement information, multiple candidate images corresponding to each of the multiple preliminary placement information are obtained; and The placement state of the electronic device is changed based on the candidate image corresponding to the target image among the plurality of candidate images.

Images