Electronic device for performing capturing by predicting capturing composition and moving line, and control method therefor

Abstract

An electronic device comprises: a camera; a driver; at least one memory that stores one or more instructions; and one or more processors operatively connected to the camera, the driver, and the at least one memory, wherein the one or more instructions, when executed by the one or more processors, may cause the electronic device to: acquire an image captured through the camera; and control the driver so as to change, on the basis of an object included in the image, a capturing method of the camera and an arrangement state of the electronic device.

Description

Electronic device for performing shooting by predicting shooting composition and movement line and its control method

[0001] The present disclosure relates to an electronic device and a control method thereof, and more particularly, to an electronic device that performs shooting by predicting a shooting composition and movement line, and a control method thereof.

[0002] Advances in electronic technology have led to the development of electronic devices offering a variety of functions. In particular, the proliferation of video platforms has led to an explosive increase in demand for video recording using a variety of filming equipment.

[0003] However, when individuals film videos, their ability to film and edit on their own is limited. Furthermore, individuals often lack professional filming expertise and end up filming with fixed equipment. Consequently, the resulting footage often has poor quality.

[0004] Meanwhile, most robots capable of autonomous driving are equipped with cameras, but none of them automatically record videos.

[0005] Therefore, a method is required that enables high-quality shooting while improving user convenience in video shooting.

[0006] An electronic device and a control method thereof are provided to increase user convenience by automatically identifying the optimal shooting composition and movement path and performing shooting according to the identified shooting composition and movement path.

[0007] According to one embodiment of the present disclosure for achieving the above object, an electronic device includes a camera, a driving unit, at least one memory storing one or more commands, and one or more processors operably connected to the camera, the driving unit, and the at least one memory, wherein the one or more processors execute the one or more commands to obtain an image captured by the camera, and control the driving unit to change a shooting method of the camera and a placement state of the electronic device based on an object included in the image.

[0008] Additionally, the one or more processors may control the camera to perform zoom-in or zoom-out based on an object included in the image by executing the one or more commands, and control the driving unit to perform at least one of changing the position of the electronic device or changing the direction of the camera.

[0009] And, the one or more processors can control the driving unit to execute the one or more commands to obtain a re-captured image through the camera while changing the position of the electronic device, and to adjust the shooting method of the camera and the arrangement state of the electronic device based on an object included in the re-captured image.

[0010] In addition, the at least one memory stores a first neural network model and a second neural network model, and the one or more processors execute the one or more commands to input the image into the first neural network model to obtain a target image corresponding to an object included in the image, input the image and the target image into the second neural network model to obtain arrangement information of the electronic device corresponding to the target image, obtain a plurality of candidate images each corresponding to a plurality of pieces of preliminary arrangement information based on the arrangement information, and control the driving unit to change the arrangement state of the electronic device based on a candidate image corresponding to the target image among the plurality of candidate images.

[0011] And, the arrangement information includes information about the location of the electronic device and the direction of the camera, and the one or more processors can execute the one or more commands to apply a plurality of preset values ​​to each of the location of the electronic device and the direction of the camera to obtain the plurality of preliminary arrangement information, and remap the image based on the plurality of preliminary arrangement information to obtain the plurality of candidate images.

[0012] Additionally, the at least one memory includes 3D model information for the object, and the one or more processors can execute the one or more commands to remap the image based on the 3D model information and the plurality of preliminary arrangement information to obtain the plurality of candidate images.

[0013] And, the one or more processors may execute the one or more commands to obtain a re-captured image through the camera while changing the position of the electronic device, input the re-captured image into the first neural network model to obtain an updated target image corresponding to an object included in the re-captured image, input the re-captured image and the updated target image into the second neural network model to obtain updated arrangement information of the electronic device corresponding to the updated target image, and stop the operation of the driving unit when the arrangement state of the commercial electronic device and the updated arrangement information correspond.

[0014] In addition, the one or more processors may execute the one or more commands to stop the operation of the driving unit, and then acquire a plurality of images through the camera at preset time intervals, and when an image in which the state of an object has changed is identified from the plurality of images, the identified image may be input into the first neural network model.

[0015] And, the at least one memory stores information about a movement range of the electronic device, and the one or more processors can control the driving unit to change the arrangement state of the electronic device within the movement range by executing the one or more commands.

[0016] Additionally, the device further includes a sensor, wherein the range of movement includes information about an area in which the electronic device can move, and the information about the area in which the electronic device can move may be based on an output of the sensor while the electronic device is moving to a location.

[0017] And, further comprising a user interface, wherein the one or more processors can control the driving unit to change the arrangement state of the electronic device based on a user command received through the user interface by executing the one or more commands.

[0018] Meanwhile, according to one embodiment of the present disclosure, a method for controlling an electronic device may include a step of acquiring an image captured by a camera included in the electronic device, and a step of changing a shooting method of the camera and changing a placement state of the electronic device based on an object included in the image.

[0019] Additionally, the changing step may control the camera to perform zoom in or zoom out based on an object included in the image, and perform at least one of changing the position of the electronic device or changing the direction of the camera.

[0020] And, the method may further include a step of acquiring a re-captured image through the camera while changing the position of the electronic device, and a step of adjusting a shooting method of the camera and adjusting a placement state of the electronic device based on an object included in the re-captured image.

[0021] In addition, the changing step may include a step of inputting the image into a first neural network model to obtain a target image corresponding to an object included in the image, a step of inputting the image and the target image into a second neural network model to obtain arrangement information of the electronic device corresponding to the target image, a step of obtaining a plurality of candidate images each corresponding to a plurality of pieces of preliminary arrangement information based on the arrangement information, and a step of changing the arrangement state of the electronic device based on a candidate image corresponding to the target image among the plurality of candidate images.

[0022] The above and other aspects, features and advantages of specific embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings.

[0023] FIG. 1 is a block diagram illustrating a hardware configuration of an electronic device according to an embodiment of the present disclosure.

[0024] FIG. 2 is a block diagram illustrating a software configuration of an electronic device according to an embodiment of the present disclosure.

[0025] FIG. 3 is a block diagram showing a detailed configuration of an electronic device according to an embodiment of the present disclosure.

[0026] FIG. 4 is a drawing for explaining a method of performing shooting while changing the composition according to one embodiment of the present disclosure.

[0027] FIG. 5 is a drawing for explaining the movement range of an electronic device according to one embodiment of the present disclosure.

[0028] FIG. 6 is a drawing for explaining 3D model information according to one embodiment of the present disclosure.

[0029] FIG. 7 and FIG. 8 are a flowchart and a drawing, respectively, for explaining a shooting method according to an embodiment of the present disclosure.

[0030] FIGS. 9 and 10 are drawings for explaining examples of use according to various embodiments of the present disclosure.

[0031] FIG. 11 is a flowchart for explaining a method for controlling an electronic device according to an embodiment of the present disclosure.

[0032] The embodiments of the present disclosure are susceptible to various modifications. Therefore, specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, it should be understood that the present disclosure is not limited to specific embodiments and encompasses all modifications, equivalents, and alternatives within the scope of the technical spirit of the present disclosure. Furthermore, detailed descriptions of well-known functions or configurations are omitted to avoid obscuring the content with unnecessary detail.

[0033] Hereinafter, the present disclosure will be described in detail with reference to the attached drawings.

[0034] The terms used in the embodiments of this disclosure have been selected from widely used, current terms, taking into account the functions of this disclosure. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the description of the relevant disclosure. Therefore, the terms used in this disclosure should not be defined simply as names of terms, but rather based on the meanings of the terms and the overall content of this disclosure.

[0035] In this specification, expressions such as “has,” “can have,” “includes,” or “may include” indicate the presence of a feature (e.g., a number, function, operation, or component such as a part), and do not exclude the presence of additional features.

[0036] The expressions “at least one of A and B” and “at least one of A or B” should be understood to refer to either “A” or “B” or “A and B”.

[0037] As used herein, the expressions “first,” “second,” “first,” or “second,” etc., may describe various components, regardless of order and / or importance, and are only used to distinguish one component from another, but do not limit the components.

[0038] Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, terms such as "comprise" or "consist of" are intended to indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but should be understood not to preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0039] In this specification, the term user may refer to a person using an electronic device or a device using an electronic device (e.g., an artificial intelligence electronic device).

[0040] Various embodiments of the present disclosure will be described in more detail with reference to the attached drawings below.

[0041] FIG. 1 is a block diagram illustrating a hardware configuration of an electronic device (100) according to one embodiment of the present disclosure.

[0042] The electronic device (100) may be a device that captures images. For example, the electronic device (100) may include a means of transportation and a means of photographing, and may be a device that controls the means of transportation to move the location of the electronic device (100) and captures images through the means of photographing. Here, the means of transportation and the means of photographing may be implemented as separate devices, and the electronic device (100) may be implemented in a form in which the means of transportation and the means of photographing are combined. However, the present invention is not limited thereto, and the electronic device (100) may be any device that includes a means of transportation and a means of photographing.

[0043] According to FIG. 1, the electronic device (100) may include a camera (110), a driving unit (120), and a processor (130). However, the present invention is not limited thereto, and the electronic device (100) may be implemented in a form in which some components are excluded. For example, the electronic device (100) may include only the camera (110) and the processor (130), and may be used in a form in which it is coupled to a separately implemented driving device and moved. Alternatively, the electronic device (100) may include only the driving unit (120) and the processor (130), and may be used in a form in which it is coupled to a separately implemented photographing device and performs photographing.

[0044] The camera (110) may be configured to capture still images or moving images. The camera (110) may capture still images at a specific point in time, but may also capture still images continuously.

[0045] The camera (110) may include a lens, a shutter, an aperture, a solid-state image sensor, an AFE (Analog Front End), and a TG (Timing Generator). The shutter controls the time at which light reflected from a subject enters the camera (110), and the aperture controls the amount of light incident on the lens by mechanically increasing or decreasing the size of the opening through which light enters. The solid-state image sensor outputs a photoelectric charge when light reflected from a subject is collected. The TG outputs a timing signal for reading out pixel data of the solid-state image sensor, and the AFE samples and digitizes an electrical signal output from the solid-state image sensor.

[0046] The driving unit (120) may be configured to move the position of the electronic device (100) under the control of the processor (130), and may include wheels, a motor, a transmission, etc. However, the present invention is not limited thereto, and the driving unit (120) may have any configuration as long as it can move the position of the electronic device (100).

[0047] The processor (130) can control the overall operation of the electronic device (100). Specifically, the processor (130) is connected to each component of the electronic device (100) and can control the overall operation of the electronic device (100). For example, the processor (130) can be connected to components such as a camera (110), a driving unit (120), a memory, etc. and can control the operation of the electronic device (100).

[0048] The one or more processors (130) may include one or more of a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), an Accelerated Processing Unit (APU), a Many Integrated Core (MIC), a Digital Signal Processor (DSP), a Neural Processing Unit (NPU), a hardware accelerator, or a machine learning accelerator. The one or more processors (130) may control one or any combination of other components of the electronic device (100) and perform operations related to communication or data processing. The one or more processors (130) may execute one or more programs or instructions stored in a memory. For example, the one or more processors (130) may perform a method according to an embodiment of the present disclosure by executing one or more instructions stored in a memory.

[0049] When a method according to an embodiment of the present disclosure includes multiple operations, the multiple operations may be performed by one processor or by multiple processors. For example, when a first operation, a second operation, and a third operation are performed by a method according to an embodiment, the first operation, the second operation, and the third operation may all be performed by the first processor, or the first operation and the second operation may be performed by the first processor (e.g., a general-purpose processor) and the third operation may be performed by the second processor (e.g., an artificial intelligence-specific processor).

[0050] One or more processors (130) may be implemented as a single core processor including one core, or may be implemented as one or more multicore processors including multiple cores (e.g., homogeneous multicores or heterogeneous multicores). When one or more processors (130) are implemented as a multicore processor, each of the multiple cores included in the multicore processor may include an internal processor memory, such as a cache memory or an on-chip memory, and a common cache shared by the multiple cores may be included in the multicore processor. In addition, each of the multiple cores (or some of the multiple cores) included in the multicore processor may independently read and execute a program instruction for implementing a method according to an embodiment of the present disclosure, or all (or some) of the multiple cores may be linked to read and execute a program instruction for implementing a method according to an embodiment of the present disclosure.

[0051] When a method according to an embodiment of the present disclosure includes a plurality of operations, the plurality of operations may be performed by one core among the plurality of cores included in a multi-core processor, or may be performed by the plurality of cores. For example, when a first operation, a second operation, and a third operation are performed by a method according to an embodiment, the first operation, the second operation, and the third operation may all be performed by a first core included in the multi-core processor, or the first operation and the second operation may be performed by a first core included in the multi-core processor, and the third operation may be performed by a second core included in the multi-core processor.

[0052] In one or more embodiments of the present disclosure, one or more processors (130) may refer to a system on a chip (SoC) in which one or more processors and other electronic components are integrated, a single-core processor, a multi-core processor, or a core included in a single-core processor or a multi-core processor, wherein the core may be implemented as a CPU, a GPU, an APU, a MIC, a DSP, an NPU, a hardware accelerator, or a machine learning accelerator, but the embodiments of the present disclosure are not limited thereto. However, for convenience of explanation, the operation of the electronic device (100) is described below using the expression processor (130).

[0053] The processor (130) can control the camera (110) and the driving unit (120) to obtain an image captured by the camera (110), and change the shooting method of the camera (110) and the arrangement state of the electronic device (100) (e.g., change the position of the electronic device (100)) based on an object included in the image.

[0054] For example, the processor (130) may control the driving unit (120) to perform zoom-in or zoom-out of the camera (110) based on an object included in an image, and to perform at least one of movement of the position of the electronic device (100) or change of the direction of the camera (110). Alternatively, the processor (130) may control the camera (110) or the driving unit (120) to change the shooting method of the camera (110) or change the arrangement state of the electronic device (100).

[0055] However, the present invention is not limited thereto, and the processor (130) may change the shooting method of the camera (110) in any number of ways. For example, the processor (130) may change the shooting method by changing at least one of the aperture value, focus, angle of view, white balance, image size, flash, filter, or shake correction of the camera (110).

[0056] Through this action, the processor (130) can capture an object with a shooting composition optimized for the object.

[0057] The processor (130) can control the driving unit (120) to obtain a re-captured image re-captured through the camera (110) while moving the position of the electronic device (100), and to adjust the shooting method of the camera (110) and the arrangement state of the electronic device (100) based on the object included in the re-captured image. That is, the processor (130) can change the shooting angle of the object, the distance between the electronic device (100) and the object, etc. as at least one of the shooting method or the arrangement state is changed, and can control the driving unit (120) to re-position the camera (110), adjust the shooting method of the camera (110), and adjust the arrangement state of the electronic device (100) for a new shooting composition optimized for the changed object. Alternatively, the processor (130) may control the driving unit (120) to reposition the camera, adjust the shooting method of the camera (110), and adjust the placement state of the electronic device (100) to obtain a new shooting composition optimized for the changed object, even if the shooting method and placement state do not change, as the shooting angle of the object, the distance between the electronic device (100), etc. may change due to movement of the object.

[0058] In the same manner as above, the processor (130) can change the shooting method and change the arrangement state at preset time intervals, and can further improve the shooting quality by reducing the preset time intervals.

[0059] Additionally, the processor (130) may change the preset time interval based on the degree of change in the object photographed at the preset time interval. For example, if the position of the object changes by a preset distance or more or the size of the object changes by a threshold ratio or more in the first image and the second image photographed 1 second after the first image is photographed, the processor (130) may photograph a third image 0.3 seconds after the second image is photographed.

[0060] Meanwhile, the operation of this processor (130) is explained in more detail through the various modules of FIG. 2.

[0061] FIG. 2 is a block diagram illustrating a software configuration of an electronic device (100) according to an embodiment of the present disclosure. In FIG. 2, the positioning of multiple modules within a processor (130) indicates a state in which the multiple modules are loaded (or executed) by the processor (130) and operated by the processor (130), and the multiple modules may be pre-stored in a memory (140).

[0062] Memory (140) may refer to hardware that stores information such as data in an electrical or magnetic form so that the processor (130) or the like can access it. To this end, the memory (140) may be implemented as at least one piece of hardware from among non-volatile memory, volatile memory, flash memory, hard disk drive (HDD), solid state drive (SSD), RAM, ROM, etc.

[0063] The memory (140) may store at least one instruction required for the operation of the electronic device (100) or the processor (130). Here, the instruction is a unit of code that instructs the operation of the electronic device (100) or the processor (130), and may be written in machine language, which is a language that a computer can understand. Alternatively, the memory (140) may store a plurality of instructions for performing a specific task of the electronic device (100) or the processor (130) as an instruction set.

[0064] The memory (140) may store data in bit or byte units that can represent characters, numbers, images, etc. For example, the memory (140) may store a target image acquisition module (130-1), a placement information acquisition module (130-2), a candidate image acquisition module (130-3), a drive unit control module (130-4), along with a neural network module, a movement range module, 3D model information, etc.

[0065] The memory (140) is accessed by the processor (130), and reading / writing / modifying / deleting / updating instructions, instruction sets, or data can be performed by the processor (130).

[0066] A first neural network model and a second neural network model may be stored in the memory (140). Here, the first neural network model may be a model that has learned data characteristics through deep reinforcement learning (DRL). For example, the first neural network model is a model that has learned objects and shooting compositions from multiple sample videos and has learned multiple sample videos. When an acquired image is input, the first neural network model can output a target image according to the optimal shooting composition.

[0067] However, it is not limited to this, and the first neural network model may be a model that learns multiple sample videos and learns multiple objects and shooting compositions from each video.

[0068] The second neural network model can be implemented as a recurrent neural network (RNN), and when the acquired image and target image are input, it can output a pose for a shooting composition corresponding to the target image.

[0069] The memory (140) may store 3D model information about an object. For example, 3D model information about an object that is a subject of a photograph by a user may be stored in the memory (140). Alternatively, the processor (130) may obtain 3D model information about an object from a plurality of images captured during the movement of the electronic device (100) and store the obtained 3D model information in the memory (140).

[0070] The memory (140) may store information about the range of movement within which the electronic device (100) can move. For example, information about the range of movement within which the electronic device (100) can move may be stored in the memory (140) by a user. Alternatively, the processor (130) may obtain information about the range of movement within which the electronic device (100) can move during the movement process of the electronic device (100) and store information about the range of movement within which the electronic device (100) can move in the memory (140).

[0071] The processor (130) can control the overall operation of the electronic device (100) by executing modules or instructions stored in the memory (140). Specifically, the processor (130) can read and interpret modules or instructions and determine a sequence for data processing, and accordingly transmit control signals that control the operation of other components, such as the memory (140), thereby controlling the operation of other components.

[0072] The processor (130) can input the acquired image into the first neural network model by executing the target image acquisition module (130-1) to acquire a target image having a composition corresponding to the object included in the acquired image. Here, the target image may be an image predicted to be captured with an optimal shooting composition based on the object included in the acquired image.

[0073] The processor (130) can input the acquired image and target image into a second neural network model by executing the placement information acquisition module (130-2) to acquire placement information of the electronic device (100) corresponding to the target image. For example, the placement information may include information on at least one of the position of the electronic device (100) or the shooting direction of the camera (110). For example, the placement information may include information on the position of the electronic device (100) capable of taking pictures like the target image and the shooting direction of the camera (110). As another example, the placement information may include information on a change in the position of the electronic device (100) capable of taking pictures like the target image and a change in the shooting direction of the camera (110) by comparing the captured image and the target image.

[0074] The processor (130) may acquire a plurality of candidate images, each corresponding to a plurality of pieces of preliminary arrangement information, based on the arrangement information by executing the candidate image acquisition module (130-3). For example, the processor (130) may acquire a plurality of pieces of preliminary arrangement information by applying a plurality of preset values ​​to the position of the electronic device (100) and the shooting direction of the camera (110), and may acquire a plurality of candidate images by remapping the acquired images based on the plurality of pieces of preliminary arrangement information. For example, by applying a plurality of preset values ​​to the position (A) of the electronic device (100) and the shooting direction (B) of the camera (110), a plurality of preliminary arrangement information such as (A, B), (A, B+1), (A+1, B), (A+1, B+1), (A, B-1), (A-1, B), (A-1, B-1), (A+1, B-1), (A-1, B+1) may be obtained, and nine candidate images may be obtained by remapping the obtained images based on the plurality of preliminary arrangement information.

[0075] The processor (130) may also acquire multiple candidate images by remapping the acquired image based on 3D model information and multiple preliminary arrangement information about the object stored in the memory (140). In this case, more precise candidate images can be acquired.

[0076] The processor (130) can control the driving unit (120) to change the arrangement state of the electronic device (100) based on a candidate image corresponding to a target image among a plurality of candidate images by executing the driving unit control module (130-4). In the above-described example, the processor (130) can change the arrangement state of the electronic device (100) based on a candidate image that is most similar to the target image among nine candidate images. For example, when a candidate image based on (A+1, B+1) among the nine candidate images is most similar to the target image, the processor (130) can change the position of the electronic device (100) based on A+1 and change the shooting direction of the camera (110) based on B+1.

[0077] The processor (130) obtains a re-captured image re-captured through the camera (110) while moving the position of the electronic device (100), inputs the re-captured image into a first neural network model to obtain an updated target image with an optimal composition corresponding to an object included in the re-captured image, inputs the re-captured image and the updated target image into a second neural network model to obtain updated arrangement information of the electronic device (100) corresponding to the target image, and may stop the operation of the driving unit (120) when the arrangement state of the electronic device (100) corresponds to the updated arrangement information. That is, the processor (130) may repeat an operation of acquiring an acquired image, an operation of acquiring a target image, an operation of acquiring arrangement information, an operation of acquiring a plurality of candidate images, and an operation of changing the arrangement state based on the candidate images even while moving the position of the electronic device (100), and through these operations, a change in the optimal composition in real time is possible. Afterwards, if the arrangement status and the updated arrangement information correspond, the processor (130) may identify that the shooting is done in an optimal composition and stop the operation of the driving unit (120).

[0078] The processor (130) may acquire images through the camera (110) at preset time intervals after stopping the operation of the driving unit (120), and if an image in which the state of the object has changed is identified from the images acquired at preset time intervals, the identified image may be input into the first neural network model. That is, even while performing shooting with an optimal composition, the optimal composition may be changed due to changes in the shooting angle of the object, the distance between the electronic device (100) and the object, etc., due to movement of the object, etc. Therefore, the processor (130) may maintain the operation of acquiring a shooting image, the operation of acquiring a target image, the operation of acquiring an image of arrangement information, the operation of acquiring a plurality of candidate images, and the operation of comparing the arrangement state of the electronic device (100) with the arrangement information corresponding to the candidate images even after the operation of the driving unit (120) has stopped.

[0079] The processor (130) can control the driving unit (120) to change the arrangement state of the electronic device (100) within the movement range based on information about the movement range in which the electronic device (100) can move stored in the memory (140).

[0080] Here, the electronic device (100) further includes a sensor, and the range of movement may include information about an area in which the electronic device (100) can move, detected through the sensor, while moving the electronic device (100) to an arbitrary location.

[0081] The electronic device (100) further includes a user interface, and the processor (130) may control the driving unit (120) to change the arrangement of the electronic device (100) based on a user command received through the user interface.

[0082] The functions related to artificial intelligence according to the present disclosure can be operated through a processor (130) and a memory (140).

[0083] The processor (130) may be composed of one or more processors. In this case, one or more processors may be a general-purpose processor such as a CPU, AP, DSP, etc., a graphics-only processor such as a GPU or VPU (Vision Processing Unit), or an artificial intelligence-only processor such as an NPU.

[0084] One or more processors are controlled to process input data according to predefined operating rules or artificial intelligence models stored in memory (140). Alternatively, if one or more processors are dedicated artificial intelligence processors, the dedicated artificial intelligence processors may be designed with a hardware structure specialized for processing a specific artificial intelligence model. The predefined operating rules or artificial intelligence models are characterized by being created through learning.

[0085] Here, "created through learning" means that a basic artificial intelligence model is learned using a learning algorithm using a plurality of learning data, thereby creating a predefined set of operating rules or an artificial intelligence model set to perform a desired characteristic (or a desired purpose). This learning may be performed on the device itself on which the artificial intelligence according to the present disclosure is performed, or may be performed through a separate server and / or system. Examples of learning algorithms include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0086] An artificial intelligence model may be composed of multiple neural network layers. Each of the multiple neural network layers has multiple weight values ​​and performs neural network operations by calculating the results of previous layers and the multiple weights. The multiple weights of the multiple neural network layers can be optimized based on the learning results of the artificial intelligence model. For example, the multiple weights may be updated during the learning process to reduce or minimize the loss or cost values ​​obtained by the artificial intelligence model.

[0087] Artificial neural networks may include deep neural networks (DNNs), such as, but not limited to, convolutional neural networks (CNNs), deep neural networks (DNNs), recurrent neural networks (RNNs), restricted boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), generative adversarial networks (GANs), or deep Q-networks.

[0088] FIG. 3 is a block diagram showing a detailed configuration of an electronic device (100) according to one embodiment of the present disclosure.

[0089] The electronic device (100) may include a camera (110), a driving unit (120), a processor (130), and a memory (140). In addition, according to FIG. 3, the electronic device (100) may further include a sensor (150), a user interface (160), a microphone (170), a communication interface (180), a display (190), and a speaker (195). Among the components illustrated in FIG. 3, a detailed description of parts that overlap with the components illustrated in FIGS. 1 and 2 will be omitted.

[0090] The sensor (150) may include a cliff sensor, a time-of-flight (ToF) sensor, etc. The cliff sensor may be a sensor that outputs light toward the floor and receives light reflected from the floor. The ToF sensor may be a three-dimensional sensor that calculates the distance that light emitted from an object via an infrared wavelength bounces back and returns, and may be a sensor that recognizes the three-dimensionality, spatial information, and movement of an object. The processor (130) may measure the distance from the floor based on the information received from the sensor (150). In addition, the processor (130) may identify a cliff when reflected light is not received, and may identify the boundary of the movable area of ​​the electronic device (100).

[0091] The user interface (160) may be implemented with buttons, a touch pad, a mouse, a keyboard, etc., or may be implemented with a touch screen capable of performing both display and operation input functions. Here, the buttons may be various types of buttons, such as mechanical buttons, touch pads, wheels, etc., formed on any area of ​​the front, side, or back of the main body of the electronic device (100).

[0092] The microphone (170) is configured to receive sound and convert it into an audio signal. The microphone (170) is electrically connected to the processor (130) and can receive sound under the control of the processor (130).

[0093] For example, the microphone (170) may be formed as an integrated unit integrated into the upper side, front side, side side, etc. of the electronic device (100). Alternatively, the microphone (170) may be provided in a remote control, etc., separate from the electronic device (100). In this case, the remote control may receive sound through the microphone (170) and provide the received sound to the electronic device (100).

[0094] The microphone (170) may include various configurations such as a microphone that collects sound in analog form, an amplifier circuit that amplifies the collected sound, an A / D conversion circuit that samples the amplified sound and converts it into a digital signal, and a filter circuit that removes noise components from the converted digital signal.

[0095] The microphone (170) may be implemented in the form of a sound sensor, and any configuration capable of collecting sound may be used.

[0096] The communication interface (180) is a component that performs communication with various types of external devices according to various types of communication methods. For example, the electronic device (100) can perform communication with a content server or a user terminal device through the communication interface (180).

[0097] The communication interface (180) may include a Wi-Fi module, a Bluetooth module, an infrared communication module, a wireless communication module, etc. Here, each communication module may be implemented in the form of at least one hardware chip.

[0098] Wi-Fi and Bluetooth modules communicate via Wi-Fi and Bluetooth, respectively. When using a Wi-Fi or Bluetooth module, connection information, such as the SSID and session key, is first transmitted and received. This information is then used to establish a communication connection before various other information can be transmitted and received. Infrared communication modules use infrared data association (IrDA) technology, which wirelessly transmits data over short distances using infrared light, which lies between visible light and millimeter waves.

[0099] In addition to the above-described communication method, the wireless communication module may include at least one communication chip that performs communication according to various wireless communication standards such as zigbee, 3G (3rd Generation), 3GPP (3rd Generation Partnership Project), LTE (Long Term Evolution), LTE-A (LTE Advanced), 4G (4th Generation), 5G (5th Generation), etc.

[0100] Alternatively, the communication interface (180) may include a wired communication interface such as HDMI, DP, Thunderbolt, USB, RGB, D-SUB, DVI, etc.

[0101] In addition, the communication interface (180) may include at least one of a LAN (Local Area Network) module, an Ethernet module, or a wired communication module that performs communication using a pair cable, a coaxial cable, or an optical fiber cable.

[0102] The display (190) is a configuration that displays an image and can be implemented as a display of various forms such as an LCD (Liquid Crystal Display), an OLED (Organic Light Emitting Diodes) display, a PDP (Plasma Display Panel), etc. The display (190) may also include a driving circuit, a backlight unit, etc. that can be implemented as a form such as an a-si TFT, an LTPS (low temperature poly silicon) TFT, an OTFT (organic TFT), etc. The display (190) may be implemented as a touch screen combined with a touch sensor, a flexible display, a 3D display, etc.

[0103] The speaker (195) is a component that outputs various audio data processed by the processor (130) as well as various notification sounds and voice messages.

[0104] As described above, the electronic device (100) can automatically capture images with an optimal composition according to the object, and user convenience can be improved.

[0105] Hereinafter, the operation of the electronic device (100) will be described in more detail with reference to FIGS. 4 to 10. For convenience of explanation, individual embodiments are described in FIGS. 4 to 10. However, the individual embodiments of FIGS. 4 to 10 may be implemented in any combination.

[0106] FIG. 4 is a drawing for explaining a method of performing shooting while changing the composition according to one embodiment of the present disclosure.

[0107] The processor (130) can acquire an image by taking a picture with an angle of view of 410 from the left side of the object, as shown on the left side of FIG. 4.

[0108] The processor (130) can input the acquired image into the first neural network model to acquire a target image with a composition corresponding to the object included in the acquired image. For example, the processor (130) can acquire a target image when photographing from the right to the left of the object, as shown in the upper center of FIG. 4. Here, the target image may be an image predicted to be captured with an optimal shooting composition based on the object included in the acquired image.

[0109] The processor (130) inputs the acquired image and the target image into a second neural network model to obtain arrangement information of the electronic device (100) corresponding to the target image, and as shown in the lower center of FIG. 4, a plurality of candidate images (420-1, 420-2, 420-3, 420-4) corresponding to each of the plurality of preliminary arrangement information can be obtained based on the arrangement information.

[0110] The processor (130) can control the camera (110) and the driving unit (120) to change the shooting method of the camera (110) and change the arrangement state of the electronic device (100) based on the candidate image that is most similar to the target image among the plurality of candidate images (420-1, 420-2, 420-3, 420-4). For example, the processor (130) can control the camera (110) and the driving unit (120) to change the shooting method of the camera (110) and change the arrangement state of the electronic device (100) based on the candidate image (420-4) that is most similar to the target image among the plurality of candidate images (420-1, 420-2, 420-3, 420-4).

[0111] Through the above operation, the processor (130) can control the camera (110) and the driving unit (120) to change the shooting method of the camera (110) and change the arrangement state of the electronic device (100), as shown in 430 of FIG. 4.

[0112] Thereafter, the processor (130) may obtain an updated image, an updated target image, updated placement information, and a plurality of updated candidate images, as shown in 440 of FIG. 4, and adjust the placement state based on the plurality of updated candidate images, and may perform an operation as shown in 430 of FIG. 4 accordingly.

[0113] The processor (130) can repeat the operations 440 and 430 at preset intervals, thereby enabling shooting with an optimal composition in real time.

[0114] Meanwhile, although FIG. 4 illustrates a single object, the processor (130) may also operate by considering multiple objects. For example, the processor (130) may control the camera (110) and the driving unit (120) to change the shooting method of the camera (110) and the arrangement state of the electronic device (100) by considering not only the elephant object of FIG. 4 but also the user's hand.

[0115] FIG. 5 is a drawing for explaining the movement range of an electronic device (100) according to one embodiment of the present disclosure.

[0116] As illustrated in FIG. 5, the processor (130) acquires information about the movable area of ​​the electronic device (100) detected through the sensor (150) while moving the electronic device (100) to an arbitrary location, and stores the information about the final movable area as the movement range in the memory (140).

[0117] Thereafter, the processor (130) can only move within the movement range during the process of performing the operation as in FIG. 4. In the example of FIG. 4, when the processor (130) identifies that the movement according to the candidate image (420-4) most similar to the target image among the plurality of candidate images (420-1, 420-2, 420-3, 420-4) is out of the movement range, the processor (130) may control the camera (110) and the driving unit (120) to change the shooting method of the camera (110) and change the arrangement state of the electronic device (100) based on the candidate image that is second most similar to the target image among the plurality of candidate images (420-1, 420-2, 420-3, 420-4).

[0118] However, the present invention is not limited thereto, and the memory (140) may not store information on the range of movement. In this case, the processor (130), while performing an operation as shown in FIG. 4, may identify whether movement is possible through the sensor (150), and if movement is identified as impossible, may operate in a manner of changing the candidate image described above.

[0119] FIG. 6 is a drawing for explaining 3D model information according to one embodiment of the present disclosure.

[0120] For more sophisticated operations, the user may also obtain 3D model information about the object using the electronic device (100). For example, when a user command to obtain 3D model information about the object is received, the processor (130) may move around the object (610) and capture multiple images (620-1, 620-2, 620-3), as illustrated in FIG. 6, and obtain 3D model information about the object based on the multiple images (620-1, 620-2, 620-3). However, the present invention is not limited thereto, and the user may also directly provide the 3D model information to the electronic device (100).

[0121] The processor (130) can identify an object included in the acquired image and determine whether the identified object corresponds to 3D model information stored in the memory (140). If the identified object corresponds to the 3D model information, the processor (130) can remap the acquired image based on the 3D model information and a plurality of pieces of preliminary arrangement information to acquire a plurality of candidate images.

[0122] These movements allow for more precise shooting compositions.

[0123] FIG. 7 and FIG. 8 are flowcharts for explaining a shooting method according to one embodiment of the present disclosure.

[0124] First, the processor (130) can obtain an image (S710).

[0125] The processor (130) can acquire a target image based on the acquired image (S720). For example, as illustrated in FIG. 8, the processor (130) can input the acquired image (current scene, 810) into a first neural network model (DRL, 820-1) to acquire a target image (composition data, 830). Thereafter, the processor (130) can input the acquired image and the target image into a second neural network model (RNN, 820-2) to acquire arrangement information (predicted area, 840) of the electronic device (100) corresponding to the target image.

[0126] The processor (130) can obtain a plurality of candidate images each corresponding to a plurality of pieces of preliminary batch information based on the batch information (S730).

[0127] The processor (130) can obtain the target placement state of the electronic device (100) based on the candidate image most similar to the target image among a plurality of candidate images (S740), and control the driving unit (120) to change the placement state of the electronic device (100) based on the target placement state (S750).

[0128] FIGS. 9 and 10 are drawings for explaining examples of use according to various embodiments of the present disclosure.

[0129] The electronic device (100) may further include a projection unit, and the processor (130) may control the projection unit to project an image as shown in the upper part of FIG. 9. In this case, the processor (130) may identify a user as an object in the same manner as in FIG. 5, and, when the user's position moves, control the driving unit (120) to change the projection area as shown in the lower part of FIG. 9.

[0130] Alternatively, the electronic device (100) may be implemented as a serving robot. For example, when users are eating, as shown in the upper part of FIG. 10, the processor (130) may identify a first user among the users as an object in the same manner as in FIG. 5 and perform serving based on the first user. Thereafter, as shown in the lower part of FIG. 10, the processor (130) may identify a second user among the users as an object in the same manner as in FIG. 5 and perform serving based on the second user.

[0131] FIG. 11 is a flowchart for explaining a method for controlling an electronic device according to an embodiment of the present disclosure.

[0132] First, an image captured by a camera included in an electronic device is acquired (S1110). Then, based on the object included in the acquired image, the camera's shooting method is changed and the arrangement of the electronic device is changed (S1120).

[0133] Additionally, the changing step (S1120) may perform at least one of zooming in or zooming out of the camera, moving the position of the electronic device, or changing the shooting direction of the camera based on an object included in the acquired image.

[0134] And, the method may further include a step of acquiring a re-captured image re-captured through a camera while moving the position of the electronic device, and a step of adjusting a shooting method of the camera and adjusting a placement state of the electronic device based on an object included in the re-captured image.

[0135] In addition, the changing step (S1120) may include a step of inputting the acquired image into a first neural network model to obtain a target image having a composition corresponding to an object included in the acquired image, a step of inputting the acquired image and the target image into a second neural network model to obtain arrangement information of an electronic device corresponding to the target image, a step of obtaining a plurality of candidate images each corresponding to a plurality of pieces of preliminary arrangement information based on the arrangement information, and a step of changing the arrangement state of the electronic device based on a candidate image corresponding to the target image among the plurality of candidate images.

[0136] In addition, the arrangement information may include information about the position of the electronic device and the shooting direction of the camera, and the step of obtaining a plurality of candidate images may include obtaining a plurality of preliminary arrangement information by applying a plurality of preset values ​​to the position of the electronic device and the shooting direction of the camera, respectively, and remapping the obtained images based on the plurality of preliminary arrangement information to obtain a plurality of candidate images.

[0137] Additionally, the step of acquiring multiple candidate images can acquire multiple candidate images by remapping the acquired images based on 3D model information for the object and multiple preliminary arrangement information.

[0138] And, the method may further include a step of obtaining a re-captured image re-captured through a camera while moving the location of the electronic device, a step of inputting the re-captured image into a first neural network model to obtain an updated target image having a composition corresponding to an object included in the re-captured image, a step of inputting the re-captured image and the updated target image into a second neural network model to obtain updated arrangement information of the electronic device corresponding to the updated target image, and a step of stopping the movement of the location of the electronic device when the arrangement status of the commercial electronic device and the updated arrangement information correspond.

[0139] In addition, after stopping the movement of the electronic device, the method may further include a step of acquiring images through a camera at preset time intervals, and when an image in which the state of an object has changed is identified from the images acquired at preset time intervals, a step of inputting the identified image into a first neural network model.

[0140] And, the changing step (S1120) can change the arrangement state of the electronic device within the movable range of the electronic device.

[0141] Additionally, the range of movement may include information about the area within which the electronic device can move as detected by sensors included in the electronic device while moving the electronic device to any location.

[0142] And, it may further include a step of receiving a user command and a step of changing the arrangement of the electronic device based on the user command.

[0143] According to various embodiments of the present disclosure as described above, an electronic device can automatically capture images with an optimal composition according to an object, and user convenience can be improved.

[0144] According to an exemplary embodiment of the present disclosure, the various embodiments described above may be implemented as software including instructions stored in a machine-readable storage medium that can be read by a machine (e.g., a computer). The device may include an electronic device (e.g., electronic device (A)) according to the disclosed embodiments, which is a device that can call instructions stored in the storage medium and operate according to the called instructions. When an instruction is executed by a processor, the processor may directly or under the control of the processor perform a function corresponding to the instruction using other components. The instruction may include code generated or executed by a compiler or interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' means that the storage medium does not contain a signal and is tangible, but does not distinguish between data being stored semi-permanently or temporarily in the storage medium.

[0145] Furthermore, according to one embodiment of the present disclosure, the method according to the various embodiments described above may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or online through an application store (e.g., Play Store™). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0146] Furthermore, according to one embodiment of the present disclosure, the various embodiments described above may be implemented in a computer-readable recording medium or a similar device using software, hardware, or a combination thereof. In some cases, the embodiments described herein may be implemented by the processor itself. In a software implementation, embodiments such as the procedures and functions described herein may be implemented as separate software. Each software may perform one or more functions and operations described herein.

[0147] Computer instructions for performing processing operations of a device according to the various embodiments described above may be stored in a non-transitory computer-readable medium. The computer instructions stored in such a non-transitory computer-readable medium, when executed by a processor of a specific device, cause the specific device to perform processing operations of the device according to the various embodiments described above. A non-transitory computer-readable medium refers to a medium that permanently stores data and can be read by a device, rather than a medium that stores data for a short period of time, such as a register, cache, or memory. Specific examples of non-transitory computer-readable media may include a CD, DVD, hard disk, Blu-ray disk, USB, memory card, or ROM.

[0148] In addition, each of the components (e.g., modules or programs) according to the various embodiments described above may be composed of a single or multiple entities, and some of the corresponding sub-components described above may be omitted, or other sub-components may be further included in various embodiments. Alternatively or additionally, some components (e.g., modules or programs) may be integrated into a single entity, which may perform the same or similar functions as those performed by each of the corresponding components prior to integration. Operations performed by modules, programs or other components according to various embodiments may be executed sequentially, in parallel, iteratively or heuristically, or at least some operations may be executed in a different order, omitted, or other operations may be added.

[0149] Although exemplary embodiments have been illustrated and described, the present disclosure is not limited to the specific embodiments described above, and various modifications may be made by those skilled in the art to which the present disclosure pertains without departing from the spirit or scope of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical idea or prospect of the present disclosure.

Claims

1. In electronic devices, camera; drive unit; At least one memory storing one or more instructions; and comprising one or more processors operably connected to the camera, the driving unit and the at least one memory; The one or more processors execute the one or more instructions, Obtaining an image captured through the above camera, An electronic device that controls the driving unit to change the shooting method of the camera and change the arrangement state of the electronic device based on an object included in the image.

2. In paragraph 1, The one or more processors execute the one or more instructions, An electronic device that controls the camera to perform zoom in or zoom out based on an object included in the image, and controls the driving unit to perform at least one of changing the position of the electronic device or changing the direction of the camera.

3. In paragraph 1, The one or more processors execute the one or more instructions, Acquire re-captured images through the camera while changing the position of the electronic device; An electronic device that controls the driving unit to adjust the shooting method of the camera and adjust the arrangement state of the electronic device based on the object included in the recaptured image.

4. In paragraph 1, At least one of the above memories, Save the first neural network model and the second neural network model, The one or more processors execute the one or more instructions, Inputting the above image into the first neural network model to obtain a target image corresponding to the object included in the image, By inputting the above image and the target image into the second neural network model, arrangement information of the electronic device corresponding to the target image is obtained, Based on the above placement information, a plurality of candidate images each corresponding to a plurality of preliminary placement information are obtained, An electronic device that controls the driving unit to change the arrangement state of the electronic device based on a candidate image corresponding to the target image among the plurality of candidate images.

5. In paragraph 4, The above placement information is, Contains information about the location of the electronic device and the direction of the camera; The one or more processors execute the one or more instructions, By applying a plurality of preset values ​​to each of the position of the electronic device and the direction of the camera, the plurality of preliminary arrangement information are obtained, An electronic device that obtains the plurality of candidate images by remapping the image based on the plurality of preliminary arrangement information.

6. In paragraph 5, At least one of the above memories, Contains 3D model information for the above object, The one or more processors execute the one or more instructions, An electronic device that obtains the plurality of candidate images by remapping the image based on the 3D model information and the plurality of preliminary arrangement information.

7. In paragraph 4, The one or more processors execute the one or more instructions, Acquire re-captured images through the camera while changing the position of the electronic device; By inputting the above re-captured image into the first neural network model, an updated target image corresponding to an object included in the above re-captured image is obtained, By inputting the above re-captured image and the above updated target image into the second neural network model, updated arrangement information of the electronic device corresponding to the above updated target image is obtained, An electronic device that stops the operation of the driving unit when the arrangement status of the commercial electronic device and the updated arrangement information correspond.

8. In paragraph 7, The one or more processors execute the one or more instructions, After stopping the operation of the above driving unit, multiple images are acquired through the camera at preset time intervals, An electronic device that, when an image in which an object's state has changed is identified from the plurality of images, inputs the identified image into the first neural network model.

9. In paragraph 1, At least one of the above memories, Stores information about the movement range of the electronic device; The one or more processors execute the one or more instructions, An electronic device that controls the driving unit to change the arrangement state of the electronic device within the above movement range.

10. In paragraph 9, sensors; including more; The above range of movement is, Contains information about the area in which the electronic device can move, Information about the area in which the above electronic device can move, An electronic device based on the output of said sensor while said electronic device is moving to a location.

11. In paragraph 1, further including a user interface; The one or more processors execute the one or more instructions, An electronic device that controls the driving unit to change the arrangement state of the electronic device based on a user command received through the user interface.

12. In a method for controlling an electronic device, A step of acquiring an image captured by a camera included in the electronic device; and A control method, comprising: a step of changing a shooting method of the camera and changing a placement state of the electronic device based on an object included in the image.

13. In paragraph 12, The above changing steps are: A control method for controlling the camera to perform zoom in or zoom out based on an object included in the image, and performing at least one of changing the position of the electronic device or changing the direction of the camera.

14. In paragraph 12, A step of acquiring a re-captured image through the camera while changing the position of the electronic device; and A control method further comprising: a step of adjusting a shooting method of the camera and adjusting a placement state of the electronic device based on an object included in the re-captured image.

15. In paragraph 12, The above changing steps are: A step of inputting the above image into a first neural network model to obtain a target image corresponding to an object included in the image; A step of inputting the above image and the target image into a second neural network model to obtain arrangement information of the electronic device corresponding to the target image; A step of obtaining a plurality of candidate images each corresponding to a plurality of preliminary batch information based on the above batch information; and A control method, comprising: a step of changing the arrangement state of the electronic device based on a candidate image corresponding to the target image among the plurality of candidate images.

Images