Electronic device for providing image in which motion of object is corrected, and control method therefor

Abstract

An electronic device is disclosed. The electronic device may comprise: a memory for storing a first neural network model trained to receive a two-dimensional video and output a three-dimensional video, a second neural network model trained to change the motion of an object included in the three-dimensional video, and instructions; and one or more processors including processing circuitry, wherein, when executed individually or collectively by the one or more processors, the instructions can: input a first image and a second image into the first neural network model so as to acquire each of a first three-dimensional video and a second three-dimensional video when a first user command requesting comparison of the first image captured by a user and the second image, which is a reference image, is received; acquire, on the basis of a first object included in the first three-dimensional image and a second object included in the second three-dimensional image, a correction guide for correcting the motion of the first object; and input the first three-dimensional image and the correction guide into the second neural network model so as to acquire a corrected image in which the motion of the first object is corrected in the first three-dimensional image.

Description

Electronic device for providing an image with corrected motion of an object and a method for controlling the same

[0001] The present disclosure relates to an electronic device and a method for controlling the same, and more specifically, to an electronic device and a method for controlling the same that provides an image in which the motion of an object is corrected.

[0002] Thanks to advancements in electronic technology, various types of electronic devices are being developed. In particular, most electronic devices recently come equipped with cameras, enabling users to acquire images or videos using these cameras.

[0003] The user can film their daily life through a camera. For example, as shown in FIG. 1, the user can film their exercise scene and analyze their movements by comparing the filmed video with a professional's video.

[0004] However, there is a problem in that accurate comparative analysis is difficult due to differences in composition and other factors between the filmed video and the expert's video.

[0005] According to one embodiment of the present disclosure for achieving the above objectives, an electronic device comprises one or more processors including a first neural network model trained to receive a two-dimensional video and output a three-dimensional video, a second neural network model trained to change the motion of an object included in the three-dimensional video, a memory for storing instructions, and processing circuitry. When the instructions are executed individually or collectively by the one or more processors, when a first user command is received requesting a comparison between a captured first video and a reference second video, the first video and the second video are each input into the first neural network model to obtain a first three-dimensional video and a second three-dimensional video, a correction guide for correcting the motion of the first object is obtained based on a first object included in the first three-dimensional video and a second object included in the second three-dimensional video, and the first three-dimensional video and the correction guide are input into the second neural network model. A corrected image in which the motion of the first object is corrected can be obtained in the first three-dimensional image.

[0006] Additionally, when the above instructions are executed individually or collectively by the one or more processors, the correction guide can be obtained based on at least one of a point in time when correction is required during the motion of the first object, a correction point, or a difference from the motion of the second object based on the first object and the second object.

[0007] And, the memory further stores a third neural network model trained to output descriptive information corresponding to an image, and when the instructions are executed individually or collectively by the one or more processors, a frame corresponding to the point in time when correction is needed in the first three-dimensional image is input into the third neural network model to obtain descriptive information corresponding to the frame, and the correction guide can be updated based on the descriptive information.

[0008] In addition, when the above instructions are executed individually or collectively by the one or more processors, the above frame and a preset number of additional frames before and after the above frame can be input into the third neural network model to obtain the above explanatory information.

[0009] In addition, when the above instructions are executed individually or collectively by one or more processors, the correction guide can be updated by deleting parts that overlap with the description information in the correction guide.

[0010] Additionally, when the above instructions are executed individually or collectively by the one or more processors, if a second user command specifying a region of interest in the first image is received, the first object and the second object are compared based on the region of interest, and the correction guide can be obtained based on the comparison result.

[0011] And, when the above instructions are executed individually or collectively by the one or more processors, a region of interest in the first image is identified based on the motion of the first object, the first object and the second object are compared based on the region of interest, and the correction guide is obtained based on the comparison result.

[0012] Additionally, when the above instructions are executed individually or collectively by the one or more processors, at least one of viewpoint matching, object normalization, or sync matching is performed on the first three-dimensional image and the second three-dimensional image based on the first object and the second object, and then the correction guide can be obtained.

[0013] And, further including a display, the instructions, when executed individually or collectively by one or more processors, can convert the correction image into a two-dimensional correction image and display the two-dimensional correction image through the display.

[0014] In addition, when the above instructions are executed individually or collectively by the one or more processors, a UI may be provided indicating that correction has been performed in the two-dimensional correction image at a point in time and screen area corresponding to the correction guide.

[0015] And, when the above instructions are executed individually or collectively by the one or more processors, at least one of changing the viewpoint, zooming in on a part of the area, or changing the playback speed in the correction image is performed to obtain a two-dimensional additional correction image, and the two-dimensional additional correction image can be displayed through the display.

[0016] In addition, when the above instructions are executed individually or collectively by one or more processors, the second image and the two-dimensional correction image can be displayed through the display in a time series.

[0017] And, when the above instructions are executed individually or collectively by the one or more processors, they can provide a UI including at least one of a correction point or a difference from the motion of the second object based on the first object and the second object.

[0018] Meanwhile, according to one embodiment of the present disclosure, a control method for an electronic device comprises, when a first user command is received requesting a comparison between a first image captured by a user and a second image which is a reference image, inputting the first image and the second image into a first neural network model to obtain a first three-dimensional image and a second three-dimensional image, respectively; obtaining a correction guide for correcting the motion of the first object based on a first object included in the first three-dimensional image and a second object included in the second three-dimensional image; and inputting the first three-dimensional image and the correction guide into a second neural network model to obtain a corrected image in which the motion of the first object in the first three-dimensional image is corrected, wherein the first neural network model is a neural network model trained to receive a two-dimensional image and output a three-dimensional image, and the second neural network model may be a neural network model trained to change the motion of an object included in the three-dimensional image.

[0019] Additionally, the step of obtaining the correction guide may be based on at least one of a point in time requiring correction during the motion of the first object, a correction point, or a difference from the motion of the second object, based on the first object and the second object.

[0020] And, the step of obtaining the correction guide may be to input a frame corresponding to the point in time when correction is required in the first three-dimensional image into a third neural network model to obtain explanatory information corresponding to the frame, and to update the correction guide based on the explanatory information, and the third neural network model may be a neural network model trained to output explanatory information corresponding to the image.

[0021] In addition, the step of obtaining the correction guide can obtain the explanatory information by inputting the frame and a preset number of additional frames before and after the frame into the third neural network model.

[0022] In addition, the step of obtaining the correction guide may update the correction guide by deleting parts that overlap with the explanatory information in the correction guide.

[0023] Additionally, the step of obtaining the correction guide may involve, when a second user command specifying a region of interest in the first image is received, comparing the first object and the second object based on the region of interest, and obtaining the correction guide based on the comparison result.

[0024] In addition, the step of obtaining the correction guide may involve identifying a region of interest in the first image based on the motion of the first object, comparing the first object and the second object based on the region of interest, and obtaining the correction guide based on the comparison result.

[0025] Additionally, the step of obtaining the correction guide may involve performing at least one of viewpoint matching, object normalization, or sync matching on the first three-dimensional image and the second three-dimensional image based on the first object and the second object, and then obtaining the correction guide.

[0026] In addition, the method may further include the step of converting the correction image into a two-dimensional correction image and the step of displaying the two-dimensional correction image.

[0027] In addition, the displaying step may provide a UI indicating that correction has been performed in a viewpoint and screen area corresponding to the correction guide in the two-dimensional correction image.

[0028] And, the method may further include the step of obtaining a two-dimensional additional correction image by performing at least one of changing the viewpoint, enlarging a part of the area, or changing the playback speed in the correction image, and the step of displaying the two-dimensional additional correction image.

[0029] FIG. 1 is a drawing for explaining a captured image to aid in understanding the present disclosure.

[0030] FIG. 2 is a block diagram showing the configuration of an electronic device according to one embodiment of the present disclosure.

[0031] FIG. 3 is a block diagram showing the detailed configuration of an electronic device according to one embodiment of the present disclosure.

[0032] FIG. 4 is a flowchart illustrating a method for obtaining a correction image according to one embodiment of the present disclosure.

[0033] FIG. 5 is a drawing for explaining a first user command according to one embodiment of the present disclosure.

[0034] FIG. 6 is a flowchart illustrating a method for processing natural language input according to one embodiment of the present disclosure.

[0035] FIGS. 7 to 11 are drawings for explaining a comparison between images according to an embodiment of the present disclosure.

[0036] FIGS. 12 to 17 are drawings for explaining a method of obtaining a calibration guide according to one embodiment of the present disclosure.

[0037] FIGS. 18 to 21 are drawings for explaining a method for generating a correction image according to one embodiment of the present disclosure.

[0038] FIGS. 22 and FIGS. 23 are drawings for illustrating a correction image and an additional correction image according to an embodiment of the present disclosure.

[0039] FIG. 24 is a drawing for explaining a method of using a wearable device according to one embodiment of the present disclosure.

[0040] FIG. 25 is a drawing for explaining various fields to which an embodiment of the present disclosure is applied.

[0041] FIG. 26 is a flowchart illustrating a method for controlling an electronic device according to one embodiment of the present disclosure.

[0042] The object of the present disclosure is to provide an electronic device that provides a video in which the motion of an object is corrected, and a method for controlling the same.

[0043] The present disclosure will be described in detail below with reference to the attached drawings.

[0044] The terms used in the embodiments of this disclosure have been selected to be as widely used as possible, taking into account their functions within this disclosure; however, these terms may vary depending on the intent of those skilled in the art, case law, the emergence of new technologies, etc. Additionally, in specific cases, terms have been arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the relevant explanatory section of this disclosure. Therefore, terms used in this disclosure should be defined not merely by their names, but based on their meanings and the overall content of this disclosure.

[0045] In this specification, expressions such as “have,” “may have,” “include,” or “may include” indicate the presence of such features (e.g., numerical values, functions, operations, or components such as parts) and do not exclude the presence of additional features.

[0046] The expression "at least one of A or / and B" should be understood as representing either "A" or "B" or "A and B".

[0047] Expressions such as "first," "second," "first," or "second" used in this specification may modify various components regardless of order and / or importance, and are used only to distinguish one component from another and do not limit said components.

[0048] The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, terms such as "comprising" or "consisting of" are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0049] In this specification, the term "user" may refer to a person using an electronic device or a device using an electronic device (e.g., an artificial intelligence electronic device).

[0050] Various embodiments of the present disclosure will be described in more detail below with reference to the attached drawings.

[0051] FIG. 2 is a block diagram showing the configuration of an electronic device (100) according to one embodiment of the present disclosure.

[0052] The electronic device (100) is a device for comparing multiple images and can be implemented as a device such as a desktop PC, laptop, smartphone, tablet PC, server, TV, projector, smart glasses, smart watch, speaker, soundbar, etc. However, it is not limited thereto, and the electronic device (100) may be any device capable of comparing multiple images.

[0053] According to FIG. 2, the electronic device (100) includes memory (110) and a processor (120).

[0054] Memory (110) may refer to hardware that stores information, such as data, in an electrical or magnetic form so that a processor (120), etc., can access it. To this end, memory (110) may be implemented as at least one piece of hardware among non-volatile memory, volatile memory, flash memory, hard disk drive (HDD) or solid-state drive (SSD), RAM, ROM, etc.

[0055] At least one instruction required for the operation of an electronic device (100) or a processor (120) may be stored in the memory (110). Here, the instruction is a unit of code that directs the operation of the electronic device (100) or the processor (120), and may be written in machine language, which is a language that a computer can understand. Alternatively, a plurality of instructions that perform a specific task of the electronic device (100) or the processor (120) may be stored in the memory (110) as an instruction set.

[0056] Data that is information in bit or byte units capable of representing characters, numbers, images, etc. can be stored in the memory (110). For example, multiple images, neural network models, etc. can be stored in the memory (110).

[0057] Here, the neural network model may include at least one of a first neural network model trained to receive a two-dimensional video input and output a three-dimensional video, a second neural network model trained to change the motion of an object included in the three-dimensional video, or a third neural network model trained to output descriptive information corresponding to the image.

[0058] Additionally, the neural network model may further include at least one of a neural network model trained to identify a region of interest or a neural network model trained to generate a correction guide.

[0059] The memory (110) is accessed by the processor (120), and the processor (120) may perform read / write / modify / delete / update, etc. on instructions, instruction sets, or data.

[0060] The processor (120) controls the overall operation of the electronic device (100). Specifically, the processor (120) can control the overall operation of the electronic device (100) by being connected to each component of the electronic device (100). For example, the processor (120) can control the operation of the electronic device (100) by being connected to a component such as a memory (110).

[0061] One or more processors (120) may include one or more of a CPU, a GPU (Graphics Processing Unit), an APU (Accelerated Processing Unit), a MIC (Many Integrated Core), a NPU (Neural Processing Unit), a hardware accelerator, or a machine learning accelerator. One or more processors (120) may control one or any combination of other components of the electronic device (100) and may perform operations or data processing related to communication. One or more processors (120) may execute one or more programs or instructions stored in memory (110). For example, one or more processors (120) may perform a method according to one embodiment of the present disclosure by executing one or more instructions stored in memory (110).

[0062] When a method according to one embodiment of the present disclosure includes a plurality of operations, the plurality of operations may be performed by a single processor or by a plurality of processors. For example, when a first operation, a second operation, and a third operation are performed by a method according to one embodiment, the first operation, the second operation, and the third operation may all be performed by a first processor, or the first operation and the second operation may be performed by a first processor (e.g., a general-purpose processor) and the third operation may be performed by a second processor (e.g., an artificial intelligence dedicated processor).

[0063] One or more processors (120) may be implemented as a single-core processor including one core, or as one or more multicore processors including multiple cores (e.g., homogeneous multicore or heterogeneous multicore). When one or more processors (120) are implemented as multicore processors, each of the multiple cores included in the multicore processor may include internal processor memory such as cache memory or on-chip memory, and a common cache shared by multiple cores may be included in the multicore processor. Additionally, each of the multiple cores included in the multicore processor (or some of the multiple cores) may independently read and execute program instructions for implementing a method according to one embodiment of the present disclosure, or all (or some) of the multiple cores may be linked together to read and execute program instructions for implementing a method according to one embodiment of the present disclosure.

[0064] When a method according to one embodiment of the present disclosure includes a plurality of operations, the plurality of operations may be performed by one of the plurality of cores included in a multi-core processor, or may be performed by a plurality of cores. For example, when a first operation, a second operation, and a third operation are performed by a method according to one embodiment, the first operation, the second operation, and the third operation may all be performed by a first core included in a multi-core processor, or the first operation and the second operation may be performed by a first core included in a multi-core processor and the third operation may be performed by a second core included in a multi-core processor.

[0065] In the embodiments of the present disclosure, one or more processors (120) may refer to a system-on-chip (SoC) in which one or more processors and other electronic components are integrated, a single-core processor, a multi-core processor, or a core included in a single-core processor or a multi-core processor, wherein the core may be implemented as a CPU, GPU, APU, MIC, NPU, hardware accelerator, or machine learning accelerator, but the embodiments of the present disclosure are not limited thereto. However, for convenience of explanation, the operation of the electronic device (100) is described below using the expression "processor (120)."

[0066] The processor (120) may receive a first user command requesting a comparison between a first video captured by the user and a second video which is a reference video. For example, the processor (120) may select the first video and the second video and receive a first user command requesting a comparison between the first video and the second video. Alternatively, the processor (120) may display a menu for selecting a reference video when a content analysis command is received from the user while the user is displaying the first video captured by the user, and may identify that a first user command requesting a comparison between the first video and the second video has been received when the user selects the second video through the menu.

[0067] However, it is not limited thereto, and the processor (120) may receive the first user command in various ways. For example, the processor (120) may receive a first image and a second image from a user terminal device and receive a first user command requesting a comparison of the first image and the second image.

[0068] When the processor (120) receives a first user command, it can input the first image and the second image into the first neural network model, respectively, to obtain the first three-dimensional image and the second three-dimensional image. Here, the first neural network model may be a neural network model trained to receive a two-dimensional image as input and output a three-dimensional image. For example, the first neural network model may include a NeRF (neural radiance fields) model, which is a generative AI based on a multi-layer perception (MLP) architecture that uses images captured from various views, camera information, and lighting information to predict the color and density of three-dimensional points and reconstruct them into a three-dimensional image. Alternatively, the first neural network model may include a large reconstruction model (LRM) model based on a vision transformer (ViT) architecture that divides the image into patches and uses tokens of each patch to convert the image into a vector to generate a three-dimensional image. However, it is not limited thereto, and the first neural network model may be implemented in any form as long as it can generate a three-dimensional image from a two-dimensional image. Additionally, when the processor (120) receives the first user command, it may input the first image and the second image into the rule-based transformation model, respectively, to obtain the first three-dimensional image and the second three-dimensional image.

[0069] The processor (120) can obtain a correction guide for correcting the motion of the first object based on the first object included in the first three-dimensional image and the second object included in the second three-dimensional image.

[0070] For example, the processor (120) may obtain a correction guide based on at least one of a point in time when correction is required during the motion of the first object, a point of correction, or a difference from the motion of the second object based on the first object and the second object. However, it is not limited thereto, and the processor (120) may obtain a correction guide by inputting the first 3D image and the second 3D image into a neural network model trained to generate a correction guide.

[0071] In one embodiment, the processor (120) may obtain a correction guide after performing at least one of viewpoint matching, object normalization, or sync matching on the first 3D image and the second 3D image based on the first object and the second object before comparing the first object and the second object. For example, if the first object in the first 3D image is captured as the back view of a person, but the second object in the second 3D image is captured as the front view of a person, the processor (120) may change the viewpoint of the first 3D image to include the front view of the first object in the first 3D image. The processor (120) may obtain a correction guide by comparing the first object in the first 3D image with the changed viewpoint and the second object in the second 3D image. Additionally, the processor (120) may perform object normalization based on the size of the first object and the size of the second object, or perform sync matching based on the motion of the first object and the motion of the second object.

[0072] The processor (120) inputs a frame corresponding to the point in time when correction is needed in the first three-dimensional image into a third neural network model to obtain explanatory information corresponding to the frame, and can update the correction guide based on the explanatory information. For example, the processor (120) can update the correction guide by deleting parts that overlap with the explanatory information in the correction guide. Here, the third neural network model may be a neural network model trained to output explanatory information corresponding to an image.

[0073] Alternatively, the processor (120) may input a pre-set number of additional frames before and after the frame into the third neural network model to obtain explanatory information, and update the correction guide by deleting parts that overlap with the explanatory information in the correction guide. The processor (120) may obtain explanatory information reflecting the motion of an object by inputting not only the frame corresponding to the time when correction is needed into the third neural network model, but also a pre-set number of additional frames before and after the frame corresponding to the time when correction is needed.

[0074] The processor (120) can identify the number of frames to be input into the third neural network model based on the resources of the electronic device (100). For example, if the value is less than a preset value in the resources of the electronic device (100), the processor (120) inputs a frame corresponding to the time when correction is needed into the third neural network model to obtain explanatory information corresponding to the frame, and if the value is greater than or equal to the preset value in the resources of the electronic device (100), the processor (120) inputs a frame corresponding to the time when correction is needed and a preset number of additional frames before and after the frame into the third neural network model to obtain explanatory information corresponding to the frame.

[0075] However, it is not limited thereto, and the processor (120) may identify the number of frames to be input to the third neural network model based on the motion value of the frame corresponding to the time when correction is needed. For example, the processor (120) may obtain the motion value of the frame corresponding to the time when correction is needed based on the surrounding frames of the frame corresponding to the time when correction is needed, and if the motion value is less than a preset value, input the frame corresponding to the time when correction is needed into the third neural network model to obtain explanatory information corresponding to the frame, and if the motion value is greater than or equal to the preset value, input the frame corresponding to the time when correction is needed and a preset number of additional frames before and after the frame to the third neural network model to obtain explanatory information corresponding to the frame.

[0076] When the processor (120) receives a second user command specifying a region of interest in the first image, it may compare the first object and the second object based on the region of interest and obtain a correction guide based on the comparison result. For example, when the processor (120) receives a second user command specifying a region of interest in the first image, it may obtain only the comparison result within the region of interest among the comparison results of the first object and the second object, and obtain a correction guide based on the obtained comparison result.

[0077] Alternatively, the processor (120) may identify a region of interest in the first image based on the motion of the first object, compare the first object and the second object based on the region of interest, and obtain a correction guide based on the comparison result. For example, the processor (120) may divide the first object into multiple regions and identify the region with the largest motion among the motions of each of the multiple regions as the region of interest. However, it is not limited thereto, and the processor (120) may compare the first object and the second object and identify the region with the largest difference as the region of interest.

[0078] The processor (120) can input the first three-dimensional image and the correction guide into the second neural network model to obtain a corrected image in which the motion of the first object in the first three-dimensional image is corrected. Here, the second neural network model may be a neural network model trained to change the motion of an object included in the three-dimensional image.

[0079] The electronic device (100) further includes a display, and the processor (120) can convert the correction image into a two-dimensional correction image and display the two-dimensional correction image through the display.

[0080] The processor (120) may provide a UI indicating that correction has been performed in a viewpoint and screen area corresponding to the correction guide in a two-dimensional correction image. Alternatively, the processor (120) may obtain an additional two-dimensional correction image by performing at least one of changing the viewpoint, zooming in on a part of the area, or changing the playback speed in the correction image, and display the additional two-dimensional correction image through a display.

[0081] Alternatively, the processor (120) may display the second image and the two-dimensional correction image through the display (130) in a time series. Alternatively, the processor (120) may provide a UI that includes at least one of a correction point or a difference from the motion of the second object based on the first object and the second object. For example, the processor (120) may provide a UI for a point where the difference between the first object and the second object is greater than or equal to a threshold value. Alternatively, the processor (120) may provide a UI for the pose of the region of interest.

[0082] Meanwhile, the artificial intelligence-related functions according to the present disclosure can be operated through the processor (120) and memory (110).

[0083] The processor (120) may be composed of one or more processors. In this case, the one or more processors may be a general-purpose processor such as a CPU, AP, DSP, etc., a graphics-dedicated processor such as a GPU, VPU (Vision Processing Unit), or an artificial intelligence-dedicated processor such as an NPU.

[0084] One or more processors control the processing of input data according to predefined operation rules or artificial intelligence models stored in memory (110). Alternatively, if one or more processors are dedicated artificial intelligence processors, the dedicated artificial intelligence processors may be designed with a hardware structure specialized for processing a specific artificial intelligence model. The predefined operation rules or artificial intelligence models are characterized by being created through learning.

[0085] Here, "created through learning" means that a basic artificial intelligence model is trained using multiple learning data by a learning algorithm, thereby creating a predefined rule of operation or an artificial intelligence model configured to perform a desired characteristic (or objective). Such learning may be performed on the device itself where the artificial intelligence according to the present disclosure is executed, or it may be performed through a separate server and / or system. Examples of learning algorithms include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0086] An artificial intelligence model can be composed of multiple neural network layers. Each of the multiple neural network layers has multiple weight values ​​and performs neural network operations through calculations between the results of previous layers and the multiple weights. The multiple weights possessed by the multiple neural network layers can be optimized based on the learning results of the artificial intelligence model. For example, the multiple weights can be updated during the learning process so that the loss or cost values ​​obtained by the artificial intelligence model are reduced or minimized.

[0087] Artificial neural networks may include deep neural networks (DNNs), such as, but are not limited to, Convolutional Neural Networks (CNNs), Deep Neural Networks (DNNs), Recurrent Neural Networks (RNNs), Restricted Boltzmann Machines (RBMs), Deep Belief Networks (DBNs), Bidirectional Recurrent Deep Neural Networks (BRDNNs), Generative Adversarial Networks (GANs), or Deep Q-Networks.

[0088] FIG. 3 is a block diagram showing the detailed configuration of an electronic device (100) according to one embodiment of the present disclosure. The electronic device (100) may include a memory (110) and a processor (120). Additionally, according to FIG. 3, the electronic device (100) may further include a display (130), a communication interface (140), a user interface (150), a microphone (160), a speaker (170), and a camera (180). Detailed descriptions of parts of the components shown in FIG. 3 that overlap with the components shown in FIG. 2 are omitted.

[0089] The display (130) is configured to display an image and can be implemented as various types of displays such as an LCD (Liquid Crystal Display), an OLED (Organic Light Emitting Diodes) display, and a PDP (Plasma Display Panel). The display (130) may also include a driving circuit, a backlight unit, etc., which can be implemented in the form of an a-si TFT, an LTPS (low temperature poly silicon) TFT, or an OTFT (organic TFT). Meanwhile, the display (130) can be implemented as a touch screen combined with a touch sensor, a flexible display, a 3D display, etc.

[0090] The communication interface (140) is a configuration that performs communication with various types of external devices according to various types of communication methods. For example, an electronic device (100) can perform communication with a user terminal device or a server through the communication interface (140).

[0091] The communication interface (140) may include a Wi-Fi module, a Bluetooth module, an infrared communication module, and a wireless communication module. Here, each communication module may be implemented in the form of at least one hardware chip.

[0092] The Wi-Fi module and Bluetooth module perform communication using the Wi-Fi and Bluetooth methods, respectively. When using the Wi-Fi or Bluetooth module, various connection information, such as the SSID and session key, is transmitted and received first; after establishing a communication connection using this information, various data can be transmitted and received. The infrared communication module performs communication according to infrared communication (IrDA, Infrared Data Association) technology, which wirelessly transmits data over short distances using infrared rays that lie between visible light and millimeter waves.

[0093] In addition to the communication method described above, the wireless communication module may include at least one communication chip that performs communication according to various wireless communication standards such as Zigbee, 3G (3rd Generation), 3GPP (3rd Generation Partnership Project), LTE (Long Term Evolution), LTE-A (LTE Advanced), 4G (4th Generation), and 5G (5th Generation).

[0094] Alternatively, the communication interface (140) may include a wired communication interface such as HDMI, DP, Thunderbolt, USB, RGB, D-SUB, DVI, etc.

[0095] In addition, the communication interface (140) may include at least one of a LAN (Local Area Network) module, an Ethernet module, or a wired communication module that performs communication using a pair cable, a coaxial cable, or a fiber optic cable.

[0096] The user interface (150) may be implemented as a button, touchpad, mouse, and keyboard, or as a touch screen capable of performing display functions and operation input functions. Here, the button may be a various type of button, such as a mechanical button, touchpad, or wheel, formed in any area of ​​the exterior of the main body of the electronic device (100), such as the front, side, or back.

[0097] The microphone (160) is configured to receive sound input and convert it into an audio signal. The microphone (160) is electrically connected to the processor (120) and can receive sound under the control of the processor (120).

[0098] For example, the microphone (160) may be formed as an integrated unit on the upper side, front side, or side side of the electronic device (100). Alternatively, the microphone (160) may be provided in a remote control or the like, separate from the electronic device (100). In this case, the remote control may receive sound through the microphone (160) and provide the received sound to the electronic device (100).

[0099] The microphone (160) may include various configurations such as a microphone that collects analog sound, an amplifier circuit that amplifies the collected sound, an A / D conversion circuit that samples the amplified sound and converts it into a digital signal, and a filter circuit that removes noise components from the converted digital signal.

[0100] Meanwhile, the microphone (160) may be implemented in the form of a sound sensor, and any configuration capable of collecting sound is acceptable.

[0101] The speaker (170) is a component that outputs various audio data processed by the processor (120), as well as various notification sounds or voice messages.

[0102] The camera (180) is configured to capture still images or video. The camera (180) can capture a still image at a specific point in time, but can also capture a series of still images.

[0103] The camera (180) can capture the front of the electronic device (100) to capture the actual environment in front of the electronic device (100). The processor (120) may also identify a region of interest from the image captured through the camera (180).

[0104] The camera (180) includes a lens, a shutter, an aperture, a solid-state image sensor, an AFE (Analog Front End), and a TG (Timing Generator). The shutter controls the time when light reflected from a subject enters the camera (180), and the aperture controls the amount of light incident on the lens by mechanically increasing or decreasing the size of the opening through which light enters. When light reflected from a subject accumulates as photocharge, the solid-state image sensor outputs an image based on the photocharge as an electrical signal. The TG outputs a timing signal for reading out pixel data from the solid-state image sensor, and the AFE samples and digitizes the electrical signal output from the solid-state image sensor.

[0105] As described above, the electronic device (100) can provide information for correcting the motion of an object by comparing two images and providing a video in which the motion of the object is corrected.

[0106] The operation of the electronic device (100) will be described in more detail below through FIGS. 4 to 25. FIGS. 4 to 25 describe individual embodiments for convenience of explanation. However, the individual embodiments of FIGS. 4 to 25 may be implemented in any combination.

[0107] FIG. 4 is a flowchart illustrating a method for obtaining a correction image according to one embodiment of the present disclosure.

[0108] First, the processor (120) receives a request for analysis of user-selected content (S410) and can receive input information for content to be compared (S420). For example, when the processor (120) receives a content analysis command from the user while the user is displaying a first video that has been captured, it can display a menu for selecting a reference video, and if the user selects a second video through the menu, it can identify that a first user command requesting a comparison between the first video and the second video has been received. Here, the menu for selecting a reference video may include thumbnails of videos stored in the electronic device (100).

[0109] Alternatively, when the processor (120) receives a content analysis command from the user while the user is displaying the first video that was captured, it may display a recommended reference video based on the user's motion included in the first video. For example, when the processor (120) receives a content analysis command from the user while the user is displaying the first video that was captured, it may identify the user's motion included in the first video to identify that the user is practicing golf and display a recommended reference video of a golf player hitting golf. Here, the processor (120) may identify the recommended reference video through a server or an internet search and display the identified recommended reference video.

[0110] Alternatively, the processor (120) may receive a first user command through the user interface (150) requesting a comparison between a first image captured by the user and a second image which is a reference image.

[0111] However, it is not limited to this, and the processor (120) may receive a first user command requesting a comparison between a first video taken by another user and a second video which is a reference video.

[0112] The processor (120) can convert content according to user commands into three-dimensional content and compare the state values ​​of each body point (S430). For example, the processor (120) can convert the first image and the second image into a first three-dimensional image and a second three-dimensional image, respectively, and compare the state values ​​of each point of the first object included in the first three-dimensional image and the second object included in the second three-dimensional image.

[0113] The processor (120) derives a correction guide based on the comparison result (S440), generates a correction image by reconstructing user-selected content based on the correction guide (S450), and can display the correction image (S460).

[0114] For example, the processor (120) may obtain a correction guide for points of the first object that require correction based on the comparison result. Alternatively, the processor (120) may obtain a correction guide by further considering the region of interest, information about the corresponding frame, etc. Based on the correction guide, the processor (120) may obtain a correction image in which the motion of the first object in the first three-dimensional image has been changed, convert the correction image into a two-dimensional correction image, and display the two-dimensional correction image.

[0115] FIG. 5 is a drawing for explaining a first user command according to one embodiment of the present disclosure.

[0116] The processor (120) can display a first image and a content analysis icon (510) as shown at the top of FIG. 5. When the user touches the content analysis icon (510), the processor (120) can display a guidance message such as "Please select content to use for analysis" and a plurality of applications for searching for content to use for analysis, as shown at the middle and left of FIG. 5.

[0117] When one of the multiple applications is selected, the processor (120) can provide a list of contents registered to the selected application, as shown in the middle right of FIG. 5.

[0118] When the second image (520) is selected from the content list, the processor (120) can display a guidance message (530) such as “What analysis do you want?” as shown at the bottom of FIG. 5. When the processor (120) receives a natural language input (540) from the user saying “I want to improve my golf swing to be like the swing in the content,” it can identify a region of interest based on the natural language input (540) and compare the first image and the second image (520) based on the region of interest.

[0119] In FIG. 5, for convenience of explanation, the processor (120) is described as identifying a region of interest based on natural language input (540), but is not limited thereto. For example, when a second image (520) is selected from a content list, the processor (120) may identify a region of interest based on the motion of the first image and compare the first image and the second image (520) based on the region of interest. In this case, the processor (120) may not display a guidance message (530) such as the bottom of FIG. 5.

[0120] Alternatively, the processor (120) may compare the first image and the second image (520) without identifying the region of interest. In this case as well, the processor (120) may not display a guidance message (530) such as the bottom of FIG. 5.

[0121] FIG. 6 is a flowchart illustrating a method for processing natural language input according to one embodiment of the present disclosure.

[0122] The processor (120) can receive natural language input (S610). For example, the processor (120) can receive natural language input containing content such as the subject of analysis, goals, interests, etc. For instance, the processor (120) can receive natural language input such as "I want to improve my golf swing to be like the swing in the content."

[0123] The processor (120) can perform text preprocessing (S620). For example, the processor (120) can remove unnecessary symbols, control stop words, etc., for natural language input.

[0124] The processor (120) can perform natural language processing (NLP) on the preprocessed natural language input (S630). For example, the processor (120) can input the preprocessed natural language input into a natural language processing model to perform tokenization, sentence vectorization, etc.

[0125] The processor (120) can perform topic analysis and classification on the natural language input processed by natural language processing (S640). For example, the processor (120) can extract topics and classify the natural language input processed by natural language processing using a model such as LDA, BERT, etc.

[0126] The processor (120) can identify a region of interest (ROI) (S650). For example, the processor (120) can derive specific regions, topics, etc. based on keywords. For instance, the processor (120) can derive topics such as golf, swing, grip, and weight transfer, and identify a region of interest based on the topics.

[0127] FIGS. 7 to 11 are drawings for explaining a comparison between images according to an embodiment of the present disclosure.

[0128] First, as illustrated in FIG. 7, the processor (120) can convert the first image and the second image into a first three-dimensional image and a second three-dimensional image, respectively (S710). For example, as illustrated in FIG. 8, the processor (120) can input the first image at the top left and the second image at the bottom left into a first neural network model, respectively, to obtain a first three-dimensional image at the top right and a second three-dimensional image at the bottom right.

[0129] The processor (120) may perform at least one of viewpoint matching, object normalization, or sync matching on the first three-dimensional image and the second three-dimensional image based on the first object included in the first three-dimensional image and the second object included in the second three-dimensional image. For example, the processor (120) may identify that viewpoint matching is required by comparing the first object and the second object in the first three-dimensional image and the second three-dimensional image of FIG. 8. For instance, the processor (120) may identify that the first three-dimensional image was taken from the right side of the first object and that the second three-dimensional image was taken from the right rear side of the second object. In this case, the processor (120) may change the viewpoint of looking at the first object in the first three-dimensional image to the right rear side, or change the viewpoint of looking at the second object in the second three-dimensional image to the right side.

[0130] The processor (120) can compare the state values ​​of each body point in the first object and the second object (S720). For example, the processor (120) can compare at least one of the angle, position, or velocity of each body point in the first object and the second object. For instance, the processor (120) can compare the angle formed by the thigh and calf of each of the first object and the second object, as shown in FIG. 9. Alternatively, the processor (120) can compare the positions of the head, neck, and shoulder of each of the first object and the second object, as shown in FIG. 10. Alternatively, the processor (120) can compare the velocity of the club head of each of the first object and the second object, as shown in FIG. 11.

[0131] The processor (120) can identify whether the state value comparison result exceeds a threshold (S730), and can identify whether the body point exceeding the threshold is associated with the area of ​​interest (S740).

[0132] If a body point exceeding a threshold is associated with a region of interest, the processor (120) can extract time information and the difference in state value of the point (S750).

[0133] FIGS. 12 to 17 are drawings for explaining a method of obtaining a calibration guide according to one embodiment of the present disclosure.

[0134] First, the processor (120) can obtain a correction guide by inputting the difference between body points and state values ​​into a neural network model (S1210) as shown in FIG. 12 (S1220). For example, as shown in FIG. 13, the processor (120) can obtain time information of 02:47, club head as body points, coordinate information (+2, +22, +3), and angle (-27) as the difference between state values ​​by comparing the first 3D image and the second 3D image. The processor (120) can obtain a correction guide as shown in FIG. 14 by inputting the obtained information into a neural network model. For instance, the processor (120) can obtain a correction guide for five items, and each item may include a problem situation and a solution. In FIG. 14, only the problem situation and solution of item 2 are shown for convenience of explanation.

[0135] Here, the neural network model may be a neural network model trained to generate a correction guide. However, it is not limited to this, and the processor (120) may obtain a correction guide through a rule-based model without using a neural network model.

[0136] The processor (120) can generate extended text of the correction guide (S1230). For example, the processor (120) can generate extended text of the correction guide with added content such as “rotated less than 75 degrees”, “elbow raised”, “left shoulder positioned over right foot”, etc., to the correction guide of FIG. 14, as shown in FIG. 15.

[0137] The processor (120) can input the correction guide into a neural network model to generate expanded text of the correction guide. Here, the neural network model may be a neural network model trained to expand the content of the correction guide. However, it is not limited to this, and the processor (120) may obtain expanded text of the correction guide through a rule-based model without using a neural network model.

[0138] The processor (120) performs recaptioning on a frame-by-frame basis of the user content converted into three dimensions (S1240) and can compare the recaptioning result with the expanded text of the correction guide (S1250). For example, the processor (120) can input frames in which the difference in state values ​​in the first three-dimensional image is greater than or equal to a preset value and surrounding frames into a third neural network model to obtain descriptive information for each frame as shown in FIG. 16. As shown in FIG. 17, the processor (120) can compare the descriptive information (1710) of the frame in which the difference in state values ​​is greater than or equal to a preset value and the expanded text (1720) of the correction guide among the descriptive information for each frame.

[0139] The processor (120) can compare whether the expanded text is fully included in the recaptioning (S1260). If the expanded text is not fully included in the recaptioning, the processor (120) can include the correction guide in the priority list (S1270). Alternatively, if the expanded text is fully included in the recaptioning, the processor (120) can exclude the expanded text. That is, if the expanded text is fully included in the recaptioning, the processor (120) can identify that the first object has performed an action corresponding to the expanded text.

[0140] FIGS. 18 to 21 are drawings for explaining a method for generating a correction image according to one embodiment of the present disclosure.

[0141] First, the processor (120) can derive a correction value from the extended text of the correction guide that does not overlap with the recaptioning and the second three-dimensional image (reference content) as shown in FIG. 18 (S1810). For example, the processor (120) can derive a state value correction value for the body point corresponding to the extended text of the correction guide that does not overlap with the recaptioning to correspond to the second object as shown in FIG. 19.

[0142] The processor (120) can derive a specific prompt for generating a corrected image by performing a recaptioning technique based on a natural language input (natural language prompt) (S1820). For example, as illustrated in FIG. 20, the processor (120) can obtain a specific prompt for generating a corrected image by inputting a natural language input, "I want to improve my golf swing to be like the swing in the content," an extended text of a correction guide, and a correction value of a body point's state value into a neural network model. Here, the neural network model may be a neural network model trained to receive at least one of the natural language input, the correction guide, or the correction value of a body point's state value and output a specific prompt for generating a corrected image.

[0143] The processor (120) can generate a reconstructed correction image by using a recaptioning prompt (recaptioned prompt) for generating a correction image (S1830). For example, the processor (120) can acquire a correction image by inputting a recaptioning prompt for generating a correction image and a first three-dimensional image such as the left side of FIG. 21 into a neural network model, and convert the correction image into a two-dimensional correction image such as the right side of FIG. 21.

[0144] FIGS. 18 to 21 describe a specification prompt for generating a correction image, but is not limited thereto. For example, the processor (120) may obtain a correction image by inputting a first three-dimensional image and a correction guide into a second neural network model. Alternatively, the processor (120) may obtain a correction value for a body point based on the correction guide and obtain a correction image by inputting the first three-dimensional image, the correction guide, and the correction value for a body point into a second neural network model. Alternatively, the processor (120) may obtain a correction value for a body point based on the correction guide and obtain a correction image by inputting natural language input, the first three-dimensional image, the correction guide, and the correction value for a body point into a second neural network model.

[0145] FIGS. 22 and FIGS. 23 are drawings for illustrating a correction image and an additional correction image according to an embodiment of the present disclosure.

[0146] The processor (120) can convert a correction image into a two-dimensional correction image and display the two-dimensional correction image through the display (130). For example, the processor (120) can display an image (2210) including a two-dimensional correction image and a second image through the display (130), as shown in FIG. 22. Additionally, the processor (120) may further display an improvement method guide (2220) and an expected improvement result (2230) through the display (130).

[0147] Alternatively, the processor (120) may display an image (2310) including a first image before correction and a two-dimensional correction image through the display (130), as shown in FIG. 23.

[0148] Additionally, the processor (120) may obtain a two-dimensional additional correction image by performing at least one of changing the viewpoint in the correction image, enlarging a part of the area, or changing the playback speed, and may display the two-dimensional additional correction image through the display (130). For example, the processor (120) may obtain a first additional correction image with a changed viewpoint by changing the viewpoint in the correction image, obtain a second additional correction image by enlarging a part of the first additional correction image, convert the first additional correction image and the second additional correction image into a two-dimensional first additional correction image and a two-dimensional second additional correction image, respectively, and may additionally display the two-dimensional first additional correction image (2320-1) and the two-dimensional second additional correction image (2320-2) through the display (130).

[0149] Additionally, the processor (120) may additionally display a progress bar (2330) that guides the playback point through the display (130).

[0150] FIG. 24 is a drawing for explaining a method of using a wearable device according to one embodiment of the present disclosure.

[0151] In the above description, it is assumed that a first video including a user has been captured, but this is not limited thereto. For example, the processor (120) may display a video capture and content analysis icon through the display (130), and when the icon is selected, capture may be performed through the camera (180).

[0152] Alternatively, the processor (120) may display a wearable device-linked video shooting and content analysis icon through the display (130), and when the icon is selected, shooting may be performed through the camera (180).

[0153] In this case, the processor (120) may receive sensing information related to the user's actions from at least one of the wearable devices worn by the user, as shown in FIG. 24.

[0154] When the processor (120) completes the image capture and obtains the first image, it can obtain a first three-dimensional image based on the sensing information and the first image, and since the sensing information is used, the accuracy can be improved.

[0155] FIG. 25 is a drawing for explaining various fields to which an embodiment of the present disclosure is applied.

[0156] Although correction videos related to golf have been described above, they are not limited thereto. For example, the processor (120) may generate correction videos for improving dance movements, as shown in FIG. 25. However, it is not limited thereto, and the processor (120) may generate correction videos such as playing musical instruments. Alternatively, the processor (120) may generate correction videos for objects other than users. For instance, the processor (120) may generate correction videos from a first video of a robot and a second video of a user, and the robot may change its motion control logic based on the correction videos.

[0157] FIG. 26 is a flowchart illustrating a method for controlling an electronic device according to one embodiment of the present disclosure.

[0158] First, when a first user command is received requesting a comparison between a first captured image and a second reference image, the first image and the second image are input into a first neural network model, respectively, to obtain a first three-dimensional image and a second three-dimensional image (S2610). Then, a correction guide for correcting the motion of the first object is obtained based on the first object included in the first three-dimensional image and the second object included in the second three-dimensional image (S2620). Then, the first three-dimensional image and the correction guide are input into a second neural network model to obtain a corrected image in which the motion of the first object in the first three-dimensional image is corrected (S2630). Here, the first neural network model may be a neural network model trained to receive a two-dimensional image and output a three-dimensional image, and the second neural network model may be a neural network model trained to change the motion of an object included in the three-dimensional image.

[0159] Additionally, the step of obtaining a correction guide (S2620) may obtain a correction guide based on at least one of a point in time when correction is required during the motion of the first object, a correction point, or a difference from the motion of the second object based on the first object and the second object.

[0160] And, the step of obtaining a correction guide (S2620) inputs a frame corresponding to the point where correction is needed in the first three-dimensional image into a third neural network model to obtain explanatory information corresponding to the frame, updates the correction guide based on the explanatory information, and the third neural network model may be a neural network model trained to output explanatory information corresponding to the image.

[0161] Additionally, the step of obtaining a correction guide (S2620) can obtain explanatory information by inputting a frame and a preset number of additional frames before and after the frame into a third neural network model.

[0162] In addition, the step of obtaining a correction guide (S2620) can update the correction guide by deleting parts that overlap with the explanatory information in the correction guide.

[0163] Additionally, the step of obtaining a correction guide (S2620) can compare the first object and the second object based on the region of interest when a second user command specifying a region of interest in the first image is received, and obtain a correction guide based on the comparison result.

[0164] And, the step of obtaining a correction guide (S2620) can identify a region of interest in the first image based on the motion of the first object, compare the first object and the second object based on the region of interest, and obtain a correction guide based on the comparison result.

[0165] Additionally, the step of obtaining a correction guide (S2620) may obtain a correction guide after performing at least one of viewpoint matching, object normalization, or sync matching on the first three-dimensional image and the second three-dimensional image based on the first object and the second object.

[0166] In addition, it may further include the step of converting the correction image into a two-dimensional correction image and the step of displaying the two-dimensional correction image.

[0167] In addition, the display step may provide a UI indicating that correction has been performed in a viewpoint and screen area corresponding to the correction guide in a two-dimensional correction image.

[0168] And, it may further include the step of obtaining a two-dimensional additional correction image by performing at least one of changing the viewpoint, zooming in on a part of the area, or changing the playback speed in the correction image, and the step of displaying the two-dimensional additional correction image.

[0169] An electronic device according to one embodiment as described above comprises one or more processors including a first neural network model trained to receive a two-dimensional video and output a three-dimensional video, a second neural network model trained to change the motion of an object included in the three-dimensional video, a memory for storing instructions, and processing circuitry. When the instructions are executed individually or collectively by the one or more processors, if a first user command is received requesting a comparison between a captured first video and a reference second video, the first video and the second video are each input into the first neural network model to obtain a first three-dimensional video and a second three-dimensional video, a correction guide for correcting the motion of the first object is obtained based on a first object included in the first three-dimensional video and a second object included in the second three-dimensional video, and the first three-dimensional video and the correction guide are input into the second neural network model to obtain the first three-dimensional A corrected image can be obtained in which the motion of the first object is corrected in the video.

[0170] According to one example, when the instructions are executed individually or collectively by the one or more processors, the correction guide can be obtained based on at least one of a point in time when correction is required during the motion of the first object, a correction point, or a difference from the motion of the second object based on the first object and the second object.

[0171] According to one example, the memory further stores a third neural network model trained to output descriptive information corresponding to an image, and when the instructions are executed individually or collectively by the one or more processors, a frame corresponding to the point in time when correction is needed in the first three-dimensional image is input into the third neural network model to obtain descriptive information corresponding to the frame, and the correction guide can be updated based on the descriptive information.

[0172] According to one example, when the instructions are executed individually or collectively by the one or more processors, the frame and a predetermined number of additional frames before and after the frame can be input into the third neural network model to obtain the explanatory information.

[0173] According to one example, when the above instructions are executed individually or collectively by one or more processors, the correction guide can be updated by deleting parts that overlap with the description information in the correction guide.

[0174] According to one example, when the instructions are executed individually or collectively by the one or more processors, if a second user command specifying a region of interest in the first image is received, the first object and the second object are compared based on the region of interest, and the correction guide can be obtained based on the comparison result.

[0175] According to one example, when the instructions are executed individually or collectively by one or more processors, a region of interest in the first image is identified based on the motion of the first object, the first object and the second object are compared based on the region of interest, and the correction guide is obtained based on the comparison result.

[0176] According to one example, when the instructions are executed individually or collectively by one or more processors, at least one of viewpoint matching, object normalization, or sync matching is performed on the first three-dimensional image and the second three-dimensional image based on the first object and the second object, and then the correction guide can be obtained.

[0177] According to one example, the system further includes a display, and when the instructions are executed individually or collectively by one or more processors, the correction image can be converted into a two-dimensional correction image and the two-dimensional correction image can be displayed through the display.

[0178] According to one example, when the instructions are executed individually or collectively by one or more processors, a UI may be provided indicating that correction has been performed in a point in time and screen area corresponding to the correction guide in the two-dimensional correction image.

[0179]

[0180] According to one example, when the above instructions are executed individually or collectively by one or more processors, at least one of changing the viewpoint, zooming in on a part of the area, or changing the playback speed in the correction image is performed to obtain a two-dimensional additional correction image, and the two-dimensional additional correction image can be displayed through the display.

[0181] A control method for an electronic device according to one embodiment includes the steps of: when a first user command is received requesting a comparison between a first image captured by a user and a second image which is a reference image, inputting the first image and the second image into a first neural network model to obtain a first three-dimensional image and a second three-dimensional image, respectively; obtaining a correction guide for correcting the motion of the first object based on a first object included in the first three-dimensional image and a second object included in the second three-dimensional image; and inputting the first three-dimensional image and the correction guide into a second neural network model to obtain a corrected image in which the motion of the first object in the first three-dimensional image is corrected, wherein the first neural network model is a neural network model trained to receive a two-dimensional image and output a three-dimensional image, and the second neural network model may be a neural network model trained to change the motion of an object included in the three-dimensional image.

[0182] According to one example, the step of obtaining the correction guide may be based on at least one of a point in time when correction is required during the motion of the first object, a correction point, or a difference from the motion of the second object based on the first object and the second object.

[0183] According to one example, the step of obtaining the correction guide may be to input a frame corresponding to the point in time when correction is required in the first three-dimensional image into a third neural network model to obtain explanatory information corresponding to the frame, and to update the correction guide based on the explanatory information, and the third neural network model may be a neural network model trained to output explanatory information corresponding to the image.

[0184] According to one example, the step of obtaining the correction guide may involve inputting the frame and a predetermined number of additional frames before and after the frame into the third neural network model to obtain the explanatory information.

[0185] According to one example, the step of obtaining the correction guide may update the correction guide by deleting parts that overlap with the explanatory information in the correction guide.

[0186] According to one example, the step of obtaining the correction guide may involve, when a second user command specifying a region of interest in the first image is received, comparing the first object and the second object based on the region of interest, and obtaining the correction guide based on the comparison result.

[0187] According to one example, the step of obtaining the correction guide may involve identifying a region of interest in the first image based on the motion of the first object, comparing the first object and the second object based on the region of interest, and obtaining the correction guide based on the comparison result.

[0188] According to one example, the step of obtaining the correction guide may be to obtain the correction guide after performing at least one of viewpoint matching, object normalization, or sync matching on the first three-dimensional image and the second three-dimensional image based on the first object and the second object.

[0189] According to one example, the method may further include the steps of converting the correction image into a two-dimensional correction image and displaying the two-dimensional correction image.

[0190] According to one example, the displaying step may provide a UI indicating that correction has been performed in a viewpoint and screen area corresponding to the correction guide in the two-dimensional correction image.

[0191] According to one example, the method may further include the step of obtaining a two-dimensional additional correction image by performing at least one of changing the viewpoint, enlarging a part of the area, or changing the playback speed in the correction image, and the step of displaying the two-dimensional additional correction image.

[0192] According to various embodiments of the present disclosure as described above, an electronic device can provide information for correcting the motion of an object by comparing two images and providing a video in which the motion of the object is corrected.

[0193] The electronic device according to one or more embodiments disclosed in this disclosure may be a device of various forms. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a consumer electronics device. The electronic device according to the embodiments of this disclosure is not limited to the devices described above.

[0194] One or more embodiments of the present disclosure and the terms used therein are not intended to limit the technical features described in the present disclosure to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In the present disclosure, each of phrases such as “A or B”, “at least one of A and B”, “at least one of A or B”, “A, B or C”, “at least one of A, B and C”, and “at least one of A, B, or C” may include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as “first,” “second,” or “first” or “second” may be used simply to distinguish a component from another component and do not limit the components in any other aspect (e.g., importance or order). Where any (e.g., first) component is referred to as “coupled” or “connected” to another (e.g., second) component, with or without the terms “functionally” or “communicationally,” it means that said component may be connected to said other component directly (e.g., wired), wirelessly, or through a third component.

[0195] The term “module” as used in one or more embodiments of the present disclosure may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0196] One or more embodiments of the present disclosure may be implemented as software (e.g., program (140)) comprising one or more instructions stored in a storage medium (e.g., internal memory (136) or external memory (138)) readable by a machine (e.g., electronic device (101)). For example, a processor (e.g., processor (120)) of the machine (e.g., electronic device (101)) may call at least one of the one or more instructions stored in the storage medium and execute it. This enables the machine to be operated to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily.

[0197] According to one embodiment, the method according to one or more embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or an application store (e.g., Play Store). TM It can be distributed online (e.g., downloaded or uploaded) through ) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0198] According to one or more embodiments, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to one or more embodiments, one or more of the components or operations among the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to one or more embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

In electronic devices, A memory storing a first neural network model trained to receive a 2D video as input and output a 3D video, a second neural network model trained to change the motion of an object included in the 3D video, and instructions; and One or more processors including processing circuitry; and When the above instructions are executed individually or collectively by the one or more processors, When a first user command is received requesting a comparison between a first captured image and a second reference image, the first image and the second image are each input into the first neural network model to obtain a first 3D image and a second 3D image, and A correction guide for correcting the motion of a first object is obtained based on a first object included in the first three-dimensional image and a second object included in the second three-dimensional image, and An electronic device that inputs the first three-dimensional image and the correction guide into the second neural network model to obtain a corrected image in which the motion of the first object in the first three-dimensional image is corrected. In paragraph 1, When the above instructions are executed individually or collectively by the one or more processors, An electronic device that obtains a correction guide based on at least one of a point in time when correction is required during the motion of the first object, a correction point, or a difference from the motion of the second object, based on the first object and the second object. In paragraph 2, The above memory is, A third neural network model trained to output explanatory information corresponding to an image is further stored, and When the above instructions are executed individually or collectively by the one or more processors, A frame corresponding to the point in time requiring correction in the first three-dimensional image is input into the third neural network model to obtain explanatory information corresponding to the frame, and An electronic device that updates the calibration guide based on the above-described information. In paragraph 3, When the above instructions are executed individually or collectively by the one or more processors, An electronic device that obtains the explanatory information by inputting the above frame and a predetermined number of additional frames before and after the above frame into the third neural network model. In paragraph 3, When the above instructions are executed individually or collectively by the one or more processors, An electronic device that updates the correction guide by deleting parts that overlap with the description information in the correction guide. In paragraph 2, When the above instructions are executed individually or collectively by the one or more processors, When a second user command specifying a region of interest in the first image is received, the first object and the second object are compared based on the region of interest, and An electronic device that obtains the calibration guide based on the above comparison results. In paragraph 2, When the above instructions are executed individually or collectively by the one or more processors, Identifying a region of interest in the first image based on the motion of the first object, and Based on the above region of interest, the first object and the second object are compared, An electronic device that obtains the calibration guide based on the above comparison results. In paragraph 1, When the above instructions are executed individually or collectively by the one or more processors, An electronic device that obtains a correction guide after performing at least one of viewpoint matching, object normalization, or sync matching on the first three-dimensional image and the second three-dimensional image based on the first object and the second object. In paragraph 1, It further includes a display; When the above instructions are executed individually or collectively by the one or more processors, Convert the above correction image into a two-dimensional correction image, and An electronic device that displays the above two-dimensional correction image through the above display. In Paragraph 9, When the above instructions are executed individually or collectively by the one or more processors, An electronic device that provides a UI indicating that correction has been performed in a viewpoint and screen area corresponding to the correction guide in the above two-dimensional correction image. In Paragraph 9, When the above instructions are executed individually or collectively by the one or more processors, At least one of changing the viewpoint, zooming in on a part of the area, or changing the playback speed in the above-mentioned correction image is performed to obtain an additional two-dimensional correction image, and An electronic device that displays the above-mentioned two-dimensional additional correction image through the above-mentioned display. In Paragraph 9, When the above instructions are executed individually or collectively by the one or more processors, An electronic device that displays the second image and the two-dimensional correction image in a time series through the display. In Paragraph 9, When the above instructions are executed individually or collectively by the one or more processors, An electronic device that provides a UI including at least one of a correction point or a difference from the motion of the second object based on the first object and the second object. In a method for controlling an electronic device, When a first user command is received requesting a comparison between a first captured image and a second reference image, the first image and the second image are input into a first neural network model, respectively, to obtain a first three-dimensional image and a second three-dimensional image; A step of obtaining a correction guide for correcting the motion of a first object based on a first object included in the first three-dimensional image and a second object included in the second three-dimensional image; The method includes the step of inputting the first three-dimensional image and the correction guide into a second neural network model to obtain a corrected image in which the motion of the first object in the first three-dimensional image is corrected; The above first neural network model is, It is a neural network model trained to take a 2D image as input and output a 3D image, and The above second neural network model is, A control method, which is a neural network model trained to change the motion of an object included in a three-dimensional image. In Paragraph 14, The step of obtaining the above correction guide is, A control method for obtaining a correction guide based on at least one of a point in time when correction is required during the motion of the first object, a correction point, or a difference from the motion of the second object, based on the first object and the second object.

Images