Electronic device and control method thereof
The electronic device provides real-time exercise feedback using neural networks to identify symmetrical and holding movements, addressing the inefficiencies of prior technologies that require pre-learning.
Patent Information
- Application Number
- CN202380039051.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-05-10
- Filing Date
- 2023-05-09
- Publication Date
- 2024-12-06
AI Technical Summary
Existing technologies require pre-learning of each exercise movement posture to provide feedback, incurring significant costs and inefficiencies.
An electronic device that analyzes exercise motions in real time using neural networks to identify symmetrical and holding movements without prior learning, determining frame intervals and providing feedback based on exercise pattern information.
Enables real-time feedback on exercise movements without the need for pre-learning, reducing costs and improving efficiency in exercise analysis.
Smart Images

Figure CN119096269A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to an electronic device and a control method thereof. For example, the present disclosure relates to an electronic device that provides information about an exercise motion by analyzing an exercise motion of a user and a control method thereof. Background Art
[0002] In the related art, body information corresponding to a person's exercise movement posture has been obtained through pre-learning, and feedback for the current exercise movement has been provided based on the obtained body information.
[0003] In order to obtain the above-mentioned body information, in the related art, an exercise movement posture corresponding to each of all exercise movements must be learned in advance, and there is a problem that too much cost is spent in the advance learning. Summary of the invention
[0004] Technical issues
[0005] The present disclosure solves at least the above-mentioned problems and / or disadvantages of the related art, and embodiments of the present disclosure provide an electronic device and a control method thereof that provides information about a person's exercise movements in real time without learning body information of each exercise movement posture in advance.
[0006] Problems to be solved by the present disclosure are not limited to the above-mentioned problems and / or disadvantages, and other problems not mentioned herein may be clearly understood by those having ordinary skill in the art from the following description.
[0007] Technical Solutions
[0008] According to an example embodiment, an electronic device includes: a memory including at least one instruction; and a processor coupled to the memory and configured to control the electronic device, wherein the processor is configured to: obtain a moving image, identify a person and posture data of the person from multiple frames in the moving image, obtain exercise pattern information corresponding to the multiple frames using the posture data of the identified person, identify exercise movements of the identified person by inputting at least one frame of the moving image into at least one neural network model, obtain exercise feature information corresponding to the identified exercise movement, identify a first frame interval and a second frame interval different from the first frame interval from the multiple frames in the moving image based on the exercise feature information and the exercise pattern information, and provide information about the exercise movement corresponding to the second frame interval by comparing the first frame interval with the second frame interval.
[0009] The processor can be configured to: input at least one frame of the moving image into a first neural network model configured to identify whether the person's exercise movement is a symmetrical movement, and identify whether the person's exercise movement is a symmetrical movement, and input at least one frame of the moving image into a second neural network model configured to identify whether the person's exercise movement is a holding movement, and identify whether the person's exercise movement is a holding movement.
[0010] The first neural network model may be configured to be trained based on a learning moving image that captures a person's exercise movement and information about whether the person's exercise movement corresponding to the learning moving image is a symmetrical movement, and the second neural network model may be configured to be trained based on a learning moving image that captures a person's exercise movement and information about whether the person's exercise movement corresponding to the learning moving image is a holding movement.
[0011] The processor may be configured to identify, from a plurality of frames in the moving image, an interval from a start frame to a first time as a first frame interval based on the person's exercise motion being a symmetrical motion and a holding motion.
[0012] The processor can be configured to: based on the person's exercise movement being a symmetrical movement and a non-maintained movement, identify each exercise repetition interval from multiple frames in the moving image based on the exercise pattern information, and based on the identified exercise repetition interval, identify, from the multiple frames in the moving image, an interval from a starting frame to a frame corresponding to a first number of exercise repetition intervals as a first frame interval.
[0013] The processor may be configured to: based on the person's exercise motion being an asymmetric motion and maintaining motion exercise, identify from a plurality of frames in the moving image based on the exercise pattern information a first frame interval of a frame corresponding to an interval from a frame in which a first symmetrical exercise motion is first started to a second time and an interval from a frame in which a second symmetrical exercise motion is first started to a second time.
[0014] The processor may be configured to: based on the person's exercise motion being an asymmetric motion and a non-holding motion, identify each frame corresponding to the first symmetric exercise motion and the second symmetric exercise motion from a plurality of frames in the moving image based on the exercise pattern information, and identify, from a plurality of frames in the moving image, a frame corresponding to a second number of the first symmetric exercise motion and a frame corresponding to the second number of the second symmetric exercise motion from a start frame as a first frame interval.
[0015] The processor may be configured to identify a frame at a designated interval from a frame being played as a second frame based on the moving image being played.
[0016] The processor may be configured to identify a state of physical exhaustion based on a difference between exercise pattern information corresponding to a first frame interval and exercise pattern information of a second frame interval being greater than or equal to a preset value, and to provide information about an exercise movement of the person corresponding to the second frame interval based on being identified as a state of physical exhaustion.
[0017] The processor may be configured to obtain an original moving image, and obtain a moving image by identifying a frame corresponding to a person's exercise motion from among a plurality of frames included in the original moving image based on an exercise database (DB) storing a plurality of moving images including a plurality of exercise motions.
[0018] According to an example embodiment, a method for controlling an electronic device includes: obtaining a moving image, identifying a person and posture data of the person from multiple frames in the moving image, obtaining exercise pattern information corresponding to the multiple frames using the posture data of the identified person, recognizing exercise movements of the identified person by inputting at least one frame of the moving image into at least one neural network model, obtaining exercise feature information corresponding to the recognized exercise movement, identifying a first frame interval and a second frame interval different from the first frame interval from the multiple frames in the moving image based on the exercise feature information and the exercise pattern information, and providing information about the exercise movement corresponding to the second frame by comparing the first frame interval with the second frame interval.
[0019] Obtaining exercise characteristic information may include: inputting at least one frame of a moving image into a first neural network model that identifies whether a person's exercise movement is a symmetrical movement, and identifying whether the person's exercise movement is a symmetrical movement, and inputting at least one frame of a moving image into a second neural network model that identifies whether the person's exercise movement is a holding movement, and identifying whether the person's exercise movement is a holding movement.
[0020] The first neural network model can be trained based on the learning moving image that captures the exercise movement of a person and information about whether the exercise movement of the person corresponding to the learning moving image is a symmetrical movement, and the second neural network model can be trained based on the learning moving image that captures the exercise movement of the person and information about whether the exercise movement of the person corresponding to the learning moving image is a holding movement.
[0021] Identifying a first frame interval and a second frame interval different from the first frame interval may include identifying an interval from a start frame to a first time from a plurality of frames in the moving image as the first frame interval based on the person's exercise motion being a symmetrical motion and a holding motion.
[0022] According to an example embodiment, a non-transitory computer-readable recording medium may have stored thereon a program that, when executed by a processor, causes an electronic device to perform operations corresponding to a method of controlling an electronic device, the operations including: obtaining a moving image, identifying a person and posture data of the person from multiple frames in the moving image, using the posture data of the identified person to obtain exercise pattern information corresponding to the multiple frames, identifying exercise movements of the identified person by inputting at least one frame of the moving image into at least one neural network model and obtaining exercise feature information corresponding to the identified exercise movement, identifying a first frame interval and a second frame interval different from the first frame interval from multiple frames in the moving image based on the exercise feature information and the exercise pattern information, and providing information about the exercise movement corresponding to the second frame by comparing the first frame interval with the second frame interval. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent through the following detailed description in conjunction with the accompanying drawings, in which:
[0024] Figure 1 is a block diagram illustrating an example configuration of an electronic device according to various embodiments;
[0025] 2A is a flow chart illustrating example operations of an electronic device for providing information about a person's exercise motion within a moving image in accordance with various embodiments;
[0026] 2B is a flow chart illustrating example operations of an electronic device for providing information about a person's exercise motion within a moving image in accordance with various embodiments;
[0027] Figure 3 is a flowchart illustrating an example method of obtaining a moving image according to various embodiments;
[0028] Figure 4 is a graph illustrating an example method of identifying a first frame interval from a symmetric motion exercise and a hold motion exercise according to various embodiments;
[0029] Figure 5 is a graph illustrating an example method of identifying a first frame interval from a symmetric motion exercise and a non-hold motion exercise according to various embodiments;
[0030] Figure 6 is a graph illustrating an example method of identifying whether an interval between each exercise exceeds a set value from a non-hold motion exercise according to various embodiments;
[0031] Figure 7 is a graph illustrating an example method of identifying a first frame interval from an asymmetric motion exercise and a hold motion exercise according to various embodiments;
[0032] Figure 8 is a graph illustrating an example method of identifying a first frame interval from an asymmetric motion exercise and a non-hold motion exercise according to various embodiments;
[0033] Fig. 9 is a graph illustrating an example method of identifying whether an interval between each exercise exceeds a set value from a non-hold motion exercise according to various embodiments;
[0034] Fig.10 is a diagram illustrating an example method in which an electronic device provides information about an exercise motion according to various embodiments;
[0035] Fig.11 is a diagram illustrating an example method in which an electronic device provides information about an exercise motion according to various embodiments; and
[0036] Fig.12 is a flow chart illustrating an example method of controlling an electronic device according to various embodiments. DETAILED DESCRIPTION
[0037] Various modifications may be made to the various example embodiments of the present disclosure, and there may be various types of embodiments. Therefore, the example embodiments will be shown in the drawings and described in more detail in the specific embodiments. However, it should be noted that the embodiments are not intended to limit the scope of the present disclosure to specific embodiments, but should be interpreted as including various modifications, equivalents and / or alternatives of the embodiments included herein. With regard to the description of the drawings, the same reference numerals may be used to indicate the same elements.
[0038] In describing the present disclosure, in the case where it is determined that a detailed description of a related known art may unnecessarily obscure the gist of the present disclosure, the detailed description thereof may be omitted.
[0039] Furthermore, the following embodiments may be modified into various different forms, and it should be understood that the scope of the technical spirit of the present disclosure is not limited to the following embodiments.
[0040] The terms used herein are only used to describe specific embodiments and are not intended to limit the scope of protection. Unless otherwise specified, a singular expression includes a plural expression.
[0041] In the present disclosure, expressions such as “having”, “may have”, “including”, “may include”, etc. are used to specify the existence of corresponding characteristics (for example, elements such as values, functions, operations, or components), and do not exclude the existence or possibility of additional characteristics.
[0042] In the present disclosure, expressions such as "A or B", "at least one of A and / or B", or "one or more of A and / or B" may include all possible combinations of the items listed together. For example, "A or B", "at least one of A and B", or "at least one of A or B" may refer to all cases including (1) at least one A, (2) at least one B, or (3) both at least one A and at least one B.
[0043] Expressions such as "first", "second", "1st", "2nd", etc. used herein may be used to refer to various elements, regardless of order and / or importance. In addition, it should be noted that these expressions are only used to distinguish one element from another element, rather than limiting the relevant elements.
[0044] When an element (e.g., a first element) is indicated as being “operably or communicatively coupled with / coupled to” or “connected to” another element (e.g., a second element), it may be understood that the element is directly coupled with / coupled to the other element or coupled through other elements (e.g., a third element).
[0045] On the other hand, when a certain element (for example, a first element) is indicated as being “directly coupled / directly coupled to” or “directly connected to” another element (for example, a second element), it can be understood that another element (for example, a third element) does not exist between the certain element and the other element.
[0046] The expression "configured to (or set to)" used in the present disclosure may be used interchangeably with, for example, "suitable to?" "capable of?" "designed to?" "adapted to?" "made to?" or "capable of", depending on the context. The term "configured to (or set to)" may not necessarily refer to, for example, "specifically designed to" in terms of hardware.
[0047] In contrast, in certain contexts, the expression “a device configured to…” may refer to something such as a device together with another device or component “can perform…” For example, the phrase “a subprocessor configured (or set) to perform A, B, or C” may refer to, for example, a dedicated processor (e.g., an embedded processor) for performing the corresponding operation or a general-purpose processor (e.g., a central processing unit (CPU) or an application processor) capable of performing the corresponding operation by executing one or more software programs stored in a memory device.
[0048] The term "module" or "component" used in the embodiments of this document performs at least one function or operation, and can be implemented in hardware or software, or in a combination of hardware and software. In addition, in addition to the "module" or "component" that needs to be implemented as specific hardware, multiple "modules" or multiple "components" can be integrated into at least one module and implemented in at least one processor.
[0049] The various elements and regions of the drawings have been schematically shown. Therefore, the present disclosure is not limited by the relative sizes and distances shown in the drawings.
[0050] The embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings to help those skilled in the art to understand.
[0051] Figure 1 is a block diagram illustrating an example configuration of an electronic device according to various embodiments.
[0052] Referring to the configuration of the electronic device according to the example embodiment, Figure 1 , the electronic device 100 may include a memory 110 and a processor (eg, including a processing circuit) 120 .
[0053] The memory 110 may store at least one instruction or data associated with at least one other element of the electronic device 100. For example, the memory 110 may be implemented as a nonvolatile memory, a volatile memory, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), etc. The memory 110 may be accessed by the processor 120, and reading / writing / modifying / deleting / updating of data may be performed by the processor 120.
[0054] The term “memory” used herein may include the memory 110 , a read-only memory (ROM; not shown), a random access memory (RAM; not shown) in the processor 120 , or a memory card (not shown; e.g., micro SD card, memory stick) installed to the electronic device 100 .
[0055] As described above, the memory 110 may store at least one instruction. The instruction may be used to control the electronic device 100. For example, in the memory 110, an instruction associated with an information providing function of a person's exercise motion within a moving image may be stored. For example, the memory 110 may include a plurality of configurations (or modules) for providing information about a person's exercise motion within a moving image according to an embodiment, and the above content will be described below.
[0056] The processor 120 may be electrically coupled to the memory 110, may include various processing circuits, and control the overall operation and functions of the electronic device 100. For example, the processor 120 may provide information about the exercise motion of a person included in the moving image. Figure 1 As shown, the processor 120 may include a moving image acquisition module (e.g., including various processing circuits and / or executable program instructions) 10, an exercise mode information acquisition module (e.g., including various processing circuits and / or executable program instructions) 20, an exercise feature information acquisition module (e.g., including various processing circuits and / or executable program instructions) 30, a first frame interval identification module (e.g., including various processing circuits and / or executable program instructions) 40, and an information providing module (e.g., including various processing circuits and / or executable program instructions) 50, and the program instructions of each module may be stored in the memory 110.
[0057] Then, the plurality of modules 10 to 50 may be loaded into a memory (e.g., a volatile memory) included in the processor 120 to provide information about the exercise motion of the person within the moving image. The processor 120 may load the plurality of modules 10 to 50 from the non-volatile memory to the volatile memory and perform the corresponding functions of the plurality of modules 10 to 50. Loading may refer to an operation of calling data stored in the non-volatile memory to the volatile memory and storing the data for access by the processor 120.
[0058] In an embodiment, although information about exercise movements may be stored in Figure 1 The plurality of modules 10 to 50 in the memory 110 are shown to be provided, but the embodiment is not limited thereto and may be implemented by an external device coupled to the electronic apparatus 100 .
[0059] The plurality of modules 10 to 50 may be implemented with corresponding software, but the embodiment is not limited thereto, and a portion of the module may be implemented by a combination of hardware and software. In an embodiment, the plurality of modules 10 to 50 may be implemented with one software. In addition, a portion of the module may be implemented in the electronic device 100, and the other portion of the module may be implemented in an external device.
[0060] The moving image obtaining module 10 may be a configuration for obtaining a moving image capturing a person's exercise motion.
[0061] For example, the moving image acquisition module 10 may acquire an original moving image, which may include a person's warm-up exercise, exercise exercise, and exercise relaxation exercise, and the moving image may be a moving image in which only interval frames corresponding to the person's exercise exercise are extracted from the original moving image.
[0062] In an embodiment, the moving image acquisition module 10 may acquire the original moving image through a camera or sensor included in the electronic device 100, or acquire the original moving image through an external device or an external server. In an example, based on the electronic device 100 being a device including a camera or a sensor, the electronic device 100 may obtain the original moving image by capturing a person through the camera or the sensor. In an example, if the electronic device 100 is implemented as a server, the electronic device 100 may receive the original moving image from an external device including a camera or a sensor.
[0063] The moving image obtaining module 10 may obtain a moving image by extracting a frame corresponding to a person's exercise motion from a plurality of frames included in an original moving image based on an exercise database (DB) in which a plurality of moving images including a plurality of exercise motions are stored.
[0064] The exercise DB may be a database storing moving images including various exercise movements. The exercise DB may store moving images in which a specific person performing various exercise movements is captured at various angles. The exercise DB may be stored with moving images including exercise movements, but is not limited thereto, and may also be stored with specific images corresponding to the exercise movements.
[0065] The moving image acquisition module 10 can obtain a moving image including only the exercise movement intervals by excluding the warm-up movement intervals and the exercise relaxation intervals of the person from the original moving image. Specifically, the moving image acquisition module 10 can compare the exercise movements of the person included in the exercise DB with the original moving image leaving only similar frames, and obtain the moving image by cutting the remaining frames (e.g., frames corresponding to the warm-up movement and the exercise relaxation movement).
[0066] The exercise pattern information obtaining module 20 may be a configuration for obtaining pattern information of an exercise motion of a person included in a moving image. According to the present disclosure, the pattern information may include speed information, cycle information, number of times information, etc. of the exercise motion of the person.
[0067] For example, the exercise pattern information acquisition module 20 can perform image analysis (e.g., posture estimation) on the moving image, and identify a person and the posture data of the person from multiple frames in the moving image. The posture data can be data that identifies the joint movement of the person by tracking specific body parts of the person included in the moving image (e.g., joint areas such as elbows, shoulders, knees, etc.). In the example, the exercise pattern information acquisition module 20 can identify each specific body part of the person in the moving image as a key point, and obtain the posture data by identifying the movement of the identified key points. In the example, the exercise pattern information acquisition module 20 can obtain the posture data by inputting the moving image into a posture estimation neural network model that can identify the posture data. Then, the exercise pattern information acquisition module 20 can analyze the obtained posture data and obtain exercise pattern information corresponding to multiple frames in the moving image (e.g., speed information, cycle information, number of times information, etc.).
[0068] The exercise characteristic information obtaining module 30 may be a configuration for recognizing the exercise motion of a person included in a moving image and obtaining exercise characteristic information corresponding to the recognized exercise motion. Here, the exercise characteristic information may be information on whether the exercise motion of a person is a symmetrical motion or a holding motion.
[0069] According to the present disclosure, a symmetrical exercise may be an exercise in which one exercise is performed, and for example, a push-up exercise, a squat exercise, and a plank exercise may be included in the symmetrical exercise. In addition, an asymmetrical exercise may be an exercise in which at least two exercises are performed, and for example, a lunge exercise and a one-hand plank exercise may be included in the asymmetrical exercise.
[0070] In addition, according to the present disclosure, a hold exercise may be an exercise in which a specific movement is held for a certain period of time, and for example, a plank exercise and a one-hand plank exercise may be included in the hold exercise. In addition, a non-hold exercise may be an exercise in which a specific movement is repeated without being held for a certain period of time, and for example, a push-up exercise and a squat exercise may be included in the non-hold exercise.
[0071] In an embodiment, the exercise feature information acquisition module 30 may use, for example but not limited to, a first neural network model for identifying whether the exercise movement of a person included in a moving image is a symmetrical movement and a second neural network model for identifying whether it is a sustaining movement to identify whether the exercise movement of a person included in a moving image is a symmetrical movement or a sustaining movement.
[0072] The first neural network model may be a configuration for identifying whether a person's exercise motion is a symmetrical motion, and, for example, may perform learning using a learning moving image capturing a person's exercise motion as input data and information (false / true) on whether a person's exercise motion corresponding to the relevant learning moving image is a symmetrical motion as output data. The exercise feature information acquisition module 30 may input at least one frame of the moving image to the first neural network model for learning execution, and identify whether a person's exercise motion in the moving image is a symmetrical motion or an asymmetrical motion.
[0073] The second neural network model may be a configuration for identifying whether a person's exercise motion is a hold motion, and, for example, may perform learning using a learning moving image including a person's exercise motion as input data and information (false / true) as to whether a person's exercise motion corresponding to the relevant learning moving image is a hold motion as output data. The exercise feature information acquisition module 30 may input the moving image to the second neural network model for learning execution, and identify whether the person's exercise motion in the moving image is a hold motion or a non-hold motion.
[0074] In the example, if the exercise motion of the person in the moving image is a plank exercise motion, the exercise feature information acquisition module 30 may input at least one frame of the moving image into the first neural network model and identify the exercise motion of the person in the moving image as a symmetrical motion, and input at least one frame of the moving image into the second neural network model and identify the exercise motion of the person in the moving image as a holding motion. However, if the exercise motion of the person in the moving image is a one-handed plank exercise motion, the exercise feature information acquisition module 30 may input at least one frame of the moving image into the first neural network model and identify the exercise motion of the person in the moving image as an asymmetrical motion, and input at least one frame of the moving image into the second neural network model and identify the exercise motion of the person in the moving image as a holding motion.
[0075] In an example, if the exercise motion of the person in the moving image is a lunge exercise motion, the exercise feature information obtaining module 30 may input at least one frame of the moving image into the first neural network model and identify the exercise motion of the person in the moving image as an asymmetric motion, and input at least one frame of the moving image into the second neural network model and identify the exercise motion of the person in the moving image as a non-hold motion.
[0076] In the above-mentioned embodiment, the exercise characteristic information obtained by two neural network models (the first neural network model and the second neural network model) is described, but the embodiment is not limited thereto, and the exercise characteristic information may be obtained by one neural network model or at least three neural network models. In the example, if one neural network model is used, the exercise characteristic information obtaining module 30 may input at least one frame of a moving image to the neural network model, and identify whether the exercise motion of a person in the moving image is a symmetrical motion or an asymmetrical motion, and whether it is a holding motion or a non-holding motion.
[0077] The first frame interval identification module 40 may be a configuration for identifying a frame interval corresponding to an exercise motion of a reference state from a plurality of frame intervals in a moving image. For example, the first frame interval identification module 40 may identify the exercise feature information of the moving image and a frame interval (first frame interval) corresponding to an exercise motion of a reference state in the moving image based on the exercise pattern information. The first frame interval identification module 40 may identify a frame interval corresponding to an exercise motion of a reference state of a person from a plurality of frames included in the moving image as a first frame interval based on the exercise feature information and the exercise pattern information of the moving image.
[0078] In the example, the frame interval corresponding to the exercise motion of the reference state may be a frame interval corresponding to the initial exercise motion of the person in the frame included in the moving image, but it may be identified differently based on whether the exercise motion of the person in the moving image is a symmetrical motion or an asymmetrical motion and is a holding motion or a non-holding motion.
[0079] If the exercise motion of the person in the moving image is a symmetrical motion and a holding motion, the first frame interval recognition module 40 may recognize the interval from the start frame to the first time in the moving image as a frame corresponding to the first frame interval. The first time may be an initial setting value, but is not limited thereto, and may be a value set by the user of the electronic device 100. In an example, if the exercise motion of the person in the moving image is a plank exercise motion, the exercise characteristic information acquisition module 30 may recognize the exercise motion of the person as a symmetrical motion and a holding motion, and the first frame interval recognition module 40 may recognize the interval from the start frame to the first time (e.g., five seconds) from the moving image as the first frame interval.
[0080] If the exercise motion of the person in the moving image is a symmetrical motion and a non-holding motion, the first frame interval identification module 40 may identify the exercise repetition interval of each of the multiple frames in the moving image based on the exercise mode information. The first frame interval identification module 40 may identify the exercise repetition interval corresponding to one of the exercise motions of the person in the moving image based on the exercise mode information including the speed information, cycle information, number of times information, etc. of the exercise motion of the person in the moving image. In an example, if the exercise motion of the person in the moving image is a push-up exercise motion, the first frame interval identification module 40 may identify each frame corresponding to the interval in which the push-up exercise motion is performed once from the multiple frames in the moving image based on the exercise mode information of the relevant moving image. The first frame interval identification module 40 may identify the interval from the start frame of the moving image to the frame corresponding to the exercise repetition interval of the first number (e.g., n times) as the frame corresponding to the first frame interval based on the identified exercise repetition interval. The first number may be the number of times initially set, but is not limited thereto, and may be the number of times set by the user of the electronic device 100. For example, if the exercise motion of the person in the moving image is a push-up exercise motion, the first frame interval identification module 40 may identify an interval from a start frame of the moving image to a frame in which the push-up motion is performed a first number of times (e.g., five times) as a first frame interval.
[0081] If the exercise motion of the person in the moving image is an asymmetrical motion and a holding motion, the first frame interval identification module 40 can identify frames corresponding to an interval from a frame where the first symmetrical exercise motion is first started to a second time (e.g., m seconds) and an interval from a frame where the second symmetrical exercise motion is first started to the second time (e.g., m seconds) in the frames of the moving image as the first frame interval based on the exercise pattern information.
[0082] For example, if the exercise movement of the person in the moving image is an asymmetric movement and a maintaining movement, the first frame interval identification module 40 can respectively identify the first exercise interval corresponding to the first symmetrical exercise movement and the second exercise interval corresponding to the second symmetrical exercise movement from the exercise movements included in the moving image based on the exercise mode information including speed information, cycle information, number information, etc. of the exercise movement of the person in the moving image.
[0083] In an example, if the exercise motion of the person in the moving image is a moving image of a one-handed plank exercise motion, the first frame interval recognition module 40 may recognize a plank exercise motion performed by supporting the right hand to the ground as a first symmetrical exercise motion, and recognize a plank exercise motion performed by supporting the left hand to the ground as a second symmetrical exercise motion from the exercise motion of the person in the moving image based on the exercise pattern information. The first frame interval recognition module 40 may recognize frames of a first exercise interval corresponding to the first symmetrical exercise motion and frames of a second exercise interval corresponding to the second symmetrical exercise motion in the moving image.
[0084] The first frame interval identification module 40 may identify frames corresponding to an interval from a frame where the first symmetrical exercise motion first starts to a second time (e.g., 3 seconds) and an interval from a frame where the second symmetrical exercise motion first starts to the second time (e.g., 3 seconds) from a plurality of frames in the moving image as first frame intervals.
[0085] If the exercise motion of the person in the moving image is an asymmetric motion and a non-holding motion, the first frame interval identification module 40 may respectively identify frame intervals corresponding to the first symmetric exercise motion and the second symmetric exercise motion from a plurality of frames in the moving image based on the exercise pattern information. The first frame interval identification module 40 may identify, from a plurality of frames in the moving image, a frame corresponding to the first symmetric exercise motion of the second number from the start frame and a frame corresponding to the second symmetric exercise motion of the second number as the first frame interval. The second number may be an initially set number, but is not limited thereto, and may be a number set by a user of the electronic device 100.
[0086] In an example, if the person's exercise motion in the moving image is a lunge exercise motion, the first frame interval identification module 40 may identify, based on the exercise pattern information, frames corresponding to a first symmetrical exercise motion of the person's right foot bending forward and frames corresponding to a second symmetrical exercise motion of the person's left foot bending forward from a plurality of frames in the moving image, respectively. The first frame interval identification module 40 may identify, from a plurality of frames in the moving image, a frame in which the first symmetrical exercise motion is performed a second number of times and a frame in which the second symmetrical exercise motion is performed a second number of times as a first frame interval.
[0087] In an example, if a person in a moving image performs the first symmetrical exercise motion and the second symmetrical exercise motion alternately each time (the first symmetrical exercise motion is performed first), the first frame interval identifying module 40 may identify an interval from a start frame of the moving image to a frame in which the second symmetrical exercise motion is first performed a second number of times as the first frame interval.
[0088] In an example, if a person in a moving image performs a second symmetrical exercise motion a specific number of times after performing a first symmetrical exercise motion a specific number of times, the first frame interval identification module 40 may identify, as the first frame interval, an interval from a start frame of the moving image to a frame at which the first symmetrical exercise motion is performed a second number of times and an interval from a frame at which the second symmetrical exercise motion is first started to a frame at which the second symmetrical exercise motion is performed a second number of times.
[0089] The information providing module 50 may be a configuration for providing information on an exercise motion being performed by comparing the first frame interval with the exercise motion being performed by a person in a current interval in the moving image.
[0090] For example, the information providing module 50 may identify a difference between the exercise pattern information of the first frame interval identified by the first frame interval identifying module 40 and the exercise pattern information of the second frame interval as the current frame.
[0091] According to the present disclosure, if a moving image is being played, the second frame interval may be a frame of a preset (eg, specified) interval from a frame currently being played.
[0092] In the example, if the exercise motion of the person in the moving image is a hold motion, the second frame interval may be an interval of 10% of one hold interval of the person in the hold motion moving image stored in the exercise DB. For example, if one hold interval of the person in the hold motion moving image stored in the exercise DB is one minute, the second frame interval corresponding to the hold motion may be six seconds.
[0093] In an example, if the exercise motion of the person in the moving image is a symmetrical motion, the second frame interval may be a time interval required for the person in the symmetrical motion moving image stored in the exercise DB to repeat the motion three times on average, and may be, for example, three seconds.
[0094] The information providing module 50 may use a distance algorithm to identify the difference between the two exercise pattern information to compare the exercise pattern information of the first frame interval and the exercise pattern information of the second frame interval. In an example, the information providing module 50 may convert a plurality of key points included in the exercise pattern information into vectors, identify the closest vector between the converted vectors, and use a cosine distance algorithm for comparing the exercise pattern information to identify the difference between the two exercise pattern information.
[0095] The information providing module 50 can identify the person as being in a state of physical exhaustion based on the difference between the exercise pattern information of the first frame interval and the exercise pattern information of the second frame interval being greater than or equal to a preset (e.g., specified) value. In an example, the preset values can be set differently according to the exercise feature information corresponding to the exercise motion being performed by the person in the moving image.
[0096] Based on being identified as being in a physically exhausted state, the information providing module 50 may provide information about the second frame interval. The information about the second frame interval may include information for guiding the person's exercise movement in the second frame interval, and may be message information such as "Lift your hips higher" when, for example, the person's exercise movement is a plank exercise movement.
[0097] In an example, if the electronic device 100 includes a display, the information providing module 50 may display a user interface (UI) including information about the second frame interval on the display of the electronic device 100. In an example, if the electronic device 100 includes a speaker, the information providing module 50 may output a voice including information about the second frame interval through the speaker.
[0098] However, the embodiment is not limited thereto, and even when the difference between the exercise mode information of the first frame interval and the exercise mode information of the second frame interval is less than a preset value, information about the second frame interval may be provided. In this case, the information providing module 50 may provide message information notifying that the exercise motion of the second frame interval is being well performed.
[0099] In the related art, body information corresponding to a person's exercise movement posture is obtained by pre-learning, and feedback on the current exercise movement is provided based on the body information. In order to obtain the above-mentioned body information, the exercise movement posture corresponding to each of all exercise movements must be pre-learned in the related art, and there is a problem of spending too much cost in pre-learning. Therefore, the electronic device 100 according to the present disclosure can provide information about a person's exercise movement in real time based on the above-mentioned embodiment without pre-learning the body information of each exercise movement posture.
[0100] 2A and 2B are flowcharts illustrating example operations of the electronic device 100 for providing information about exercise motion of a person within a moving image according to various embodiments.
[0101] 2A , the electronic device 100 may obtain a moving image ( S201 ). For example, the moving image may be a moving image having only frames of intervals corresponding to a person's exercise motion extracted from a plurality of frames in an original moving image obtained by the electronic device 100 .
[0102] Reference Figure 3 The above contents are described in more detail.
[0103] For example, the electronic device 100 may obtain an original moving image associated with a person's exercise motion ( S310 ).
[0104] The electronic device 100 can obtain a moving image associated with an exercise motion by removing outliers from the original moving image via outlier analysis. An outlier may refer to data that exceeds the general characteristics of given data. The electronic device 100 can identify outliers through outlier analysis and remove outliers from the original moving image.
[0105] For example, the original moving image may include an exercise movement interval, a warm-up exercise movement interval associated with the exercise movement, and an exercise relaxation interval. At this time, the electronic device 100 can remove the frames of the exercise movement interval and the exercise relaxation interval included in the original moving image through outlier analysis, and obtain a moving image including the exercise movement interval.
[0106] The electronic device 100 can calculate the degree of similarity with the exercise movement included in the exercise DB by comparing the original moving image associated with the exercise movement of the person and the exercise movement included in the exercise DB, and obtain a moving image that only includes frames whose degree of similarity is greater than or equal to a preset value (S320). The electronic device 100 can use an algorithm (e.g., dynamic time warping) to compare the original moving image and the exercise movement included in the exercise DB, by which the pattern of time series data can be compared. The electronic device 100 can obtain exercise pattern information based on the moving image (S202). For example, the electronic device 100 can perform image analysis (e.g., posture estimation) on the moving image, and obtain posture data of the person in the moving image. Then, the exercise pattern information acquisition module 20 can analyze the obtained posture data and obtain pattern information of the person's exercise movement (e.g., speed information, cycle information, number of times information, etc.).
[0107] The electronic device 100 may obtain user input for analyzing posture data from the user. For example, the body shape of a person performing an exercise included in the exercise DB and the body shape of the user included in the original moving image may vary. At this time, depending on the difference in body shape, a difficult problem of obtaining accurate posture data may occur.
[0108] Therefore, the electronic device 100 may provide a UI for receiving input of information about the body shape of a person performing an exercise included in the exercise DB through a display, and receive input of information about the body shape from the user through the provided UI. Alternatively, the electronic device 100 may provide a UI for receiving input of information about the body shape of a person included in a moving image through a display, and receive input of information about the body shape from the user through the provided UI. At this time, the body shape may refer to height, weight, gender, etc.
[0109] The electronic device 100 may obtain exercise characteristic information based on the moving image (S203). The exercise characteristic information may include information about whether the exercise motion of the person in the moving image is a symmetrical motion or a holding motion. For example, the electronic device 100 may use a first neural network model for identifying whether the exercise motion included in the moving image is a symmetrical motion and a second neural network model for identifying whether the exercise motion included in the moving image is a holding motion to identify whether the exercise motion included in the moving image is a symmetrical motion and a holding motion.
[0110] The electronic device 100 may identify whether the exercise motion is a symmetrical motion exercise ( S204 ).
[0111] If the exercise motion is the symmetrical motion exercise (S204-Y), the electronic device 100 may identify whether the exercise motion is the holding motion (S205).
[0112] If the exercise motion is a hold motion exercise (S205-Y), that is, if the exercise motion is a symmetrical motion exercise and a hold motion exercise, the electronic device 100 may identify a hold interval (S206). The electronic device 100 may identify an earlier portion of the hold interval portion of N seconds as a first frame interval (S207). For example, based on the exercise pattern information, the electronic device 100 may identify a hold interval of the exercise motion from the moving image, and identify an interval from the start frame to N seconds (first time) of the first hold interval as the first frame interval.
[0113] For example, the exercise motion may be a symmetrical motion exercise such as a "plank support" and a holding motion exercise. Information about the user's exercise motion in the moving image may be as follows: Figure 4 As shown. The x-axis may represent time, the y-axis may represent information about the user's movement, and the graph may represent information about the user's movement. The identified holding interval may be the first interval 410. The first interval may be an interval in which a plank support as a symmetrical movement exercise and a holding movement exercise is being performed. The electronic device 100 may identify a second interval 420 corresponding to the initial N seconds of the first interval as a first frame interval.
[0114] If the exercise motion is a symmetrical motion exercise or the exercise motion is not a hold motion (S205-N), that is, if the exercise motion is a symmetrical motion exercise and a non-hold motion exercise, the electronic device 100 may identify an exercise repetition interval (S208). The electronic device 100 may obtain an exercise interval between exercise repetition intervals (S209). The electronic device 100 may identify the earlier part of Y times as the first frame interval (S210). For example, the electronic device 100 may identify the exercise repetition interval based on exercise pattern information. The electronic device 100 may identify an exercise repetition interval corresponding to one exercise motion from the exercise motion included in the moving image based on the exercise pattern information including speed information, cycle information, number information, etc. of the person's exercise motion. The electronic device 100 may obtain an exercise interval between one exercise motion, and identify the interval from the first exercise motion to Y times (the first number) as the first frame interval.
[0115] For example, the exercise motion may be a symmetrical motion exercise such as "squat" and a non-hold motion exercise. Information about the user's exercise motion in the moving image may be as follows: Figure 5 As shown. The x-axis may represent time, the y-axis may represent information about the user's movement, and the graph may represent information about the user's movement. The identified exercise repetition intervals may be a first interval 510, a second interval 520, a third interval 530, and a fourth interval 540. Each interval may be an interval in which the exercise motion is performed once. If Y times is three times, the electronic device 100 may identify the initial three intervals 510, 520, and 530 of the exercise repetition interval as a first frame interval 550.
[0116] When the first frame interval is identified through operations S207 to S210, the electronic device 100 may identify whether the exercise motion in the first frame interval is continued (S211). For example, the electronic device 100 may compare the exercise mode information of the first frame interval with the exercise mode information of the currently performed exercise motion (the second frame interval), and identify whether the currently performed exercise motion (the exercise motion in the second frame interval) is the same exercise motion as that in the first frame interval.
[0117] If the exercise motion in the first frame interval is identified as not to continue (S211-N), the electronic device 100 may return to step S202 to obtain exercise pattern information of the current exercise motion and perform subsequent steps.
[0118] If the exercise motion in the first frame interval is identified as continuing (S211-Y), the electronic device 100 may identify whether the difference between the exercise pattern information of the first frame interval and the current frame interval (the second frame interval) exceeds the first setting value (threshold A) (S212). For example, the electronic device 100 may identify the difference between the exercise pattern information of the first frame interval and the exercise pattern information of the current frame interval (the second frame interval).
[0119] If the difference between the exercise pattern information of the first frame interval and the current frame interval (second frame interval) exceeds the first setting value (threshold A), the electronic device 100 may recognize that the person in the moving image is physically exhausted ( S227 ).
[0120] If the difference of the exercise pattern information of the first frame interval and the current frame interval does not exceed the first setting value (threshold A) (S212-N), the electronic device 100 may identify whether the exercise motion corresponding to the first frame interval is a hold motion exercise (S213).
[0121] If the exercise motion corresponding to the first frame interval is the maintain motion exercise (S213-Y), the electronic device 100 may return to S211 to identify whether the exercise motion in the current first frame interval continues (S211).
[0122] If the exercise motion corresponding to the first frame interval is not a hold motion exercise (S213-N), that is, if the exercise motion corresponding to the first frame interval is a non-hold motion, the electronic device 100 may obtain (e.g., calculate) an interval between each exercise (S214). The electronic device 100 may identify whether the interval between each exercise exceeds a second setting value (threshold B) (S215).
[0123] Even if there is not much difference between the user's exercise posture in the first frame interval and the user's exercise posture in the second frame interval, the electronic device 100 can also identify whether the time spent by the user to perform the same movement in the first frame interval and the time spent by the user to perform the same movement in the second frame interval exceed the preset setting value.
[0124] For example, the exercise motion may be a symmetrical motion and a non-hold motion exercise such as "squat". Information about the user's exercise motion in the moving image may be as follows: Figure 6As shown. The first interval 610, the second interval 620, the third interval 630, the fourth interval 640, and the fifth interval 650 may be intervals for performing one exercise movement, respectively. In addition, the first frame interval 660 may be three intervals 610, 620, and 630 of the initial interval, and the second frame interval may be the fifth interval 650. The time taken to perform one exercise movement in each of the first interval 610, the second interval 620, the third interval 630, the fourth interval 640, and the fifth interval 650 may be five seconds, three seconds, four seconds, seven seconds, and eight seconds. In addition, the second setting value may be three seconds.
[0125] The interval between each exercise may refer to the time taken to perform one exercise movement in each interval. Alternatively, the interval between each exercise may refer to the difference in time taken to perform the exercise movement once just before.
[0126] For example, the electronic device 100 may compare the average time taken to perform one exercise movement in the first frame interval 660 and the time taken to perform one exercise movement in the second frame interval 650. Then, the electronic device 100 may compare the difference between the average time taken to perform one exercise movement in the first frame interval 660 and the time taken to perform one exercise movement in the second frame interval 650 with a preset second setting value.
[0127] At this time, the average time taken to perform one exercise motion in the first frame interval 660 may be four seconds, which is the average time of five seconds, three seconds, and four seconds. The time taken to perform one exercise motion in the second frame interval 650 may be eight seconds. Because the difference between the average time taken to perform one exercise motion in the first frame interval 660 and the time taken to perform one exercise motion in the second frame interval 650 is four seconds, and the second setting value is three seconds, the electronic device 100 may recognize that the interval between each exercise exceeds the second setting value.
[0128] Alternatively, the electronic device 100 may compare a difference between a minimum time among times taken to perform one exercise motion in the first frame interval 660 and a time taken to perform one exercise motion in the second frame interval with the second setting value.
[0129] For example, the minimum time among the time taken to perform one exercise movement in the first frame interval 660 may be three seconds. In addition, the time taken to perform one exercise movement in the second frame interval may be eight seconds. In addition, the second setting value may be three seconds. At this time, because the difference between three seconds, which is the minimum time among the time taken to perform one exercise movement in the first frame interval 660, and eight seconds, which is the time taken to perform one exercise movement in the second frame interval 650, is five seconds, the electronic device 100 may recognize that the interval between each exercise exceeds the second setting value.
[0130] Alternatively, the electronic device 100 may compare the difference between the maximum time taken to perform one exercise motion in the first frame interval 660 and the time taken to perform one exercise motion in the second frame interval with the second setting value. Similar to the above method, the electronic device 100 may identify whether the interval between each exercise exceeds the second setting value.
[0131] If the interval between each exercise does not exceed the second setting value (threshold B) ( S215 -N), the electronic device 100 may return to step S211 to identify whether the same exercise motion as the exercise motion corresponding to the first frame interval continues ( S211 ).
[0132] If the interval between each exercise exceeds the second setting value (threshold B) ( S215 -Y), the electronic device 100 may recognize that the person in the moving image is physically exhausted ( S227 ).
[0133] In operation 204 , referring to FIG. 2B , if the exercise motion is not the symmetrical motion exercise ( S204 -N), that is, if the exercise motion is the asymmetrical motion exercise, the electronic device 100 may identify whether the exercise motion is a holding motion ( S216 ).
[0134] If the exercise motion is a hold motion exercise (S216-Y), that is, if the exercise motion is an asymmetrical motion exercise and is a hold motion exercise, the electronic device 100 may identify a hold interval (S217). The electronic device 100 may identify an earlier portion m seconds and a relative portion m seconds of the hold interval portion as a first frame interval (S218). For example, the electronic device 100 may identify frames corresponding to an interval from a frame where the first symmetrical exercise motion is first started to m seconds (second time) and an interval from a frame where the second symmetrical exercise motion is first started to m seconds (second time) from the moving image based on the exercise pattern information as the first frame interval.
[0135] For example, the exercise motion may be an asymmetric motion exercise and a holding motion exercise such as "one-hand plank support". Information about the user's exercise motion in the moving image may be as follows: Figure 7As shown. The x-axis may represent time, the y-axis may represent information about the user's movement, and the graph may represent information about the user's movement. The asymmetric motion exercise may include a first exercise motion and a second exercise motion. The identified earlier portion of the hold interval (first exercise motion interval) may be the first interval 710, and the later portion of the hold interval (second exercise motion interval) may be the second interval 720. The exercise from the first interval (first exercise motion) and the exercise from the second interval (second exercise motion) may be asymmetric motion. The electronic device 100 may identify a third interval 730 corresponding to m seconds in the earlier portion of the hold interval and a fourth interval 740 corresponding to m seconds in the later portion of the hold interval as the first frame interval.
[0136] If the exercise motion is not the hold motion exercise (S216-N), that is, if the exercise motion is the asymmetric motion exercise and the non-hold motion exercise, the electronic device 100 may identify the exercise repetition interval (S219). The electronic device 100 may obtain the exercise interval between the exercise repetition intervals (S220). The electronic device 100 may identify the earlier portion x times and the opposite side x times as the first frame interval (S221). Specifically, the electronic device 100 may identify the frame corresponding to the first symmetric exercise motion x times (second number) from the start frame and the frame corresponding to the second symmetric exercise motion x times (second number) from the moving image as the first frame interval.
[0137] For example, the exercise motion may be an asymmetric motion exercise, and a non-hold motion exercise such as a "lunge". Information about the user's exercise motion in the moving image may be as follows: Figure 8 As shown. The x-axis may represent time, the y-axis may represent information about the user's movement, and the graph may represent information about the user's movement. The asymmetric motion exercise may include a first exercise motion and a second exercise motion. The identified exercise repetition intervals may be a first interval 810, a second interval 820, a third interval 830, a fourth interval 840, and a fifth interval 850. The repetition intervals of the first exercise motion may be the first interval 810, the third interval 830, and the fifth interval 850. In addition, the repetition intervals of the second exercise motion may be the second interval 820 and the fourth interval 840. Here, if x times is two times, the electronic device 100 may identify the sixth interval 860 including two first exercise motion intervals and two second exercise motion intervals as the first frame interval.
[0138] When the first frame interval is identified through operations S218 and S221, the electronic device 100 may identify whether the exercise motion in the current first frame interval continues (S222). For example, the electronic device 100 may compare the exercise mode information of the first frame interval with the exercise mode information of the current frame interval (second frame interval), and identify whether the exercise motion of the current frame interval is the same exercise motion as that in the first frame interval.
[0139] If the exercise motion in the first frame interval is identified as not to continue (S222-N), the electronic device 100 may return to operation S202 to obtain exercise pattern information of the current exercise motion and perform subsequent steps.
[0140] If the exercise motion in the first frame interval is identified as continuing (S222-Y), the electronic device 100 may identify whether the difference between the exercise pattern information of the first frame interval and the current frame interval exceeds a third setting value (threshold C) (S223). For example, the electronic device 100 may identify the difference between the exercise pattern information of the first frame interval and the exercise pattern information of the current frame interval (second frame interval).
[0141] If it is identified that the difference of the exercise pattern information of the first frame interval and the current frame interval exceeds the third setting value (S223-Y), the electronic device may identify that the person in the moving image is physically exhausted (S227).
[0142] If the difference between the exercise mode information of the first frame interval and the exercise mode information of the current frame interval (second frame interval) does not exceed the third setting value (threshold C) (S223-N), the electronic device 100 may identify whether the exercise motion of the first frame interval is a hold motion exercise (S224).
[0143] If the exercise motion of the first frame interval is the maintain motion exercise (S224-Y), the electronic device 100 may return to operation S222 to identify whether the exercise motion in the first frame interval continues (S222).
[0144] If the exercise motion of the first frame interval is not a hold motion exercise (S224-N), that is, if the exercise motion of the first frame interval is a non-hold motion, the electronic device 100 may obtain (e.g., calculate) an interval between each exercise (S225). The electronic device 100 may identify whether the interval between each exercise exceeds a fourth setting value (threshold D) (S226).
[0145] For example, the exercise motion may be an asymmetric motion and a non-hold motion exercise, such as a "lunge". The asymmetric motion may include a first symmetric exercise motion and a second symmetric exercise motion. At this time, the information about the user's exercise motion in the moving image may be as follows: Fig. 9As shown. The first interval 910, the second interval 911, the third interval 912, the fourth interval 913, the fifth interval 914, the sixth interval 915, the seventh interval 916, the eighth interval 917 and the eighth interval 918 may be intervals for performing one first symmetrical exercise movement or one second symmetrical exercise movement, respectively. In addition, the first frame interval may be an interval 920 in which the asymmetrical movement is initially performed twice, and the second frame interval may be an interval 930. The time spent performing one first symmetrical exercise movement or one second symmetrical exercise movement in each of the first interval 910, the second interval 911, the third interval 912, the fourth interval 913 and the fifth interval 914, the sixth interval 915, the seventh interval 916, the eighth interval 917 and the ninth interval 918 may be five seconds, four seconds, four seconds, three seconds, five seconds, six seconds, seven seconds, six seconds and eight seconds. In addition, the fourth setting value may be three seconds.
[0146] The interval between each exercise may refer to the time spent in performing an asymmetrical exercise movement (eg, a first symmetrical exercise movement and a second symmetrical exercise movement).
[0147] For example, the electronic device 100 may compare the average time taken to perform one asymmetric exercise motion in the first frame interval 920 with the time taken to perform one asymmetric exercise motion in the second frame interval 930. The electronic device 100 may compare the difference between the average time taken to perform one asymmetric exercise motion in the first frame interval 760 and the time taken to perform one asymmetric exercise motion in the second frame interval 750 with a preset fourth setting value.
[0148] The average time taken to perform one asymmetric exercise motion in the first frame interval 760 may be nine sections, which is an average time of nine seconds, seven seconds, and eleven seconds. Then, the time taken to perform one asymmetric exercise motion in the second frame interval 930 may be fifteen seconds. Because the difference between the average time taken to perform one exercise motion in the first frame interval 760 and the time taken to perform one exercise motion in the second frame interval 750 is six seconds, and the fourth setting value is three seconds, the electronic device 100 may recognize the interval between each exercise as exceeding the fourth setting value.
[0149] Alternatively, the electronic device 100 may compare a difference between a minimum time among times taken to perform one asymmetric exercise motion in the first frame interval 760 and a time taken to perform one asymmetric exercise motion in the second frame interval with the fourth setting value.
[0150] For example, the minimum time among the times taken to perform one asymmetric exercise motion in the first frame interval 760 may be seven seconds. In addition, the time taken to perform one asymmetric exercise motion in the second frame interval 750 may be fifteen seconds. Since the difference between seven seconds, which is the minimum time among the times taken to perform one asymmetric exercise motion in the first frame interval 760, and fifteen seconds, which is the time taken to perform one asymmetric exercise motion in the second frame interval 750, is eight seconds, the electronic device 100 may recognize the interval between each exercise as exceeding the fourth setting value.
[0151] Alternatively, the electronic device 100 may compare the difference between the maximum time of the time taken to perform one exercise motion in the first frame interval 760 and the time taken to perform one exercise motion in the second frame interval with the fourth setting value. As in the above method, the electronic device 100 may identify whether the interval between each exercise exceeds the fourth setting value. If the interval between each exercise does not exceed the fourth setting value (threshold D) (S226-N), the electronic device may return to step S222 to identify whether the exercise motion in the current first frame interval continues (S222).
[0152] If the interval between each exercise exceeds the fourth setting value (threshold D) ( S226 -Y), the electronic device 100 may recognize that the person in the moving image is physically exhausted ( S227 ).
[0153] If the physical strength of the person in the moving image is identified as exhausted, the electronic device 100 may provide information about the exercise movement. At this time, the information about the exercise movement may be information about the second frame interval. The information about the second frame interval may include information for guiding the person's exercise movement in the second frame interval.
[0154] For example, if the physical strength is recognized as exhausted, the electronic device 100 may provide information for guiding the person's exercise movement in the second frame interval. For example, if the person's exercise movement is "squatting", the electronic device 100 may provide a voice such as "open your chest." through a speaker included in the electronic device 100. Fig.10 Alternatively, the electronic device 100 may provide a notification sound for proving that a problem has occurred in the posture of the exercise movement due to exhaustion of the user's physical strength.
[0155] Alternatively, the electronic device 100 may provide a moving image captured through a display included in the electronic device 100 while capturing the user's exercise moving image. At this time, if it is recognized that the user is physically exhausted while performing exercise, the electronic device 100 may provide information on a posture guiding the exercise movement in a screen provided through the display.
[0156] The information of the posture guiding the exercise movement may be information for showing a specific point of a person's body.
[0157] For example, if the person's exercise is "plank support", the electronic device 100 may provide an exercise image 1111 of the person captured through the display 1110, such as Fig.11 If the physical strength is identified as exhausted, the electronic device 100 may provide a mark 1120 for a designated body part on the display 1110 together to guide the user's exercise. At this time, the electronic device 100 may provide a message 1130 on the display 1110 together to guide the designated body part.
[0158] Alternatively, the electronic device 100 may provide a mark 1120 for specifying a body part to guide the user's exercise movement through the display 1110 , and together provide voice information guiding the user's exercise movement (eg, “the hips should be lifted higher”) through a speaker included in the electronic device 100 .
[0159] Fig.12 is a flow chart illustrating an example method of controlling an electronic device according to various embodiments.
[0160] The electronic device 100 may obtain a moving image (S1210). For example, the electronic device 100 may obtain an original moving image and obtain the moving image by identifying a frame corresponding to a person's exercise motion from among a plurality of frames included in the original moving image based on an exercise database (DB) stored together with a plurality of moving images including a plurality of exercise motions.
[0161] The electronic device 100 may recognize a person and gesture data of the person from a plurality of frames in the moving image (S1220). For example, the electronic device 100 may obtain gesture data by inputting the moving image into a neural network model capable of recognizing gesture data.
[0162] The electronic device 100 may obtain exercise pattern information corresponding to a plurality of frames using the recognized posture data of the person (S1230). For example, the electronic device 100 may recognize each specific body part of the person included in the moving image as a key point and obtain posture data by recognizing the movement of the recognized key point.
[0163] The electronic device 100 may recognize the exercise motion of the identified person by inputting at least one frame of the moving image to at least one neural network model, and obtain exercise feature information corresponding to the recognized exercise motion (S1240).
[0164] The electronic device 100 may identify a first frame interval and a second frame interval different from the first frame interval from a plurality of frames in the moving image based on the exercise characteristic information and the exercise pattern information (S1250). For example, if the person's exercise motion is an asymmetric motion and a non-holding motion, frames corresponding to the first symmetric exercise motion and the second symmetric exercise motion may be respectively identified from a plurality of frames in the moving image based on the exercise pattern information. A frame corresponding to the second number of the first symmetric exercise motion and a frame corresponding to the second number of the second symmetric exercise motion from the start frame among the plurality of frames in the moving image may be identified as the first frame interval.
[0165] The electronic device 100 may compare the first frame interval with the second frame interval and provide information about the exercise motion corresponding to the second frame (S1260). The information about the exercise motion may be information for guiding the exercise motion of a person in the second frame interval.
[0166] Functions associated with artificial intelligence according to the present disclosure may be operated by the processor 120 and the memory 110 of the electronic device 100 .
[0167] The processor 120 may be configured by one or more processors. The one or more processors may be general-purpose processors such as a central processing unit (CPU) and an application processor (AP), graphics-specific processors such as a graphics processing unit (GPU) and a visual processing unit (VPU), or artificial intelligence-specific processors such as a neural processing unit (NPU) and a tensor processing unit (TPU).
[0168] According to an embodiment, if a plurality of processors are included in a system on chip (SoC) included in the electronic device 100, the electronic device 100 may use a graphics-dedicated processor or an artificial intelligence-dedicated processor among the plurality of processors to perform calculations associated with artificial intelligence (e.g., calculations associated with learning or inference of an artificial intelligence model), and use a general-purpose processor among the plurality of processors to perform typical calculations of the electronic device. For example, the electronic device 100 may use at least one of a GPU, a VPU, an NPU, and a TPU dedicated to convolution calculations among the plurality of processors to perform calculations associated with artificial intelligence, and use at least one of a CPU and an AP among the plurality of processors to perform typical calculations of the electronic device 100.
[0169] In addition, the electronic device 100 can use multiple cores (e.g., dual cores, quad cores, etc.) included in one processor to perform calculations on functions associated with artificial intelligence. For example, the electronic device can use multiple cores included in the processor to perform convolution calculations in parallel. One or more processors can be controlled to process input data according to predefined operating rules or artificial intelligence models stored in the memory. The predefined operating rules or artificial intelligence models are characterized by being created through learning.
[0170] The creation by learning mentioned here refers to creating the predefined operating rules of the desired features or artificial intelligence model by applying the learning algorithm to multiple learning data. The learning can be performed in the machine itself that executes the artificial intelligence according to the present disclosure, or by a separate server and / or system.
[0171] The artificial intelligence model may be configured by multiple neural network layers. Each layer may have multiple weight values, and the processing of the layer may be performed by processing the processing result of the previous layer and the multiple weight values. Examples of neural networks may include convolutional neural networks (CNNs), deep neural networks (DNNs), recurrent neural networks (RNNs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), and deep-Q networks, and unless otherwise specified, the neural networks of the present disclosure are not limited to the above examples.
[0172] The learning algorithm may be a method for a predetermined target machine to use a plurality of learning data to make decisions or predictions by itself to train the predetermined target machine (e.g., a robot). Examples of the learning algorithm may include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, and unless otherwise specified, the learning algorithm of the present disclosure is not limited to the above examples.
[0173] The term "component" or "module" used in the present disclosure may include a unit configured as hardware, software or firmware or any combination thereof, and may be used interchangeably with terms such as, for example, but not limited to, logic, logic block, component, circuit, etc. A "component" or "module" may be a minimum unit or part of a component formed as a whole or a component that performs one or more functions. For example, a module may be configured as an application specific integrated circuit (ASIC).
[0174] Various embodiments of the present disclosure may be implemented with software including instructions stored in a machine-readable storage medium (e.g., a computer). The machine may call the stored instructions from the storage medium, and as a device operable according to the called instructions, may include the electronic device 100 according to the above-described embodiments. Based on the instructions executed by the processor, the processor may use other elements directly or under the control of the processor to perform functions corresponding to the instructions. The instructions may include code generated by a compiler or executed by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. "Non-transitory" storage media are tangible and may not include signals, and the term does not distinguish between data stored semi-permanently or temporarily in the storage medium.
[0175] According to an embodiment, a method according to various embodiments described in the present disclosure including a computer program product may be provided. The computer program product may be exchanged between a seller and a buyer as a commodity. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disk read-only memory (CD-ROM)), or distributed online through an application store (e.g., PLAYSTORE). In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored in a storage medium such as a manufacturer's server, an application store's server, or a memory of a relay server, or may be temporarily generated.
[0176] Each element (e.g., module or program) according to various embodiments may be formed as a single entity or multiple entities, and some of the above-mentioned sub-elements may be omitted, or other sub-elements may be further included in various embodiments. Alternatively or additionally, some elements (e.g., modules or programs) may be integrated into one entity to perform the same or similar functions performed by the corresponding elements before integration. According to various embodiments, the operations performed by a module, program, or another element may be performed sequentially, in parallel, repeatedly, or in a heuristic manner, or at least some operations may be performed in a different order, omitted, or different operations may be added.
[0177] Although the present disclosure has been shown and described with reference to various exemplary embodiments, it should be understood that the various exemplary embodiments are intended to be illustrative rather than restrictive. Those skilled in the art will further understand that various changes in form and details may be made without departing from the true spirit and full scope of the present disclosure (including the appended claims and their equivalents). It should also be understood that any embodiment described herein may be used in combination with any other embodiment described herein.
Claims
1. An electronic device, include: a memory including at least one instruction; as well as a processor coupled to the memory and configured to control the electronic device, The processor is configured to: Get moving images, identifying a person and gesture data of the person from a plurality of frames in the moving image, obtaining exercise pattern information corresponding to the plurality of frames using the recognized person's posture data, recognizing an exercise movement of the identified person by inputting at least one frame of the moving image into at least one neural network model, and obtaining exercise feature information corresponding to the recognized exercise movement, identifying a first frame interval and a second frame interval different from the first frame interval from the plurality of frames in the moving image based on the exercise characteristic information and the exercise pattern information, and Information on an exercise motion corresponding to the second frame interval is provided by comparing the first frame interval with the second frame interval.
2. The electronic device according to claim 1, wherein The processor is configured to: inputting at least one frame of the moving image into a first neural network model configured to identify whether a person's exercise motion is a symmetrical motion, and identifying whether the person's exercise motion is a symmetrical motion, and At least one frame of the moving image is input into a second neural network model configured to identify whether the person's exercise motion is a hold motion, and identify whether the person's exercise motion is a hold motion.
3. The electronic device according to claim 2, in: The first neural network model is configured to be trained based on a learning moving image capturing an exercise movement of a person and information about whether the exercise movement of the person corresponding to the learning moving image is a symmetrical movement, and The second neural network model is configured to be trained based on a learning moving image capturing an exercise motion of a person and information about whether an exercise motion of the person corresponding to the learning moving image is a holding motion.
4. The electronic device according to claim 1, wherein The processor is configured to: Based on the person's exercise motion being a symmetrical motion and a holding motion, an interval from a start frame to a first time is identified from the plurality of frames in the moving image as the first frame interval.
5. The electronic device according to claim 1, wherein The processor is configured to: identifying each exercise repetition interval from the plurality of frames in the moving image based on the exercise pattern information based on the person's exercise motion being a symmetrical motion and a non-hold motion, and Based on the identified exercise repetition interval, an interval from a start frame to a frame corresponding to a first number of exercise repetition intervals is identified from among the plurality of frames in the moving image as the first frame interval.
6. The electronic device according to claim 1, wherein The processor is configured to: Based on the fact that the person's exercise motion is an asymmetric motion and a holding motion exercise, frames corresponding to an interval from a frame where a first symmetrical exercise motion is first started to a second time and an interval from a frame where a second symmetrical exercise motion is first started to a second time are identified from the multiple frames in the moving image based on the exercise pattern information as the first frame interval.
7. The electronic device according to claim 1, wherein The processor is configured to: Based on the person's exercise motion being an asymmetric motion and a non-holding motion, identifying each frame corresponding to a first symmetric exercise motion and a second symmetric exercise motion from the plurality of frames in the moving image based on the exercise pattern information, and A frame corresponding to a second number of first symmetrical exercise movements from a start frame and a frame corresponding to a second number of second symmetrical exercise movements are identified from the plurality of frames in the moving image as the first frame interval.
8. The electronic device according to claim 1, wherein The processor is configured to: Based on the moving image being played, a frame at a specified interval from the played frame is identified as the second frame.
9. The electronic device according to claim 1, wherein The processor is configured to: A physical exhaustion state is identified based on a difference between exercise pattern information corresponding to the first frame interval and exercise pattern information of the second frame interval being greater than or equal to a specified value, and information about the person's exercise movement corresponding to the second frame interval is provided based on the identification of the physical exhaustion state.
10. The electronic device according to claim 1, wherein The processor is configured to: Get the original moving image, and The moving image is obtained by identifying a frame corresponding to the person's exercise motion from among a plurality of frames included in the original moving image, based on an exercise database (DB) storing a plurality of moving images including a plurality of exercise motions.
11. A method for controlling an electronic device, the method include: Obtaining moving images; identifying a person and gesture data of the person from a plurality of frames in the moving image; obtaining exercise pattern information corresponding to the plurality of frames using the recognized posture data of the person; recognizing an exercise motion of the identified person by inputting at least one frame of the moving image into at least one neural network model, and obtaining exercise feature information corresponding to the recognized exercise motion; identifying, from the plurality of frames in the moving image, a first frame interval and a second frame interval different from the first frame interval based on the exercise characteristic information and the exercise pattern information; as well as Information about an exercise motion corresponding to the second frame is provided by comparing the first frame interval with the second frame interval.
12. The method according to claim 11, wherein Get exercise characteristics information include: inputting at least one frame of the moving image into a first neural network model for identifying whether a person's exercise motion is a symmetrical motion, and identifying whether the person's exercise motion is a symmetrical motion, and At least one frame of the moving image is input into a second neural network model for identifying whether the person's exercise motion is a hold motion, and identifying whether the person's exercise motion is a hold motion.
13. The method according to claim 12, in: The first neural network model is configured to be trained based on a learning moving image capturing an exercise movement of a person and information about whether the exercise movement of the person corresponding to the learning moving image is a symmetrical movement, and The second neural network model is configured to be trained based on a learning moving image capturing an exercise motion of a person and information about whether an exercise motion of the person corresponding to the learning moving image is a holding motion.
14. The method according to claim 11, wherein identifying a first frame interval and a second frame interval different from the first frame interval include: Based on the person's exercise motion being a symmetrical motion and a holding motion, an interval from a start frame to a first time is identified from the plurality of frames in the moving image as the first frame interval.
15. The method according to claim 11, wherein identifying a first frame interval and a second frame interval different from the first frame interval include: identifying each exercise repetition interval from the plurality of frames in the moving image based on exercise pattern information based on the person's exercise motion being a symmetrical motion and a non-hold motion, and Based on the identified exercise repetition interval, an interval from a start frame to a frame corresponding to a first number of exercise repetition intervals is identified from among the plurality of frames in the moving image as the first frame interval.
