Electronic device for estimating posture and control method therefor

Abstract

This method for controlling an electronic device comprises the steps of: acquiring a plurality of pieces of sensing data acquired from a plurality of sensors while a user moves; acquiring a plurality of embedding vectors by embedding the plurality of pieces of sensing data at a plurality of time intervals; estimating the posture of the user on the basis of a correlation between the plurality of embedding vectors; and guiding the motion of the user on the basis of the estimated posture of the user.

Description

Electronic device for attitude estimation and its control method

[0001] The present disclosure relates to an electronic device and a control method thereof, and more particularly, to an electronic device for estimating a user's posture and guiding the user's posture, and a control method thereof.

[0002] Existing exercise recommendation systems monitor the user's heart rate and suggest exercise times, routes, and target walking speeds to help maintain an optimal heart rate.

[0003] In particular, for exercises recommended for the elderly, such as walking, the importance of stability and walking posture is increasing.

[0004] There is a growing need to analyze users' posture and exercise intensity while exercising and recommend customized exercises to users.

[0005] A method for controlling an electronic device according to one or more embodiments of the present disclosure includes the steps of acquiring a plurality of sensing data obtained from a plurality of sensors while a user moves, embedding the plurality of sensing data according to a plurality of time intervals to acquire a plurality of embedding vectors, estimating a posture of the user based on a correlation between the plurality of embedding vectors, and guiding a movement of the user based on the estimated posture of the user.

[0006] The step of estimating the posture of the user may estimate the posture of the user by using at least one embedding vector among the plurality of embedding vectors, the value indicating the degree of correlation with the posture of the user being greater than or equal to a preset value.

[0007] Among the plurality of embedding vectors, a first embedding vector may correspond to a first time interval, and a second embedding vector may correspond to a second time interval different from the first time interval.

[0008] The above control method may further include a step of training an attention layer to learn correlations between sensing data acquired from a plurality of sensors.

[0009] Each of the plurality of sensing data corresponds to each of the plurality of domains, and the control method may further include a step of obtaining a correlation between embedding vectors belonging to the same domain among the plurality of domains and a step of obtaining a correlation between embedding vectors belonging to different domains among the plurality of domains.

[0010] The above control method may further include a step of identifying whether the plurality of sensing data satisfies a preset condition, and a step of estimating a posture of the user corresponding to the condition if the plurality of sensing data satisfies the preset condition.

[0011] The step of guiding the user's movement may include guiding the user's movement by comparing the estimated user's posture with reference data, and the reference data may be determined based on the user's age and gender.

[0012] Each of the plurality of sensors can sense the plurality of sensing data at a different sampling rate.

[0013] The above plurality of sensors may be acceleration sensors.

[0014] An electronic device according to one or more embodiments of the present disclosure includes a sensor for detecting a movement of the electronic device, a communication interface, a memory, and a processor, wherein the processor obtains a plurality of sensing data obtained from a plurality of sensors including the sensor while a user moves, embeds the plurality of sensing data according to a plurality of time intervals to obtain a plurality of embedding vectors, estimates a posture of the user based on a correlation between the plurality of embedding vectors, and guides an action of the user based on the estimated posture of the user.

[0015] The processor can estimate the user's posture by using at least one embedding vector among the plurality of embedding vectors, the value indicating the degree of correlation with the user's posture being greater than or equal to a preset value.

[0016] Among the plurality of embedding vectors, a first embedding vector may correspond to a first time interval, and a second embedding vector may correspond to a second time interval different from the first time interval.

[0017] The above processor can train an attention layer to learn correlations between sensing data acquired from multiple sensors.

[0018] Each of the plurality of sensing data corresponds to each of the plurality of domains, and the processor can obtain a correlation between embedding vectors belonging to the same domain among the plurality of domains and obtain a correlation between embedding vectors belonging to different domains among the plurality of domains.

[0019] The processor can identify whether the plurality of sensing data satisfies a preset condition, and if the plurality of sensing data satisfies the preset condition, estimate the user's posture corresponding to the condition.

[0020] The processor guides the user's movements by comparing the estimated user's posture with reference data, and the reference data can be determined based on the user's age and gender.

[0021] Each of the plurality of sensors can sense the plurality of sensing data at a different sampling rate.

[0022] The above plurality of sensors may be acceleration sensors.

[0023] A non-transitory computer-readable recording medium comprising a program for executing a control method of an electronic device according to one or more embodiments of the present disclosure, wherein the control method comprises the steps of: acquiring a plurality of sensing data obtained from a plurality of sensors while a user moves; acquiring a plurality of embedding vectors by embedding the plurality of sensing data according to a plurality of time intervals; estimating a posture of the user based on a correlation between the plurality of embedding vectors; and guiding a movement of the user based on the estimated posture of the user.

[0024] The step of estimating the posture of the user may estimate the posture of the user by using at least one embedding vector among the plurality of embedding vectors, the value indicating the degree of correlation with the posture of the user being greater than or equal to a preset value.

[0025] FIG. 1 is a drawing illustrating a user posture guide system according to one or more embodiments of the present disclosure.

[0026] FIG. 2 is a block diagram illustrating a configuration of an electronic device according to one or more embodiments of the present disclosure.

[0027] FIG. 3 is a diagram illustrating the structure of an artificial intelligence model according to one or more embodiments of the present disclosure.

[0028] FIG. 4 is a diagram illustrating an embedding layer and a first attention layer of an artificial intelligence model according to one or more embodiments of the present disclosure.

[0029] FIG. 5 is a diagram illustrating the correlation of embedding vectors according to one or more embodiments of the present disclosure.

[0030] FIG. 6 is a diagram illustrating a second attention layer of an artificial intelligence model according to one or more embodiments of the present disclosure.

[0031] FIG. 7 is a diagram illustrating the correlation of embedding vectors according to one or more embodiments of the present disclosure.

[0032] FIG. 8 is a diagram illustrating the correlation of embedding vectors according to one or more embodiments of the present disclosure.

[0033] FIG. 9 is a flowchart illustrating a method for an electronic device to train an artificial intelligence model according to one or more embodiments of the present disclosure.

[0034] FIG. 10 and FIG. 11 are diagrams illustrating a second artificial intelligence model according to one or more embodiments of the present disclosure.

[0035] FIG. 12 is a flowchart illustrating a method for an electronic device according to one or more embodiments of the present disclosure to guide a user's posture using an artificial intelligence model.

[0036] FIG. 13 is a diagram illustrating a method for guiding a user's posture by an electronic device according to one or more embodiments of the present disclosure.

[0037] FIG. 14 is a flowchart illustrating a method for estimating a user's posture based on whether a condition of acquired sensing data is satisfied by an electronic device according to one or more embodiments of the present disclosure.

[0038] FIG. 15 is a flowchart illustrating a method for controlling an electronic device according to one or more embodiments of the present disclosure.

[0039] The present embodiments may be modified and have various embodiments. Therefore, specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the scope to specific embodiments, but should be understood to encompass various modifications, equivalents, and / or alternatives of the embodiments of the present disclosure. In connection with the description of the drawings, similar reference numerals may be used for similar components.

[0040] In describing the present disclosure, if it is determined that a specific description of a related known function or configuration may unnecessarily obscure the gist of the present disclosure, a detailed description thereof will be omitted.

[0041] Additionally, the following embodiments may be modified in various other forms, and the scope of the technical concepts of the present disclosure is not limited to the following embodiments. Rather, these embodiments are provided to further faithfully and completely convey the technical concepts of the present disclosure to those skilled in the art.

[0042] The terminology used in this disclosure is for the purpose of describing specific embodiments only and is not intended to limit the scope of the rights. Singular expressions include plural expressions unless the context clearly dictates otherwise.

[0043] In this disclosure, expressions such as “has,” “can have,” “includes,” or “may include” indicate the presence of a corresponding feature (e.g., a component such as a number, function, operation, or part), and do not exclude the presence of additional features.

[0044] In this disclosure, expressions such as “A or B,” “at least one of A and / or B,” or “one or more of A or / and B” can include all possible combinations of the listed items. For example, “A or B,” “at least one of A and B,” or “at least one of A or B” can all refer to (1) including at least one A, (2) including at least one B, or (3) including both at least one A and at least one B.

[0045] The expressions “first,” “second,” “first,” or “second,” etc., used in this disclosure can describe various components, regardless of order and / or importance, and are only used to distinguish one component from another, but do not limit the components.

[0046] When it is said that a component (e.g., a first component) is “(operatively or communicatively) coupled with / to” or “connected to” another component (e.g., a second component), it should be understood that said component may be directly coupled to said other component, or may be coupled via another component (e.g., a third component).

[0047] On the other hand, when it is said that a component (e.g., a first component) is "directly connected" or "directly connected" to another component (e.g., a second component), it can be understood that no other component (e.g., a third component) exists between said component and said other component.

[0048] The expression "configured to" used in the present disclosure may be used interchangeably with, for example, "suitable for," "having the capacity to," "designed to," "adapted to," "made to," or "capable of." The term "configured to" may not necessarily mean only "specifically designed to" in terms of hardware.

[0049] Instead, in some contexts, the phrase "a device configured to" may mean that the device is "capable of" doing something in conjunction with other devices or components. For example, the phrase "a processor configured (or set) to perform A, B, and C" may mean a dedicated processor (170) for performing the actions, or a general-purpose processor (170) (e.g., a CPU or application processor) that can perform the actions by executing one or more software programs stored in a memory device.

[0050] In the embodiments, a 'module' or 'part' performs at least one function or operation, and may be implemented as hardware or software, or as a combination of hardware and software. Furthermore, a plurality of 'modules' or 'parts' may be integrated into at least one module and implemented as at least one processor, except for a 'module' or 'part' that needs to be implemented as a specific hardware.

[0051] Meanwhile, the various elements and areas in the drawings are schematically drawn. Therefore, the technical concept of the present invention is not limited by the relative sizes or spacing depicted in the attached drawings.

[0052]

[0053] *Hereinafter, with reference to the attached drawings, embodiments according to the present disclosure are described in detail so that a person having ordinary knowledge in the technical field to which the present disclosure pertains can easily carry out the present disclosure.

[0054] FIG. 1 is a drawing illustrating a user posture guide system according to one or more embodiments of the present disclosure.

[0055] A user posture guide system (1) according to one or more embodiments of the present disclosure may include an electronic device (100), a first external device (200), and a second external device (300).

[0056] According to one or more embodiments of the present disclosure, the electronic device (100) may be a smartphone, the first external device (200) may be a smart watch, and the second external device (300) may be earphones, but is not limited thereto, and the electronic device (100), the first external device (200), and the second external device (300) may be implemented as various types of devices.

[0057] Each of the electronic device (100), the first external device (200), and the second external device (300) may include a first sensor, a second sensor, and a third sensor for detecting movement of the device. The sensors included in each of the devices (100, 200, 300) may detect movement of the device (100, 200, 300) while the user moves while carrying the device (100, 200, 300) (e.g., while the user is exercising by walking).

[0058] In the present disclosure, the movement of the device may correspond to the movement of a specific body part of the user. For example, if the electronic device (100) is a smartphone located in the user's lower body pocket, the movement of the electronic device (100) may indicate the movement of one of the user's legs. In addition, if the first external device (200) is a smartwatch worn on the user's left arm, the movement of the first external device (200) may indicate the movement of the user's left arm. In addition, if the second external device (300) is earphones worn on both ears of the user, the movement of the second external device (300) may indicate the movement of both ears of the user. That is, in the present disclosure, "movement of the device" may be replaced with "movement of the user."

[0059] In the present disclosure, a sensor for detecting the movement of a device (100, 200, 300) may be an inertial measurement unit (IMU) sensor. Here, the inertial measurement unit may include at least one of an acceleration sensor, a gyroscope sensor, and a geomagnetic sensor. The inertial measurement unit may include a three-axis accelerometer and / or a three-axis gyrometer. The sensor for detecting the movement of the device (100, 200, 300) is not limited to an inertial measurement unit, and may be implemented as various types of sensors for detecting the movement of the device (100, 200, 300).

[0060] Additionally, at least one of the electronic device (100), the first external device (200), and the second external device (300) may include a heart rate sensor for detecting the user's heart rate. For example, the first external device (200) implemented as a smartwatch may include a fourth sensor for detecting the user's heart rate.

[0061] In the present disclosure, a heart rate sensor can detect biosignals generated by a user's heartbeat and convert them into electrical signals. The heart rate sensor can detect changes in blood flow due to the heartbeat using optical, electrical, or mechanical sensing methods. In the present disclosure, the heart rate sensor may be a photoplethysmogram (PPG) sensor, but is not limited thereto, and may be implemented as various types of sensors for sensing the user's heart rate.

[0062] The electronic device (100) can sense the movement of the electronic device (100) through the first sensor and receive first sensing data.

[0063] In addition, the electronic device (100) can receive second sensing data from the first external device (200) that detects the movement of the first external device (200) through the second sensor.

[0064] In addition, the electronic device (100) can receive third sensing data from the second external device (300) that detects the movement of the second external device (300) through the third sensor.

[0065] In addition, the electronic device (100) can receive fourth sensing data detecting the user's heart rate from the first external device (200).

[0066] That is, the electronic device (100) can obtain multiple sensing data sensed by multiple sensors.

[0067] In the present disclosure, the plurality of sensing data may be time series data.

[0068] Additionally, each of the plurality of sensors can acquire sensing data at a different sampling rate.

[0069] For example, the electronic device (100) can detect the movement of the electronic device (100) at a 50 Hz sampling rate to obtain first sensing data. The first external device (200) can detect the movement of the first external device (200) at a 100 Hz sampling rate to obtain second sensing data. The second external device (300) can detect the movement of the second external device (300) at a 150 Hz sampling rate to obtain third sensing data. In addition, the first external device (200) can detect the user's heart rate at a 120 Hz sampling rate to obtain fourth sensing data.

[0070] An electronic device (100) according to the present disclosure can estimate a user's posture using a plurality of acquired sensing data.

[0071] In addition, the electronic device (100) according to the present disclosure can estimate the user's heart rate using sensing data that detects the user's heart rate.

[0072] In addition, the electronic device (100) according to the present disclosure can estimate the user's heart rate more accurately by using sensing data that detects the user's heart rate and a plurality of sensing data that detects the user's movement. In the present disclosure, the user's posture may mean, but is not limited to, the position of the user's main joints (elbows, shoulders, knees, or neck), the user's gait judgment index, the user's posture state, the user's movement state, or the user's gait state.

[0073] According to one or more embodiments of the present disclosure, the electronic device (100) can input a plurality of sensing data into an artificial intelligence model (111) to acquire the positions of the user's major joints. In addition, the electronic device (100) can acquire a gait determination index of the user using the positions of the user's major joints.

[0074] Alternatively, the electronic device (100) can input multiple sensing data into an artificial intelligence model (111) to obtain a user's gait judgment index.

[0075] In the present disclosure, the gait determination index may include at least one of walking symmetry, arm swing magnitude, stride time, stance time, swing time, step time, stride length, step length, gait speed, and gait variability. Based on the estimated user's posture, the electronic device (100) may provide information guiding the user's posture or motion. In addition, based on the estimated user's posture and the user's heart rate, the electronic device (100) may provide information guiding the user's posture or motion.

[0076] FIG. 2 is a block diagram illustrating the configuration of an electronic device (100) according to one or more embodiments of the present disclosure.

[0077] Referring to FIG. 2, the electronic device (100) may include at least one of a memory (110), a communication interface (120), a user interface (130), a display (140), a speaker (150), a sensor (160), and a processor (170). Some of the above components may be omitted, and the electronic device (100) may further include other components in addition to the above components. For example, components other than the memory (110), the communication interface (120), and the processor (170) may be omitted.

[0078] As described above, the electronic device (100) may be implemented as a smartphone, but this is only one embodiment, and may be implemented in various forms, such as a server, a TV, a smart TV, a set-top box, a mobile phone, a PDA (personal digital assistant), a laptop, a media player, an e-book reader, a digital broadcasting terminal, a navigation device, a kiosk, an MP3 player, a wearable device, a home appliance, and other mobile or non-mobile computing devices.

[0079] The memory (110) can store at least one instruction regarding the electronic device (100). The memory (110) can store an operating system (O / S) for driving the electronic device (100). In addition, the memory (110) can store various software programs or applications for operating the electronic device (100) according to various embodiments of the present disclosure. In addition, the memory (110) can include a semiconductor memory such as a flash memory (110) or a magnetic storage medium such as a hard disk.

[0080] Specifically, the memory (110) can store various software modules for operating the electronic device (100) according to various embodiments of the present disclosure, and the processor (170) can control the operation of the electronic device (100) by executing various software modules stored in the memory (110). That is, the memory (110) is accessed by the processor (170), and data reading / recording / modifying / deleting / updating, etc. can be performed by the processor (170).

[0081] Meanwhile, in the present disclosure, the term memory (110) may be used to mean a memory (110), a ROM (not shown), a RAM (not shown) in a processor (170), or a memory card (not shown) (e.g., a micro SD card, a memory stick) mounted on an electronic device (100).

[0082] According to one or more embodiments of the present disclosure, the memory (110) may store an artificial intelligence model (111). In the present disclosure, the artificial intelligence model (111) may be a model that estimates the movement of a user when sensing data sensing the movement of a device carried by the user is input.

[0083] According to one or more embodiments of the present disclosure, the memory (110) may store information about the user. In this case, the information about the user may include at least one of the user's age, the user's gender, the user's height, the user's weight, the user's body fat percentage, and the user's BMI.

[0084] And, the communication interface (120) includes a circuitry and is a configuration capable of communicating with external devices and servers. The communication interface (120) can communicate with external devices or servers based on a wired or wireless communication method. The communication interface (120) may include a Bluetooth module (not shown), a Wi-Fi module (not shown), an IR (infrared) module, a LAN (Local Area Network) module, an Ethernet module, etc. Here, each communication module may be implemented in the form of at least one hardware chip. In addition to the above-described communication method, the wireless communication module may include at least one communication chip that performs communication according to various wireless communication standards such as Zigbee, USB (Universal Serial Bus), MIPI CSI (Mobile Industry Processor Interface Camera Serial Interface), 3G (3rd Generation), 3GPP (3rd Generation Partnership Project), LTE (Long Term Evolution), LTE-A (LTE Advanced), 4G (4th Generation), 5G (5th Generation), etc. However, this is only one embodiment, and the communication interface (120) can utilize at least one communication module among various communication modules.

[0085] According to one or more embodiments of the present disclosure, the communication interface (120) can receive sensing data from an external device.

[0086] According to one or more embodiments of the present disclosure, the communication interface (120) can receive sensing data sensing movement of the external device from an external device including a sensor capable of detecting movement of the external device.

[0087] The sensing data may be sensing data that senses the movement of the device.

[0088] According to one or more embodiments of the present disclosure, the communication interface (120) can receive sensing data sensing a user's heart rate from an external device including a sensor capable of sensing a user's heart rate.

[0089] The user interface (130) may be implemented as a device such as a button, a touch pad, a mouse, and a keyboard, or as a touch screen capable of performing the aforementioned display and operation input functions. Here, the button may be a mechanical button, a touch pad, a wheel, or any other type of button formed on any area of ​​the front, side, or back of the main body of the electronic device (100).

[0090] According to one or more embodiments of the present disclosure, the user interface (130) may obtain user input for entering information about the user.

[0091] The display (140) may be implemented as a display of various forms, such as a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display panel (PDP), etc. The display (140) may also include a driving circuit, a backlight unit, etc., which may be implemented as a form, such as an a-si TFT (amorphous silicon thin film transistor), an LTPS (low temperature poly silicon) TFT, an OTFT (organic TFT), etc. Meanwhile, the display (140) may be implemented as a touch screen combined with a touch sensor, a flexible display (140), a three-dimensional display (140), etc. In addition, according to an embodiment of the present disclosure, the display (140) may include a bezel that houses the display panel as well as a display panel that outputs an image. In particular, according to an embodiment of the present disclosure, the bezel may include a touch sensor for detecting user interaction.

[0092] According to one or more embodiments of the present disclosure, the display (140) may display information about the user's posture, information for guiding the user's posture, or information for guiding the user's movements.

[0093] The speaker (150) is a component for outputting an audio signal. In particular, the speaker (150) may include an audio output mixer, an audio signal processor, and an audio output module. The audio output mixer may synthesize a plurality of audio signals to be output into at least one audio signal. For example, the audio output mixer may synthesize an analog audio signal and another analog audio signal (e.g., an analog audio signal received from an external source) into at least one analog audio signal. The audio output module may include a speaker or an output terminal.

[0094] According to one or more embodiments of the present disclosure, the speaker (150) may output audio about the user's posture, audio to guide the user's posture, or audio to guide the user's movements.

[0095] The sensor (160) may include a sensor for sensing the movement of the electronic device (100). The sensor (160) may include an acceleration sensor, a gyroscope sensor, or a geomagnetic sensor. The sensor (160) may sense the linear acceleration, rotation, and angular velocity of the device.

[0096] The sensor (160) may include a sensor for sensing the user's heart rate. The sensor (160) may include a heart rate sensor or a PPG sensor. The sensor (160) may sense the user's heart rate using an optical signal or an electrical signal.

[0097] Meanwhile, the sensor (160) according to the present disclosure may be included in the first external device (200) as well as the second external device (300) in addition to the electronic device (100).

[0098] The processor (170) can control the overall operation and function of the electronic device (100). Specifically, the processor (170) is connected to the configuration of the electronic device (100) including the memory (110), and can control the overall operation of the electronic device (100) by executing at least one command stored in the memory (110) as described above.

[0099] The processor (170) may be implemented in various ways. For example, the processor (170) may be implemented as at least one of an application specific integrated circuit (ASIC), a logic integrated circuit, an embedded processor, a microcomputer (Micom), a microprocessor, hardware control logic, a hardware finite state machine (FSM), and a digital signal processor (170).

[0100] In particular, the processor (170) may include one or more processors. Specifically, the one or more processors may include one or more of a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), an Accelerated Processing Unit (APU), a Many Integrated Core (MIC), a Digital Signal Processor (DSP), a Neural Processing Unit (NPU), a Main Processing Unit (MPU), a hardware accelerator, or a machine learning accelerator. The one or more processors may control one or any combination of other components of the electronic device, and may perform operations related to communication or data processing. The one or more processors may execute one or more programs or instructions stored in a memory. For example, the one or more processors may perform a method according to an embodiment of the present disclosure by executing one or more instructions stored in a memory.

[0101] When a method according to one or more embodiments of the present disclosure includes multiple operations, the multiple operations may be performed by one processor or by multiple processors. That is, when a first operation, a second operation, and a third operation are performed by a method according to one or more embodiments, the first operation, the second operation, and the third operation may all be performed by the first processor, or the first operation and the second operation may be performed by the first processor (170) and the third operation may be performed by the second processor (170).

[0102] One or more processors may be implemented as a single-core processor (170) including one core, or may be implemented as one or more multi-core processors (170) including multiple cores (e.g., homogeneous multi-cores or heterogeneous multi-cores). When one or more processors are implemented as a multi-core processor, each of the multiple cores included in the multi-core processor may include internal processor memory, such as cache memory or on-chip memory, and a common cache shared by the multiple cores may be included in the multi-core processor. In addition, each of the multiple cores (or some of the multiple cores) included in the multi-core processor may independently read and execute a program instruction for implementing a method according to one or more embodiments of the present disclosure, or all (or some) of the multiple cores may be linked to read and execute a program instruction for implementing a method according to one or more embodiments of the present disclosure.

[0103] When a method according to one or more embodiments of the present disclosure includes a plurality of operations, the plurality of operations may be performed by one core among the plurality of cores included in a multi-core processor, or may be performed by the plurality of cores. For example, when a first operation, a second operation, and a third operation are performed by a method according to one or more embodiments, the first operation, the second operation, and the third operation may all be performed by a first core included in the multi-core processor, or the first operation and the second operation may be performed by a first core included in the multi-core processor, and the third operation may be performed by a second core included in the multi-core processor.

[0104] In embodiments of the present disclosure, the processor (170) may mean a system on a chip (SoC) in which one or more processors and other electronic components are integrated, a single-core processor, a multi-core processor, or a core included in a single-core processor or a multi-core processor, wherein the core may be implemented as a CPU, a GPU, an APU, a MIC, a DSP, an NPU, a hardware accelerator, or a machine learning accelerator, but embodiments of the present disclosure are not limited thereto.

[0105] The operation of the processor (170) for implementing various embodiments of the present disclosure may be implemented through multiple modules.

[0106] Specifically, data for a plurality of modules according to the present disclosure can be stored in a memory (110), and the processor (170) can access the memory (110) to load the data for the plurality of modules into a memory or buffer within the processor (170), and then implement various embodiments according to the present disclosure using the plurality of modules.

[0107] However, at least one of the plurality of modules according to the present disclosure may be implemented in hardware and included in the processor (170) in the form of a system on chip.

[0108] Alternatively, at least one of the plurality of modules according to the present disclosure may be implemented as a separate external device, and the electronic device (100) and each module may communicate and perform operations according to the present disclosure.

[0109] According to one or more embodiments of the present disclosure, the processor (170) may control the operation of the electronic device (100) described with reference to the drawings below. At this time, the processor (170) may control at least one of the components of the electronic device (100) so that the electronic device (100) performs each operation.

[0110] Hereinafter, the operation of the processor (170) according to the present disclosure will be described in detail with reference to the attached drawings.

[0111] FIG. 3 is a diagram illustrating the structure of an artificial intelligence model according to one or more embodiments of the present disclosure.

[0112] The artificial intelligence model (111) of the present disclosure may include an embedding layer (112), a first attention layer (113), a second attention layer (114), and an output layer (115).

[0113] The embedding layer (112) can embed data input to the embedding layer (112) and output an embedding vector. Here, embedding can refer to an operation of mapping high-dimensional data into a low-dimensional vector space. In other words, the embedding layer can output an embedding vector that includes the characteristics of the input data.

[0114] In the present disclosure, “embedding layer” may be replaced with an expression representing the same / similar concept, such as “input layer.”

[0115]

[0116] *In this disclosure, “embedding vector” may be replaced with expressions representing the same / similar concept, such as “embedding,” “feature vector,” or “feature.”

[0117] The electronic device (100) can divide sensing data (10, 20, 30), which is time series data, into multiple segments according to multiple time intervals.

[0118] Here, the sensing data may be sensing data that senses the movement of the device or sensing data that senses the user's heart rate.

[0119] In the present disclosure, “time interval” may be replaced with expressions representing the same / similar concepts, such as “window size”, “interval”, and “frame size”.

[0120] At this time, the first sensing data (10), the second sensing data (20), and the third sensing data (30) may each belong to different domains. That is, the domain of the sensing data may correspond to the type of device that collects the sensing data.

[0121] At this time, the sensing data sensed by the first type of device may belong to the first domain, the sensing data sensed by the second type of device may belong to the second domain, and the sensing data sensed by the third type of device may belong to the third domain.

[0122] That is, the first domain corresponds to sensing data sensed in the first type of device, the second domain corresponds to sensing data sensed in the second type of device, and the third domain may correspond to sensing data sensed in the third type of device.

[0123] According to one or more embodiments of the present disclosure, the first type of device may mean a smartphone, the second type of device may mean a smartwatch, and the third type of device may mean earphones, but is not limited thereto.

[0124] Meanwhile, the domain of sensing data can correspond to the type of data sensed by the sensor.

[0125] For example, in a first type of device, sensing data detecting the movement of the device may belong to a first domain, sensing data detecting the user's heart rate in the first type of device may belong to a second domain, and sensing data detecting the movement of the device in the second type of device may belong to a third domain.

[0126] An electronic device (100) can input a plurality of segments into an embedding layer (112) to obtain a plurality of embedding vectors. At this time, the embedding vectors can include features of the plurality of segments.

[0127] Meanwhile, according to one or more embodiments of the present disclosure, the electronic device (100) may input three pieces of sensing data into the artificial intelligence model (111) as illustrated in FIG. 3, but this is only one embodiment, and the electronic device (100) may input less than three pieces of sensing data or more than three pieces of sensing data into the artificial intelligence model (111).

[0128] In the present disclosure, an attention layer may mean a layer that outputs a correlation between input data.

[0129] In the present disclosure, “correlation” may be replaced with expressions of the same / similar concept, such as “correlation,” “attention,” “association,” “relevance,” “interrelation,” “dependency,” or “interdependency.”

[0130] The first attention layer (113) can output a time-series correlation between input data.

[0131] Specifically, the first attention layer (113) can output a time-series correlation between multiple embedding vectors when multiple segment embedding vectors of a specific domain are input.

[0132] That is, when a plurality of embedding vectors corresponding to sensing data acquired from a specific sensor are input, the first attention layer (113) can output a time-series correlation between the plurality of embedding vectors.

[0133] In the present disclosure, the first attention layer (113) can be implemented as a multi-head attention structure in which multiple self-attention layers are overlapped.

[0134] For example, as illustrated in FIG. 4, the electronic device (100) can divide the sensing data (10) of the first domain, which is time-series data, into a first time interval (e.g., 1-second interval) to obtain a plurality of first segments (410). Then, the electronic device (100) can divide the sensing data (10) of the first domain into a second time interval (e.g., 2-second interval) to obtain a plurality of second segments (412). Then, the electronic device (100) can divide the time-series data (10) of the first domain into a third time interval (e.g., 4-second interval) to obtain a plurality of third segments (413).

[0135] The electronic device (100) can input each of a plurality of first segments (411), a plurality of second segments (412), and a plurality of third segments (413) into the embedding layer (112) to obtain an embedding vector (421, 422, 423) of each segment.

[0136] The electronic device can input a plurality of first embedding vectors (421) corresponding to a plurality of first segments (411) into a first attention layer (113) to obtain a first attention vector (431) including a correlation between the plurality of first embedding vectors (421), and input a plurality of second embedding vectors (422) into the first attention layer (113) to obtain a second attention vector (432) including a correlation between the plurality of second segments (412). In addition, the electronic device (100) can input a plurality of third embedding vectors (423) into the first attention layer (113) to obtain a third attention vector (433) including a correlation between the plurality of third embedding vectors (423).

[0137] At this time, the electronic device (100) may sequentially input each of a plurality of first embedding vectors (421), a plurality of second embedding vectors (422), and a plurality of third embedding vectors (423) into the first attention layer (113), but is not limited thereto.

[0138] Specifically, when a plurality of first embedding vectors (421) are input, the first attention layer (113) can output a correlation between the plurality of first embedding vectors (421). And, when a plurality of second embedding vectors (422) are input, the first attention layer (113) can output a correlation between the plurality of second embedding vectors (422).

[0139] That is, when a plurality of first embedding vectors (421) are input, the first attention layer (113) can output a first correlation between the first embedding vector among the plurality of first embedding vectors (421) and each of the plurality of first embedding vectors (421), a second correlation between the second embedding vector among the plurality of first embedding vectors (421) and each of the plurality of first embedding vectors (421), …, an n-th correlation between the n-th embedding vector among the plurality of first embedding vectors (421) and each of the plurality of first embedding vectors (421).

[0140] For example, as illustrated in FIG. 5, the first attention vector (431) may include information about a correlation between one of the plurality of first embedding vectors (421) and the vector (510) and each of the plurality of first embedding vectors (421).

[0141] The electronic device (100) can perform the same operation for the second sensing data (20) belonging to the second domain and the third sensing data (30) belonging to the third domain as for the first sensing data (20).

[0142] Accordingly, the electronic device (100) can obtain an attention vector including information on the correlation between the embedding vectors of the second sensing data (20). In addition, the electronic device (100) can include an attention vector including information on the correlation between the embedding vectors of the third sensing data (30).

[0143] Referring to FIG. 6, the electronic device (100) can input the output (611, 612, 613) of the first attention layer to the second attention layer (114).

[0144] Here, the outputs (611, 612, 613) of the first attention layer (113) may include an attention vector (611) in which first sensing data (10) belonging to the first domain is output through the embedding layer (112) and the first attention layer (113), an attention vector (612) in which second sensing data (20) belonging to the second domain is output through the embedding layer (112) and the first attention layer (113), and an attention vector (613) in which third sensing data (30) belonging to the third domain is output through the embedding layer (112) and the first attention layer (113).

[0145] The second attention layer (114) can receive the output of the first attention layer (113) for each of multiple domains and output the correlation between the multiple domains.

[0146] At this time, the second attention layer (114) can be implemented as a multi-head attention structure in which multiple attention layers are overlapped.

[0147] Specifically, the second attention layer (114) can output a correlation between an embedding vector corresponding to a specific time section in a specific domain among multiple domains and embedding vectors in other domains.

[0148] Alternatively, the second attention layer (114) may output a correlation between an attention vector corresponding to a specific time interval in a specific domain among multiple domains and attention vectors in other domains.

[0149] Specifically, when a first attention vector of a first domain, a second attention vector of a second domain, and a third attention vector of a third domain are input, the second attention layer (114) can output a correlation between a plurality of first embedding vectors of the first domain, a plurality of second embedding vectors of the second domain, and a plurality of third embedding vectors of the third domain.

[0150] Alternatively, when a first attention vector of a first domain, a second attention vector of a second domain, and a third attention vector of a third domain are input, the second attention layer (114) can output a correlation between the attention vector of the first domain, the attention vector of the second domain, and the attention vector of the third domain.

[0151] For example, as illustrated in FIG. 7, the output (621) of the second attention layer may include a correlation between the embedding vector (430a) of the first domain and the embedding vectors (430a, 430b, 430c) of multiple domains.

[0152] At this time, if the time interval corresponding to a specific embedding vector of the first domain is 0 to 4 seconds, the output (621) of the second attention layer may include, but is not limited to, a correlation between the specific embedding vector of the first domain and each of the corresponding embedding vectors within 0 to 4 seconds in multiple domains.

[0153] Referring again to FIG. 3, the electronic device (100) can input the output of the second attention layer (114) to the output layer (115).

[0154] The output layer (115) can output data of a preset type when data is input. Here, the preset type of data may refer to data for estimating the user's posture. That is, the output layer (115) can output data for estimating the user's posture.

[0155] At this time, the estimated user's posture may mean, but is not limited to, the position of the user's major joints (elbows, shoulders, knees, or neck), the user's gait judgment indicator, the user's posture state, the user's movement state, or the user's gait state, as described above.

[0156] According to one or more embodiments of the present disclosure, the output layer (115) can output a gait determination index of the user.

[0157] Alternatively, the output layer (115) may output the positions of the user's major joints. Then, the electronic device (100) may obtain an index of the user's gait using the positions of the user's major joints.

[0158] According to one or more embodiments of the present disclosure, the gait determination indicator may be an indicator for measuring the user's gait.

[0159] Specifically, the output layer (115) can output an index for measuring the user's gait. Here, the index for measuring the gait may be an index for evaluating the user's walking posture.

[0160] In the present disclosure, the gait judgment index may include at least one of walking symmetry, arm swing magnitude, stride time, stance time, swing time, step time, stride length, step length, gait speed, and gait variability, as described above.

[0161] For example, the output layer (115) may output an indicator representing walking symmetry. The symmetry indicator may be expressed as a value between 0 and 1. The closer the symmetry indicator is to 1, the higher the symmetry.

[0162] Alternatively, the output layer (115) may output an indicator representing the magnitude of arm swing during walking. The indicator representing the magnitude of arm swing may be a value estimating the degree to which the arm swings in degrees.

[0163] Alternatively, the output layer (115) may output an indicator representing the stride length when walking. The indicator representing the stride length may be an estimated value representing the distance taken in one step when walking.

[0164] Alternatively, the output layer (115) may output an indicator representing the walking speed (cadence) during walking. The indicator representing the walking speed may be an estimated value of the gait speed during walking.

[0165] Alternatively, the output layer (115) may output an indicator representing the double support time during walking. The indicator representing the double support time may be an estimated value representing the time when both feet touch the ground simultaneously during walking.

[0166] Meanwhile, the output layer (115) can output various indicators for measuring the user's gait in addition to the indicators described above. Furthermore, the output layer (115) can output one or more of the indicators described above together.

[0167] The electronic device (100) of the present disclosure can obtain data for evaluating a user's posture by comparing the aforementioned gait determination indicators with reference data. Further details regarding this will be described later with reference to FIG. 10.

[0168] Meanwhile, an artificial intelligence model (111) that has learned the time-series correlation and inter-domain correlation of embedding vectors can estimate output data using embedding vectors that have a high correlation with the output data.

[0169] Meanwhile, in the present disclosure, the electronic device (100) is described as inputting the output of the first attention layer (113) to the second attention layer (114), but this is only one embodiment, and the electronic device (100) may input the output of the embedding layer (112) to the second attention layer (114) and input the output of the second attention layer (114) to the first attention layer. Thereafter, the electronic device (100) may input the output of the first attention layer to the output layer (115).

[0170] Alternatively, the first attention layer (113) and the second attention layer (114) may be configured in parallel. In this case, the electronic device (100) may input the output of the embedding layer (112) to each of the first attention layer (113) and the second attention layer (114), and may obtain an attention vector by performing a concatenation operation on the outputs of the first attention layer (113) and the second attention layer (114). In addition, the electronic device (100) may input the obtained attention vector to the output layer (115).

[0171] As described above, the electronic device (100) of the present disclosure can identify embedding vectors having a high correlation with the output of the output layer (115) through the learned artificial intelligence model (111), and estimate the user's posture using the identified embedding vectors.

[0172] Referring to FIG. 8, the artificial intelligence model (111) of the present disclosure can estimate output data using an embedding vector (810, 820, 830, 840) whose value indicating a correlation with the output data is greater than or equal to a preset first value.

[0173] At this time, the value indicating the correlation between the embedding vectors (810, 820, 830, 840) may also be greater than the second value.

[0174] Information about the above-described preset first value and preset second value may be stored in the memory (110).

[0175] Since the electronic device (100) according to the present disclosure obtains a plurality of embedding vectors by embedding a plurality of sensing data at a plurality of time intervals, the time intervals of the embedding vectors having a high correlation with the output data may be different for each domain, as illustrated in FIG. 8.

[0176] That is, the electronic device (100) according to the present disclosure can estimate output data using an embedding vector of a time interval optimized for each domain. Accordingly, the electronic device (100) can estimate the user's posture more accurately and efficiently.

[0177] In addition, the electronic device (100) can train the artificial intelligence model (111) so that the difference between the output data output by the output layer and the data labeled in the sensing data is minimized. Here, the labeled data may be data on the user's posture measured through a separate sensor attached to the user's body while the learning data acquisition data is being acquired.

[0178] At this time, the loss function representing the difference between the output data and the labeled data can be implemented in the form of a cross-score function, but is not limited thereto.

[0179] At this time, the electronic device (100) can learn the weights of at least one of the embedding layer (112), the first attention layer (113), the second attention layer (114), and the output layer (115) using a loss function.

[0180] The operation of the electronic device (100) to train the artificial intelligence model (111) will be described with reference to FIG. 9.

[0181] FIG. 9 is a flowchart illustrating a method for an electronic device (100) to train an artificial intelligence model (111) according to one or more embodiments of the present disclosure.

[0182] Referring to FIG. 9, the electronic device (100) can obtain learning data including a plurality of sensing data (S910).

[0183] The training data may include multiple sensing data points, each of which senses the movements of multiple devices while the user is moving while carrying multiple devices. For example, the training data may include sensing data from a smartphone, a smartwatch, and earphones, which sense the movements of the devices for 180 seconds.

[0184] Additionally, the training data may include sensing data that senses the user's heart rate. For example, the training data may include sensing data that senses the user's heart rate for 180 seconds from a smartwatch.

[0185] The training data may include sensing data belonging to multiple domains. As described above, the domains may correspond to the type of device that sensed the sensing data. Furthermore, the sampling rates at which the sensing data is sensed may vary across domains.

[0186] In addition, the learning data may be labeled data regarding the user's posture measured while multiple sensing data are sensed. At this time, the labeled data may be the type of data that the artificial intelligence model (111) intends to output. For example, the data labeled in the learning data may be one of the indices for measuring gait. For example, the indices for measuring gait may be at least one of an indice indicating symmetry during walking, an indice indicating the size of arm swing during walking, an indice indicating stride length during walking, an indice indicating walking speed during walking, and an indice indicating double support time during walking.

[0187] In addition, the electronic device (100) can train an artificial intelligence model (111) to learn time-series correlations between sensing data of a specific domain using learning data.

[0188] Specifically, the electronic device (100) can divide each of the plurality of sensing data into a plurality of segments according to a plurality of time intervals, and input the divided segments into the embedding layer (112) to obtain an embedding vector (S920).

[0189] And, the electronic device (100) can input the embedding vector into the attention layer (S930).

[0190] Specifically, the electronic device (100) can input an embedding vector into a first attention layer (113) to obtain a vector including a time-series correlation between embedding vectors within a specific domain. In addition, the electronic device (100) can input a vector output from the first attention layer (113) into a second attention layer (114) to obtain a vector including a correlation between embedding vectors in different domains.

[0191] And, the electronic device (100) can input the output of the attention layer to the output layer (S940).

[0192] And, the electronic device (100) can train the artificial intelligence model so that the difference between the output of the output layer (115) and the data labeled in the learning data is minimized when learning data is input to the artificial intelligence model (111) (S950).

[0193] Accordingly, the electronic device (100) can learn the weights of at least one of the embedding layer (112), the first attention layer (113), the second attention layer (114), and the output layer (115).

[0194] Accordingly, the first attention layer (113) can be trained to output correlations between time series in a specific domain.

[0195] Additionally, the second attention layer (114) can be trained to output correlations between domains. That is, the second attention layer (114) can be trained to output correlations between embedding vectors of different domains.

[0196] Meanwhile, the attention layer according to the present disclosure can be trained to output the correlation between the output value of the output layer (115) and the embedding vector.

[0197] Meanwhile, the electronic device (100) according to the present disclosure can increase the number of learning data by data augmentation of acquired learning data, and train an artificial intelligence model (111) using the increased number of learning data.

[0198] Meanwhile, the electronic device (100) according to the present disclosure can delete some of the learning data and train the artificial intelligence model (111) using the learning data from which some of the data has been deleted. Compared to the learning data, sensing data acquired in an actual environment may have a problem in which sensing data is missing in some time sections for various reasons. Since the electronic device (100) of the present disclosure trains the artificial intelligence model (111) by deleting data corresponding to some time sections of the learning data, even when sensing data from which some time sections are missing are acquired, the electronic device (100) can estimate the user's posture with high accuracy.

[0199] Additionally, the electronic device (100) can estimate the user's heart rate with high accuracy even when the user's heart rate sensing data is acquired with some time intervals missing.

[0200] Referring to FIG. 10, in the case of the heart rate sensor, since it is sensitive to the user's movement, there may be a problem in which the heart rate sensing data measuring the user's heart rate while the user is exercising is omitted.

[0201] At this time, the electronic device (100) inputs the user's heart rate data (1020) with some time sections missing and the sensing data (1030) that detects the user's movement into the second artificial intelligence model (111b), so that the electronic device (100) can estimate the user's heart rate data (1010) in the missing time sections. That is, the electronic device (100) can estimate the user's heart rate data in the time sections with missing heart rate through the second artificial intelligence model (111b). The second artificial intelligence model (111b) may be stored in the memory (110). Meanwhile, the artificial intelligence model (111) described with reference to FIGS. 1 to 9 may be referred to as the first artificial intelligence model.

[0202] FIG. 11 is a diagram illustrating a second artificial intelligence model according to one or more embodiments of the present disclosure.

[0203] Referring to FIG. 11, the second artificial intelligence model (111b) may include multiple first layers (1110) for extracting characteristics of the user's movement from sensing data (1030) that senses the user's movement. In this case, the characteristics of the user's movement may be variance, frequency, and shape.

[0204] That is, the second AI model (111b) may include a layer for measuring variance from motion detection data. Variance represents the degree to which the device's movement changes, and may be an important factor in predicting a user's heart rate.

[0205] At this time, the window size for inputting sensing data detecting the user's movement into the neural network to measure the variance may vary from 1 second to 60 seconds. Here, the window size may refer to the time interval during which the sensing data detecting the user's movement is measured. For example, if the window size is 2 seconds, the electronic device (100) may input sensing data detecting the user's movement from 4 seconds to 6 seconds into the neural network. Accordingly, the electronic device (100) may detect the variance of the user's movement from various aspects.

[0206] Additionally, the second artificial intelligence model (111b) may include a layer for detecting the frequency of the user's movements.

[0207] At this time, the layer for detecting frequency can use linear-frequency cepstrum coefficients (LFCC) derived from the short-time Fourier Transform method.

[0208] Additionally, the second artificial intelligence model (111b) may include a layer for detecting the shape of the user's movement.

[0209] Because the user's wrist movements often exhibit irregular and unique patterns at the beginning or end of exercise, utilizing the shape of the user's movements can predict the user's heart rate more accurately than using only the magnitude or frequency of the movements.

[0210] At this time, the layer for detecting the shape of the user's movement can process data that senses the user's movement using SincNet, a deep learning architecture that can be used for voice recognition.

[0211] In addition, the second artificial intelligence model (111b) may include a plurality of second layers (1120) that estimate a missing value of a heart rate sensing value when a characteristic of the user's movement is input. Here, each of the plurality of second layers may be a linear layer. In addition, the second artificial intelligence model (111b) may include a plurality of second layers (1120) that estimate a missing value of a heart rate sensing value when heart rate data with some time intervals missing is input. In this case, each of the plurality of second layers (1120) may receive heart rate data, one of the characteristics of the user's movement, and data sensing the user's movement, respectively.

[0212] In addition, the second artificial intelligence model (111b) may include a third layer (1130) that estimates a missing value of a heart rate sensing value when the outputs of multiple second layers (1120) are input. In this case, the third layer (1130) may be an ensemble layer that performs a more accurate prediction by combining the outputs of multiple second layers (1120).

[0213] Accordingly, the electronic device (100) of the present disclosure can accurately estimate the user's heart rate. That is, the electronic device (100) can enable compensation through a sensor that senses the user's movement in the case of a sensing value that is missing or has a large amount of noise due to the user's movement, such as a sensing value by a heart rate sensor.

[0214] FIG. 12 is a flowchart illustrating a method for an electronic device according to one or more embodiments of the present disclosure to guide a user's posture using an artificial intelligence model.

[0215] Referring to FIG. 10, the electronic device (100) can obtain multiple sensing data from multiple sensors (S1210).

[0216] For example, the plurality of sensing data may include at least one of first sensing data sensing movement of the electronic device (100), second sensing data sensing movement of the first external device (200), third sensing data sensing movement of the second external device (300), and fourth sensing data sensing the user's heart rate by the first external device (200).

[0217] The electronic device (100) can obtain multiple embedding vectors by embedding multiple pieces of sensing data according to multiple time intervals (S1220). That is, the electronic device (100) can obtain embedding vectors by dividing each of the multiple pieces of sensing data into multiple segments according to multiple time intervals and inputting the divided segments into the embedding layer (112).

[0218] Based on the correlation between multiple embedding vectors, the electronic device (100) can estimate the user's posture (S1230).

[0219] Specifically, the electronic device (100) can input a plurality of embedding vectors into a first attention layer (113) and input the output value of the first attention layer (113) into a second attention layer (114).

[0220] The electronic device (100) can estimate the user's posture by inputting the output value of the second attention layer (114) into the output layer.

[0221] Specifically, the electronic device (100) can obtain data on the user's posture by inputting the output value of the second attention layer (114) into the output layer (115).

[0222] In the present disclosure, data on the user's posture may be data on an indicator for measuring the user's gait.

[0223] Based on the estimated user's posture, the electronic device (100) can provide information guiding the user's posture or the user's movement (S1240).

[0224] According to one or more embodiments of the present disclosure, the electronic device (100) can obtain information indicating the state of the user's posture or movement by comparing the reference data with the estimated user's posture. That is, the electronic device (100) can obtain information for evaluating the quality of the user's posture or movement by comparing the reference data with the output of the artificial intelligence model (111).

[0225] Here, the reference data may include information on at least one of symmetry during walking, magnitude of arm swing during walking, stride length during walking, walking speed, double support time during walking, and reference heart rate.

[0226] That is, the reference data may include information on at least one reference value of symmetry during walking, size of arm swing during walking, stride length during walking, walking speed, migration support time during walking, and heart rate during walking.

[0227] At this time, the electronic device (100) can compare the reference data with the estimated posture and classify the user's posture or motion (e.g., walking motion) as “good”, “normal”, “bad”, or “dangerous”.

[0228] Alternatively, the electronic device (100) may obtain a value (e.g., 0 to 10) indicating the suitability of the user's posture or motion (e.g., walking motion) by comparing the reference data with the estimated posture. Meanwhile, the reference data may be determined by information about the user. Here, the information about the user may include at least one of the user's age, the user's gender, the user's height, the user's weight, the user's body fat percentage, and the user's BMI.

[0229] Specifically, the electronic device (100) can compare indicators for measuring the user's gait with reference data. At this time, the electronic device (100) can identify the degree to which each indicator is close to the reference value.

[0230] Here, the reference value may be the average value of each indicator. For example, if the user of the electronic device (100) is a male in his 20s, the preset value may be the average value of each indicator measured for a male in his 20s.

[0231] Alternatively, the reference value may be the ideal value of each indicator. For example, the ideal value of an indicator representing symmetry may be 1. And the ideal value of an indicator representing double support time may be 0.

[0232] The electronic device (100) can evaluate the quality of the user's posture or movement as higher the closer the indicator for measuring the user's gait approaches the reference value. For example, the electronic device (100) can classify the user's posture or movement as "good." Alternatively, the electronic device (100) can evaluate the suitability of the user's posture or movement as approaching "10."

[0233] According to one or more embodiments of the present disclosure, the electronic device (100) may obtain information for evaluating the quality of the user's posture or movement by comparing an indicator indicating the user's motion or posture with reference data. However, this is only one embodiment, and the operation of comparing the indicator indicating the above-described motion or posture with reference data may also be performed on the output layer of the artificial intelligence model (111). In this case, the electronic device (100) may directly obtain information for evaluating the quality of the user's motion or posture from the output of the artificial intelligence model (111).

[0234] According to one or more embodiments of the present disclosure, the electronic device (100) may provide information for guiding the user's posture or motion based on at least one of the current user's exercise intensity, the current user's posture or motion status, the current user's heart rate, and information about the user.

[0235] According to one or more embodiments of the present disclosure, if it is determined that the user's exercise intensity is greater than a preset value, the electronic device (100) may provide information guiding the user to reduce the exercise intensity.

[0236] For example, if the difference between the user's heart rate and the estimated user's heart rate included in the reference data is greater than or equal to a threshold value, the electronic device (100) may provide information guiding the user to reduce the intensity (or speed) of the movement.

[0237] For example, if the difference between the value representing the acquired step symmetry and the reference value is greater than a threshold value, the electronic device (100) can provide information guiding the user to reduce the intensity (or speed) of the movement.

[0238] For example, if the user's exercise intensity is greater than or equal to a preset first value, but a value indicating the suitability of the user's posture or movement state is greater than or equal to a preset second value, the user's heart rate is within a preset value range, and the user's age is less than or equal to a preset age, the electronic device (100) may provide information guiding the user to maintain the intensity (or speed) of the movement.

[0239] According to one or more embodiments of the present disclosure, the information guided by the electronic device (100) may include information guiding the user to maintain the current movement, information guiding the user to reduce the intensity of the current movement, or information guiding the user to increase the intensity of the current movement. Alternatively, the information guided by the electronic device (100) may include information recommending a movement (or exercise) more suitable for the user.

[0240] For example, as illustrated in FIG. 13, the electronic device (100) can display text (1110) such as “Your left-right symmetry is unbalanced when walking. Walk more slowly.” through the display (140).

[0241] Alternatively, the electronic device (100) may output audio through the speaker (150), such as “Your left-right symmetry is unbalanced when walking. Please walk more slowly.”

[0242] According to one or more embodiments of the present disclosure, the electronic device (100) can identify whether the acquired sensing data satisfies a preset condition, and if the condition is satisfied, estimate the user's posture and provide information for guiding the user's actions.

[0243] FIG. 14 is a flowchart illustrating a method for estimating a user's posture based on whether a condition of acquired sensing data is satisfied by an electronic device (100) according to one or more embodiments of the present disclosure.

[0244] Referring to FIG. 14, the electronic device (100) can obtain sensing data from multiple sensors (S1410). At this time, the operation of S1210 may be the same as the operation of S910.

[0245] The electronic device (100) can identify whether the acquired sensing data satisfies a preset condition (S1420).

[0246] Here, the preset condition may be a condition in which the similarity between the pattern of the sensing data and the preset pattern is greater than a threshold value.

[0247] Alternatively, the preset condition may be a condition in which the difference between the pattern of the sensing data and the preset pattern is less than a preset value.

[0248] Alternatively, the preset condition may be a condition in which the time at which sensing data is acquired is greater than or equal to a preset time.

[0249] Alternatively, the preset condition may be a condition under which sensing data from a specific domain is acquired. For example, the preset condition may be a condition under which sensing data sensed through earphones is acquired.

[0250] Alternatively, the preset condition may be a condition that satisfies multiple conditions among the conditions described above.

[0251] If the sensing data does not satisfy the preset conditions (S1220-N), the electronic device (100) can continue to acquire the sensing data (S1410).

[0252] If the sensing data satisfies the preset conditions (S1420-Y), the electronic device (100) can input the sensing data into an artificial intelligence model to estimate the user's posture (S1430).

[0253] At this time, the user's posture estimated by the electronic device (100) may be different depending on the satisfied conditions.

[0254] Specifically, if the first condition (e.g., sensing data is received from a smartwatch) is satisfied, the electronic device (100) can estimate a first posture (e.g., hand amplitude) among the user's postures. And, if the second condition (e.g., sensing data is received through earphones) is satisfied, the electronic device (100) can estimate a second posture (e.g., gait symmetry) among the user's postures.

[0255] Based on the estimated user's posture, the electronic device (100) can provide guide information to guide the user's posture or movement (S1440).

[0256] In the present disclosure, the electronic device (100) can provide information in various ways.

[0257] According to one or more embodiments of the present disclosure, the electronic device (100) can control the display (140) to display a screen including information.

[0258] According to one or more embodiments of the present disclosure, the electronic device (100) can control the speaker (150) to output audio including information.

[0259] According to one or more embodiments of the present disclosure, an electronic device (100) may transmit data for outputting a screen or audio containing information to a user terminal device via a communication interface (120). Based on the received data, the user terminal device may output a screen or audio containing information.

[0260] FIG. 15 is a flowchart for explaining a method of controlling an electronic device (100) according to one or more embodiments of the present disclosure.

[0261] Referring to FIG. 15, the electronic device (100) can acquire multiple sensing data obtained from multiple sensors while the user moves (S1510).

[0262] At this time, each of the plurality of sensors can sense multiple sensing data at different sampling rates.

[0263] The plurality of sensors may be acceleration sensors.

[0264] The electronic device (100) can obtain multiple embedding vectors by embedding multiple sensing data at multiple time intervals (S1520).

[0265] Among the plurality of embedding vectors, a first embedding vector may correspond to a first time interval, and a second embedding vector may correspond to a second time interval different from the first time interval.

[0266] The electronic device (100) can estimate the user's posture based on the correlation between multiple embedding vectors (S1530).

[0267] Meanwhile, the electronic device (100) can identify whether a plurality of sensing data satisfy a preset condition.

[0268] And, if multiple sensing data satisfy a preset condition, the electronic device (100) can estimate the user's posture corresponding to the condition.

[0269] Specifically, the electronic device (100) can estimate the user's posture by using at least one embedding vector among a plurality of embedding vectors, the value of which indicates the degree of correlation with the user's posture being greater than or equal to a preset value.

[0270] Each of the multiple sensing data can correspond to each of the multiple domains.

[0271] The electronic device (100) can obtain a correlation between embedding vectors belonging to the same domain among multiple domains.

[0272] The electronic device (100) can obtain a correlation between embedding vectors belonging to different domains among a plurality of domains.

[0273] The electronic device (100) can guide the user's movements based on the estimated user's posture (S1540).

[0274] Specifically, the electronic device (100) can guide the user's movements by comparing the estimated user's posture with reference data. At this time, the reference data can be determined based on the user's age and gender.

[0275] Although various embodiments have been described above, each embodiment is not necessarily implemented individually, and may be implemented together in a single product by being combined in whole or in part with at least one other embodiment.

[0276] Meanwhile, the terms "part" or "module" used in the present disclosure include units composed of hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A "part" or "module" may be an integrally composed component, a minimum unit performing one or more functions, or a portion thereof. For example, a module may be composed of an application-specific integrated circuit (ASIC).

[0277] Various embodiments of the present disclosure may be implemented as software including instructions stored in a machine-readable storage medium that can be read by a machine (e.g., a computer). The device, which is a device capable of calling instructions stored in the storage medium and operating according to the called instructions, may include an electronic device (100) according to the disclosed embodiments. When the instructions are executed by a processor, the processor may directly or under the control of the processor perform a function corresponding to the instructions using other components. The instructions may include code generated or executed by a compiler or interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, "non-transitory" means that the storage medium does not contain signals and is tangible, but does not distinguish between data being stored semi-permanently or temporarily in the storage medium.

[0278] According to one or more embodiments, the methods according to the various embodiments disclosed herein may be provided as a computer program product. The computer program product may be traded as a commodity between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or online through an application store (e.g., Play Store™). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0279] Each component (e.g., a module or a program) according to various embodiments may be composed of one or more entities, and some of the aforementioned sub-components may be omitted, or other sub-components may be further included in various embodiments. Alternatively or additionally, some components (e.g., a module or a program) may be integrated into a single entity, which may perform the same or similar functions as those performed by each of the respective components prior to integration. Operations performed by a module, program, or other component according to various embodiments may be executed sequentially, in parallel, iteratively, or heuristically, or at least some operations may be executed in a different order, omitted, or other operations may be added.

Claims

1. In a method for controlling an electronic device, A step of acquiring multiple sensing data obtained from multiple sensors while the user moves; A step of obtaining a plurality of embedding vectors by embedding the plurality of sensing data according to a plurality of time intervals; A step of estimating the user's posture based on the correlation between the plurality of embedding vectors; and A control method, comprising: a step of guiding a user's movement based on the estimated user's posture.

2. In paragraph 1, The step of estimating the user's posture is as follows: A control method for estimating the posture of the user by using at least one embedding vector among the plurality of embedding vectors, the value indicating the degree of correlation with the posture of the user being greater than or equal to a preset value.

3. In paragraph 1, A control method, wherein among the plurality of embedding vectors, a first embedding vector corresponds to a first time interval, and a second embedding vector corresponds to a second time interval different from the first time interval.

4. In paragraph 1, The above control method is, A control method further comprising: a step of training an artificial intelligence model to learn correlations between sensing data acquired from a plurality of sensors.

5. In paragraph 1, Each of the above multiple sensing data corresponds to each of the multiple domains, The above control method is, A step of obtaining a correlation between embedding vectors belonging to the same domain among the above multiple domains; and A control method further comprising: a step of obtaining a correlation between embedding vectors belonging to different domains among the plurality of domains.

6. In paragraph 1, The above control method is, A step of identifying whether the plurality of sensing data satisfies a preset condition; and A control method further comprising: a step of estimating a posture of the user corresponding to the condition when the plurality of sensing data satisfies the preset condition.

7. In paragraph 1, The steps for guiding the above user's actions are: By comparing the estimated user's posture with the reference data, the user's movements are guided, A control method wherein the above reference data is determined based on the age and gender of the user.

8. In paragraph 1, A control method, wherein each of the plurality of sensors senses the plurality of sensing data at a different sampling rate.

9. In paragraph 1, A control method wherein the plurality of sensors are at least one of an acceleration sensor, a gyroscope sensor, and a geomagnetic sensor.

10. In electronic devices, A sensor for detecting movement of the electronic device; communication interface; memory; and a processor; including; The above processor, Acquire multiple sensing data obtained from multiple sensors including the above sensor while the user moves, Embedding the above multiple sensing data according to multiple time intervals to obtain multiple embedding vectors, Based on the correlation between the plurality of embedding vectors, the user's posture is estimated, An electronic device that guides the user's movements based on the estimated user's posture.

11. In paragraph 10, The above processor, An electronic device that estimates the posture of the user by using at least one embedding vector among the plurality of embedding vectors, the value indicating the degree of correlation with the posture of the user being greater than or equal to a preset value.

12. In paragraph 10, An electronic device, wherein among the plurality of embedding vectors, a first embedding vector corresponds to a first time interval, and a second embedding vector corresponds to a second time interval different from the first time interval.

13. In paragraph 10, The above processor, An electronic device that trains an artificial intelligence model to learn correlations between sensing data acquired from multiple sensors.

14. In paragraph 10, Each of the above multiple sensing data corresponds to each of the multiple domains, The above processor, Obtaining the correlation between embedding vectors belonging to the same domain among the above multiple domains, An electronic device that obtains correlations between embedding vectors belonging to different domains among the plurality of domains.

15. In paragraph 10, The above processor, Identify whether the above plurality of sensing data satisfy a preset condition, An electronic device that estimates the user's posture corresponding to the condition when the plurality of sensing data satisfies the preset condition.

Images