Electronic device, control method thereof, and non-transitory computer-readable recording medium

Abstract

An electronic device disclosed herein comprises: a memory; a display; an image sensor; a communication circuit connected to a wearable device and a mobile robot; and at least one processor connected to the memory, the display, the image sensor, and the communication circuit, wherein the at least one processor controls the display to display content for guiding a first posture, receives sensing data related to the posture of a user from the wearable device by requesting the wearable device for the sensing data through the communication circuit, identifies, on the basis of the first posture and the sensing data, whether a predetermined event has occurred, acquires a first captured image including the user through the image sensor when the predetermined event is identified, receives a second captured image including the user from the mobile robot by requesting, from the mobile robot, a captured image including the user through the communication circuit, and provides guide information on the basis of the content including the first posture, the first captured image, the sensing data, and the second captured image.

Description

Electronic device, control method thereof and non-transitory computer-readable recording medium

[0001] The present disclosure relates to an electronic device and a control method thereof, and more particularly, to an electronic device that provides content guiding a user to a specific posture, a control method thereof, and a non-transitory computer-readable recording medium.

[0002] Home fitness services can provide customized exercise coaching to users using television, wearable devices, and smart home robots. Home fitness services can recognize users' exercise postures in a home fitness environment and provide exercise guidance.

[0003] The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above-described matters constitute prior art related to the present disclosure.

[0004] The present disclosure is designed to improve the above-described problem, and an object of the present disclosure is to provide an electronic device and a control method thereof that guides a user to a specific posture based on sensing data acquired from a wearable device, a first captured image captured by an electronic device displaying content, and a second captured image captured by a mobile robot.

[0005] According to one embodiment, an electronic device includes a memory, a display, an image sensor, a wearable device, and a communication circuit connected to a mobile robot, and at least one processor connected to the memory, the display, the image sensor, and the communication circuit, wherein the at least one processor controls the display to display content for guiding a first posture, requests sensing data related to a posture of a user from the wearable device through the communication circuit, thereby receiving the sensing data from the wearable device, identifies whether a predetermined event has occurred based on the first posture and the sensing data, and when the predetermined event is identified, obtains a first captured image including the user through the image sensor, requests a captured image including the user from the mobile robot through the communication circuit, thereby receiving a second captured image including the user from the mobile robot, and provides guide information based on the content including the first posture, the first captured image, the sensing data, and the second captured image.

[0006] The at least one processor can identify the user's measured posture information based on the sensing data, obtain a similarity between the measured posture information and the first posture, and if the similarity is less than a threshold value, identify that the predetermined event has occurred.

[0007] The at least one processor may obtain reference posture information corresponding to the first posture based on the content, and obtain the similarity by comparing the reference posture information and the measured posture information.

[0008] The at least one processor may obtain the guide information for guiding the first posture based on the reference posture information, the first captured image, the measured posture information, and the second captured image, and control the display to display the content representing the first posture together with the guide information.

[0009] The above guide information is obtained based on an AI (artificial intelligence) model that receives the reference posture information, the first photographed image, the measured posture information, and the second photographed image as input and generates output data, and the AI ​​model may include a large health model (LHM).

[0010] The at least one processor may control the display to display the content when a user input for outputting the content is received, and may identify the measured posture information of the user based on the sensing data when the first posture included in the content is displayed.

[0011] The at least one processor may, when the user input for outputting the content is received, transmit a first control signal to the wearable device through the communication circuit for requesting sensing data related to the user's posture, and receive the sensing data from the wearable device through the communication circuit.

[0012] The at least one processor may receive the sensing data from the wearable device through the communication circuit while the first posture is displayed on the display, and identify the measured posture information of the user based on the sensing data received while the first posture is displayed.

[0013] The at least one processor, when the predetermined event is identified, acquires a shooting position corresponding to the first posture, transmits a second control signal to the mobile robot through the communication circuit for requesting a shooting image including the user based on the shooting position, and receives the second shooting image including the user captured based on the shooting position from the mobile robot through the communication circuit.

[0014] When the guide information is acquired, the at least one processor can acquire a projection surface position corresponding to the first posture and transmit a third control signal for outputting the guide information based on the projection surface position to the mobile robot through the communication circuit.

[0015] According to one embodiment, a control method of an electronic device connected to a wearable device and a mobile robot includes the steps of displaying content for guiding a first posture, requesting sensing data related to a user's posture from the wearable device to thereby receive the sensing data, identifying whether a predetermined event has occurred based on the first posture and the sensing data, obtaining a first captured image including the user through the electronic device when the predetermined event is identified, requesting a captured image including the user from the mobile robot to thereby receive a second captured image including the user, and providing guide information based on the content including the first posture, the first captured image, the sensing data, and the second captured image.

[0016] The step of identifying whether the above event has occurred may include identifying the user's measurement posture information based on the sensing data, obtaining a similarity between the measurement posture information and the first posture, and identifying that the determined event has occurred if the similarity is less than a threshold value.

[0017] The step of obtaining the above similarity may include obtaining reference posture information corresponding to the first posture based on the content, and comparing the reference posture information with the measured posture information to obtain the similarity.

[0018] The step of providing the above guide information may include obtaining the guide information for guiding the first posture based on the reference posture information, the first photographed image, the measured posture information, and the second photographed image, and displaying the content representing the first posture together with the guide information.

[0019] The above guide information is obtained based on an AI (artificial intelligence) model that receives the reference posture information, the first photographed image, the measured posture information, and the second photographed image as input and generates output data, and the AI ​​model may include a large health model (LHM).

[0020] The step of displaying the content may further include a step of displaying the content when a user input for outputting the content is received, and the control method may further include a step of identifying the measured posture information of the user based on the sensing data when the first posture included in the content is displayed.

[0021] The step of receiving the sensing data may include, when the user input for outputting the content is received, transmitting a first control signal for requesting sensing data related to the user's posture to the wearable device, and receiving the sensing data from the wearable device.

[0022] The step of identifying the above measurement posture information may include receiving the sensing data from the wearable device while the first posture is displayed on the electronic device, and identifying the user's measurement posture information based on the sensing data received while the first posture is displayed.

[0023] The step of receiving the second captured image may include, when the predetermined event is identified, acquiring a captured position corresponding to the first posture, transmitting a second control signal to the mobile robot for requesting a captured image including the user based on the captured position, and receiving the second captured image including the user captured based on the captured position from the mobile robot.

[0024] The step of providing the above guide information may include, when the guide information is obtained, obtaining a projection surface position corresponding to the first posture, and transmitting a third control signal for outputting the guide information based on the projection surface position to the mobile robot.

[0025] A non-transitory computer-readable recording medium storing computer instructions that, when executed by a processor of an electronic device, cause an electronic device connected to a wearable device and a mobile robot to perform an operation, the operation comprises: a step of displaying content for guiding a first posture; a step of requesting sensing data related to a posture of a user from the wearable device and thereby receiving the sensing data from the wearable device; a step of identifying whether a predetermined event has occurred based on the first posture and the sensing data; a step of obtaining a first captured image including the user through the electronic device when the predetermined event is identified; a step of requesting a captured image including the user from the mobile robot and thereby receiving a second captured image including the user from the mobile robot; and a step of providing guide information based on the content including the first posture, the first captured image, the sensing data, and the second captured image.

[0026] In connection with the description of the drawings, the same or similar reference numerals may be used for the same or similar components.

[0027] FIG. 1 is a drawing for explaining an embodiment of providing guide information to a user according to one embodiment.

[0028] FIG. 2 is a block diagram illustrating an electronic device according to one embodiment.

[0029] FIG. 3 is a block diagram illustrating a specific configuration of the electronic device of FIG. 2, according to one embodiment.

[0030] FIG. 4 is a flowchart illustrating an operation of providing guide information related to a user's posture, according to one embodiment.

[0031] FIG. 5 is a drawing for explaining an operation of capturing a user's posture according to one embodiment.

[0032] FIG. 6 is a drawing for explaining an operation of providing guide information for correcting a user's posture, according to one embodiment.

[0033] FIG. 7 is a drawing for explaining an image screen for correcting a user's posture according to one embodiment.

[0034] FIG. 8 is a diagram illustrating an operation of generating guide information related to a user's posture, according to one embodiment.

[0035] FIG. 9 is a diagram for explaining an operation of calculating similarity according to one embodiment.

[0036] FIG. 10 is a drawing for explaining an operation of requesting a photograph from a mobile robot according to one embodiment.

[0037] FIG. 11 is a drawing for explaining shooting information according to one embodiment.

[0038] FIG. 12 is a diagram for explaining an operation of managing data required for guide information according to one embodiment.

[0039] FIG. 13 is a diagram illustrating an AI model that generates guide information according to one embodiment.

[0040] FIG. 14 is a diagram for explaining an operation of obtaining guide information using an AI model stored in an electronic device, according to one embodiment.

[0041] FIG. 15 is a diagram for explaining an operation of obtaining guide information using an AI model stored in a server, according to one embodiment.

[0042] FIG. 16 is a diagram for explaining an operation of comparing detailed information according to one embodiment.

[0043] FIG. 17 is a drawing for explaining an operation of providing guide information through a mobile robot according to one embodiment.

[0044] FIG. 18 is a drawing for explaining an operation of a mobile robot outputting guide information according to one embodiment.

[0045] FIG. 19 is a drawing for explaining an operation of providing a projection surface position for outputting guide information by a mobile robot according to one embodiment.

[0046] FIG. 20 is a drawing for explaining projection information according to one embodiment.

[0047] FIG. 21 is a drawing for explaining an operation of guiding a user's initial position and an initial position of a mobile robot according to one embodiment.

[0048] FIG. 22 is a drawing for explaining an operation of guiding a user's initial position and an initial position of a mobile robot according to one embodiment.

[0049] FIG. 23 is a diagram for explaining an operation of requesting sensing data according to a set event, according to one embodiment.

[0050] FIG. 24 is a drawing for explaining an operation of estimating a user's posture according to one embodiment.

[0051] FIG. 25 is a drawing for explaining an operation for estimating a user's posture according to one embodiment.

[0052] FIG. 26 is a flowchart for explaining a method for controlling an electronic device, according to one embodiment.

[0053] Hereinafter, the present disclosure will be described in detail with reference to the attached drawings.

[0054] The terms used in the embodiments of this disclosure have been selected from widely used, current terms, taking into account the functions of this disclosure. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the description of the relevant disclosure. Therefore, the terms used in this disclosure should not be defined simply as names of terms, but rather based on the meanings of the terms and the overall content of this disclosure.

[0055] In this specification, expressions such as “has,” “can have,” “includes,” or “may include” indicate the presence of a feature (e.g., a number, function, operation, or component such as a part), and do not exclude the presence of additional features.

[0056] The expression "at least one of A and / or B" should be understood to mean either "A" or "B" or "A and B".

[0057] As used herein, the expressions “first,” “second,” “first,” or “second,” etc., may describe various components, regardless of order and / or importance, and are only used to distinguish one component from another, but do not limit the components.

[0058] When it is said that a component (e.g., a first component) is “(operatively or communicatively) coupled with / to” or “connected to” another component (e.g., a second component), it should be understood that the component may be directly coupled to the other component, or may be connected through another component (e.g., a third component).

[0059] Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, terms such as "comprise" or "consist of" are intended to indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but should be understood not to preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0060] In the present disclosure, a "module" or "part" performs at least one function or operation and may be implemented in hardware or software, or a combination of hardware and software. Furthermore, multiple "modules" or multiple "parts" may be integrated into at least one module and implemented as at least one processor, excluding any "modules" or "parts" that need to be implemented as specific hardware.

[0061] In this specification, the term user may refer to a person using an electronic device or a device using an electronic device (e.g., an artificial intelligence electronic device).

[0062] An embodiment of the present disclosure will be described in more detail with reference to the attached drawings below.

[0063] FIG. 1 is a drawing for explaining an embodiment of providing guide information to a user according to one embodiment.

[0064] The system (1000) of FIG. 1 may include at least one of an electronic device (100), a wearable device (200), or a mobile robot (300).

[0065] The electronic device (100) may be a device that provides content. The electronic device (100) may include a display (140). The electronic device (100) may be described as a display device. The electronic device (100) may be a content providing device that includes a display module. For example, the electronic device (100) may be implemented as a TV, home appliance, laptop, tablet, smartphone, or signage device. The electronic device (100) may provide a health application.

[0066] The electronic device (100) can be connected to at least one of a wearable device (200) or a mobile robot (300) using a preset communication method. The preset communication method may be a wireless communication method. The wireless communication method may include Bluetooth, Bluetooth low energy (BLE), ultra-wideband (UWB), or Wi-Fi.

[0067] A wearable device (200) may refer to an electronic device worn by a user. The wearable device (200) may include at least one sensor. The wearable device (200) may sense data related to the user through the at least one sensor. The at least one sensor may include at least one of an inertial sensor and a biometric sensor. The inertial sensor may be implemented as an acceleration sensor or a gyro sensor. The biometric sensor may be a sensor for sensing biological data of the user. For example, the wearable device (200) may be implemented as a smart watch, a smart ring, a head mounted display (HMD), or earbuds.

[0068] A mobile robot (300) may be an electronic device including a movable member. The mobile robot (300) controls the movable member by driving a motor and can move to a specific location. The mobile robot (300) may include an image sensor (e.g., a camera). The movable member may include wheels for movement. The mobile robot (300) may acquire a photographed image including a user through the image sensor. The mobile robot (300) may include a projection unit that projects a projected image. For example, the mobile robot (300) may be implemented as a service robot, a smart home robot, or a mobile projector.

[0069] According to one embodiment, the electronic device (100) may be communicatively connected to each of a wearable device (200) and a mobile robot (300). The electronic device (100) may include a communication circuit (160). The electronic device (100) may be connected to the wearable device (200) and the mobile robot (300) through the communication circuit (160).

[0070] According to one embodiment, the wearable device (200) may be communicatively connected to each of the electronic device (100) and the mobile robot (300). The wearable device (200) may include a communication circuit.

[0071] According to one embodiment, the mobile robot (300) may be communicatively connected to each of the electronic device (100) and the wearable device (200). The mobile robot (300) may include a communication circuit.

[0072] The image information collected through the TV's camera can be used to identify the user's current posture. However, it can be difficult to accurately determine a person's posture based on the captured image alone.

[0073] The user's exercise posture can be recognized by acquiring sensor information from the wearable device the user is wearing. However, accurately determining a person's posture based solely on sensor information can be difficult. If sensor information is similar across different movements, it can be difficult to accurately determine the person's current posture.

[0074] Additionally, even if both data are used, blind spots may occur, making it difficult to accurately determine a person's posture.

[0075] To guide users to the correct posture, a guide image can be displayed on a TV or robot display. However, if the user adopts a posture (or orientation) that prevents them from viewing the display, the guidance may not be accurately conveyed to the user.

[0076] FIG. 2 is a block diagram illustrating an electronic device (100) according to one embodiment.

[0077] Referring to FIG. 2, the electronic device (100) may include at least one of a processor (110), a memory (120), a display (140), an image sensor (150), and a communication circuit (160).

[0078] The memory (120) can store content provided to the user. The memory (120) can store various information necessary to provide guide information. For example, the memory (120) can store shooting information (see FIG. 11) or projection information (see FIG. 20).

[0079] The display (140) can output image information or an image signal.

[0080] The image sensor (150) can acquire data corresponding to a shooting function. The image sensor (150) can acquire a first shooting image according to a user command.

[0081] The communication circuit (160) can be connected to a wearable device (200) and a mobile robot (300). The communication circuit (160) can be described as a communication interface.

[0082] At least one processor (110) may be connected to a memory (120), a display (140), an image sensor (150), and a communication circuit (160).

[0083] The number of processors (110) may be one or more. For example, the processor (110) may have a multi-core processor structure such as a dual core, quad core, or hexa core.

[0084] The processor (110) can control the operations of the electronic device (100) by executing instructions stored in the memory (120). For example, the processor (110) can correspond to a plurality of processors that collectively perform a plurality of operations by dividing them among the processors.

[0085] At least one processor (110) controls the display (140) to display content for guiding a first posture, and receives sensing data from the wearable device (200) by requesting sensing data related to the posture of the user from the wearable device (200) through the communication circuit (160), identifies whether a predetermined event has occurred based on the first posture and the sensing data, and when the predetermined event is identified, obtains a first captured image including the user through the image sensor (150), requests a captured image including the user from the mobile robot (300) through the communication circuit (160), receives a second captured image including the user from the mobile robot (300), and provides guide information based on the content including the first posture, the first captured image, the sensing data, and the second captured image.

[0086] The content may include images to guide a specific posture (e.g., the first posture). The content may include at least one of home training content or exercise lecture content. The content may include images depicting people or characters in specific postures to guide the user to adopt a specific posture. The content may be described as exercise videos, home fitness content, or exercise guide content.

[0087] For example, content may be pre-stored in the memory (120) of the electronic device (100).

[0088] For example, content can be provided to an electronic device (100) from an external server in real time.

[0089] At least one processor (110) can obtain reference posture information corresponding to a first posture. At least one processor (110) can identify the first posture included in the content. At least one processor (110) can identify reference posture information corresponding to the first posture among a plurality of reference posture pieces of information.

[0090] Reference posture information can be information indicating a posture for guidance to the user. Reference posture information can be compared to the user's actual posture.

[0091] At least one processor (110) may generate a first control signal requesting sensing data representing the user's posture. At least one processor (110) may transmit the generated first control signal to the wearable device (200) via the communication circuit (160).

[0092] The wearable device (200) can receive a first control signal requesting sensing data from the electronic device (100). The wearable device (200) can obtain the sensing data. The wearable device (200) can transmit the sensing data to the electronic device (100) in response to the first control signal.

[0093] At least one processor (110) can receive sensing data from a wearable device (200) via a communication circuit (160). At least one processor (110) can measure (or estimate or evaluate) the user's posture based on the sensing data. At least one processor (110) can obtain the user's measured posture information based on the sensing data.

[0094] The measured posture information may be information indicating the user's current posture. For example, at least one processor (110) may obtain the information based on sensing data provided by the wearable device (200).

[0095] According to one embodiment, the measurement posture information may be updated based on the first captured image and the second captured image. A description thereof is provided in FIG. 16.

[0096] At least one processor (110) can obtain user's measured posture information based on sensing data received from the wearable device (200). At least one processor (110) can determine whether the actual posture (measured posture information) taken by the user is similar to the first posture included in the content.

[0097] At least one processor (110) can identify whether a predetermined event has occurred. The predetermined event may include an event in which the similarity between the first posture and the user's posture is identified as being below a threshold value.

[0098] At least one processor (110) can identify the user's measurement posture information based on the sensing data, obtain a similarity between the measurement posture information and the first posture, and if the similarity is less than a threshold value, identify that a predetermined event has occurred.

[0099] At least one processor (110) may compare reference posture information related to a first posture included in the content with measured posture information related to a posture taken by an actual user. Based on the comparison result, at least one processor (110) may determine whether a predetermined event has occurred.

[0100] At least one processor (110) can obtain reference posture information corresponding to a first posture based on content, and compare the reference posture information with the measured posture information to obtain a similarity. The operation of determining a predetermined event based on the similarity is described in FIG. 9.

[0101] At least one processor (110) controls the display (140) to display the content when a user input for outputting the content is received, and when a first posture included in the content is displayed, the processor can identify the user's measured posture information based on the sensing data.

[0102] At least one processor (110) can receive user input for outputting content. At least one processor (110) can display content corresponding to the user input. At least one processor (110) can display a screen (or image) including a first pose included in the content.

[0103] At least one processor (110) can identify measurement posture information corresponding to the sensing data for comparison with the displayed first posture.

[0104] At least one processor (110) may, when a user input for outputting content is received, transmit a first control signal to the wearable device (200) through the communication circuit (160) to request sensing data related to the user's posture, and obtain the sensing data from the wearable device (200) through the communication circuit (160).

[0105] At least one processor (110) can receive sensing data from the wearable device (200) through the communication circuit (160) while the first posture is displayed on the display (140), and identify the user's measured posture information based on the sensing data received while the first posture is displayed.

[0106] At least one processor (110) may receive sensing data from a wearable device (200) while displaying content including a first posture on a display (140). To analyze the user's posture in real time, reference posture information corresponding to the first posture displayed on the display (140) may be compared with measured posture information corresponding to the posture taken by the user.

[0107] When a given event is identified, at least one processor (110) may perform a photographing function via the image sensor (150). The at least one processor (110) may acquire a first photographed image as a result of the photographing function. The at least one processor (110) may acquire the first photographed image by performing a photograph of the front of the electronic device (100). The first photographed image may include a user. The first photographed image may include a human object representing the user.

[0108] When a given event is identified, at least one processor (110) may request a second captured image from the mobile robot (300). At least one processor (110) may request a captured image from the mobile robot (300). The mobile robot (300) may acquire the second captured image. The second captured image may include a user. The second captured image may include a human object representing the user in a specific pose. The mobile robot (300) may transmit the second captured image to the electronic device (100) in response to the request.

[0109] At least one processor (110) can obtain guide information. At least one processor (110) can obtain guide information for guiding a first posture based on reference posture information, a first captured image, measured posture information, and a second captured image, and control a display (140) to display content indicating the first posture together with the guide information. An operation for providing guide information is described in FIG. 8.

[0110] Guide information is acquired based on an AI (artificial intelligence) model that receives reference posture information, a first photographed image, measured posture information, and a second photographed image as input and generates output data, and the AI ​​model may include a large health model (LHM).

[0111] A large health model (LHM) may include features that generate output data based on natural language or images from a large language model (LLM) or large multimodal models (LMM).

[0112] For example, the electronic device (100) may include an AI model to which knowledge learning based on health information is applied to reduce hallucinations.

[0113] AI models can receive not only existing learned knowledge information but also biosensor data as input data (or input parameters).

[0114] AI models can infer and predict (or plan). By learning from more health information (or health knowledge), AI models can provide accurate and reliable information.

[0115] AI models can determine which actions to perform on which devices based on input sensor information, device capabilities (or device function information), and device capabilities. AI models can also determine which feedback to provide to which devices.

[0116] The AI ​​model can receive information related to the robot used by the user and information related to the terminal device (e.g., TV) used by the user in the form of prompt data.

[0117] The AI ​​model can store learned information about how data input from the robot will be processed on the TV. Based on this learned information, the AI ​​model can generate output data for planning and execution. The AI ​​model can generate output data expressed in modalities (e.g., images, language, voice). The electronic device (100) can obtain the output data generated by the AI ​​model.

[0118] At least one processor (110) can obtain guide information using an AI model. At least one processor (110) can input at least one of reference posture information, a first captured image, measured posture information, or a second captured image as input data to the AI ​​model. At least one processor (110) can obtain guide information as output data through the AI ​​model. The operation of obtaining guide information through the AI ​​model is described in FIGS. 12 and 13 .

[0119] At least one processor (110) may, when a predetermined event is identified, acquire a shooting position corresponding to a first posture, transmit a second control signal to the mobile robot (300) through the communication circuit (160) for requesting a shooting image including the user based on the shooting position, and receive a second shooting image including the user based on the shooting position from the mobile robot (300) through the communication circuit (160). Specific operations related to the shooting position are described in FIGS. 10 and 11.

[0120] At least one processor (110) can acquire a projection surface position corresponding to the first posture when guide information is acquired, and transmit a third control signal to the mobile robot (300) for outputting guide information based on the projection surface position through the communication circuit (160). A description related to the projection surface position is described in FIGS. 19 and 20.

[0121] The electronic device (100) can additionally utilize a wearable device (200) and a mobile robot (300) to provide home fitness services. By utilizing these additional devices, the user's posture can be more accurately recognized. The mobile robot (300) can capture blind spots that are difficult for the electronic device (100) to capture, allowing for a more accurate analysis of the user's posture.

[0122] FIG. 3 is a block diagram for explaining a specific configuration of the electronic device (100) of FIG. 2, according to one embodiment.

[0123] FIG. 3 is a block diagram of an exemplary electronic device (100) capable of performing the operations described in this document.

[0124] Referring to FIG. 3, the electronic device (100) may be one of various types of electronic devices, such as similar computing devices (not shown). The components, their relationships, and their functions illustrated in FIG. 3 are merely exemplary and do not limit the implementations described or claimed in this document. The electronic device (100) may be referred to as a mobile device, a user device, a multi-function device, a portable device, or a server.

[0125] The electronic device (100) may include components including at least one processor (110) (hereinafter referred to as processor (110)), at least one memory (120) (hereinafter referred to as memory (120)), at least one display (140) (hereinafter referred to as display (140)), at least one image sensor (150) (hereinafter referred to as image sensor (150)), at least one communication circuit (160) (hereinafter referred to as communication circuit (160)), and / or at least one sensor (170) (hereinafter referred to as sensor (170)). The above components are merely exemplary. For example, the electronic device (100) may include other components (e.g., power management integrated circuitry (PMIC), audio processing circuitry, an antenna, a rechargeable battery, or an input / output interface). For example, some components may be omitted from the electronic device (100). For example, some components may be integrated into one component.

[0126] The processor (110) may be implemented as one or more IC (integrated circuit (or circuitry)) chips and may perform various data processing. The processor (110) may include at least one electrical circuit and may individually or collectively perform distributed processing of instructions (or programs, data) stored in the memory (120). The processor (110) may include a processor assembly including one or more processing circuits. The processor (110) may include any processing circuit operative to control the performance and operations of one or more components (e.g., the memory (120), the display (140), the image sensor (150), the communication circuit (160), and / or the sensor (170)) of the electronic device (100). For example, the processor (110) (e.g., the application processor (AP)) may be implemented as a system on chip (SoC) (e.g., a single chip or a chipset). For example, the processor (110) may be implemented with multiple cores (or at least one core circuit), multiple chips, or multiple chipsets. For example, the processor (110) may include one or more processing circuits. For example, the processor (110) may include one or more processing circuits configured to individually and / or collectively perform various functions of the present disclosure. As a non-limiting example, at least a portion of the processor (110) may be included in a first chip of the electronic device (100), and at least another portion of the processor (110) may be included in a second chip of the electronic device (100) that is different from the first chip of the electronic device (100).

[0127] For example, the processor (110) may include a central processing unit (CPU) (111), a graphics processing unit (GPU) (112), a neural processing unit (NPU) (113), an image signal processor (ISP) (114), a display controller (115), a memory controller (116), a storage controller (117), a communication processor (CP) (118), and / or a sensor interface (119). These components of the processor (110) are merely exemplary. For example, the processor (110) may further include other components. For example, some components of the processor (110) may be omitted from the processor (110). For example, some components of the processor (110) may be included as separate components of the electronic device (100) outside the processor (110). For example, some components of the processor (110) (e.g., memory controller (116)) may be included within other components (e.g., at least a portion of memory (120), an interface (e.g., available for connection to at least one component of the electronic device (100)), a display (140) and / or an image sensor (150)).

[0128] The processor (110) may cause other components of the electronic device (100) to perform various operations by executing instructions stored in the memory (120). The CPU (111) (or central processing circuit) may be configured to control components of the processor (110) based on the execution of instructions stored in the memory (120) (e.g., volatile memory (121) and / or non-volatile memory (122)). The GPU (112) (or graphics processing circuit) may be configured to execute parallel operations (e.g., rendering). The NPU (113) (or neural processing circuit, or artificial intelligence (AI) chip) may be configured to execute operations for an artificial intelligence model (e.g., convolution computation). The ISP (114) (or image signal processing circuit) may be configured to process a raw image acquired through the image sensor (150) into a format suitable for a component within the electronic device (100) or a component of the processor (110). The display controller (115) (or display control circuit, or display processing unit (DPU)) may be configured to process an image acquired from the CPU (111), the GPU (112), the ISP (114), or the memory (120) (e.g., the volatile memory (121)) into a format suitable for the display (140). The memory controller (116) (or memory control circuit) may be configured to control reading data from the volatile memory (121) and writing data to the volatile memory (121). The storage controller (117) (or storage control circuit) may be configured to control reading data from the nonvolatile memory (122) and writing data to the nonvolatile memory (122).The CP (118) (communication processing circuit) may be configured to process data acquired from a component of the processor (110) into a format suitable for transmission to another electronic device via the communication circuit (160), or to process data acquired from another electronic device via the communication circuit (160) into a format suitable for processing by the component of the processor (110). For example, the communication circuit (160) may include one or more communication circuits. The sensor interface (119) (or sensing data processing circuit, sensor hub) may be configured to process data about the state of the electronic device (100) and / or the state of the surroundings of the electronic device (100), acquired via the sensor (170), into a format suitable for the component of the processor (110).

[0129] The memory (120) may include one or more storage media (or one or more storage devices). For example, the memory (120) may include a memory assembly including one or more storage media. For example, the one or more storage media may include permanent memory (e.g., non-volatile memory (122)) such as a hard drive, flash memory, read-only memory (ROM), semi-permanent memory (e.g., volatile memory (121)) such as random access memory (RAM), any other suitable type of storage (or storage assembly), or any combination thereof. The memory (120) may include cache memory, which is one or more different types of memory used to temporarily store data for a function or feature of the electronic device (100). As a non-limiting example, the cache memory may be included within the processor (110). The memory (120) may be fixedly embedded within the electronic device (100) or incorporated into one or more suitable types of components (e.g., a subscriber identity module (SIM) card and / or a secure digital (SD) card) that may be repeatedly inserted into and removed from the electronic device (100).

[0130] For example, the memory (120) may store one or more software applications, such as an operating system (or system) software application, a firmware software application, a driver software application, a plug-in (e.g., add-in, add-on, and / or applet) software application, and / or any other suitable software applications. For example, the one or more software applications may include instructions executable by the processor (110). For example, the memory (120) may store instructions callable by an application programming interface (API). For example, the memory (120) may store instructions within a library.

[0131] FIG. 4 is a flowchart illustrating an operation of providing guide information related to a user's posture, according to one embodiment.

[0132] In the following examples, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0133] According to one embodiment, S410 to S450 may be understood to be performed in a processor (e.g., processor (110) of FIG. 3) of an electronic device (e.g., electronic device (100) of FIG. 3).

[0134] Referring to FIG. 4, the electronic device (100) can display content (S410). The content may include images to guide a specific posture (e.g., a first posture). The content may include at least one of home training content and exercise lecture content. The content may include multiple images to guide at least one posture. The content may be implemented in the form of a video.

[0135] According to one embodiment, the electronic device (100) can obtain sensing data from the wearable device (200) (S420). The electronic device (100) can identify (or estimate) the user's posture based on the sensing data.

[0136] The electronic device (100) can acquire a first captured image from an image sensor (150) included in the electronic device (100) (S430). The electronic device (100) can perform a capture function with respect to the front direction of the electronic device (100) through the image sensor (150). The electronic device (100) can acquire the first captured image by performing the capture function. The first captured image can include a user (or a user object) located around the electronic device (100).

[0137] The electronic device (100) can acquire a second captured image from an image sensor included in the mobile robot (300) (S440). The mobile robot (300) can perform a capture function with respect to the surroundings of the mobile robot (300) through the image sensor. The mobile robot (300) can acquire a second captured image by performing the capture function. The second captured image can include a user (or a user object) located around the mobile robot (300).

[0138] The mobile robot (300) can acquire a second photographed image. The mobile robot (300) can acquire the second photographed image at a preset location. The preset location may vary depending on the user's settings or initial settings. The preset location may be described as an initial photographing location, an initial setting location, a fixed location, etc.

[0139] According to one embodiment, the preset position may be determined based on a captured image (or test image). The mobile robot (300) may acquire the captured image by capturing the surroundings. The mobile robot (300) may identify a preset object in the captured image. The mobile robot (300) may determine the preset position based on the identified object. The identified object may be a human object or an electronic device (TV) object. For example, the mobile robot (300) may determine a position located a threshold distance away from the position of the identified object in a threshold direction as the preset position.

[0140] According to one embodiment, the preset position may be an absolute position. The mobile robot (300) may store spatial map information. The mobile robot (300) may navigate using the spatial map information. The preset position may be one of multiple positions included in the spatial map information. For example, the preset position may be a position located in the spatial map information at a threshold distance in a threshold direction from the position where the electronic device (TV) is placed.

[0141] According to one embodiment, the mobile robot (300) may receive a first captured image from the electronic device (100). Upon obtaining the first captured image, the electronic device (100) may transmit the first captured image to the mobile robot (300). The mobile robot (300) may identify a preset location based on the first captured image received from the electronic device (100). For example, the mobile robot (300) may identify a location of a human object based on the first captured image. The mobile robot (300) may store spatial map information. The mobile robot (300) may identify a location of the identified human object based on the first captured image and the spatial map information. The mobile robot (300) may determine a location that is a threshold distance away from the location of the human object in a threshold direction as a preset location in the spatial map information.

[0142] The location at which a threshold distance is located in a critical direction from a specific object (e.g., a TV, a person) can vary depending on the settings.

[0143] For example, the preset location may be a location that is a critical distance (3 m) away from the electronic device (TV) in the direction in which the electronic device (TV) displays the screen.

[0144] For example, the preset position may be a position that is a critical distance (3 m) from the electronic device (TV) to the left of the critical angle (45 degrees) from the direction in which the electronic device (TV) displays the screen.

[0145] For example, the preset position may be a position that is a critical distance (3 m) away from the electronic device (TV) at a critical angle (45 degrees) to the right of the direction in which the electronic device (TV) displays the screen.

[0146] Critical angle, critical direction, and critical distance can be changed depending on the settings.

[0147] When a preset event is identified, the mobile robot (300) can move to a preset location.

[0148] According to one embodiment, the preset event may include a control command for movement received from the electronic device (100). When the control command is received from the electronic device (100), the mobile robot (300) may move to the preset location.

[0149] According to one embodiment, the preset event may include a notification received from the electronic device (100).

[0150] For example, a preset event may include a change in content provided by the electronic device (100). When the content provided (or displayed) by the electronic device (100) changes, the electronic device (100) may generate a notification indicating the content change. The electronic device (100) may transmit the notification indicating the content change to the mobile robot (300). When the notification indicating the content change is received from the electronic device (100), the mobile robot (300) may move to a preset location.

[0151] The electronic device (100) can provide guide information based on at least one of content (including the first posture), sensing data, the first captured image, or the second captured image (S450). The electronic device (100) can determine whether the user is accurately assuming the first posture based on at least one of content (including the first posture), sensing data, the first captured image, or the second captured image. The electronic device (100) can generate guide information indicating whether the user is accurately assuming the first posture. The electronic device (100) can provide the guide information to the user. The electronic device (100) can compare the current posture of the user with the first posture to obtain a similarity. If the similarity is greater than or equal to a threshold value, the electronic device (100) can determine that the user is accurately assuming the first posture.

[0152] According to one embodiment, the guide information may be provided to the user in the form of image information. The image information may be output based on the display (140) of the electronic device (100), the display of the wearable device (200), or the display of the mobile robot (300).

[0153] According to one embodiment, the guide information may be provided to the user in the form of audio information. The audio information may be output based on the speaker of the electronic device (100), the speaker of the wearable device (200), or the speaker of the mobile robot (300).

[0154] According to one embodiment, the electronic device (100) may input a first captured image and a second captured image as input data to an AI model. The AI ​​model may include a large health model (LHM). The electronic device (100) may obtain guide information as output data through the AI ​​model. The AI ​​model may generate guide information corresponding to the input data including the first captured image and the second captured image as output data.

[0155] FIG. 5 is a drawing for explaining an operation of capturing a user's posture according to one embodiment.

[0156] Referring to the embodiment (510) of FIG. 5, the electronic device (100) can display content including a first posture. A user wearing the wearable device (200) can view the content displayed on the electronic device (100). The user can follow the first posture guided by the content.

[0157] The mobile robot (300) can capture images of the user from the side and rear directions. The electronic device (100) can be fixed in a shooting direction toward the front of the electronic device (100). Therefore, the first captured image acquired through the image sensor (150) of the electronic device (100) can include only a portion of the user. The mobile robot (300) can include other parts of the user that are not included in the first captured image. The mobile robot (300) can capture images of the user's blind spots that are not included in the first captured image. The first captured image and the second captured image can be used complementarily.

[0158] Referring to the embodiment (520) of FIG. 5, the mobile robot (300) can capture a predetermined part of the user. It is assumed that capture information corresponding to the first posture is stored in the memory (120). The electronic device (100) can obtain the capture information corresponding to the first posture. The capture information may include at least one of a capture area or a capture target. A description related to this is provided in FIG. 11.

[0159] The electronic device (100) can acquire a photographing target (waist, hip) corresponding to the first posture. The electronic device (100) can acquire a photographing area to focus on photographing the photographing target. The electronic device (100) can generate a control command to move the mobile robot (300) to a specific location included in the photographing area. The electronic device (100) can generate a control command to photograph the photographing target after the mobile robot (300) moves to the specific location. The electronic device (100) can transmit the control command to the mobile robot (300). The electronic device (100) can perform a zoom-in function to focus on photographing the photographing target.

[0160] The mobile robot (300) can move to a shooting position and take pictures of a shooting target (waist, hips) based on a control command transmitted by the electronic device (100). The mobile robot (300) can obtain a second shooting image as a result of the shooting.

[0161] FIG. 6 is a drawing for explaining an operation of providing guide information for correcting a user's posture, according to one embodiment.

[0162] Referring to the embodiment (600) of FIG. 6, the electronic device (100) can display content including a first posture. The user can view the content including the first posture. It is assumed that the user is in a second posture different from the first posture. It is assumed that the mobile robot (300) captures the user in the second posture from the side or rear of the user.

[0163] The mobile robot (300) can acquire a second captured image by capturing a user in a second posture different from the first posture. If the user is identified as being in the second posture different from the first posture, the electronic device (100) can output guide information. The guide information can include at least one of information indicating that the user is not in the first posture (e.g., the hand positions are different) or a guide motion for the first posture (e.g., please move both hands forward). The guide information can include a second captured image of the user in the second posture. Through the second captured image, the user can easily recognize that he or she is in the second posture different from the first posture.

[0164] The electronic device (100) may display a screen (610) including guide information including a second captured image and content including a first pose. The screen (610) may include guide information (a second captured image) indicating a user assuming the second pose.

[0165] FIG. 7 is a drawing for explaining an image screen for correcting a user's posture according to one embodiment.

[0166] Referring to the embodiment (700) of FIG. 7, the embodiment of FIG. 6 can be applied as is. Additionally, the electronic device (100) can generate guide information including an image (711) representing the current user's posture and an image (712) representing the first posture. The electronic device (100) can provide a screen (710) including the guide information. The image (711) representing the current user's posture and the image (712) representing the first posture can include only a specific part of the user.

[0167] The electronic device (100) can perform a posture comparison function to generate guide information including only images (711, 712) that include only parts of the entire user's body that are different from the first posture.

[0168] According to one embodiment, the electronic device (100) may provide guide information while continuing to display content including the first posture.

[0169] FIG. 8 is a diagram illustrating an operation of generating guide information related to a user's posture, according to one embodiment.

[0170] In the following examples, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0171] Referring to FIG. 8, the electronic device (100) can receive a user input for displaying content (S805). When the user input is received, the electronic device (100) can display (or output) content including a first posture through the display (140) (S810).

[0172] According to one embodiment, the electronic device (100) can obtain reference posture information corresponding to the first posture (S815).

[0173] The electronic device (100) can request sensing data from the wearable device (200) (S820). The wearable device (200) can receive a sensing data request from the electronic device (100). The wearable device (200) can obtain sensing data in response to the request (S825). The wearable device (200) can transmit the sensing data to the electronic device (100).

[0174] The electronic device (100) can receive sensing data from the wearable device (200). The electronic device (100) can obtain the user's measurement posture information based on the sensing data (S835).

[0175] The electronic device (100) can determine whether a predetermined event is identified (S840). The predetermined event may include at least one of the following: an event in which a predetermined amount of time elapses from the time content including the first posture is displayed; an event in which a user command (or user input) for generating guide information is received; and an event in which posture similarity is identified as being below a threshold value. Events related to similarity are described in FIG. 9.

[0176] If a given event is not identified (S840-N), the electronic device (100) and the wearable device (200) may repeat steps S810, S815, S820, S825, S830, S835, and S840.

[0177] When a predetermined event is identified (S840-Y), the electronic device (100) can acquire a first captured image including the user (S845). The electronic device (100) can acquire the first captured image through the image sensor (150).

[0178] The electronic device (100) can request a captured image from the mobile robot (300) (S850). The mobile robot (300) can receive a request for a captured image from the electronic device (100).

[0179] When a request for a shooting image is received, the mobile robot (300) can move to the shooting position (S851).

[0180] The shooting location can be described in various embodiments.

[0181] For example, the shooting location may be a preset location. When a request for a captured image is received, the mobile robot (300) may move to the preset location. The preset location may be a fixed location. The preset location may be a predetermined location that allows the user's posture to be easily recognized relative to the electronic device (100).

[0182] For example, the shooting position may vary depending on the exercise posture being guided (e.g., the first posture). The appropriate shooting position may vary depending on the exercise posture. While the shooting angle of the electronic device (100) is fixed, the mobile robot (300) may perform shooting at different positions. The mobile robot (300) may acquire the second shooting image at different shooting positions depending on the exercise posture.

[0183] When a request for a captured image is received, the mobile robot (300) can acquire a shooting position corresponding to the movement posture. There may be various methods for acquiring a shooting position corresponding to the movement posture.

[0184] For example, a shooting position can be acquired based on a previously stored shooting position table. A shooting position table that maps different shooting positions for each exercise posture can be stored in the mobile robot (300). Once the exercise posture is specified, the mobile robot (300) can identify a shooting position corresponding to the specified exercise posture based on the shooting position table.

[0185] For example, the shooting location can be acquired based on at least one of the user's location or the user's posture. When a request for a shooting image is received, the mobile robot (300) can identify at least one of the user's location or the user's posture by capturing the user at the current location or capturing the user at a location where the user can be recognized. The mobile robot (300) can acquire the shooting location based on at least one of the identified user's location or the user's posture. The mobile robot (300) can identify a location where the user's posture can be accurately captured as the shooting location based on at least one of the identified user's location or the user's posture.

[0186] When the mobile robot (300) directly determines the shooting location, the mobile robot (300) can receive information about the movement posture (first posture) from the electronic device (100) together with the shooting image request.

[0187] The shooting location can be determined based on the user's current location. The shooting location can be a location facing the user's front, a location facing the user's side, or a location facing the user's back.

[0188] The mobile robot (300) can acquire a second captured image including the user in response to a request for a captured image transmitted by the electronic device (100) (S855). The mobile robot (300) can transmit the second captured image to the electronic device (100) (S860).

[0189] For example, it is assumed that the mobile robot (300) is positioned in front of the user. When a request to capture a squat exercise is received, the mobile robot (300) can scan the user. The mobile robot (300) can identify that the current location is in front of the user. The mobile robot (300) can identify a location corresponding to the side of the user as a capture location corresponding to the squat exercise posture. The mobile robot (300) can move to a location corresponding to the side of the user. The mobile robot (300) can capture the user at a location corresponding to the side of the user. When the side of the user is identified in the captured image, the mobile robot (300) can transmit the captured image to the electronic device (100).

[0190] The electronic device (100) can receive a second captured image from the mobile robot (300). The electronic device (100) can provide guide information based on at least one of the reference posture information, the measured posture information, the first captured image, or the second captured image (S865).

[0191] According to one embodiment, the mobile robot (300) may acquire a second captured image before moving to the shooting location and transmit the second captured image to the electronic device (100). If the mobile robot (300) is identified as having moved to the shooting location, the electronic device (100) may provide guide information. If the mobile robot (300) is identified as having moved to the shooting location, the electronic device (100) may calculate (or start) a movement count.

[0192] FIG. 9 is a diagram for explaining an operation of calculating similarity according to one embodiment.

[0193] In the following examples, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0194] Steps S905, S910, S915, S920, S925, S930, S935, S945, S950, S955, S960, and S965 of FIG. 9 may correspond to steps S805, S810, S815, S820, S825, S830, S835, S845, S850, S855, S860, and S865 of FIG. 8. Duplicate explanations are omitted.

[0195] According to one embodiment, after measurement posture information is acquired based on sensing data, the electronic device (100) can acquire a similarity between the reference posture information and the measurement posture information (S941). The similarity can be described as a similarity value, a similarity value, or a comparison value.

[0196] The electronic device (100) can identify whether the similarity is below a threshold value (S942). If the similarity is above the threshold value (S942-N), the electronic device (100) and the wearable device (200) can repeat steps S910, S915, S920, S925, S930, S935, S941, and S942.

[0197] If the similarity is less than the threshold value (S942-Y), the electronic device (100) and the mobile robot (300) can perform steps S945, S950, S955, S960, and S965.

[0198] Similarity can be calculated using techniques such as dynamic time warping (DTW), cosine similarity, and Pearson correlation coefficient. The calculation operation can be performed in an electronic device (100), a wearable device (200), or a mobile robot (300).

[0199] In the above description, it was described that the similarity is obtained based on sensing data acquired from a wearable device (200). Various methods may exist for estimating a user's posture.

[0200] According to one embodiment, the electronic device (100) can obtain measurement posture information based on a captured image rather than sensing data. The captured image may include at least one of a first captured image captured by the electronic device (100) or a second captured image captured by the mobile robot (300). The electronic device (100) can obtain measurement posture information indicating the user's posture through image analysis rather than sensing data. The electronic device (100) can obtain similarity based on reference posture information and measurement posture information (obtained by the image analysis result).

[0201] According to one embodiment, the electronic device (100) may obtain reference posture information based on content including a first posture, and then convert the reference posture information into inertial sensor information. The electronic device (100) may obtain inertial sensor information corresponding to the reference posture information using a preset conversion method. The electronic device (100) may obtain a similarity by comparing the inertial sensor information obtained based on the content with sensing data received from the wearable device (200). The similarity may be more easily calculated by comparing only the data itself without defining a posture.

[0202] FIG. 10 is a drawing for explaining an operation of requesting a photograph from a mobile robot (300) according to one embodiment.

[0203] In the following examples, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0204] Steps S1040 and S1065 of Fig. 10 may correspond to steps S840 and S865 of Fig. 8. Duplicate explanations are omitted.

[0205] When a given event is identified (S1040-Y), the electronic device (100) can identify the user's location (S1050-1).

[0206] According to one embodiment, the electronic device (100) can identify the location of the user based on a communication signal transmitted by the wearable device (200). The electronic device (100) can request location information from the wearable device (200). The wearable device (200) can transmit the location information to the electronic device (100). The electronic device (100) can identify (or obtain) the location of the user based on the location information transmitted by the wearable device (200).

[0207] According to one embodiment, the electronic device (100) can identify the location of the user based on the captured image. The electronic device (100) can capture a user positioned in front of the electronic device (100) through the image sensor (150). The electronic device (100) can identify a human object representing the user based on the captured image, and identify the location of the user based on the location of the human object.

[0208] The electronic device (100) can acquire shooting information corresponding to the first posture (S1050-2). The shooting information may include at least one of a shooting area or a shooting target. The shooting information may vary depending on the posture. A description of the shooting information is provided in FIG. 11.

[0209] The electronic device (100) can store map information about the space in which the electronic device (100) is placed.

[0210] For example, map information may be generated by a mobile robot (300). The mobile robot (300) may generate map information about a space through a driving function. The mobile robot (300) may transmit the map information to an electronic device (100). The electronic device (100) may receive the map information from the mobile robot (300). The electronic device (100) may store the map information.

[0211] The electronic device (100) can identify the location of a user (or person) based on map information and a captured image. The electronic device (100) can identify a current location among a plurality of locations included in the map information. The electronic device (100) can obtain a captured image capturing the surroundings. The electronic device (100) can identify a human object based on the captured image. The electronic device (100) can identify the location of the human object based on the location of the electronic device (100) that obtained the captured image. The electronic device (100) can identify the location of the human object by considering both the location of the electronic device (100) and the shooting direction. The electronic device (100) can identify the location of the human object as the user's location.

[0212] For example, map information can be generated based on user input. A user can directly generate map information for a space. The electronic device (100) can store the map information.

[0213] The electronic device (100) can identify a shooting location based on the user's location, map information, and shooting information (S1050-3). The shooting information may not indicate a specific, specific location. The electronic device (100) can identify at least one of the shooting location and the shooting target to move the mobile robot (300) to a specific location. The electronic device (100) can identify a shooting location corresponding to the user's location and shooting area among a plurality of locations included in the map information.

[0214] The electronic device (100) can transmit at least one of a control command requesting a photographed image, a photographing location, and a photographing target to the mobile robot (300) (S1050-4).

[0215] The mobile robot (300) can receive at least one of a control command requesting a photographed image, a photographing location, and a photographing target from the electronic device (100). The mobile robot (300) can obtain a second photographed image including the user based on at least one of the photographing location and the photographing target (S1055).

[0216] The mobile robot (300) can move to a shooting location and then photograph the subject. After moving to the shooting location, the mobile robot (300) can determine detailed settings of the image sensor of the mobile robot (300) to photograph the subject. For example, the mobile robot (300) can determine the focus and shooting direction of the image sensor to intensively photograph the subject.

[0217] The mobile robot (300) can obtain a second photographed image by performing a photographing function. The mobile robot (300) can transmit the second photographed image to the electronic device (100) (S1060).

[0218] The electronic device (100) can receive a second captured image from the mobile robot (300). The electronic device (100) can provide guide information based on at least one of the reference posture information, the measured posture information, the first captured image, or the second captured image (S1065).

[0219] FIG. 11 is a drawing for explaining shooting information according to one embodiment.

[0220] According to one embodiment, the shooting information (1100) of FIG. 11 may include predetermined information indicating a shooting method of the mobile robot (300). The shooting information (1100) may include at least one of a plurality of postures, a shooting area corresponding to each of the plurality of postures, and a shooting target corresponding to each of the plurality of postures. The shooting area may indicate an area where the mobile robot (300) should be positioned to shoot a user in the first posture. The shooting target may indicate which part of the entire body of the user in the first posture is to be shot.

[0221] For example, the shooting information may include a shooting area (user's side) corresponding to a first posture (squat) and a shooting target (full body) corresponding to the first posture (squat). The mobile robot (300) may acquire a second shooting image by shooting the user's full body from the user's side based on the shooting information.

[0222] For example, the shooting information may include a shooting area (back of the user) corresponding to the second posture (back extension) and a shooting target (full body) corresponding to the second posture (back extension). The mobile robot (300) may acquire a second shooting image by shooting the user's full body from the side of the user based on the shooting information.

[0223] For example, the shooting information may include a shooting area (user's side) corresponding to the third posture (plank) and a shooting target (waist, shoulder) corresponding to the third posture (plank). The mobile robot (300) may acquire a second shooting image by shooting the user's waist and shoulder from the user's side based on the shooting information.

[0224] For example, the shooting information may include a shooting area (user's side) corresponding to a first posture (sit-up) and a shooting target (neck) corresponding to the first posture (sit-up). The mobile robot (300) may acquire a second shooting image by shooting the user's neck from the user's side based on the shooting information.

[0225] FIG. 12 is a diagram for explaining an operation of managing data required for guide information according to one embodiment.

[0226] Referring to the embodiment (1200) of FIG. 12, the electronic device (100) may include at least one of a control module (1210), a data management module (1220), a posture similarity calculation module (1230), and an AI model (1240).

[0227] The electronic device (100) can receive user input for selecting specific content. The content may be exercise-related content for guiding the user's posture. Upon receiving user input for selecting content, the control module (1210) can control various devices required to generate guidance information. The various devices may include the electronic device (100), a wearable device (200), and a mobile robot (300).

[0228] The control module (1210) can control the display (140), image sensor (150), and speaker of the electronic device (100).

[0229] The control module (1210) can control a sensor (e.g., an inertial sensor or a biometric sensor) of the wearable device (200). The control module (1210) can generate a control signal for acquiring sensing data through the sensor of the wearable device (200). The control module (1210) can transmit the control signal to the wearable device (200).

[0230] The control module (1210) can control the display, motor (or moving member), image sensor, projection unit, and speaker of the mobile robot (300). The control module (1210) can generate a control signal for controlling the hardware configuration of the mobile robot (300). The control module (1210) can transmit the control signal to the mobile robot (300).

[0231] The control module (1210) can transmit data acquired as a result of the control operation to the data management module (1220). The data management module (1220) can be a module for storing and managing various types of data.

[0232] The data management module (1220) may include at least one of a content storage module (1221), a sensing data storage module (1222), a first captured image storage module (1223), or a second captured image storage module (1224).

[0233] The content storage module (1221) may be a module that stores content for guiding at least one posture.

[0234] The sensing data storage module (1222) may be a module that stores sensing data transmitted from a wearable device (200).

[0235] The first captured image storage module (1223) may be a module that stores the first captured image acquired by the image sensor (150) of the electronic device (100).

[0236] The second captured image storage module (1224) may be a module that stores the second captured image acquired by the image sensor of the mobile robot (300).

[0237] The data management module (1220) can transmit at least one of the stored information to another module according to a predetermined method.

[0238] The data management module (1220) can transmit content and sensing data to the posture similarity calculation module (1230). The posture similarity calculation module (1230) can receive content and sensing data from the data management module (1220).

[0239] The posture similarity calculation module (1230) can calculate (or obtain) a similarity indicating how similar the user's posture is to the posture included in the content based on content and sensing data. The posture similarity calculation module (1230) can compare the similarity with a threshold value. If the similarity is less than the threshold value, the posture similarity calculation module (1230) can transmit a notification indicating that a predetermined event has occurred to the data management module (1220).

[0240] When a notification indicating that a given event has occurred is received, the data management module (1220) can transmit data stored in each storage module to the AI ​​model (1240). The data management module (1220) can transmit content, sensing data, the first captured image, and the second captured image to the AI ​​model (1240).

[0241] The AI ​​model (1240) may be a model that receives content, sensing data, and at least one image as input and generates guide information to guide the user's posture as output data. The AI ​​model (1240) may include an LHM (large health model) model.

[0242] The AI ​​model (1240) can output guide information. The guide information can be provided in at least one of image, audio, or text formats. The AI ​​model (1240) can be implemented in a multimodal structure.

[0243] According to one embodiment, the AI ​​model (1240) may generate exercise assessment information. The exercise assessment information may represent an evaluation index related to the user's exercise ability.

[0244] An LHM can be a generative AI model that typically generates new forms of data based on user input. An LHM may include models that generate images and / or models that generate language.

[0245] LHM can be implemented using a generative adversarial network (GAN) or a variational autoencoder (VAE). Diffusion-based generative models using VAEs and transformer structures can be applied to LHM.

[0246] Large multimodal models (LMMs) that can recognize various types of data input, such as text, images, and voice, and generate corresponding new data can be applied to LHMs.

[0247] An LHM can include a large vision model (LVM), which takes image data as input and produces output, and a large language model (LLM), which takes language data as input and produces output. An LHM can also include an LMM, which can simultaneously input multiple modalities, such as images or language.

[0248] The LHM can receive sensor data (biometric data, activity data) and other modality data (images, language, contextual information, environmental information) as input data. It can generate data for evaluating a specified performance status as output data. It can also generate guidance information (voice, text, images) as output data. It can also generate control signals or control-related data as output data.

[0249] LHM can evaluate the user's exercise performance status (exercise posture, exercise count, exercise duration, exercise performance score).

[0250] LHM can generate guidance information (voice, text, or image guidance) as output data. Guidance information can be provided in different ways depending on the user's exercise posture.

[0251] The LHM may include generative AI based on specialized health information. The LHM can match the behavioral patterns performed based on the generative AI's functions and content. The LHM may perform an inference function that predicts (or plans) the behavior of connected devices. The LHM may also perform an action function that executes the predicted inference function. The LHM may include an AI model capable of performing both inference and action functions.

[0252] The inference function may include a function that identifies the user's actions. The inference function may include an action that determines substantially the same posture even if the user's physical information (height, weight, or body shape) differs.

[0253] The inference function may include an action to determine whether the mobile robot (300) took the picture at a preset location after repositioning (an action to find the location again).

[0254] The LHM can generate information to determine where to move the mobile robot (300) based on the content provided to the user. The LHM can generate information indicating what guidance the mobile robot (300) will provide to the user. The LHM can generate at least one of the location of the mobile robot (300) or guidance information to be displayed by the mobile robot (300). The generated information can be transmitted to the mobile robot (300).

[0255] According to the default settings, the electronic device (100) can display guide information through the display (140) of the electronic device (100). If the electronic device (100) determines that the user is in a position where the display (140) of the electronic device (100) cannot be viewed, the electronic device (100) can control the mobile robot (300) to output the guide information (see FIG. 18).

[0256] FIG. 13 is a diagram illustrating an AI model that generates guide information according to one embodiment.

[0257] Embodiment (1300) of Fig. 13 may correspond to embodiment (1200) of Fig. 12. The AI ​​model (1340) may include at least one of a first extraction module (1341), a second extraction module (1342), a third extraction module (1343), a fourth extraction module (1344), an encoder (1345), or a connection layer (1346).

[0258] The first extraction module (1341) may be a module that analyzes content to obtain reference posture information. The first extraction module (1341) may be a module that obtains characteristics of a guide posture included in the content.

[0259] The second extraction module (1342) may be a module that analyzes sensing data to obtain measurement posture information. The second extraction module (1342) may be a module that obtains characteristics of the posture currently being taken by the user.

[0260] The third extraction module (1343) may be a module that acquires the posture features of the user object acquired from the first captured image.

[0261] The fourth extraction module (1344) may be a module that acquires the posture features of the user object acquired from the second captured image.

[0262] The encoder (1345) can receive feature data (or feature information) extracted from multiple extraction modules (1341, 1342, 1343, 1344) and output a result value. The encoder (1345) can be described as a transformer.

[0263] The connection layer (1346) may be a module that performs a transformation (or calculation) to express the result value output from the encoder (1345) in a predetermined manner. The connection layer may be described as a dense layer.

[0264] The data output through the connection layer (1346) may include at least one of guide information or exercise evaluation information.

[0265] FIG. 14 is a diagram for explaining an operation of obtaining guide information using an AI model stored in an electronic device (100), according to one embodiment.

[0266] In the following examples, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0267] Referring to FIG. 14, according to steps S815, S835, S845, and S855 of FIG. 8, the electronic device (100) can obtain reference posture information, measurement posture information, a first captured image, and a second captured image (S1470). FIG. 14 specifically describes operation S865 of FIG. 8.

[0268] According to one embodiment, the electronic device (100) can obtain guide information for guiding the first posture by inputting at least one of reference posture information, measured posture information, a first captured image, and a second captured image into an AI model (S1475). The AI ​​model may be a model that generates guide information based on input data. The AI ​​model is described in FIGS. 12 and 13. According to one embodiment, the AI ​​model may be stored in the electronic device (100). The AI ​​model may be provided directly from the electronic device (100) in an on-device form.

[0269] The electronic device (100) can display content including the first posture along with guide information (S1480). The electronic device (100) can display content including the previously displayed first posture on the display (140) while outputting the guide information.

[0270] According to one embodiment, the guide information may be displayed in the form of an image. The electronic device (100) may display a screen including content including the first posture and guide information on the display (140). Various methods may be used to display the guide information in the form of an image.

[0271] For example, guide information may be displayed on the display (140) in a pop-up form separately from the content including the first posture.

[0272] For example, the electronic device (100) can generate a new screen by combining content including guide information and a first posture. The electronic device (100) can display the newly generated screen on the display (140).

[0273] According to one embodiment, the guide information may be output in audio format. The electronic device (100) may output the guide information through a speaker. While the audio guide information is output through the speaker, the electronic device (100) may display content including the first posture on the display (140).

[0274] FIG. 15 is a diagram for explaining an operation of obtaining guide information using an AI model stored in a server, according to one embodiment.

[0275] In the following examples, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0276] According to one embodiment, steps S1570 and S1580 of FIG. 15 may correspond to steps S1470 and S1480 of FIG. 14. Duplicate descriptions are omitted. In FIG. 14, it is described that the AI ​​model is stored in the electronic device (100). If the hardware resources of the electronic device (100) are insufficient, the processing time required to generate guide information in the electronic device (100) may take a relatively long time. To shorten the processing time and centralize data, the AI ​​model may be stored in the server (400) in FIG. 15.

[0277] The electronic device (100) can obtain at least one of reference posture information, measurement posture information, a first captured image, and a second captured image (S1570).

[0278] The electronic device (100) can transmit at least one of the reference posture information, the measured posture information, the first captured image, and the second captured image to the server (400) (S1571).

[0279] The server (400) can receive at least one of reference posture information, measured posture information, a first captured image, and a second captured image from the electronic device (100). The server (400) can obtain guide information for guiding the first posture by inputting at least one of the reference posture information, measured posture information, the first captured image, and the second captured image into the AI ​​model (S1575). The server (400) can transmit the guide information to the electronic device (100) (S1576).

[0280] The electronic device (100) can receive guide information from the server (400). The electronic device (100) can display content including the first posture together with the guide information (S1580).

[0281] FIG. 16 is a diagram for explaining an operation of comparing detailed information according to one embodiment.

[0282] In the following examples, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0283] According to one embodiment, steps S1670 and S1680 of FIG. 16 may correspond to steps S1470 and S1480 of FIG. 14. Duplicate descriptions are omitted. The electronic device (100) may obtain at least one of reference posture information, measured posture information, a first captured image, and a second captured image (S1670).

[0284] The electronic device (100) can obtain first auxiliary posture information of a human object included in a first captured image (S1671). The electronic device (100) can identify the human object included in the first captured image by analyzing the first captured image. The electronic device (100) can identify first auxiliary posture information corresponding to the human object. The electronic device (100) can obtain first skeletal data based on the joint positions of the human object. The electronic device (100) can obtain first auxiliary posture information based on the first skeletal data. A description related to the skeletal data is described in FIG. 25.

[0285] The electronic device (100) can obtain second auxiliary posture information of a human object included in a second captured image (S1672). The electronic device (100) can identify the human object included in the second captured image by analyzing the second captured image. The electronic device (100) can identify second auxiliary posture information corresponding to the human object. The electronic device (100) can obtain second skeletal data based on the joint positions of the human object. The electronic device (100) can obtain second auxiliary posture information based on the second skeletal data.

[0286] The electronic device (100) can update the measurement posture information based on the first auxiliary posture information and the second auxiliary posture information (S1673). The electronic device (100) can obtain updated measurement posture information (second measurement posture information) by updating the measurement posture information based on the first auxiliary posture information and the second auxiliary posture information (S1674).

[0287] The electronic device (100) can use the first auxiliary posture information and the second auxiliary posture information to increase the accuracy of the already acquired measurement posture information.

[0288] The measured posture information may be information acquired using a sensor of the wearable device (200). The first auxiliary posture information may be information acquired using an image sensor (150) of the electronic device (100). The second auxiliary posture information may be information acquired using an image sensor of the mobile robot (300). It may be difficult to accurately determine the user's posture using only the sensing data of the wearable device (200). The electronic device (100) may additionally determine the user's posture using a photographed image captured by the electronic device (100) and / or the mobile robot (300).

[0289] The user's measurement posture information can be finally obtained by additionally using the first captured image captured by the electronic device (100) and the second captured image captured by the mobile robot (300). The electronic device (100) can obtain measurement posture information (first measurement posture information). The electronic device (100) can change (or update) the measurement posture information (first measurement posture information) based on the first auxiliary posture information and the second auxiliary posture information. The changed (or updated) measurement posture information can be described as second measurement posture information.

[0290] The electronic device (100) can obtain guide information for guiding the first posture by comparing the reference posture information with the updated measurement posture information (second measurement posture information) (S1675). The electronic device (100) can compare the reference posture provided in the content with the measurement posture actually taken by the user. Based on the comparison result, the electronic device (100) can generate guide information for guiding the user to take the first posture.

[0291] The electronic device (100) can display content including a first posture along with guide information (S1680).

[0292] The above description describes an operation of first acquiring measurement posture information and then using the first and second captured images. According to one embodiment, the electronic device (100) can acquire measurement posture information based on the sensing data, the first captured image, and the second captured image.

[0293] FIG. 17 is a drawing for explaining an operation of providing guide information through a mobile robot (300) according to one embodiment.

[0294] In the following examples, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0295] Steps S1770 and S1775 of Fig. 17 may correspond to steps S1470 and S1475 of Fig. 14. Duplicate explanation is omitted.

[0296] According to one embodiment, after acquiring guide information, the electronic device (100) may transmit the guide information to the mobile robot (300) (S1776). The mobile robot (300) may receive the guide information from the electronic device (100). The mobile robot (300) may provide the guide information to the user (S1777). The method of providing the guide information may vary.

[0297] According to one embodiment, the guide information may include a guide image. The mobile robot (300) may display the guide image through the display of the mobile robot (300).

[0298] According to one embodiment, the guide information may include guide audio. The mobile robot (300) may output the guide audio through the speaker of the mobile robot (300).

[0299] According to one embodiment, the guide information may include a guide image (projection image). The mobile robot (300) may output the guide image (projection image) through the projection unit of the mobile robot (300). The projection operation is described in FIG. 18.

[0300] The electronic device (100) can display content including the first posture (S1780). The electronic device (100) can display only content including the first posture without outputting guide information.

[0301] FIG. 18 is a drawing for explaining an operation of a mobile robot (300) to output guide information according to one embodiment.

[0302] A mobile robot (300) can output guide information to a projection surface. The guide information can include a guide image. The mobile robot (300) can include a projection unit that performs a projection function. The projection unit can include at least one of a projection lens and a projection circuit. The mobile robot (300) can determine a projection surface position for projecting the guide image. The mobile robot (300) can output the guide image to the determined projection surface position through the projection unit.

[0303] According to the embodiment (1810) of Fig. 18, the mobile robot (300) can output a guide image on the wall of the projection space. The mobile robot (300) can output a guide image on the wall of the projection space according to the user's posture (posture looking at the wall).

[0304] According to the embodiment (1820) of Fig. 18, the mobile robot (300) can output a guide image on the ceiling surface of the projection space. The mobile robot (300) can output a guide image on the ceiling surface of the projection space according to the user's posture (posture looking at the ceiling surface).

[0305] According to the embodiment (1830) of Fig. 18, the mobile robot (300) can output a guide image on the floor of the projection space. The mobile robot (300) can output a guide image on the floor of the projection space according to the user's posture (posture looking at the floor).

[0306] The mobile robot (300) can determine at least one of a projection position, a projection surface position, or a projection direction based on at least one of a user position, a user posture, or a user gaze.

[0307] The projection position can indicate the position at which the projection image will be output. The mobile robot (300) can move to the projection position and output the projection image at the projection surface position.

[0308] The projection surface position can indicate the location where the projection image will be output. The mobile robot (300) can output the projection image on a specified surface.

[0309] FIG. 19 is a drawing for explaining an operation of providing a projection surface position for outputting guide information by a mobile robot (300) according to one embodiment.

[0310] In the following examples, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0311] Steps S1970, S1975, S1977, and S1980 of FIG. 19 may correspond to steps S1770, S1775, S1777, and S1780 of FIG. 17. Duplicate explanations are omitted.

[0312] After the guide information is obtained, the electronic device (100) can identify the user's location (S1976-1).

[0313] According to one embodiment, the electronic device (100) can identify the location of the user based on a communication signal transmitted by the wearable device (200). The electronic device (100) can request location information from the wearable device (200). The wearable device (200) can transmit the location information to the electronic device (100). The electronic device (100) can identify (or obtain) the location of the user based on the location information transmitted by the wearable device (200).

[0314] According to one embodiment, the electronic device (100) can identify the location of the user based on the captured image. The electronic device (100) can capture a user positioned in front of the electronic device (100) through the image sensor (150). The electronic device (100) can identify a human object representing the user based on the captured image, and identify the location of the user based on the location of the human object.

[0315] The electronic device (100) can acquire projection information corresponding to the first posture (S1976-2). The projection information may include at least one of a projection area or a projection surface area. The projection information may vary for each posture. A description of the projection information is provided in FIG. 20.

[0316] The electronic device (100) can store map information about the space in which the electronic device (100) is placed.

[0317] For example, map information may be generated by a mobile robot (300). The mobile robot (300) may generate map information about a space through a driving function. The mobile robot (300) may transmit the map information to an electronic device (100). The electronic device (100) may receive the map information from the mobile robot (300). The electronic device (100) may store the map information.

[0318] For example, map information can be generated based on user input. A user can directly generate map information for a space. The electronic device (100) can store the map information.

[0319] The electronic device (100) can identify at least one of a projection location or a projection surface location based on the user's location, map information, and projection information (S1976-3). The projection information may not indicate a specific, specific location. The electronic device (100) can identify at least one of the projection location or the projection surface location to move the mobile robot (300) to a specific location. The electronic device (100) can identify a projection location corresponding to the user's location and projection area among a plurality of locations included in the map information.

[0320] The electronic device (100) can transmit at least one of guide information, projection position, and projection surface position to the mobile robot (300) (S1976-4).

[0321] The mobile robot (300) can receive at least one of guide information, projection position, and projection surface position from the electronic device (100). The mobile robot (300) can provide guide information based on at least one of the projection position and projection surface position (S1977).

[0322] The mobile robot (300) can output guide information to the projection surface location after moving to the projection position. For example, the mobile robot (300) includes a projection unit, and the mobile robot (300) can output a guide image to the projection surface location through the projection unit.

[0323] While the mobile robot (300) provides guide information, the electronic device (100) can display content including the first posture (S1980).

[0324] FIG. 20 is a drawing for explaining projection information according to one embodiment.

[0325] According to one embodiment, the projection information (2000) of FIG. 20 may include predetermined information indicating a projection method of the mobile robot (300). The projection information (2000) may include at least one of a plurality of postures, a projection area corresponding to each of the plurality of postures, and a projection surface area corresponding to each of the plurality of postures. The projection area may indicate an area where the mobile robot (300) must be positioned to perform a function of outputting guide information. The projection surface area may indicate an area where guide information (or guide image) is output.

[0326] For example, the projection information may include a projection area (user side, front) corresponding to the first posture (squat) and a projection surface area (wall) corresponding to the first posture (squat). The mobile robot (300) may output guide information to the wall from the user's side (or front) based on the projection information.

[0327] For example, the projection information may include a projection area (user side, front) corresponding to the second posture (back extension) and a projection surface area (floor surface) corresponding to the second posture (back extension). The mobile robot (300) may output guide information to the floor from the user's side (or front) based on the projection information.

[0328] For example, the projection information may include a projection area (user side, front) corresponding to the third posture (plank) and a projection surface area (floor) corresponding to the third posture (plank). The mobile robot (300) may output guide information to the floor from the user's side (or front) based on the projection information.

[0329] For example, the projection information may include a projection area (user side, front) corresponding to the first posture (sit-up) and a projection surface area (ceiling) corresponding to the first posture (sit-up). The mobile robot (300) may output guide information to the ceiling from the user's side (or front) based on the projection information.

[0330] FIG. 21 is a drawing for explaining an operation of guiding the initial position of a user and the initial position of a mobile robot (300) according to one embodiment.

[0331] The electronic device (100) may include an image sensor (150). The electronic device (100) may acquire a photographed image of the front of the electronic device (100) through the image sensor (150). The electronic device (100) may identify a human object representing the user through the photographed image. The electronic device (100) may determine whether the user is standing at a basic position through the position of the human object.

[0332] The electronic device (100) can identify (or track) the user's location in real time through captured images. The electronic device (100) can guide the user's initial location when providing exercise-related content.

[0333] The initial location may be a fixed location. For example, the initial location may be a fixed distance from the center of the electronic device (100). If the user's location is too close or too far from the electronic device (100), it may be difficult to analyze the user's posture in the captured image. The electronic device (100) may guide the user to position themselves at the initial location.

[0334] Referring to the embodiment (2110) of FIG. 21, the user may be positioned closer to the electronic device (100) than the virtual initial location (2101). The electronic device (100) may identify the user's location based on the captured image. The electronic device (100) may provide guidance information (e.g., "You are too close to the TV. Please move back a little further") to help the user move to the initial location.

[0335] Referring to the embodiment (2120) of FIG. 21, the user may be positioned further away from the electronic device (100) than the virtual initial location (2101). The electronic device (100) may identify the user's location based on the captured image. The electronic device (100) may provide guidance information (e.g., "Move in front of the TV," "Move a little further toward the center") to help the user move to the initial location.

[0336] Referring to the embodiment (2130) of FIG. 21, a user may be located at a virtual initial location (2101). If the user is identified as being at the initial location, the electronic device (100) may provide notification information indicating that the user is at the initial location (e.g., location adjustment is complete).

[0337] If the user is identified as being at the initial location, the electronic device (100) can control the mobile robot (300) to move to an initial location other than the user's initial location. For convenience of distinction, the user's initial location may be described as the first location, and the mobile robot's (300's) initial location may be described as the second location. The electronic device (100) can generate a control command to move the mobile robot (300) to a designated location to photograph the user. The designated location may vary depending on the user's settings.

[0338] For example, the determined location may be a location of one of the areas representing the location between the electronic device (100) and the user.

[0339] For example, the determined location may be one of the locations within a threshold distance from the electronic device (100).

[0340] The reason for first determining whether the user is located at the initial location is to ensure greater freedom of movement for the user. If the location of the mobile robot (300) is determined first, it may interfere with the user's movements. Once the user is located at the first location and the mobile robot (300) is located at the second location, the electronic device (100) can display content on the display (140).

[0341] FIG. 22 is a drawing for explaining an operation of guiding the initial position of a user and the initial position of a mobile robot (300) according to one embodiment.

[0342] In the following examples, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0343] Steps S2205 and S2210 of Fig. 22 may correspond to steps S905 and S910 of Fig. 9. Duplicate explanation is omitted.

[0344] In one embodiment, when a user input for displaying content is received, the electronic device (100) can identify the user location and the mobile robot (300) location (S2206).

[0345] According to one embodiment, the electronic device (100) can identify the location of the user based on a communication signal transmitted by the wearable device (200). The electronic device (100) can request location information from the wearable device (200). The wearable device (200) can transmit the location information to the electronic device (100). The electronic device (100) can identify (or obtain) the location of the user based on the location information transmitted by the wearable device (200).

[0346] According to one embodiment, the electronic device (100) can identify the location of the user based on the captured image. The electronic device (100) can capture a user positioned in front of the electronic device (100) through the image sensor (150). The electronic device (100) can identify a human object representing the user based on the captured image, and identify the location of the user based on the location of the human object.

[0347] According to one embodiment, the electronic device (100) can identify the location of the mobile robot (300) based on a communication signal transmitted by the mobile robot (300). The electronic device (100) can request location information from the mobile robot (300). The mobile robot (300) can transmit the location information to the electronic device (100). The electronic device (100) can identify (or obtain) the location of the mobile robot (300) based on the location information transmitted by the mobile robot (300).

[0348] According to one embodiment, the electronic device (100) can identify the location of the mobile robot (300) based on a captured image. The electronic device (100) can capture the mobile robot (300) located in front of the electronic device (100) through an image sensor (150). The electronic device (100) can identify a robot object representing the mobile robot (300) based on the captured image, and can identify the location of the mobile robot (300) based on the location of the robot object.

[0349] The electronic device (100) can identify whether the user location is at a predetermined first location (S2207-1).

[0350] According to one embodiment, the first predetermined position may be determined based on the electronic device (100). The first predetermined position may indicate a position located a predetermined distance away in a predetermined direction based on the arrangement position of the electronic device (100).

[0351] According to one embodiment, the determined first location may be determined based on map information. The electronic device (100) may obtain the determined first location representing the user's initial location based on map information related to space.

[0352] If the user's location is not the first predetermined location (S2207-1-N), the electronic device (100) may provide guide information to guide the user to move to the first predetermined location (S2207-2). The electronic device (100) may repeat step S2207-1.

[0353] When the user is at a predetermined first position (S2207-1-Y), the electronic device (100) can transmit a control command to the mobile robot (300) to move to a predetermined second position (S2207-4).

[0354] According to one embodiment, the determined second position may be determined based on the electronic device (100). The determined second position may indicate a position located a determined distance away in a determined direction based on the arrangement position of the electronic device (100).

[0355] According to one embodiment, the determined second location may be determined based on map information. The electronic device (100) may obtain the determined second location representing the user's initial location based on map information related to the space.

[0356] The mobile robot (300) can receive a control command from the electronic device (100) to move to a predetermined second location. The mobile robot (300) can move to the predetermined second location (S2207-4).

[0357] When the user is in a predetermined first position (S2207-1-Y), the electronic device (100) can display content including the first posture (S2210).

[0358] FIG. 23 is a diagram for explaining an operation of requesting sensing data according to a set event, according to one embodiment.

[0359] In the following examples, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0360] Steps S2305, S2310, S2345, S2350, S2355, S2360, and S2365 of FIG. 23 may correspond to steps S805, S810, S845, S850, S855, S860, and S865 of FIG. 8. The order of steps S815, S820, S825, S830, S835, and S840 of FIG. 8 may be partially changed. Duplicate explanations are omitted.

[0361] According to one embodiment, after displaying content including a first posture, the electronic device (100) may determine whether a predetermined event is identified (S2315). The predetermined event may include at least one of an event in which a predetermined amount of time elapses from the time content including the first posture is displayed, and an event in which a user command (or user input) for generating guide information is received.

[0362] If a given event is not identified (S2315-N), the electronic device (100) may repeat steps S2310 and S2315.

[0363] When a given event is identified (S2315-Y), the electronic device (100) can obtain reference posture information corresponding to the first posture (S2320). The electronic device (100) can request sensing data from the wearable device (200) (S2325).

[0364] The wearable device (200) can receive a sensing data request from the electronic device (100). The wearable device (200) can obtain the sensing data (S2330). The wearable device (200) can transmit the sensing data to the electronic device (100) (S2335).

[0365] The electronic device (100) can receive sensing data from the wearable device (200). The electronic device (100) can obtain the user's measurement posture information based on the sensing data (S2340).

[0366] When the reference posture information and the measurement posture information are acquired, the electronic device (100) can perform steps S2345, S2350, S2355, S2360, and S2365.

[0367] FIG. 24 is a drawing for explaining an operation of estimating a user's posture according to one embodiment.

[0368] Referring to FIG. 24, the electronic device (100) can identify (or determine) the user's pose using a pose estimation technique. Specifically, the electronic device (100) can use three-dimensional rotation angles, such as roll, yaw, and pitch. The electronic device (100) can identify the user's measured pose information based on at least one of the sensing data, the first captured image, or the second captured image.

[0369] According to one embodiment (2410), roll, yaw, and pitch are defined to determine the user's posture. Roll may refer to an angle of rotation around a longitudinal axis. Yaw may refer to an angle of rotation around a vertical axis. Pitch may refer to an angle of rotation around a lateral axis.

[0370] According to one embodiment (2420), it is assumed that a user is doing a push-up motion with his / her face facing the floor. The electronic device (100) can acquire face rotation angle data including a roll of about 0 degrees, a yaw of about 90 degrees, and a ptich of about -85 degrees. The electronic device (100) can acquire the user's measured posture information based on the acquired data.

[0371] According to one embodiment (2430), it is assumed that a user is performing a push-up motion while facing the electronic device (100). When the user is facing the electronic device (100), the electronic device (100) can obtain face rotation angle data including a roll of approximately -90 degrees, a yaw of approximately 0 degrees, and a ptich of approximately 0 degrees. The electronic device (100) can obtain the user's measured posture information based on the obtained data.

[0372] Although Figure 24 describes the user's face, the same method can be applied to all parts of the user's body.

[0373] FIG. 25 is a drawing for explaining an operation for estimating a user's posture according to one embodiment.

[0374] Referring to FIG. 25, according to one embodiment, the electronic device (100) can identify a human object based on at least one of the first captured image or the second captured image. The electronic device (100) can obtain skeletal data based on the human object. Skeletal data may refer to data corresponding to joints and major parts of a human, and may be expressed as a skeleton.

[0375] Skeletal data can obtain information about vertex coordinates, lengths of connecting lines, and angles between connecting lines. Here, connecting lines can be expressed as edges.

[0376] Vertex coordinate information may refer to points of a predetermined part for forming a skeleton of a human object. The number of vertices may be predetermined according to a user's settings. Various embodiments of the present disclosure assume that there are 15 vertices. In a human object, the vertices may be v1 to v15. Specifically, v1 may refer to the neck position, v2 may refer to the head position, v3 and v6 may refer to the shoulder positions of both sides, v4 and v7 may refer to the elbow positions of both sides, v5 and v8 may refer to the wrist positions of both sides, v9 may refer to the center of the abdomen position, v10 and v13 may refer to the pelvic positions of both sides, v11 and v14 may refer to the knee positions of both sides, and v12 and v15 may refer to the ankle positions of both sides.

[0377] A connecting line may refer to a line connecting two predetermined vertices among a plurality of vertices. The two vertices may be determined in a predetermined manner. In order to form a human skeleton, two pairs of vertices may be v1-v2, v1-v3, v3-v4, v4-v5, v1-v6, v6-v7, v7-v8, v1-v9, v9-v10, v10-v11, v11-v12, v9-v13, v13-v14, v14-v15. Additionally, each connecting line consisting of two pairs of vertices can be expressed as L(v1,v2), L(v1,v3), L(v3,v4), L(v4,v5), L(v1,v6), L(v6,v7), L(v7,v8), L(v1,v9), L(v9,v10), L(v10,v11), L(v11,v12), L(v9,v13), L(v13,v14), L(v14,v15).

[0378] The angle information between connecting lines may mean the angle between two predetermined connecting lines among multiple connecting lines. The angle information is A(L(v1,v2), L(v1,v3)), A(L(v1,v3)), L(v3,v4), A(L(v3,v4), L(v4,v5)), A(L(v1,v2), L(v1,v6)), A(L(v1,v6), L(v6,v7)), A(L(v6,v7), L(v7,v8)), A(L(v1,v3), L(v1,v9)), A(L(v1,v9), L(v9,v10)), A(L(v9,v10), L(v10,v11)), A(L(v10,v11), L(v11,v12)), A(L(v1,v6), L(v1,v9)), A(L(v1,v9), L(v9,v13)), It can be expressed as A(L(v9,v13), L(v13,v14)), A(L(v13,v14), L(v14,v15)).

[0379] The electronic device (100) can obtain the user's measurement posture information based on the connection lines and the angle information between the connection lines.

[0380] FIG. 26 is a flowchart for explaining a method for controlling an electronic device, according to one embodiment.

[0381] In the following examples, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0382] According to one embodiment, S2605 to S2630 may be understood to be performed in a processor (e.g., processor (110) of FIG. 3) of an electronic device (e.g., electronic device (100) of FIG. 3).

[0383] Referring to FIG. 26, according to one embodiment, a control method of an electronic device connected to a wearable device and a mobile robot includes a step of displaying content for guiding a first posture (S2605), a step of receiving sensing data from a wearable device by requesting sensing data related to a user's posture from the wearable device (S2610), a step of identifying whether a predetermined event has occurred based on the first posture and the sensing data (S2615), a step of obtaining a first captured image including the user through the electronic device when the predetermined event is identified (S2620), a step of receiving a second captured image including the user from the mobile robot by requesting a captured image including the user from the mobile robot (S2625), and a step of providing guide information based on the content including the first posture, the first captured image, the sensing data, and the second captured image (S2630).

[0384] The step (S2615) of identifying whether an event has occurred can identify the user's measurement posture information based on sensing data, obtain the similarity between the measurement posture information and the first posture, and identify that a predetermined event has occurred if the similarity is less than a threshold value.

[0385] The step of obtaining similarity may include obtaining reference posture information corresponding to the first posture based on the content, and comparing the reference posture information with the measured posture information to obtain similarity.

[0386] The step of providing guide information (S2630) may acquire guide information for guiding the first posture based on the reference posture information, the first captured image, the measured posture information, and the second captured image, and display content indicating the first posture together with the guide information.

[0387] Guide information is acquired based on an AI (artificial intelligence) model that receives reference posture information, a first photographed image, measured posture information, and a second photographed image as input and generates output data, and the AI ​​model may include a large health model (LHM).

[0388] The step of displaying content (S2605) may further include a step of displaying content when a user input for outputting content is received, and the control method may further include a step of identifying the user's measured posture information based on sensing data when a first posture included in the content is displayed.

[0389] The step of receiving sensing data (S2610) may include, when a user input for outputting content is received, transmitting a first control signal for requesting sensing data related to the user's posture to the wearable device, and receiving the sensing data from the wearable device.

[0390] The step of identifying the measurement posture information may include receiving sensing data from a wearable device while a first posture is displayed on an electronic device, and identifying the user's measurement posture information based on the sensing data received while the first posture is displayed.

[0391] The step of receiving a second captured image (S2625) may include, when a predetermined event is identified, acquiring a captured position corresponding to the first posture, transmitting a second control signal to the mobile robot for requesting a captured image including the user based on the captured position, and receiving a second captured image including the user based on the captured position from the mobile robot.

[0392] Step (S2630) of providing guide information may include, when guide information is acquired, acquiring a projection surface position corresponding to the first posture, and transmitting a third control signal to the mobile robot for outputting guide information based on the projection surface position.

[0393] The methods according to the various embodiments of the present disclosure described above can be implemented in the form of an application that can be installed on an existing electronic device.

[0394] The methods according to the various embodiments of the present disclosure described above can be implemented only with a software upgrade or a hardware upgrade for an existing electronic device.

[0395] The various embodiments of the present disclosure described above may also be performed through an embedded server provided in an electronic device, or an external server of at least one of the electronic device and the display device.

[0396] Meanwhile, the electronic device (100) can input the user's exercise video information (captured image) and information collected through a watch-type wearable device (inertial sensor information, biometric sensor information) into the generative AI, LHM (large health model). The electronic device (100) can obtain evaluation information and guide information related to the user's exercise performance status from the LHM. The electronic device (100) can provide the generated guide information (including voice, text, and image) through the electronic device (100) or the mobile robot (300) based on the user's exercise posture. If the user is in a posture where he or she can look at the electronic device (100), the exercise guide can be provided through the electronic device (100). If the user is in a posture where he or she cannot look at the electronic device (100), the mobile robot (300) can provide the exercise guide.

[0397] The mobile robot (300) can capture a multi-angle video of the user's movement through the camera of the mobile robot (300). If the user makes a voice query or if the similarity (see FIG. 9) related to the movement posture is below a threshold value, the electronic device (100) can control the mobile robot (300) to capture the user (see step S955 of FIG. 9).

[0398] The mobile robot (300) can acquire a second captured image to supplement blind spots in the first captured image according to the exercise posture guided by the content (e.g., exercise program). The second captured image can be captured only for specific parts, such as the position of the hands, the position of the feet, or the hip posture. The mobile robot (300) can acquire the second captured image by selecting a shooting target and capturing only a part of the user's body (see FIGS. 10 and 11).

[0399] According to an example embodiment of the present disclosure, the various embodiments described above may be implemented as software including instructions stored in a machine-readable storage medium that can be read by a machine (e.g., a computer). The device may include an electronic device according to the disclosed embodiments, which is a device that can call instructions stored in the storage medium and operate according to the called instructions. When the instructions are executed by a processor, the processor may directly or under the control of the processor use other components to perform a function corresponding to the instructions. The instructions may include code generated or executed by a compiler or interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' means that the storage medium does not contain signals and is tangible, but does not distinguish between whether data is stored semi-permanently or temporarily in the storage medium.

[0400] According to one embodiment of the present disclosure, the method according to the various embodiments described above may be provided as included in a computer program product. The computer program product may be traded as a commodity between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or online through an application store. In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0401] Each of the components (e.g., modules or programs) according to the various embodiments described above may be composed of one or more entities, and some of the sub-components described above may be omitted, or other sub-components may be further included in various embodiments. Alternatively or additionally, some components (e.g., modules or programs) may be integrated into a single entity, which may perform the same or similar functions as those performed by each of the respective components prior to integration. Operations performed by modules, programs or other components according to various embodiments may be executed sequentially, in parallel, iteratively or heuristically, or at least some operations may be executed in a different order, omitted, or other operations may be added.

[0402] Although the preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above, and various modifications may be made by a person skilled in the art to which the present disclosure pertains without departing from the gist of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical idea of ​​the present disclosure.

Claims

1. In electronic devices, memory; display; image sensor; Communication circuits connected to wearable devices and mobile robots; and At least one processor connected to the memory, the display, the image sensor, and the communication circuit, At least one processor, Control the display to display content to guide the first posture, By requesting sensing data related to the user's posture from the wearable device through the communication circuit, the sensing data is received from the wearable device, Identifying whether a predetermined event has occurred based on the first posture and the sensing data, When the above-determined event is identified, a first captured image including the user is acquired through the image sensor, By requesting a photographic image including the user from the mobile robot through the communication circuit, a second photographic image including the user is received from the mobile robot, An electronic device that provides guide information based on the content including the first posture, the first captured image, the sensing data, and the second captured image.

2. In paragraph 1, At least one processor, Identifying the user's measurement posture information based on the above sensing data, Obtaining the similarity between the above measurement posture information and the first posture, An electronic device that identifies that the above-described event has occurred if the above-described similarity is less than a threshold value.

3. In paragraph 2, At least one processor, Obtain reference posture information corresponding to the first posture based on the above content, An electronic device that obtains the similarity by comparing the reference posture information and the measured posture information.

4. In paragraph 3, At least one processor, Obtaining the guide information for guiding the first posture based on the reference posture information, the first photographed image, the measured posture information, and the second photographed image, An electronic device that controls the display to display the content indicating the first posture together with the guide information.

5. In paragraph 4, The above guide information is, It is acquired based on an AI (artificial intelligence) model that receives the above reference posture information, the first photographed image, the measured posture information, and the second photographed image as input and generates output data, The above AI model is an electronic device including a large health model (LHM).

6. In paragraph 2, At least one processor, When a user input for outputting the above content is received, controlling the display to display the above content; An electronic device that identifies the user's measured posture information based on the sensing data when the first posture included in the content is displayed.

7. In paragraph 6, At least one processor, When the user input for outputting the content is received, a first control signal for requesting sensing data related to the user's posture is transmitted to the wearable device through the communication circuit, An electronic device that receives the sensing data from the wearable device through the communication circuit.

8. In paragraph 7, At least one processor, While the first posture is displayed on the display, the sensing data is received from the wearable device through the communication circuit, An electronic device that identifies the user's measured posture information based on the sensing data received while the first posture is displayed.

9. In paragraph 1, At least one processor, When the above-determined event is identified, a shooting position corresponding to the first posture is acquired, Through the communication circuit, a second control signal for requesting a photographed image including the user based on the photographing position is transmitted to the mobile robot, An electronic device that receives the second photographed image including the user, which is photographed based on the photographing location, from the mobile robot through the communication circuit.

10. In paragraph 1, At least one processor, When the above guide information is obtained, the projection surface position corresponding to the first posture is obtained, An electronic device that transmits a third control signal for outputting the guide information based on the projection surface position to the mobile robot through the communication circuit.

11. A method for controlling an electronic device connected to a wearable device and a mobile robot, wherein the control method comprises: A step for displaying content to guide the first posture; A step of receiving sensing data from a wearable device by requesting sensing data related to a user's posture from the wearable device; A step of identifying whether a predetermined event has occurred based on the first posture and the sensing data; When the above-determined event is identified, a step of acquiring a first captured image including the user through the electronic device; A step of receiving a second captured image including the user from the mobile robot by requesting the mobile robot to capture an image including the user; and A control method comprising: a step of providing guide information based on the content including the first posture, the first captured image, the sensing data, and the second captured image.

12. In paragraph 11, The step of identifying whether the above event has occurred is: Identifying the user's measurement posture information based on the above sensing data, Obtaining the similarity between the above measurement posture information and the first posture, A control method for identifying that the above-determined event has occurred if the above-described similarity is less than a threshold value.

13. In paragraph 12, The step of obtaining the above similarity is: Obtain reference posture information corresponding to the first posture based on the above content, A control method for obtaining the similarity by comparing the above reference posture information and the above measured posture information.

14. In paragraph 13, The steps for providing the above guide information are: Obtaining the guide information for guiding the first posture based on the reference posture information, the first photographed image, the measured posture information, and the second photographed image, A control method for displaying the content indicating the first posture together with the above guide information.

15. In paragraph 14, The above guide information is, It is acquired based on an AI (artificial intelligence) model that receives the above reference posture information, the first photographed image, the measured posture information, and the second photographed image as input and generates output data, A control method, wherein the above AI model includes a large health model (LHM).

Images