Method for determining state of walking terrain and electronic device for performing same

Abstract

According to an embodiment, a method for determining the state of walking terrain may comprise the operations of: generating a first image by photographing a scene with a distance sensor; identifying at least one of the user's left leg and right leg in the first image on the basis of depth information regarding the first image; setting a first RoI in the first image on the basis of at least one of the left leg and right leg; generating a first depth value pattern regarding the first RoI on the basis of the depth information regarding the first RoI; and determining the first state of walking terrain corresponding to the first RoI on the basis of the first depth value pattern.

Description

Method for determining the state of walking terrain and electronic device for performing the method

[0001] Certain embodiments relate to techniques for determining the condition of the terrain on which a user is walking or will walk.

[0002] As we enter an aging society, the number of people complaining of difficulty and pain in walking due to muscle weakness or joint problems caused by aging is increasing. Consequently, interest is growing in walking assistance devices that can help elderly people with weakened muscles or patients with muscle and joint problems walk more smoothly, or enable people to exercise.

[0003] In one embodiment, a wearable device comprises a base body positioned at a waist area of ​​a user, a waist support frame and a leg support frame for supporting at least a portion of the user's body, a thigh fastening part for fixing the leg support frame to the user's thigh, an inertial measurement unit (IMU) disposed within the base body, a drive module including a motor and / or a circuit for generating a torque applied to the user's leg, the drive module being positioned between the waist support frame and the leg support frame, an angle sensor for measuring a rotation angle of the leg support frame, a distance sensor for calculating a distance to a surrounding scene of the wearable device, and a control module including at least one processor including a processing circuit for controlling the wearable device, and a memory for storing instructions executable by the at least one processor, wherein when the instructions are individually and / or collectively executed by the at least one processor, the wearable device at least: generates a first image by capturing a scene using the distance sensor, and based on depth information about the first image, The apparatus may be configured to identify at least one of the left leg or the right leg of the user in a first image, set a first region of interest (RoI) in the first image based on at least one of the left leg or the right leg, generate a first depth value pattern for the first RoI based on depth information for the first RoI, and determine a first state of a walking terrain corresponding to the first RoI based on the first depth value pattern.

[0004] According to one embodiment, a method for determining a state of a walking terrain may include: generating a first image by capturing a scene using a distance sensor; identifying at least one of a left leg or a right leg of the user within the first image based on depth information about the first image; setting a first RoI within the first image based on at least one of the left leg or the right leg; generating a first depth value pattern for the first RoI based on depth information about the first RoI; and determining a first state of a walking terrain corresponding to the first RoI based on the first depth value pattern.

[0005] According to one embodiment, a wearable device includes a base body positioned at a waist area of ​​a user when the wearable device is worn on the user's body, a waist support frame and a leg support frame for supporting at least a part of the user's body, a thigh fastening part for fixing the leg support frame to the user's thigh, an IMU positioned within the base body, a driving module for generating a torque applied to the user's leg, the driving module being positioned between the waist support frame and the leg support frame, an angle sensor (125) for measuring a rotation angle of the leg support frame, a first distance sensor installed in the wearable device at a first preset angle for calculating a first depth for a first position of a surrounding scene of the wearable device, a second distance sensor installed in the wearable device at a second preset angle for calculating a second depth for a second position of the surrounding scene of the wearable device, and a control module including at least one processor for controlling the wearable device and a memory for storing instructions executable by the processor, and when the instructions are executed by the processor, the wearable device The device may be configured to: calculate a first height based on the first angle and the first depth, calculate a second height based on the second angle and the second depth, generate a first depth value pattern based on the first height and the second height, and determine a first state of a walking terrain corresponding to the first depth value pattern based on the first depth value pattern.

[0006] Other features and aspects will become apparent from the detailed description, drawings and claims below.

[0007] The above and other aspects, features and advantages of specific embodiments of the present disclosure will become more apparent from the following detailed description taken in conjunction with the accompanying drawings.

[0008] FIG. 1 is a drawing for explaining an overview of a wearable device worn on a user's body according to one embodiment.

[0009] FIG. 2 is a diagram illustrating an exercise management system including a wearable device and an electronic device according to one embodiment.

[0010] FIG. 3 illustrates a rear schematic diagram of a wearable device according to one embodiment.

[0011] FIG. 4 illustrates a left side view of a wearable device according to one embodiment.

[0012] FIGS. 5A and 5B are diagrams illustrating the configuration of a control system of a wearable device according to one embodiment.

[0013] FIG. 6 is a diagram illustrating the interaction between a wearable device and an electronic device according to one embodiment.

[0014] FIG. 7 is a diagram illustrating a configuration of an electronic device according to one embodiment.

[0015] FIG. 8A illustrates a distance sensor module including a distance sensor installed in a wearable device according to one embodiment.

[0016] FIG. 8b illustrates a distance sensor module disposed in a waist fastening portion according to one embodiment.

[0017] FIG. 9 is a flowchart of a method for determining the state of a walking terrain according to one embodiment.

[0018] FIG. 10 is a flowchart of a method for setting an RoI based on at least one of a user's left leg or right leg in an identified image, according to one embodiment.

[0019] FIG. 11 illustrates a method for identifying a left leg or a right leg of a user in an image and setting an RoI based on the identified leg, according to one embodiment.

[0020] FIG. 12 illustrates a method for processing information of an RoI to generate a depth value pattern for a set RoI, according to one embodiment.

[0021] FIG. 13a illustrates a depth value pattern generated for a case where the terrain condition is flat, according to one embodiment.

[0022] FIG. 13b illustrates depth value patterns generated for a case where the terrain condition is an uphill staircase, according to one embodiment.

[0023] FIG. 13c illustrates depth value patterns generated for a case where the terrain condition is a downhill staircase, according to one embodiment.

[0024] FIG. 13d illustrates depth value patterns generated for a case where the terrain condition is uphill, according to one embodiment.

[0025] FIG. 13e illustrates depth value patterns generated for a case where the terrain condition is downhill, according to one embodiment.

[0026] FIG. 13f illustrates depth value patterns generated for a case where the terrain condition includes uphill steps and flat ground, according to one embodiment.

[0027] FIG. 13g illustrates depth value patterns generated for a case where the terrain condition includes a downhill slope and a flat land, according to one embodiment.

[0028] FIG. 13h illustrates depth value patterns generated for a case where the terrain condition is an uphill slope with varying slope, according to one embodiment.

[0029] FIG. 13i illustrates depth value patterns generated for a case where the terrain condition is a downhill slope with varying slope, according to one embodiment.

[0030] FIG. 14a illustrates depth value patterns generated for a case where the terrain condition is flat and obstacles are located above, according to one embodiment.

[0031] FIG. 14b illustrates depth value patterns generated for a case where the terrain condition is flat and obstacles are located below, according to one embodiment.

[0032] FIG. 15 is a flowchart of a method for determining a first state of a walking terrain corresponding to a first RoI based on a first reference depth value pattern, according to one embodiment.

[0033] FIG. 16 is a flowchart of a method for determining a first state of a walking terrain corresponding to a first RoI based on a second reference depth value pattern generated based on a first posture of a distance sensor, according to one embodiment.

[0034] FIG. 17 is a flowchart of a method for outputting a first alarm associated with a first state, according to one embodiment.

[0035] FIG. 18 is a flowchart of a method for controlling a driving module based on a user's walking pattern, according to one embodiment.

[0036] FIG. 19 is a flowchart of a method for controlling a driving module based on a user's walking speed, according to one embodiment.

[0037] FIG. 20 illustrates a method for determining the state of a walking terrain using a plurality of distance sensors according to one embodiment.

[0038] FIG. 21 is a flowchart of a method for determining the state of a walking terrain using a plurality of distance sensors according to one embodiment.

[0039] FIG. 22 illustrates a method for determining the posture of a plurality of distance sensors according to one embodiment.

[0040] FIG. 23 is a flowchart of a method for determining a state of a walking terrain based on the postures of a plurality of distance sensors, according to one embodiment.

[0041] Hereinafter, various embodiments of the present disclosure will be described with reference to the attached drawings. However, this is not intended to limit the present disclosure to specific embodiments, and it should be understood that the present disclosure encompasses various modifications, equivalents, and / or alternatives of the embodiments.

[0042] FIG. 1 is a drawing for explaining an overview of a wearable device worn on a user's body according to one embodiment.

[0043] Referring to FIG. 1, in one embodiment, a wearable device (100) may be a device worn on a user's (110) body to assist the user's (110) walking, exercise, and / or work. In one embodiment, the wearable device (100) may also be used to measure the user's (110) physical ability (e.g., walking ability, exercise ability, exercise posture). In the embodiments, the term 'wearable device' may be replaced with 'wearable robot', 'walking assistance device', or 'exercise assistance device'. The user (110) may be a human or an animal, but is not limited thereto. A wearable device (100) may be worn on a user's (110) body (e.g., lower body (legs, ankles, knees, etc.), upper body (torso, arms, wrists, etc.), or waist) to apply external forces, such as assistance force and / or resistance force, to the body movements of the user (110). Assistance force refers to a force applied in the same direction as the body movement direction of the user (110), and represents a force that assists the body movements of the user (110). Resistance force refers to a force applied in the opposite direction to the body movement direction of the user (110), and represents a force that hinders the body movements of the user (110). The term 'resistance force' may also be referred to as 'exercise load'.

[0044] In one embodiment, the wearable device (100) may operate in a walking assistance mode to assist the walking of a user (110). In the walking assistance mode, the wearable device (100) may assist the walking of the user (110) by applying an assistive force generated from a driving module (120) of the wearable device (100) to the body of the user (110). The wearable device (100) may assist the force required for the walking of the user (110), thereby enabling the user (110) to walk independently or to walk for a long time, thereby expanding the walking ability of the user (110). The wearable device (100) may also help improve the walking of a pedestrian with abnormal walking habits or walking posture.

[0045] In one embodiment, the wearable device (100) may operate in an exercise assistance mode to enhance the exercise effect of the user (110). In the exercise assistance mode, the wearable device (100) may impede the body movement of the user (110) or provide resistance to the body movement of the user (110) by applying a resistance force generated from the driving module (120) to the body of the user (110). If the wearable device (100) is a hip-type wearable device worn on the waist (or pelvis) and legs (e.g., thighs) of the user (110), the wearable device (100) may provide an exercise load to the leg movement of the user (110) while being worn on the legs, thereby further enhancing the exercise effect on the legs of the user (110). In one embodiment, the wearable device (100) may also apply an assistive force to the body of the user (110) to assist the exercise of the user (110). For example, when a disabled person or an elderly person wears a wearable device (100) to exercise, the wearable device (100) may provide assistive force to assist body movements during the exercise. In one embodiment, the wearable device (100) may provide a combination of assistive force and resistance force by exercise section or time section, such as providing assistive force in some exercise sections and resistance force in other exercise sections.

[0046] In one embodiment, the wearable device (100) may operate in a physical ability measurement mode for measuring the physical ability of a user (110). The wearable device (100) may measure movement information of the user (110) using sensors (e.g., an angle sensor (125), an inertial measurement unit (IMU) (135)) provided in the wearable device (100) while the user (110) walks or performs exercise, and may evaluate the physical ability of the user (110) based on the measured movement information. For example, the gait index or exercise ability index (e.g., muscle strength, endurance, balance, exercise movement) of the user (110) may be estimated through the movement information of the user (110) measured by the wearable device (100). The physical ability measurement mode may include an exercise movement measurement mode for measuring the exercise movement of the user (110).

[0047] In various embodiments of the present disclosure, for convenience of explanation, a hip-type wearable device (100) as illustrated in FIG. 1 is described as an example, but is not limited thereto. As described above, the wearable device (100) may be worn on other body parts (e.g., upper arms, lower arms, hands, calves, feet) other than the waist and legs (particularly, thighs), and the shape and configuration of the wearable device (100) may vary depending on the body part on which it is worn.

[0048] According to one embodiment, the wearable device (100) may include a support frame for supporting the body of the user (110) when the wearable device (100) is worn on the body of the user (110) (e.g., leg support frame (50, 55) and waist support frame (20) of FIG. 3), a sensor module for obtaining sensor data including movement information about the body movement of the user (110) (e.g., leg movement, upper body movement) (e.g., sensor module (520) of FIG. 5A), a driving module (120) for generating a torque applied to the leg of the user (110) (e.g., driving module (35, 45) of FIG. 3), and a control module (130) for controlling the wearable device (100) (e.g., control module (510) of FIGS. 5A and 5B).

[0049] The sensor module may include an angle sensor (125) and an inertial measurement device (135). The angle sensor (125) may measure a rotation angle of a leg support frame of the wearable device (100) corresponding to a hip joint angle value of the user (110). The rotation angle of the leg support frame measured by the angle sensor (125) may be estimated to be a hip joint angle value (or leg angle value) of the user (110). The angle sensor (125) may include, for example, an encoder and / or a hall sensor. In one embodiment, the angle sensors (125) may be located near the right hip joint and the left hip joint of the user (110), respectively. The inertial measurement device (135) may include an acceleration sensor and / or an angular velocity sensor, and may measure a change in acceleration and / or angular velocity according to a movement of the user (110). The inertial measurement device (135) can measure, for example, the upper body movement value of the user (110) corresponding to the movement value of the waist support frame (or base body (base body (80) of FIG. 3)) of the wearable device (100). The movement value of the waist support frame measured by the inertial measurement device (135) can be estimated as the upper body movement value of the user (110).

[0050] In one embodiment, the control module (130) and the inertial measurement device (135) may be placed in the base body of the wearable device (100) (e.g., the base body (80) of FIG. 3). The base body may be positioned at the lumbar region (waist region) of the user (110) while the user (110) wears the wearable device (100). The base body may be formed or attached to the outside of the lumbar support frame of the wearable device (100). The base body may be mounted at the lumbar region of the user (110) to provide a cushioning feeling to the lumbar region of the user (110) and support the lumbar region of the user (110) together with the lumbar support frame.

[0051] FIG. 2 is a diagram illustrating an exercise management system including a wearable device and an electronic device according to one embodiment.

[0052] Referring to FIG. 2, the exercise management system (200) may include a wearable device (100) worn on a user's body, an electronic device (210), another wearable device (220), and a server (230). In one embodiment, the exercise management system (200) may omit at least one of these devices (e.g., another wearable device (220) or the server (230)) or may add one or more other devices (e.g., a dedicated controller device of the wearable device (100)).

[0053] In one embodiment, the wearable device (100) may be worn on the user's body in a walking assistance mode to assist the user's movements. For example, the wearable device (100) may be worn on the user's leg to generate an assistive force to assist the user's leg movements, thereby assisting the user's walking.

[0054] In one embodiment, the wearable device (100) may generate and apply to the user's body a resistance force to hinder the user's body movement or an assistive force to assist the user's body movement in order to enhance the user's exercise effect in the exercise assistance mode. In the exercise assistance mode, the user may select an exercise program (e.g., squats, split lunges, dumbbell squats, lunges and knee ups, stretching, etc.) to be exercised using the wearable device (100) through the electronic device (210) and / or an exercise intensity to be applied to the wearable device (100). The wearable device (100) may control the drive module of the wearable device (100) according to the exercise program selected by the user, and may acquire sensor data including information on the user's movement through the sensor module. The wearable device (100) may adjust the strength of the resistance force or the assistive force to be applied to the user according to the exercise intensity selected by the user. For example, the wearable device (100) can control the drive module to generate a resistance corresponding to the exercise intensity selected by the user.

[0055] In one embodiment, the wearable device (100) may be used to measure a user's physical ability in conjunction with an electronic device (210). The wearable device (100) may operate in a physical ability measurement mode, which is a mode for measuring the user's physical ability under the control of the electronic device (210), and may transmit sensor data acquired by the user's movements in the physical ability measurement mode to the electronic device (210). The electronic device (210) may analyze the sensor data received from the wearable device (100) to estimate the user's physical ability.

[0056] The electronic device (210) can communicate with the wearable device (100), remotely control the wearable device (100), or provide the user with status information about the status of the wearable device (100) (e.g., booting status, charging status, sensing status, error status). The electronic device (210) can receive sensor data acquired by a sensor of the wearable device (100) from the wearable device (100), and estimate the user's physical ability or exercise result based on the received sensor data. In one embodiment, when the user wears the wearable device (100) and exercises, the wearable device (100) can acquire sensor data including movement information of the user using the sensors, and transmit the acquired sensor data to the electronic device (210). The electronic device (210) can extract the user's movement value from the sensor data, and evaluate the user's exercise motion based on the extracted movement value. The electronic device (210) can provide the user with exercise motion measurement values ​​and exercise motion evaluation information for the user's exercise motion through a graphical user interface.

[0057] In one embodiment, the electronic device (210) may execute a program (e.g., an application) for controlling the wearable device (100), and the user may adjust the operation or setting values ​​(e.g., the torque intensity output from the driving module (e.g., the driving module (35, 45) of FIG. 3), the volume of the audio output from the sound output module (e.g., the sound output module (550) of FIGS. 5A and 5B), the brightness of the light unit (e.g., the light unit (85) of FIG. 3)) of the wearable device (100) through the program. The program executed in the electronic device (210) may provide a graphical user interface (GUI) for interaction with the user. The electronic device (210) may be a device of various forms. For example, the electronic device (210) may include, but is not limited to, a portable communication device (e.g., a smartphone), a computer device, an access point, a portable multimedia device, or a home appliance device (e.g., a television, an audio device, a projector device).

[0058] According to one embodiment, the electronic device (210) may be connected to the server (230) using short-range wireless communication or cellular communication. The server (230) may receive user profile information of a user using the wearable device (100) from the electronic device (210) and store and manage the received user profile information. The user profile information may include, for example, information on at least one of name, age, gender, height, weight, or body mass index (BMI). The server (230) may receive exercise history information on exercise performed by the user from the electronic device (210) and store and manage the received exercise history information. The server (230) may provide various exercise programs or physical ability measurement programs that may be provided to the user to the electronic device (210).

[0059] According to one embodiment, the wearable device (100) and / or the electronic device (210) may be connected to another wearable device (220). The other wearable device (220) may be, for example, wireless earphones (222), a smartwatch (224), or smartglasses (226), but is not limited to the aforementioned devices. In one embodiment, the smartwatch (224) may measure a bio-signal including heart rate information of the user, and transmit the measured bio-signal to the electronic device (210) and / or the wearable device (100). The electronic device (210) may estimate heart rate information of the user (e.g., current heart rate, maximum heart rate, average heart rate) based on the bio-signal received from the smartwatch (224), and may provide the estimated heart rate information to the user.

[0060] In one embodiment, the user's exercise result information, physical ability information, and / or exercise motion evaluation information evaluated by the electronic device (210) may be transmitted to another wearable device (220) and provided to the user through the other wearable device (220). Status information of the wearable device (100) may also be transmitted to another wearable device (220) and provided to the user through the other wearable device (220). In one embodiment, the wearable device (100), the electronic device (210), and the other wearable device (220) may be connected to each other through wireless communication (e.g., Bluetooth communication, Wi-Fi communication).

[0061] In one embodiment, the wearable device (100) may provide (or output) feedback (e.g., visual feedback, auditory feedback, tactile feedback) corresponding to the state of the wearable device (100) according to a control signal received from the electronic device (210). For example, the wearable device (100) may provide visual feedback through a light unit (e.g., light unit (85) of FIG. 3) and may provide auditory feedback through an audio output module (e.g., audio output module (550) of FIGS. 5A and 5B). The wearable device (100) may include a haptic module and may provide tactile feedback in the form of vibration to the user's body through the haptic module. The electronic device (210) may also provide (or output) feedback (e.g., visual feedback, auditory feedback, tactile feedback) corresponding to the state of the wearable device (100).

[0062] In one embodiment, the electronic device (210) may present personalized exercise goals to the user in an exercise assistance mode. The personalized exercise goals may include exercise volume targets for each of the exercise types (e.g., strength training, balance training, aerobic training) that the user wishes to perform, as determined by the electronic device (210) and / or the server (230). When the server (230) determines the exercise volume targets, the server (230) may transmit information about the determined exercise volume targets to the electronic device (210). The electronic device (210) may present exercise volume targets for the exercise types of strength training, aerobic training, and balance training in a personalized manner according to the exercise program to be performed (e.g., squats, split lunges, lunge and knee-ups) and / or the user's physical characteristics (e.g., age, height, weight, BMI). The electronic device (210) may display a GUI screen indicating the exercise volume targets for each exercise type on the display.

[0063] In one embodiment, the electronic device (210) and / or the server (230) may include a database storing information on a plurality of exercise programs that may be provided to the user through the wearable device (100). To achieve the user's exercise goal, the electronic device (210) and / or the server (230) may recommend an exercise program suitable for the user. The exercise goal may include, for example, at least one of muscle strength improvement, physical strength improvement, cardiopulmonary endurance improvement, core stability improvement, flexibility improvement, or symmetry improvement. The electronic device (210) and / or the server (230) may store and manage exercise programs performed by the user and the results of the exercise programs performed.

[0064] Figure 3 illustrates a rear schematic diagram of a wearable device according to one embodiment. Figure 4 illustrates a left side view of the wearable device according to one embodiment.

[0065] Referring to FIGS. 3 and 4, a wearable device (100) according to one embodiment may include a base body (80), a waist support frame (20), a driving module (35, 45), a leg support frame (50, 55), a thigh fastening part (1, 2), and a waist fastening part (60). The base body (80) may include a lighting unit (85). In one embodiment, the wearable device (100) may omit at least one of these components (e.g., the lighting unit (85)), or may have one or more other components (e.g., a haptic module) added.

[0066] The base body (80) can be positioned on the user's lower back while the user is wearing the wearable device (100). The base body (80) can be mounted on the user's lower back to provide a cushioning feeling to the user's lower back and support the user's lower back. The base body (80) can be hung over the user's buttocks (hip area) while the user is wearing the wearable device (100) to prevent the wearable device (100) from falling downward due to gravity or to reduce the possibility of separation. The base body (80) can distribute a portion of the weight of the wearable device (100) to the user's lower back while the user is wearing the wearable device (100). The base body (80) can be directly or indirectly connected to the lower back support frame (20). Lower back support frame connection elements (not shown) that can be directly or indirectly connected to the lower back support frame (20) can be provided at both ends of the base body (80).

[0067] In one embodiment, a lighting unit (85) may be disposed outside the base body (80). The lighting unit (85) may include a light source (e.g., a light emitting diode (LED)). The lighting unit (85) may emit light under the control of a control module (not shown) (e.g., the control module (510) of FIGS. 5A and 5B). According to an embodiment, the control module may control the lighting unit (85) so that visual feedback corresponding to the status of the wearable device (100) may be provided (or output) to the user through the lighting unit (85).

[0068] The lumbar support frame (20) may extend from both ends of the base body (80). The user's lower back may be accommodated on the inside of the lumbar support frame (20). The lumbar support frame (20) may include at least one rigid body beam. Each beam may have a curved shape having a predetermined curvature so as to surround the user's lower back. A lumbar fastening member (60) may be directly or indirectly connected to an end of the lumbar support frame (20). A driving module (35, 45) may be directly or indirectly connected to the lumbar support frame (20).

[0069] In one embodiment, a control module, an inertial measurement device (not shown) (e.g., an inertial measurement device (135) of FIG. 1, an inertial measurement device (522) of FIG. 5B), a communication module (not shown) (e.g., a communication module (516) of FIGS. 5A and 5B), and a battery (not shown) may be arranged inside the base body (80). The base body (80) may protect the control module, the inertial measurement device, the communication module, and the battery. The control module may generate a control signal for controlling the operation of the wearable device (100). The control module may include a control circuit including a processor and a memory for controlling the actuators of the drive modules (35, 45). The control module may further include a power supply module (not shown) for supplying power from the battery to each component of the wearable device (100).

[0070] In one embodiment, the wearable device (100) may include a sensor module (not shown) (e.g., sensor module (520) of FIG. 5A) that obtains sensor data from one or more sensors. The sensor module may obtain sensor data that changes according to the user's movement. In one embodiment, the sensor module may obtain sensor data including movement information of the user and / or movement information of components of the wearable device (100). The sensor module may include, but is not limited to, an inertial measurement device (e.g., inertial measurement device (135) of FIG. 1, inertial measurement device (522) of FIG. 5B) for measuring a movement value of the user's upper body or a movement value of the waist support frame (20) and an angle sensor (e.g., angle sensor (125) of FIG. 1, first angle sensor (524) and second angle sensor (524-1) of FIG. 5B) for measuring a hip joint angle value of the user or a movement value of the leg support frames (50, 55). For example, the sensor module may further include at least one of a position sensor, a temperature sensor, a biosignal sensor, or a proximity sensor.

[0071] The waist fastening member (60) can be directly or indirectly connected to the waist support frame (20) and can secure the waist support frame (20) to the user's waist. The waist fastening member (60) can include, for example, a pair of belts.

[0072] The drive module (35, 45) can generate an external force (or torque) applied to the user's body based on a control signal generated by the control module. For example, the drive module (35, 45) can generate an assistive force or a resistance force applied to the user's leg. In one embodiment, the drive module (35, 45) can include a first drive module (45) positioned corresponding to the user's right hip joint position and a second drive module (35) positioned corresponding to the user's left hip joint position. The first drive module (45) can include a first actuator and a first joint member, and the second drive module (35) can include a second actuator and a second joint member. The first actuator can provide power transmitted to the first joint member, and the second actuator can provide power transmitted to the second joint member. The first actuator and the second actuator may each include a motor that receives power from a battery and generates force (or torque). When powered and driven, the motor may generate a force to assist the user's body movements (assistive force) or a force to impede the body movements (resistive force). In one embodiment, a control module including a circuit may control the strength and direction of the force generated by the motor by adjusting the voltage and / or current supplied to the motor.

[0073] In one embodiment, the first joint member and the second joint member can receive power from the first actuator and the second actuator, respectively, and apply an external force to the user's body based on the received power. The first joint member and the second joint member can be disposed at a position corresponding to a joint of the user, respectively. One side of the first joint member can be directly or indirectly connected to the first actuator, and the other side can be directly or indirectly connected to the first leg support frame (55). The first joint member can be rotated by the power received from the first actuator. An encoder or a hall sensor that can act as an angle sensor for measuring a rotation angle of the first joint member (corresponding to the user's joint angle) can be disposed on one side of the first joint member. One side of the second joint member can be directly or indirectly connected to the second actuator, and the other side can be directly or indirectly connected to the second leg support frame (50). The second joint member can be rotated by power transmitted from the second actuator. An encoder or hall sensor that can act as an angle sensor for measuring the rotation angle of the second joint member can also be arranged on one side of the second joint member.

[0074] In one embodiment, the first actuator may be disposed laterally of the first joint member, and the second actuator may be disposed laterally of the second joint member. The rotational axis of the first actuator and the rotational axis of the first joint member may be disposed to be spaced apart from each other, and the rotational axis of the second actuator and the rotational axis of the second joint member may also be disposed to be spaced apart from each other. However, the present invention is not limited thereto, and the actuator and the joint member may share a rotational axis. In one embodiment, each actuator may be disposed to be spaced apart from the joint member. In this case, the drive module (35, 45) may further include a power transmission module (not shown) that transmits power from the actuator to the joint member. The power transmission module may be a rotating body such as a gear, or a longitudinal member such as a wire, a cable, a string, a spring, a belt, or a chain. However, the scope of the embodiment is not limited by the positional relationship between the actuator and joint member and the power transmission structure described above.

[0075] In one embodiment, the leg support frame (50, 55) can support the user's leg (e.g., thigh) when the wearable device (100) is worn on the user's leg. The leg support frame (50, 55) can transmit power (torque) generated from, for example, the driving module (35, 45) to the user's thigh, and the power can act as an external force applied to the movement of the user's leg. One end of the leg support frame (50, 55) can be directly or indirectly connected to a joint member and rotated, and the other end of the leg support frame (50, 55) can be directly or indirectly connected to a thigh fastening part (1, 2), so that the leg support frame (50, 55) can support the user's thigh while transmitting the power generated from the driving module (35, 45) to the user's thigh. For example, the leg support frame (50, 55) can push or pull the user's thigh. The leg support frame (50, 55) may extend along the length of the user's thigh. The leg support frame (50, 55) may be bent to wrap around at least a portion of the user's thigh. The leg support frame (50, 55) may include a first leg support frame (55) for supporting the user's right leg and a second leg support frame (50) for supporting the user's left leg.

[0076] The thigh fastening portions (1, 2) are directly or indirectly connected to the leg support frame (50, 55) and can secure the leg support frame (50, 55) to the thigh. The thigh fastening portions (1, 2) may include a first thigh fastening portion (2) for securing the first leg support frame (55) to the user's right thigh and a second thigh fastening portion (1) for securing the second leg support frame (50) to the user's left thigh.

[0077] In one embodiment, the first thigh fastening part (2) may include a first cover, a first fastening frame, and a first strap, and the second thigh fastening part (1) may include a second cover, a second fastening frame, and a second strap. The first cover and the second cover may apply a torque generated from the driving module (35, 45) to the user's thigh. The first cover and the second cover may be disposed on one side of the user's thigh and may push or pull the user's thigh. The first cover and the second cover may be disposed on the front side of the user's thigh, for example. The first cover and the second cover may be disposed along the circumferential direction of the user's thigh. The first cover and the second cover may extend in both directions with the other end of the leg support frame (50, 55) as the center, and may include a curved surface corresponding to the user's thigh. One end of the first cover and the second cover may be directly or indirectly connected to the fastening frame, and the other end may be directly or indirectly connected to the strap.

[0078] The first fastening frame and the second fastening frame may be arranged to, for example, surround at least a portion of the user's thigh, thereby preventing the user's thigh from being dislodged from the leg support frame (50, 55) or reducing the possibility of separation. The first fastening frame may have a fastening structure connecting the first cover and the first strap, and the second fastening frame may have a fastening structure connecting the second cover and the second strap.

[0079] The first strap may encircle the user's right thigh, the remaining portion not covered by the first cover and the first fastening frame, and the second strap may encircle the user's left thigh, the remaining portion not covered by the second cover and the second fastening frame. The first strap and the second strap may comprise, for example, an elastic material (e.g., a band).

[0080] FIGS. 5A and 5B are diagrams illustrating the configuration of a control system of a wearable device according to one embodiment.

[0081] Referring to FIG. 5A, a wearable device (100) may be controlled by a control system (500). The control system (500) may include a control module (510) including a circuit, a communication module (516), a sensor module (520), a driving module (530), an input module (540), and an audio output module (550). In one embodiment, the control system (500) may omit at least one of these components (e.g., an audio output module (550)), or may have one or more other components (e.g., a haptic module) added.

[0082] The drive module (530) may include a motor (534) capable of generating power (e.g., torque) and a motor driver circuit (532) for driving the motor (534). In the embodiment of FIG. 5A, a drive module (530) including one motor driver circuit (532) and one motor (534) is illustrated, but this is merely an example. Referring to FIG. 5B, as in the control system (500-1) illustrated in FIG. 5B, there may be a plurality of motor driver circuits (532, 532-1) and a plurality of motors (534, 534-1), respectively (e.g., two or more). A drive module (530) including a motor driver circuit (532) and a motor (534) may correspond to the first drive module (45) of FIG. 3, and a drive module (530-1) including a motor driver circuit (532-1) and a motor (534-1) may correspond to the second drive module (35) of FIG. 3. The description of each of the motor driver circuit (532) and the motor (534) described below may also be applied to the motor driver circuit (532-1) and the motor (534-1) illustrated in FIG. 5b.

[0083] Returning to FIG. 5A, the sensor module (520) may include a sensor circuit including at least one sensor. The sensor module (520) may include sensor data including movement information of the user or movement information of the wearable device (100). The sensor module (520) may transmit the acquired sensor data to the control module (510). The sensor module (520) may include an inertial measurement device (522) and an angle sensor (e.g., a first angle sensor (524) and a second angle sensor (524-1)) as illustrated in FIG. 5B. The inertial measurement device (522) may measure movement values ​​of the user's upper body. For example, the inertial measurement device (522) may sense accelerations of the X-axis, Y-axis, and Z-axis and angular velocities of the X-axis, Y-axis, and Z-axis according to the movement of the user. The inertial measurement device (522) can be used to measure, for example, at least one of forward and backward tilt, left and right tilt, or rotation of the user's body. In addition, the inertial measurement device (522) can obtain movement values ​​(e.g., acceleration values ​​and angular velocity values) of a lumbar support frame (e.g., lumbar support frame (20) of FIG. 3) of the wearable device. The movement values ​​of the lumbar support frame can correspond to movement values ​​of the user's upper body.

[0084] The angle sensor can measure a hip joint angle value according to the movement of the user's legs. Sensor data that can be measured by the angle sensor can include, for example, a hip joint angle value of the right leg, a hip joint angle value of the left leg, and information on the movement direction of the legs. For example, the first angle sensor (524) of FIG. 5B can obtain a hip joint angle value of the user's right leg, and the second angle sensor (524-1) can obtain a hip joint angle value of the user's left leg. Each of the first angle sensor (524) and the second angle sensor (524-1) can include, for example, an encoder and / or a Hall sensor. In addition, the angle sensor can obtain a movement value of the leg support frame of the wearable device (100). For example, the first angle sensor (524) can obtain a movement value of the first leg support frame (55), and the second angle sensor (524-1) can obtain a movement value of the second leg support frame (50). The movement value of the leg support frame can correspond to the hip joint angle value.

[0085] In one embodiment, the sensor module (520) may further include at least one of a position sensor for obtaining a position value of the wearable device (100), a proximity sensor for detecting the proximity of an object, a biosignal sensor for detecting a user's biosignal, or a temperature sensor for measuring an ambient temperature.

[0086] The input module (540) can receive commands or data to be used in a component of the wearable device (100) (e.g., a processor (512)) from an external source (e.g., a user) of the wearable device (100). The input module (540) can include an input component circuit. The input module (540) can include, for example, a key (e.g., a button) or a touch screen.

[0087] The audio output module (550) can output audio signals to the outside of the wearable device (100). The audio output module (550) can provide auditory feedback to the user. For example, the audio output module (550) can include a speaker that plays a guide audio signal (e.g., a driving start sound, an operation error notification sound, an exercise start notification sound), a music content, or a guide voice to audibly inform specific information (e.g., exercise result information, exercise movement evaluation information).

[0088] In one embodiment, the control system (500) may further include a battery (not shown) for supplying power to each component of the wearable device (100). The wearable device (100) may convert the power of the battery to an operating voltage of each component of the wearable device (100) and supply the converted power to each component.

[0089] The drive module (530) can generate an external force applied to the user's leg under the control of the control module (510). The drive module (530) can generate a torque applied to the user's leg based on a control signal generated by the control module (510). The control module (510) including a circuit can control to transmit the control signal to the motor driver circuit (532). The motor driver circuit (532) can control the operation of the motor (534) by generating a current signal (or voltage signal) corresponding to the control signal and supplying the current signal to the motor (534). In some cases, the current signal may not be supplied to the motor (534). The motor (534) can generate a torque for an assistive force that assists the movement of the user's leg or a resistive force that hinders the movement of the leg when the motor (534) is driven by supplying a current signal to the motor (534).

[0090] The control module (510) controls the overall operation of the wearable device (100) and can generate control signals for controlling each component (e.g., communication module (516), driving module (530)). The control module (510) can include a processor (512) including a processing circuit and a memory (514).

[0091] The processor (512) may, for example, execute software to control at least one other component (e.g., a hardware or software component) of the wearable device (100) directly or indirectly connected to the processor (512), and may perform various data processing or calculations. The software may include an application for providing a GUI. According to one embodiment, as at least a part of the data processing or calculation, the processor (512) may store instructions or data received from another component (e.g., a communication module (516)) in the memory (514), process the instructions or data stored in the memory (514), and store the result data after the processing in the memory (514). According to one embodiment, the processor (512) may include a main processor (e.g., a central processing unit or an application processor) or an auxiliary processor (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together therewith. The auxiliary processor may be implemented separately from the main processor or as part of it.

[0092] The memory (514) can store various data used by at least one component (e.g., processor (512)) of the control module (510). The data can include, for example, input data or output data for software, sensor data, and commands related thereto. The memory (514) can include volatile memory or non-volatile memory (e.g., RAM, DRAM, SRAM).

[0093] The communication module (516) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the control module (510) and other components of the wearable device (100) or an external electronic device (e.g., the electronic device (210) or another wearable device (220) of FIG. 2), and the performance of communication through the established communication channel. The communication module (516) may include a communication circuit for performing a communication function. The communication module (516) may, for example, receive a control signal from an electronic device (e.g., the electronic device (210)) and transmit sensor data acquired by the sensor module (520) to the electronic device. According to one embodiment, the communication module (516) may operate independently from the processor (512) and may include one or more communication processors (not shown) that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (516) may include a wireless communication module (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) and / or a wired communication module. Any of these communication modules may communicate with other components of the wearable device (100) and / or external electronic devices via a short-range communication network such as, for example, Bluetooth, wireless fidelity (Wi-Fi), or infrared data association (IrDA), or a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN).

[0094] In one embodiment, the control system (500, 500-1) may further include a haptic module (not shown). The haptic module may provide tactile feedback to a user under the control of the processor (512). The haptic module may convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that a user may perceive through a tactile or kinesthetic sense. The haptic module may include a motor, a piezoelectric element, or an electrical stimulation device. In one embodiment, the haptic module may be located in at least one of the base body (e.g., the base body (80)), the first thigh fastening portion (2), or the second thigh fastening portion (1).

[0095] FIG. 6 is a diagram illustrating the interaction between a wearable device and an electronic device according to one embodiment.

[0096] Referring to FIG. 6, the wearable device (100) can communicate with the electronic device (210). For example, the electronic device (210) may be a user terminal of a user using the wearable device (100) or a dedicated controller device for the wearable device (100). According to one embodiment, the wearable device (100) and the electronic device (210) may be connected to each other through short-range wireless communication (e.g., Bluetooth communication, Wi-Fi communication).

[0097] In one embodiment, the electronic device (210) may execute an application to check the status of the wearable device (100) or to control or operate the wearable device (100). By executing the application, a screen of a user interface (UI) for controlling the operation of the wearable device (100) or determining the operation mode of the wearable device (100) may be displayed on the display (212) of the electronic device (210). The UI may be, for example, a graphical user interface (GUI).

[0098] In one embodiment, a user may input a command to control the operation of the wearable device (100) (e.g., a command to execute a walking assistance mode, an exercise assistance mode, or a physical ability measurement mode) or change the settings of the wearable device (100) through a GUI screen on a display (212) of the electronic device (210). The electronic device (210) may generate a control command (or a control signal) corresponding to the motion control command or setting change command input by the user, and transmit the generated control command to the wearable device (100). The wearable device (100) may operate according to the received control command, and transmit a control result according to the control command and / or sensor data measured by a sensor module of the wearable device (100) to the electronic device (210). The electronic device (210) may analyze the control result and / or sensor data to provide the user with result information (e.g., walking ability information, exercise ability information, exercise movement evaluation information) through the GUI screen.

[0099] FIG. 7 is a diagram illustrating a configuration of an electronic device according to one embodiment.

[0100] Referring to FIG. 7, the electronic device (210) may include a processor (710), a memory (720), a communication module (730), a display module (740), an audio output module (750), and an input module (760). In one embodiment, the electronic device (210) may omit at least one of these components (e.g., an audio output module (750)), or may have one or more other components (e.g., a sensor module, a battery) added.

[0101] The processor (710) may control at least one other component (e.g., hardware or software component) of the electronic device (210) and perform various data processing or calculations. According to one embodiment, as at least a part of the data processing or calculation, the processor (710) may store commands or data received from another component (e.g., communication module (730)) in the memory (720), process the commands or data stored in the memory (720), and store result data in the memory (720).

[0102] According to one embodiment, the processor (710) may include a main processor (e.g., a central processing unit or an application processor) or an auxiliary processor (e.g., a graphics processing unit, a neural network processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or in conjunction therewith.

[0103] The memory (720) can store various data used by at least one component (e.g., the processor (710) or the communication module (730)) of the electronic device (210). The data can include, for example, input data or output data for a program (e.g., an application) and commands related thereto. The memory (720) can include at least one instruction executable by the processor (710). The memory (720) can include volatile memory or non-volatile memory.

[0104] The communication module (730) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (210) and another electronic device (e.g., wearable device (100), another wearable device (220), server (230)), and the performance of communication through the established communication channel. The communication module (730) may include a communication circuit for performing a communication function. The communication module (730) may operate independently from the processor (710) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (290) may include a wireless communication module (e.g., a Bluetooth communication module, a cellular communication module, a Wi-Fi communication module, or a GNSS communication module) that performs wireless communication, or a wired communication module (e.g., a LAN communication module or a power line communication module). The communication module (730) may, for example, transmit a control command to the wearable device (100) and receive at least one of sensor data including body movement information of a user wearing the wearable device (100), status data of the wearable device (100), or control result data corresponding to the control command from the wearable device (100).

[0105] The display module (740) can visually provide information to an external party (e.g., a user) of the electronic device (210). The display module (740) may include, for example, an LCD or OLED display, a holographic device, or a projector device. The display module (740) may further include a control circuit for controlling display operation. In one embodiment, the display module (740) may further include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0106] The audio output module (750) can output an audio signal to the outside of the electronic device (210). The audio output module (750) can include a speaker that plays a guide audio signal (e.g., a driving start sound, an operation error notification sound), music content, or a guide voice based on the status of the wearable device (100). If it is determined that the wearable device (100) is not properly worn on the user's body, for example, the audio output module (750) can output a guide voice to notify the user of an abnormal wearing or to induce normal wearing. The audio output module (750) can also output a guide voice corresponding to exercise evaluation information or exercise result information that evaluates the user's exercise, for example.

[0107] The input module (760) can receive commands or data to be used in a component of the electronic device (210) (e.g., a processor (710)) from an external source (e.g., a user) of the electronic device (210). The input module (760) can include an input component circuit and can receive user input. The input module (760) can include, for example, a key (e.g., a button) or a touch screen.

[0108] FIG. 8A illustrates a distance sensor module including a distance sensor installed in a wearable device according to one embodiment.

[0109] According to one embodiment, the wearable device (100) may include a distance sensor module. For example, the distance sensor module may include one or more distance sensors. The distance sensor may be a time of flight (ToF) sensor. The distance sensor may be a stereo camera. For example, the distance sensor module may include an auxiliary IMU. The auxiliary IMU may determine the attitude of the distance sensor module or the distance sensor.

[0110] According to one embodiment, the distance sensor module may include a first distance sensor module (810) disposed in the waist fastening portion (60). The first distance sensor module (810) may be disposed in the waist fastening portion (60) so that the distance sensor within the first distance sensor module (810) is not hindered from operating even while the user is walking while wearing the wearable device (100).

[0111] According to one embodiment, the distance sensor module may be a second distance sensor module (820) disposed in the driving module (35). The distance sensor module may be disposed in the driving module (35) so that the sensor within the distance sensor module does not operate while the user wears the wearable device (100) and walks. The second distance sensor module (820) may be disposed in a location (e.g., the housing of the driving module (35)) that does not shake significantly even when the driving module (35) generates torque. The second distance sensor module (820) may be disposed in the driving module (35) so that the angle of view of the distance sensor of the second distance sensor module (820) is secured.

[0112] FIG. 8b illustrates a distance sensor module disposed in a waist fastening portion according to one embodiment.

[0113] According to one embodiment, a first distance sensor module (810) may be placed on a waist fastening member (60). For example, the first distance sensor module (810) may be placed on a buckle of the waist fastening member (60).

[0114] According to one embodiment, the first distance sensor module (810) may include a distance sensor (814). The distance sensor (814) may calculate a distance to a surrounding scene (e.g., front) of the wearable device (100). For example, the distance sensor (814) may include a transmitter (Tx) for transmitting a signal and a receiver (Rx) for receiving the signal. The distance sensor (814) may calculate a distance to an object that reflected the signal by calculating the ToF of the received signal. For example, the distance sensor (814) may generate an image of the scene. Each pixel in the image generated by the distance sensor (814) may include distance information for a point of an object corresponding to the pixel. The image generated by the distance sensor (814) is described in detail below with reference to FIG. 12.

[0115] According to one embodiment, the first distance sensor module (810) may include an auxiliary IMU (812). The auxiliary IMU (812) may measure the attitude of the auxiliary IMU (812). The measured attitude may be used as the attitude of the first distance sensor module (810) or the distance sensor (814).

[0116] FIG. 9 is a flowchart of a method for determining the state of a walking terrain according to one embodiment.

[0117] The operations 910 to 950 below may be performed by a wearable device (e.g., the wearable device (100) of FIG. 1). For example, the operations 910 to 950 may be performed by a control module (e.g., the control module (130) of FIG. 1 or the control module (510) of FIG. 5A) including at least one processor (e.g., the processor (512) of FIG. 5) for controlling the wearable device and a memory (e.g., the memory (514) of FIG. 5) for storing instructions executable by the processor.

[0118] In operation 910, the wearable device may generate a first image by capturing a scene using a distance sensor (e.g., the distance sensor (814) of FIG. 8). For example, the distance sensor may generate the first image by capturing a scene corresponding to a field of view (FoV). For example, the scene corresponding to the FoV of the distance sensor may show the left or right leg of a walking user, and the terrain along which the user is walking.

[0119] In one embodiment, each pixel of the first image may include distance information for a corresponding point in the captured scene. For example, a distance sensor may determine distance information for a pixel using ToF for a corresponding point in the scene. For example, the first image may be a depth map. For example, the distance information may include an intensity value for each pixel. For example, the distance information may include a depth value for each pixel.

[0120] In operation 920, the wearable device may identify at least one of the user's left leg or right leg within the first image based on depth information regarding the first image. For example, the wearable device may identify at least one of the user's left leg or right leg within the first image by determining whether depth information corresponding to the left leg or right leg appears within a preset region of the first image where the user's leg is likely to appear (e.g., a lower portion of the first image).

[0121] In operation 930, the wearable device may set a first RoI within the first image based on at least one of the left leg or the right leg. The terrain on which the user intends to walk is located in front of the user, particularly, in front of the legs. The expected walking direction may be set as the first RoI, in front of the user's legs. When the first RoI is set in the expected walking direction, the state of the terrain may be determined in advance. Since data is processed only for the first RoI and not for the entire first image, the amount of data processing and processing time may be reduced compared to processing data for the entire first image. Below, a method for setting the first RoI is described in detail with reference to FIG. 10.

[0122] In operation 940, the wearable device may generate a first depth value pattern for the first RoI based on depth information for the first RoI.

[0123] According to one embodiment, a first depth value pattern may be generated to indicate coordinate values ​​of pixels located on a central axis of a first RoI and depth values ​​of the pixels within a first image. For example, the first depth value pattern may be based on a coordinate system based on a first axis that positions pixels on the first RoI based on a distance from a user and a second axis that represents depth values ​​of pixels on the first axis. In the coordinate system, the first axis may be a horizontal axis, and the second axis may be a vertical axis. Pixels on the first RoI may be positioned on the first axis such that pixels closer to the user are closer to an origin of the coordinate system. Depth values ​​corresponding to pixels positioned on the first axis may be positioned on the second axis. The first depth value pattern is described in detail below with reference to FIGS. 13A to 13I, FIGS. 14A and 14B.

[0124] In operation 950, the wearable device may determine a first state of the walking terrain corresponding to the first RoI based on the first depth value pattern. For example, the first state may be any one of flat ground, uphill, downhill, uphill stairs, downhill stairs, an upward obstacle, or a downward obstacle.

[0125] Each embodiment of the present disclosure can be used in combination with other embodiments described in the present disclosure.

[0126] According to one embodiment, the wearable device may calculate the similarity between a plurality of reference depth value patterns corresponding to a plurality of states of a walking terrain and a first depth value pattern, and determine a first state of the walking terrain based on the calculated similarity. A method for determining a first state of the walking terrain based on the similarity between depth value patterns is described in detail below with reference to FIG. 15.

[0127] According to one embodiment, the wearable device can determine a first state of the walking terrain using a model based on a pre-trained artificial intelligence neural network based on a first depth value pattern. For example, the model can determine any one of flat ground, an uphill, a downhill, an uphill staircase, a downhill staircase, an upper obstacle, or a lower obstacle as the first state of the walking terrain. For example, the model can determine a slope for an uphill or a downhill as the first state of the walking terrain.

[0128] According to one embodiment, operations 910 to 950 may be performed when movement or walking of a user wearing a wearable device is detected. For example, if the user stops walking for a preset period of time, the performance of operations 910 to 950 may be stopped. If the user resumes walking, operation 910 may be re-performed. When operation 910 is re-performed, the first RoI previously set in the first image, the first depth value pattern for the first RoI, and the first state of the walking terrain corresponding to the first RoI may be initialized. As used herein, “based on” includes at least based on.

[0129] According to one embodiment, the wearable device can control itself based on a first state of the determined walking terrain. For example, the wearable device can output an alarm regarding the first state to alert the user. A method for outputting an alarm is described in detail below with reference to FIG. 17.

[0130] According to one embodiment, the wearable device can determine the user's walking pattern or walking speed based on the left or right leg identified based on the first image, and control the wearable device based on the determined walking pattern or walking speed. A method for controlling the wearable device based on the user's walking pattern is described in detail below with reference to FIG. 18. A method for controlling the wearable device based on the user's walking speed is described in detail below with reference to FIG. 19.

[0131] FIG. 10 is a flowchart of a method for setting an RoI based on at least one of a user's left leg or right leg in an identified image, according to one embodiment.

[0132] According to one embodiment, operations 1010 and 1020 below may be related to operation 930 described above with reference to FIG. 9. For example, operation 930 may include operations 1010 and 1020. Operations 1010 and 1020 may be performed by a control module (e.g., the control module (130) of FIG. 1 or the control module (510) of FIG. 5A including a processing circuit) that includes at least one processor (e.g., the processor (512) of FIG. 5) for controlling the wearable device and a memory (e.g., the memory (514) of FIG. 5) for storing instructions executable by the processor.

[0133] In operation 1010, the wearable device may determine a central axis within the first image based on the positions of the left leg and the right leg identified based on the first image. For example, the wearable device may determine the positions of the user's left leg and the right leg within the scene based on the first image and previous images of the first image.

[0134] According to one embodiment, the wearable device may determine an expected path for the left leg based on a position of the left leg identified based on a previous image generated prior to the first image. For example, the expected path for the left leg may be set to be parallel to the user's walking direction based on the position of the identified left leg. For example, the expected path for the left leg may be set as a straight line connecting a first position and a second position of the identified left leg. The first position of the left leg may be the most recently identified position of the left leg, and the second position of the left leg may be the position of the left leg identified prior to the first position.

[0135] In one embodiment, the wearable device may set a predicted path for the left leg on the first image.

[0136] According to one embodiment, the wearable device can generate a point cloud for the scene based on the first image, and set an expected path for the left leg on the generated point cloud.

[0137] According to one embodiment, the wearable device may determine an expected path for the right leg based on the location of the right leg identified based on the first image. For example, the expected path for the right leg may be set to be parallel to the user's walking direction based on the location of the identified right leg. For example, the expected path for the right leg may be set as a straight line connecting a first location and a second location of the identified right leg. The first location of the right leg may be the location of the right leg identified based on the first image, and the second location of the right leg may be the location of the right leg identified prior to the first location.

[0138] In one embodiment, the wearable device may set a predicted path for the right leg on the first image.

[0139] According to one embodiment, the wearable device can generate a point cloud for the scene based on the first image, and set an expected path for the right leg on the generated point cloud.

[0140] In one embodiment, the wearable device may determine a center axis based on the predicted path for the left leg and the predicted path for the right leg. For example, the wearable device may determine the center axis to be located midway between the predicted path for the left leg and the predicted path for the right leg. The wearable device may set the determined center axis on the first image. The center axis may be the predicted path for the user's torso.

[0141] In operation 1020, the wearable device may set a first RoI within the first image based on a central axis. For example, the first RoI may be the central axis. For example, the first RoI may be an area having a preset size that includes the central axis.

[0142] FIG. 11 illustrates a method for identifying a left leg or a right leg of a user in an image and setting an RoI based on the identified leg, according to one embodiment.

[0143] In one embodiment, a wearable device (e.g., the wearable device (100) of FIG. 1) may identify a position (1113) of a right leg within a first image (1110). The wearable device may establish a predicted path (1114) of the right leg based on the position (1113) of the right leg. For example, the predicted path (1114) of the right leg may be parallel to the direction of walking.

[0144] In one embodiment, a left leg predicted path may be set based on a previous image of the first image (1110). The wearable device may set a left leg predicted path (1112) in the first image (1110) based on the left leg predicted path set in the previous image.

[0145] According to one embodiment, the wearable device can set a central axis (1116) in the first image (1110) based on the right leg predicted path (1114) and the left leg predicted path (1112).

[0146] FIG. 12 illustrates a method for setting an RoI based on a first image, according to one embodiment.

[0147] According to one embodiment, a wearable device (e.g., the wearable device (100) of FIG. 1) can capture a scene (1210) using a distance sensor module (e.g., the first distance sensor module (810) or the second distance sensor module (820) of FIG. 8) to generate a first image (1220) which is a depth map. The scene may be an uphill staircase. The illustrated first image (1220) may be a portion of the original first image. The user's legs may appear within the original first image.

[0148] According to one embodiment, a wearable device can capture a scene (1210) using a distance sensor module to generate a first image (1230), which is a point cloud. The illustrated first image (1230) may be a portion of the entire point cloud. The user's legs may appear within the entire point cloud.

[0149] According to one embodiment, the wearable device may set a central axis in the first image (1220) or the first image (1230) and determine an RoI (1240) based on the set central axis. The RoI (1240) may correspond to an uphill staircase in the direction in which the user walks.

[0150] FIG. 13a illustrates a depth value pattern generated for a case where the terrain condition is flat, according to one embodiment.

[0151] In one embodiment, a user may be walking on a flat ground with an uphill staircase (1305a) in front of him. A distance sensor module (e.g., the first distance sensor module (810) or the second distance sensor module (820) of FIG. 8) may generate a first image by capturing a scene (1303a) of the flat ground at a point (1301). The first image generated by the distance sensor module may have a Z-axis in a depth direction, a Y-axis in a horizontal direction, and an X-axis orthogonal to each of the X-axis and the Y-axis. The width of the first image may correspond to the Y-axis FoV of the distance sensor, and the height may correspond to the X-axis FoV of the distance sensor. Each pixel of the first image may include distance information for a corresponding point of the scene.

[0152] The wearable device may set a first RoI in the first image and generate a first depth value pattern (1331) for the first RoI. For example, the first depth value pattern (1331) may be based on a coordinate system defined by a first axis that positions pixels on the first RoI based on a distance from the user and a second axis that represents depth values ​​of pixels on the first axis.

[0153] Pixels of the first RoI set for the flat area (1303a) may have relatively larger depth values ​​as they move away from the user. Accordingly, the first depth value pattern (1331) for the flat area (1303a) may have a gradually increasing graph.

[0154] After the flat ground, an uphill staircase (1305a) appears, but since the X-axis FoV of the distance sensor does not correspond to the uphill staircase (1305a), the depth value for the uphill staircase (1305a) does not appear in the first depth value pattern (1331).

[0155] FIG. 13b illustrates depth value patterns generated for a case where the terrain condition is an uphill staircase, according to one embodiment.

[0156] In one embodiment, a user may be walking uphill stairs. A distance sensor module (e.g., the first distance sensor module (810) or the second distance sensor module (820) of FIG. 8) may capture a first scene (1303b) of the uphill stairs (1305b) while the user is walking, thereby generating a first image. The wearable device may set a first RoI in the first image and generate a first depth value pattern (1332) for the first RoI. A first section (1332a) of the first depth value pattern (1332) may correspond to a flat surface, and a second section (1332b) may correspond to an upper obstacle. Based on the first depth value pattern (1332), a first state of the walking terrain may be determined as an upper obstacle.

[0157] The distance sensor module can generate a first image by capturing a first scene (1303c) of an uphill staircase (1305b) while the user is walking. The wearable device can set a first RoI in the first image and generate a first depth value pattern (1333) for the first RoI. A first section (1333a) of the first depth value pattern (1333) may correspond to a flat surface, a second section (1333b) may correspond to an upper obstacle, and a third section (1333c) may correspond to a flat surface. Based on the first depth value pattern (1333), the first state of the walking terrain may be determined to be an uphill staircase.

[0158] The distance sensor module can generate a first image by capturing a first scene (1303d) of an uphill staircase (1305b) while the user is walking. The wearable device can set a first RoI in the first image and generate a first depth value pattern (1334) for the first RoI. A first section (1334a) of the first depth value pattern (1334) may correspond to an upper obstacle, a second section (1334b) may correspond to a flat ground, and a third section (1334c) may correspond to an upper obstacle. Based on the first depth value pattern (1334), the first state of the walking terrain may be determined to be an uphill staircase.

[0159] FIG. 13c illustrates depth value patterns generated for a case where the terrain condition is a downhill staircase, according to one embodiment.

[0160] In one embodiment, a user may be walking down a downhill staircase. A distance sensor module (e.g., the first distance sensor module (810) or the second distance sensor module (820) of FIG. 8) may capture a first scene (1303e) of the downhill staircase (1305c) while the user is walking, thereby generating a first image. The wearable device may set a first RoI in the first image and generate a first depth value pattern (1335) for the first RoI. A first section (1335a) of the first depth value pattern (1335) may correspond to a flat surface, a second section (1335b) may correspond to a downward obstacle, and a third section (1335c) may correspond to a flat surface. Based on the first depth value pattern (1335), a first state of the walking terrain may be determined as a downward obstacle.

[0161] The distance sensor module can generate a first image by capturing a first scene (1303c) of a downhill staircase (1305c) while the user is walking. The wearable device can set a first RoI in the first image and generate a first depth value pattern (1336) for the first RoI. A first section (1336a) of the first depth value pattern (1336) may correspond to a downward obstacle, a second section (1336b) may correspond to a flat ground, a third section (1336c) may correspond to a downward obstacle, and a fourth section (1336d) may correspond to a flat ground. Based on the first depth value pattern (1336), the first state of the walking terrain may be determined to be a downhill staircase.

[0162] FIG. 13d illustrates depth value patterns generated for a case where the terrain condition is uphill, according to one embodiment.

[0163] In one embodiment, a user may be walking on a flat surface with an uphill slope (1305d) ahead. A distance sensor module (e.g., the first distance sensor module (810) or the second distance sensor module (820) of FIG. 8) may capture a first scene (1303g) while the user is walking, thereby generating a first image. The wearable device may set a first RoI in the first image and generate a first depth value pattern (1337) for the first RoI. The first depth value pattern (1337) may correspond to a flat surface.

[0164] The distance sensor module can generate a first image by capturing a first scene (1303h) of an uphill (1305d) while the user is walking. The wearable device can set a first RoI in the first image and generate a first depth value pattern (1338) for the first RoI. The first depth value pattern (1338) can correspond to an uphill. Based on the first depth value pattern (1338), the first state of the walking terrain can be determined as an uphill. The slope of the uphill can be additionally determined as the first state of the walking terrain.

[0165] The distance sensor module can generate a first image by capturing a first scene (1303i) of an uphill (1305d) while the user is walking. The wearable device can set a first RoI in the first image and generate a first depth value pattern (1339) for the first RoI. The first depth value pattern (1339) can correspond to an uphill. Based on the first depth value pattern (1339), the first state of the walking terrain can be determined as an uphill. The slope of the uphill can be additionally determined as the first state of the walking terrain.

[0166] The distance sensor module can generate a first image by capturing a first scene (1303j) of an uphill (1305d) while the user is walking. The wearable device can set a first RoI in the first image and generate a first depth value pattern (1340) for the first RoI. A first section (1340a) of the first depth value pattern (1340) can correspond to an uphill, and a second section (1340b) can correspond to a flat surface. Based on the first depth value pattern (1336), the first state of the walking terrain can be determined as an uphill preceding a flat surface.

[0167] FIG. 13e illustrates depth value patterns generated for a case where the terrain condition is downhill, according to one embodiment.

[0168] In one embodiment, a user may be walking on a flat surface with a downhill slope (1305e) ahead. A distance sensor module (e.g., the first distance sensor module (810) or the second distance sensor module (820) of FIG. 8) may capture a first scene (1303k) while the user is walking, thereby generating a first image. The wearable device may set a first RoI in the first image and generate a first depth value pattern (1341) for the first RoI. A first section (1341a) of the first depth value pattern (1341) may correspond to a flat surface, and a second section (1341b) may correspond to a downhill slope. Based on the first depth value pattern (1341), a first state of the walking terrain may be determined to be a flat surface with a downhill slope ahead.

[0169] The distance sensor module can generate a first image by capturing a first scene (1303l) of a downhill (1305e) while the user is walking. The wearable device can set a first RoI in the first image and generate a first depth value pattern (1342) for the first RoI. A first section (1342a) of the first depth value pattern (1342) can correspond to a downhill. Based on the first depth value pattern (1343), the first state of the walking terrain can be determined as a downhill. The slope of the downhill can be additionally determined as the first state of the walking terrain.

[0170] The distance sensor module can generate a first image by capturing a first scene (1303m) of a downhill (1305e) while the user is walking. The wearable device can set a first RoI in the first image and generate a first depth value pattern (1343) for the first RoI. A first section (1343a) of the first depth value pattern (1343) may correspond to a downhill, and a second section (1343b) may correspond to a flat surface. Based on the first depth value pattern (1343), the first state of the walking terrain can be determined as a downhill with a flat surface ahead.

[0171] FIG. 13f illustrates depth value patterns generated for a case where the terrain condition includes uphill steps and flat ground, according to one embodiment.

[0172] In one embodiment, a user may be walking uphill stairs (1305f). A distance sensor module (e.g., the first distance sensor module (810) or the second distance sensor module (820) of FIG. 8) may capture a first scene (1303n) of the uphill stairs (1305f) while the user is walking, thereby generating a first image. The wearable device may set a first RoI in the first image and generate a first depth value pattern (1344) for the first RoI. A first section (1344a) of the first depth value pattern (1344) may correspond to an upper obstacle, a second section (1344b) may correspond to a flat ground, a third section (1344c) may correspond to an upper obstacle, and a fourth section (1344d) may correspond to a flat ground. Based on the first depth value pattern (1344), a first state of the walking terrain may be determined as uphill stairs.

[0173] The distance sensor module can generate a first image by capturing a first scene (1303o) of an uphill staircase (1305f) while the user is walking. The wearable device can set a first RoI in the first image and generate a first depth value pattern (1345) for the first RoI. A first section (1345a) of the first depth value pattern (1345) may correspond to a flat surface, a second section (1345b) may correspond to an upper obstacle, and a third section (1345c) may correspond to a flat surface. Based on the first depth value pattern (1345), the first state of the walking terrain may be determined to be an uphill staircase.

[0174] The distance sensor module can generate a first image by capturing a first scene (1303p) of an uphill staircase (1305f) while the user is walking. The wearable device can set a first RoI in the first image and generate a first depth value pattern (1346) for the first RoI. A first section (1346a) of the first depth value pattern (1346) may correspond to a flat surface, a second section (1346b) may correspond to an upper obstacle, a third section (1346c) may correspond to a flat surface, and a fourth section (1346d) may correspond to an upper obstacle. Based on the first depth value pattern (1346), the first state of the walking terrain may be determined to be an uphill staircase.

[0175] The distance sensor module can generate a first image by capturing a first scene (1303q) of an uphill staircase (1305f) while the user is walking. The wearable device can set a first RoI in the first image and generate a first depth value pattern (1347) for the first RoI. A first section (1347a) of the first depth value pattern (1347) may correspond to a flat surface, a second section (1347b) may correspond to an upper obstacle, and a third section (1347c) may correspond to a flat surface. Based on the first depth value pattern (1345), the first state of the walking terrain may be determined to be an uphill staircase.

[0176] FIG. 13g illustrates depth value patterns generated for a case where the terrain condition includes a downhill slope and a flat land, according to one embodiment.

[0177] In one embodiment, a user may be walking down a set of stairs (1305g). A distance sensor module (e.g., the first distance sensor module (810) or the second distance sensor module (820) of FIG. 8) may capture a first scene (1303r) of the downhill stairs (1305g) while the user is walking, thereby generating a first image. The wearable device may set a first RoI in the first image and generate a first depth value pattern (1348) for the first RoI. A first section (1348a) of the first depth value pattern (1348) may correspond to a flat surface, a second section (1348b) may correspond to a downward obstacle, a third section (1348c) may correspond to a flat surface, a fourth section (1348d) may correspond to a downward obstacle, and a fifth section (1348e) may correspond to a flat surface. Based on the first depth value pattern (1348), the first state of the walking terrain can be determined as a downhill staircase.

[0178] The distance sensor module can generate a first image by capturing a first scene (1303s) of a downhill staircase (1305g) while the user is walking. The wearable device can set a first RoI in the first image and generate a first depth value pattern (1349) for the first RoI. A first section (1349a) of the first depth value pattern (1349) may correspond to a flat surface, a second section (1349b) may correspond to a downward obstacle, a third section (1349c) may correspond to a flat surface, a fourth section (1349d) may correspond to a downward obstacle, a fifth section (1349e) may correspond to a flat surface, and a fourth section (1349d) may correspond to a downward obstacle. Based on the first depth value pattern (1349), a first state of the walking terrain may be determined as a downhill staircase.

[0179] The distance sensor module can generate a first image by capturing a first scene (1303t) of a downhill staircase (1305g) while the user is walking. The wearable device can set a first RoI in the first image and generate a first depth value pattern (1350) for the first RoI. A first section (1350a) of the first depth value pattern (1350) may correspond to a flat surface, a second section (1350b) may correspond to a downward obstacle, a third section (1350c) may correspond to a flat surface, a fourth section (1350d) may correspond to a downward obstacle, and a fifth section (1350e) may correspond to a flat surface. Based on the first depth value pattern (1350), a first state of the walking terrain may be determined to be a downhill staircase.

[0180] The distance sensor module can generate a first image by capturing a first scene (1303u) of a downhill staircase (1305g) while the user is walking. The wearable device can set a first RoI in the first image and generate a first depth value pattern (1351) for the first RoI. A first section (1351a) of the first depth value pattern (1351) may correspond to a flat surface, a second section (1351b) may correspond to a downward obstacle, and a third section (1351c) may correspond to a flat surface. Based on the first depth value pattern (1351), the first state of the walking terrain can be determined as a downhill staircase with a flat surface in front.

[0181] FIG. 13h illustrates depth value patterns generated for a case where the terrain condition is an uphill slope with varying slope, according to one embodiment.

[0182] In one embodiment, a user may be walking uphill (1305h). A distance sensor module (e.g., the first distance sensor module (810) or the second distance sensor module (820) of FIG. 8) may capture a first scene (1303v) while the user is walking, thereby generating a first image. The wearable device may set a first RoI in the first image and generate a first depth value pattern (1352) for the first RoI. A first section (1352a) of the first depth value pattern (1352) may correspond to an uphill having a first slope, and a second section (1352b) may correspond to an uphill having a second slope. Based on the first depth value pattern (1352), a first state of the walking terrain may be determined to be an uphill with a changing slope.

[0183] The distance sensor module can generate a first image by capturing a first scene (1303w) of an uphill (1305h) while the user is walking. The wearable device can set a first RoI in the first image and generate a first depth value pattern (1353) for the first RoI. A first section (1353a) of the first depth value pattern (1353) may correspond to an uphill with a second slope, and a second section (1353b) may correspond to a flat surface. Based on the first depth value pattern (1353), the first state of the walking terrain can be determined as an uphill preceding a flat surface.

[0184] FIG. 13i illustrates depth value patterns generated for a case where the terrain condition is a downhill slope with varying slope, according to one embodiment.

[0185] In one embodiment, a user may be walking downhill (1305i). A distance sensor module (e.g., the first distance sensor module (810) or the second distance sensor module (820) of FIG. 8) may capture a first scene (1303x) while the user is walking, thereby generating a first image. The wearable device may set a first RoI in the first image and generate a first depth value pattern (1354) for the first RoI. A first section (1354a) of the first depth value pattern (1354) may correspond to a downhill having a first slope, a second section (1354b) may correspond to a downhill having a second slope, and a third section (1354c) may correspond to a downhill having a third slope. Based on the first depth value pattern (1354), a first state of the walking terrain may be determined to be a downhill with a changing slope.

[0186] The distance sensor module can generate a first image by capturing a first scene (1303y) of a downhill (1305i) while the user is walking. The wearable device can set a first RoI in the first image and generate a first depth value pattern (1355) for the first RoI. A first section (1355a) of the first depth value pattern (1355) can correspond to a downhill with a second slope, and a second section (1355b) can correspond to a downhill with a third slope. Based on the first depth value pattern (1355), the first state of the walking terrain can be determined as a downhill with a changing slope.

[0187] FIG. 14a illustrates depth value patterns generated for a case where the terrain condition is flat and obstacles are located above, according to one embodiment.

[0188] According to one embodiment, a user may be walking on a flat surface (1405a) where obstacles are located above. A distance sensor module (e.g., the first distance sensor module (810) or the second distance sensor module (820) of FIG. 8) may capture a first scene (1403a) while the user is walking, thereby generating a first image. The wearable device may set a first RoI in the first image and generate a first depth value pattern (1431) for the first RoI. Based on the first depth value pattern (1431), the first state of the walking terrain may be determined to be a flat surface where obstacles are located above.

[0189] The distance sensor module can generate a first image by capturing a first scene (1403b) while the user is walking. The wearable device can set a first RoI in the first image and generate a first depth value pattern (1432) for the first RoI. Based on the first depth value pattern (1432), the first state of the walking terrain can be determined as a flat ground where obstacles are located above.

[0190] FIG. 14b illustrates depth value patterns generated for a case where the terrain condition is flat and obstacles are located below, according to one embodiment.

[0191] According to one embodiment, a user may be walking on a flat surface (1405b) where obstacles are located below. A distance sensor module (e.g., the first distance sensor module (810) or the second distance sensor module (820) of FIG. 8) may capture a first scene (1403c) while the user is walking, thereby generating a first image. The wearable device may set a first RoI in the first image and generate a first depth value pattern (1433) for the first RoI. Based on the first depth value pattern (1431), the first state of the walking terrain may be determined to be a flat surface where obstacles are located below.

[0192] The distance sensor module can generate a first image by capturing a first scene (1403d) while the user is walking. The wearable device can set a first RoI in the first image and generate a first depth value pattern (1434) for the first RoI. Based on the first depth value pattern (1434), the first state of the walking terrain can be determined as a flat ground where obstacles are located below.

[0193] FIG. 15 is a flowchart of a method for determining a first state of a walking terrain corresponding to a first RoI based on a first reference depth value pattern, according to one embodiment.

[0194] According to one embodiment, operations 1510 and 1520 below may be related to operation 950 described above with reference to FIG. 9. For example, operation 950 may include operations 1510 and 1520. Operations 1510 and 1520 may be performed by a control module (e.g., control module 130 of FIG. 1 or control module 510 of FIG. 5A) that includes at least one processor (e.g., processor 512 of FIG. 5) for controlling the wearable device and a memory (e.g., memory 514 of FIG. 5) for storing instructions executable by the processor.

[0195] In operation 1510, the wearable device may calculate a first similarity between a first reference depth value pattern and the first depth value pattern among a plurality of reference depth value patterns corresponding to states of a plurality of walking terrains. For example, the wearable device may calculate similarities between each of the plurality of reference depth value patterns and the first depth value pattern. Each of the plurality of reference depth value patterns may be a standardized depth value pattern for a state of the corresponding walking terrain. The plurality of reference depth value patterns may be generated in advance and stored in the wearable device.

[0196] In operation 1520, the wearable device may determine a first state of the walking terrain corresponding to the first RoI based on the calculated first similarity. For example, the wearable device may determine the state of the walking terrain corresponding to the similarity having the largest value among the similarities calculated between each of the plurality of reference depth value patterns and the first depth value pattern as the first state.

[0197] FIG. 16 is a flowchart of a method for determining a first state of a walking terrain corresponding to a first RoI based on a second reference depth value pattern generated based on a first posture of a distance sensor, according to one embodiment.

[0198] According to one embodiment, operations 1610 to 1630 below may be related to operation 950 described above with reference to FIG. 9. For example, operation 950 may include operations 1610 to 1630. Operations 1610 to 1630 may be performed by a control module (e.g., control module (130) of FIG. 1 or control module (510) of FIG. 5A) including at least one processor (e.g., processor (512) of FIG. 5) for controlling the wearable device and a memory (e.g., memory (514) of FIG. 5) for storing instructions executable by the processor.

[0199] In operation 1610, the wearable device may determine a first pose of a range sensor (e.g., range sensor (814) of FIG. 8B) at the time the first image was generated using an auxiliary IMU (e.g., auxiliary IMU (812) of FIG. 8B). For example, the pose of the auxiliary IMU may be determined as the first pose of the range sensor.

[0200] In operation 1620, the wearable device may determine whether the first depth value pattern is valid based on the first pose of the distance sensor. For example, if the tilt angle of the distance sensor with respect to the horizontal plane is α°, it may be determined whether α falls within a preset valid tilt angle range.

[0201] In operation 1630, the wearable device may determine a first state of a walking terrain corresponding to the first RoI based on the first depth value pattern, if the first depth value pattern is determined to be valid.

[0202] FIG. 17 is a flowchart of a method for outputting a first alarm associated with a first state, according to one embodiment.

[0203] According to one embodiment, operations 1710 and 1720 below may be performed after operation 950 described above with reference to FIG. 9 is performed. Operations 1710 and 1720 may be performed by a control module (e.g., control module (130) of FIG. 1 or control module (510) of FIG. 5A) including at least one processor (e.g., processor (512) of FIG. 5) for controlling the wearable device and a memory (e.g., memory (514) of FIG. 5) for storing instructions executable by the processor.

[0204] In operation 1710, the wearable device may determine whether the first state of the walking terrain is the same as the current state. For example, the current state may be a state of the walking terrain determined before the first state was determined.

[0205] For example, if the current state of the walking terrain is an uphill with a first slope, and the first state is an uphill with a second slope, the current state and the first state can be determined to be the same or different depending on a slope condition preset for state determination.

[0206] In operation 1720, if the first state of the walking terrain is different from the current state, the wearable device may output a first notification associated with the first state. The first notification may be a message to alert the user.

[0207] In one embodiment, the wearable device may output a vibration associated with the first notification so that the user can experience tactile feedback. For example, the wearable device may output the vibration using a separate vibration motor. For example, the wearable device may output the notification torque as a vibration using a motor that generates torque (e.g., motor (534) of FIG. 5A).

[0208] In one embodiment, the wearable device may output a sound associated with the first notification so that the user can feel auditory feedback.

[0209] According to one embodiment, the wearable device may output a screen associated with the first notification through a display or output light through a lighting unit (e.g., lighting unit (85) of FIG. 3) so that the user can feel visual feedback.

[0210] FIG. 18 is a flowchart of a method for controlling a driving module based on a user's walking pattern, according to one embodiment.

[0211] According to one embodiment, operations 1810 and 1820 below may be performed after operation 920 described above with reference to FIG. 9 is performed. Operations 1810 and 1820 may be performed by a control module (e.g., the control module (130) of FIG. 1 or the control module (510) of FIG. 5A) including at least one processor (e.g., the processor (512) of FIG. 5) for controlling the wearable device and a memory (e.g., the memory (514) of FIG. 5) for storing instructions executable by the processor.

[0212] In operation 1810, the wearable device may determine the user's gait pattern based on at least one of the left leg or the right leg identified based on the first image. The wearable device may determine the user's gait pattern based on the posture of the identified left leg. For example, the posture of the left leg may be the angle between the left sole or shoe and the predicted path for the left leg. For example, the user's gait pattern may be a limping of the left leg. For example, the user's gait pattern may be a straight gait. For example, the user's gait pattern may be a toe-in gait. For example, the user's gait pattern may be a figure-eight gait.

[0213] In operation 1820, the wearable device may control the driving module based on the determined user's gait pattern. For example, if the user's gait pattern is a limping of the left leg, the wearable device may control the driving module to output a stronger assistive torque for the left leg than the assistive torque for the right leg. For example, if the user's gait pattern is a toe-in gait, the wearable device may control the driving module to output a torque that can correct the toe-in gait. For example, if the user's gait pattern is a figure-eight gait, the wearable device may control the driving module to output a torque that can correct the figure-eight gait.

[0214] According to one embodiment, the wearable device may calculate a first torque to assist the user's walking, calculate a second torque based on the user's walking pattern, calculate a third torque based on the first and second torques, and control the driving module to output the third torque. For example, the first torque may be determined based on the user's walking cycle and the current state of the walking terrain. The current state of the walking terrain may be the state of the walking terrain on which the user's legs are standing.

[0215] FIG. 19 is a flowchart of a method for controlling a driving module based on a user's walking speed, according to one embodiment.

[0216] According to one embodiment, operations 1910 and 1920 below may be performed after operation 920 described above with reference to FIG. 9 is performed. Operations 1910 and 1920 may be performed by a control module (e.g., the control module (130) of FIG. 1 or the control module (510) of FIG. 5A) including at least one processor (e.g., the processor (512) of FIG. 5) for controlling the wearable device and a memory (e.g., the memory (514) of FIG. 5) for storing instructions executable by the processor.

[0217] In operation 1910, the wearable device may determine the user's walking speed based on at least one of the left leg or right leg identified based on the first image. For example, the user's walking speed may be determined based on the position of the left leg identified based on the first image and the position of the left leg identified based on a previous image of the first image. For example, the walking speed may be determined based on the time between the generation time of the previous image and the generation time of the first image and the distance traveled by the left leg.

[0218] In operation 1920, the wearable device may control the drive module based on the user's walking speed. For example, the wearable device may calculate torque based on the user's walking cycle and walking speed, and control the drive module to output the calculated torque.

[0219] A method for determining a first state of a walking terrain based on a first image generated using a distance sensor module (e.g., the first distance sensor module (810) or the second distance sensor module (820) of FIG. 8) is described with reference to FIGS. 9 to 19.

[0220] A method for determining a first state of a walking terrain based on a plurality of depths for different points of a scene calculated using a distance sensor module (e.g., the first distance sensor module (810) or the second distance sensor module (820) of FIG. 8) is described below with reference to FIGS. 20 to 23.

[0221] FIG. 20 illustrates a method for determining the state of a walking terrain using a plurality of distance sensors according to one embodiment.

[0222] According to one embodiment, a wearable device (e.g., the wearable device (100) of FIG. 1) may include a distance sensor module (2010) (e.g., the first distance sensor module (810) or the second distance sensor module (820) of FIG. 8). The distance sensor module (2010) may include a first distance sensor and a second distance sensor. The first distance sensor and the second distance sensor may be arranged side by side within the distance sensor module (2010). Each of the first distance sensor and the second distance sensor may calculate a distance to a corresponding point in a scene. The distance sensor module (2010) may further include an auxiliary IMU (e.g., the auxiliary IMU (812) of FIG. 8). The auxiliary IMU may determine a pose of the auxiliary IMU.

[0223] For example, the first distance sensor may be installed on the wearable device such that the central axis of the first distance sensor has an angle of θ1° with respect to the direction of gravity when the user wears the wearable device. For example, when the tilt angle measured by the auxiliary IMU indicates 0°, the angle formed by the central axis of the first distance sensor and the direction of gravity may be θ1°.

[0224] For example, the second distance sensor may be installed on the wearable device such that the central axis of the second distance sensor has an angle of θ2° with respect to the direction of gravity when the user wears the wearable device. For example, when the tilt angle measured by the auxiliary IMU indicates 0°, the angle formed by the central axis of the second distance sensor and the direction of gravity may be θ2°. The distance or depth to the first point of the scene calculated by the second distance sensor may be l2.

[0225] In one embodiment, when the user is located on a flat surface, the distance or depth to a first point in the scene calculated by the first distance sensor may be l1, and the distance or depth to a second point in the scene calculated by the second distance sensor may be l2.

[0226] In one embodiment, when a user is positioned in front of an uphill staircase (2020), a distance or depth l3 to a third point in the scene calculated by a first distance sensor may be measured, and a distance or depth l4 to a fourth point in the scene calculated by a second distance sensor may be measured. l3 may be equal to l1. l4 may be shortened compared to l2 by the uphill staircase (2020). The wearable device may determine a first state of the walking terrain based on the first depth calculated by the first distance sensor and the second depth calculated by the second distance sensor. A method for determining a first state of the walking terrain based on the first depth and the second depth is described in detail below with reference to FIG. 21.

[0227] FIG. 21 is a flowchart of a method for determining the state of a walking terrain using a plurality of distance sensors according to one embodiment.

[0228] The operations 2110 to 2160 below may be performed by a wearable device (e.g., the wearable device (100) of FIG. 1). For example, the operations 2110 to 2160 may be performed by a control module (e.g., the control module (130) of FIG. 1 or the control module (510) of FIG. 5A) including at least one processor (e.g., the processor (512) of FIG. 5) for controlling the wearable device and a memory (e.g., the memory (514) of FIG. 5) for storing instructions executable by the processor.

[0229] In operation 2110, the wearable device may calculate a first depth (e.g., l1 or l3 in FIG. 20) for a first location (or, first point) in a scene surrounding the wearable device using a first distance sensor installed on the wearable device at a preset first angle.

[0230] In operation 2120, the wearable device may calculate a second depth (e.g., l2 or l4 in FIG. 20) for a second location (or second point) in the surrounding scene of the wearable device using a second distance sensor installed on the wearable device at a preset second angle.

[0231] In operation 2130, the wearable device may calculate a first height based on the first angle and the first depth. If both the user's current walking terrain and the walking terrain ahead are flat, the calculated first height may be equal to the height of the first distance sensor from the ground. For example, the first height may be calculated using [Mathematical Equation 1].

[0232] [Mathematical Formula 1]

[0233]

[0234] In operation 2140, the wearable device may calculate a first height based on the second angle and the second depth. For example, the second height may be calculated using [Mathematical Formula 2].

[0235] [Equation 2]

[0236]

[0237] In operation 2150, the wearable device may generate a first depth value pattern based on the first height and the second height.

[0238] For example, the wearable device may calculate a first difference between a first height and a second height. Referring to the left drawing of FIG. 20, if the terrain conditions corresponding to the first location and the terrain conditions corresponding to the second location are not identical (i.e., if the terrain is flat), the first difference may be 0 or a value close to 0.

[0239] For example, the wearable device may calculate a second difference between the third height and the fourth height. Referring to the right drawing of FIG. 20, if the terrain conditions corresponding to the third location are different from the terrain conditions corresponding to the fourth location (i.e., if the terrain is an uphill staircase), as in the aforementioned embodiment, if the fourth depth decreases relative to the second depth, the fourth height may increase relative to the second height. The calculated second difference between the third depth and the fourth depth may be negative.

[0240] According to one embodiment, the wearable device may generate a first depth value pattern by accumulating differences that are continuously calculated according to the user's walking. For example, the wearable device may generate the first depth value pattern to include a preset number of differences. When a new difference is calculated, the wearable device may exclude the first difference calculated among the differences in the first depth value pattern from the first depth value pattern and add the new difference to the first depth value pattern.

[0241] The description of the first depth value pattern can be similarly applied to the description of FIGS. 13a to 13g and FIGS. 14a and 14.

[0242] In operation 2160, the wearable device may determine a first state of the walking terrain corresponding to the first depth value pattern based on the first depth value pattern. A description of a method for determining the first state of the walking terrain based on the first depth value pattern may be replaced with the description of operation 950 described above with reference to FIG. 9.

[0243] FIG. 22 illustrates a method for determining the posture of a plurality of distance sensors according to one embodiment.

[0244] According to one embodiment, the posture of the distance sensor module (2010) (e.g., the first distance sensor module (810) or the second distance sensor module (820) of FIG. 8) may change depending on the user's movement. As the posture of the distance sensor module (2010) changes, the angle formed by the central axis of the first distance sensor and the direction of gravity and the angle formed by the central axis of the second distance sensor and the direction of gravity may change.

[0245] The auxiliary IMU of the distance sensor module (2010) (e.g., the auxiliary IMU (812) of FIG. 8) can determine the changed attitude of the distance sensor module (2010). For example, the tilt angle of the determined attitude of the distance sensor module (2010) may be α°. If the attitude of the distance sensor module (2010) changes, the first height and the second height calculated based on the attitude of the distance sensor module (2010) may be different, and thus the generated first depth value pattern may become inaccurate. A method for generating the first depth value pattern based on the attitude of the distance sensor module (2010) will be described in detail below with reference to FIG. 23.

[0246] FIG. 23 is a flowchart of a method for determining a state of a walking terrain based on the postures of a plurality of distance sensors, according to one embodiment.

[0247] The operations 2310 to 2350 below may be performed by a wearable device (e.g., the wearable device (100) of FIG. 1). For example, the operations 2310 to 2350 may be performed by a control module (e.g., the control module (130) of FIG. 1 or the control module (510) of FIG. 5A) including at least one processor (e.g., the processor (512) of FIG. 5) for controlling the wearable device and a memory (e.g., the memory (514) of FIG. 5) for storing instructions executable by the processor.

[0248] According to one embodiment, operation 2310 may be performed after operation 2110 described above with reference to FIG. 21 is performed.

[0249] In operation 2310, the wearable device may determine a first attitude of the first distance sensor and the second distance sensor using an auxiliary IMU (e.g., an auxiliary IMU (812) of FIG. 8) of a distance sensor module (e.g., the first distance sensor module (810), the second distance sensor module (820) of FIG. 8, or the distance sensor module (2010) of FIG. 20). For example, a tilt angle of the first distance sensor and the second distance sensor may be determined as the first attitude.

[0250] In operation 2320, the wearable device may calculate a third height based on the first angle, the first posture, and the first depth. For example, the third height may be calculated using [Mathematical Formula 3].

[0251] [Equation 3]

[0252]

[0253] In operation 2330, the wearable device may calculate a fourth height based on the second angle, the first posture, and the second depth. For example, the fourth height may be calculated using [Mathematical Formula 4].

[0254] [Equation 4]

[0255]

[0256] In operation 2340, the wearable device may generate a second depth value pattern based on the third height and the fourth height. The description of the method for generating the second depth value pattern may be similarly applied to the description of the method for generating the first depth value pattern.

[0257] In operation 2350, the wearable device may determine a first state of the walking terrain corresponding to the second depth value pattern based on the second depth value pattern. A description of a method for determining the first state of the walking terrain based on the second depth value pattern may be replaced with the description of operation 950 described above with reference to FIG. 9.

[0258] According to one embodiment, a wearable device (100) includes a base body (80) positioned at a waist area of ​​a user (110) when the wearable device is worn on the body of the user, a waist support frame (20) and a leg support frame (50; 55; 810) for supporting at least a part of the user's body, a thigh fastening part (1; 2) for fixing the leg support frame to the user's thigh, an IMU (135) positioned within the base body, a drive module (35; 45; 120) for generating a torque applied to the user's leg, wherein the drive module is positioned between the waist support frame and the leg support frame, an angle sensor (125) for measuring a rotation angle of the leg support frame, a distance sensor (814) for calculating a distance to a surrounding scene of the wearable device, and a control module (130; 510) including at least one processor (512) for controlling the wearable device and a memory (514) for storing instructions executable by the processor, when the instructions are executed by the processor, The wearable device may be configured to: generate a first image by capturing a scene using a distance sensor; identify at least one of a left leg or a right leg of a user within the first image based on depth information about the first image; set a first RoI within the first image based on at least one of the left leg or the right leg; generate a first depth value pattern for the first RoI based on the depth information about the first RoI; and determine a first state of a walking terrain corresponding to the first RoI based on the first depth value pattern.

[0259] Each "processor" in this disclosure may include processing circuitry, or may include multiple processors. For example, the term "processor" as used in this disclosure, including in the claims, may include various processing circuitry, including at least one processor, wherein one or more of the at least one processors may be individually and / or collectively configured to perform various functions described in this disclosure in a distributed manner. When the terms "processor," "at least one processor," and "one or more processors" as used in this disclosure are described as being configured to perform a number of functions, these terms encompass, for example, without limitation, situations where one processor performs some of the recited functions, other processor(s) perform other of the recited functions, and situations where a single processor may perform all of the recited functions. Furthermore, the at least one processor may include a combination of processors that perform various recited / disclosed functions, for example, in a distributed manner. The at least one processor may execute program instructions to achieve or perform various functions.

[0260] According to one embodiment, each pixel of the first image can have an intensity value.

[0261] According to one embodiment, each pixel of the first image can have a depth value.

[0262] According to one embodiment, when the instructions are executed by the processor, the wearable device may be configured to at least: determine a central axis within a first image based on a position of the left leg and a position of the right leg, and set a first RoI based on the central axis.

[0263] According to one embodiment, the first depth value pattern may be based on a coordinate system defined by a first axis that positions pixels on the first RoI based on a distance from the user and a second axis that represents depth values ​​of pixels on the first axis.

[0264] According to one embodiment, when the instructions are executed by the processor, the wearable device may at least: calculate a first similarity between a first reference depth value pattern and the first depth value pattern among a plurality of reference depth value patterns corresponding to states of a plurality of walking terrains, and determine the first state of the walking terrain corresponding to the first RoI based on the first similarity.

[0265] According to one embodiment, the first state of the walking terrain may be any one of a flat surface, an uphill slope, a downhill slope, an uphill staircase, a downhill staircase, an upper obstacle, or a lower obstacle.

[0266] According to one embodiment, the wearable device further includes an auxiliary IMU (812) for measuring a pose of the distance sensor, and when the instructions are executed by the processor, the wearable device may at least: determine a first pose of the distance sensor at a time when the first image is generated using the auxiliary IMU, determine whether a first depth value pattern is valid based on the first pose, and if the first depth value pattern is determined to be valid, determine the first state of the walking terrain corresponding to the first RoI based on the first depth value pattern.

[0267] According to one embodiment, when the instructions are executed by the processor, the wearable device may at least: determine whether a first state is the same as a current state, and if the first state is different from the current state, output a first notification associated with the first state.

[0268] According to one embodiment, when the instructions are executed by the processor, the wearable device may be configured to control the drive module based on at least: a first state.

[0269] According to one embodiment, when the instructions are executed by the processor, the wearable device may determine a gait pattern of the user based on at least one of the left leg or the right leg, and control the driving module based on the gait pattern.

[0270] According to one embodiment, the instructions, when executed by the processor, may cause the wearable device to determine a walking speed of the user based on at least one of the left leg or the right leg, and control the driving module based on the walking speed.

[0271] According to one embodiment, a method for determining a state of a walking terrain performed by an electronic device (100) may include an operation of generating a first image by capturing a scene using a distance sensor (814), an operation of identifying at least one of a left leg or a right leg of a user within the first image based on depth information about the first image, an operation of setting a first RoI within the first image based on at least one of the left leg or the right leg, an operation of generating a first depth value pattern for the first RoI based on the depth information about the first RoI, and an operation of determining a first state of a walking terrain corresponding to the first RoI based on the first depth value pattern.

[0272] According to one embodiment, a wearable device (100) comprises: a base body (80) positioned at the waist area of ​​a user (110) when the wearable device is worn on the body of the user; a waist support frame (20) and a leg support frame (50; 55; 810) for supporting at least a part of the user's body; a thigh fastening part (1; 2) for fixing the leg support frame to the user's thigh; an IMU (135) positioned within the base body; a driving module (35; 45; 120) for generating a torque applied to the user's leg, wherein the driving module is positioned between the waist support frame and the leg support frame; an angle sensor (125) for measuring a rotation angle of the leg support frame; a first distance sensor installed on the wearable device at a first preset angle and calculating a first depth for a first position of a surrounding scene of the wearable device; a second distance sensor installed on the wearable device at a second preset angle and calculating a second depth for a second position of a surrounding scene of the wearable device; and a controller for controlling the wearable device. A control module (130; 510) comprising at least one processor (512) and a memory (514) storing instructions executable by the processor, wherein when the instructions are executed by the processor, the wearable device can cause at least: to calculate a first height based on a first angle and a first depth, to calculate a second height based on a second angle and a second depth, to generate a first depth value pattern based on the first height and the second height, and to determine a first state of a walking terrain corresponding to the first depth value pattern based on the first depth value pattern.

[0273] According to one embodiment, the wearable device further includes an auxiliary IMU (812) for measuring a pose of the first distance sensor and the second distance sensor, and the instructions, when executed by the processor, cause the wearable device to at least: determine a first pose of the first distance sensor and the second distance sensor using the auxiliary IMU, calculate a third height based on the first angle, the first pose and the first depth, calculate a fourth height based on the second angle, the first pose and the second depth, generate a first depth value pattern based on the third height and the fourth height, and determine a first state of a walking terrain corresponding to the first RoI based on the second depth value pattern.

[0274] According to one embodiment, when the instructions are executed by the processor, the wearable device may be configured to: calculate a first similarity between a first reference depth value pattern and the first depth value pattern among a plurality of reference depth value patterns corresponding to states of a plurality of walking terrains, and determine a first state of the walking terrain corresponding to the first RoI based on the first similarity.

[0275] According to one embodiment, when the instructions are executed by the processor, the wearable device may at least: determine whether a first state is the same as a current state, and if the first state is different from the current state, output a first notification associated with the first state.

[0276] According to one embodiment, when the instructions are executed by the processor, the wearable device may be configured to control the drive module based on at least: a first state.

[0277] According to one embodiment, the first state of the walking terrain may be any one of a flat surface, an uphill slope, a downhill slope, an uphill staircase, a downhill staircase, an upward obstacle, or a downward obstacle.

[0278] The embodiments described above may be implemented using hardware components, software components, and / or a combination of hardware components and software components. For example, the devices, methods, and components described in the embodiments may be implemented using a general-purpose computer or a special-purpose computer, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and software applications running on the OS. The processing device may also access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone; however, one of ordinary skill in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

[0279] Software may include a computer program, code, instructions, or a combination of one or more of these, and may configure a processing device to perform a desired operation or, independently or collectively, command the processing device. The software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave, for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on a computer-readable recording medium.

[0280] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program commands, data files, data structures, etc., alone or in combination, and the program commands recorded on the medium may be those specially designed and configured for the embodiment or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands such as ROMs, RAMs, and flash memories. Examples of program commands include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc.

[0281] The hardware device described above may be configured to operate as one or more software modules to perform the operations of the embodiment, and vice versa.

[0282] Although the embodiments have been described with limited drawings as described above, those skilled in the art will appreciate that various technical modifications and variations can be applied based on the embodiments. For example, appropriate results can be achieved even if the described techniques are performed in a different order than the described method, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than the described method, or are replaced or substituted with other components or equivalents. While the present disclosure has been described and illustrated with reference to various embodiments, it should be understood that the various embodiments are illustrative and not limiting. Furthermore, those skilled in the art should understand that various changes in form and detail can be made without departing from the true spirit and full scope of the disclosure, including the appended claims and their equivalents. Furthermore, it should be understood that any of the embodiments described herein can be used in conjunction with other embodiments described herein.

[0283] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims described below.

Claims

1. A wearable device (100) configured to be worn by a user, A waist support frame (20), a first leg support frame (55), and a second leg support frame (50) for supporting at least a part of the user's body; A first thigh fastening part (2) and a second thigh fastening part (1) for fixing the first leg support frame (55) and the second leg support frame (50) to the user's thigh; The wearable device (100) is configured to be worn on a user's body, and thus includes a base body (80) configured to be positioned on the user's waist area; An inertial measurement unit (IMU) (135) including a sensor supported by the base body; A first drive module (45) and a second drive module (35), each including a motor and / or circuit for generating a torque applied to each of the user's legs, wherein the first drive module (45) and the second drive module (35) are positioned between the waist support frame (20) and the first leg support frame (55) and the second leg support frame (50); An angle sensor (125) that measures the rotation angle of the first leg support frame (55) and the second leg support frame (50); distance sensor (814); and At least one processor (512) comprising a processing circuit and configured to individually and / or collectively control the wearable device (100) Including, At least one processor (512) of the above: A first image is generated by capturing a scene using the above distance sensor, Identifying at least one of the user's left leg or right leg within the first image based on depth information about the first image; Setting a first RoI (region of interest) within the first image based on at least one of the left leg or the right leg, Generating a first depth value pattern for the first RoI based on depth information for the first RoI, Determining a first state of the walking terrain corresponding to the first RoI based on the first depth value pattern; Wearable device (100).

2. In paragraph 1, Each pixel of the first image above has an intensity value, Wearable device (100).

3. In paragraph 1 or 2, Each pixel of the first image above has a depth value, Wearable device (100).

4. In any one of paragraphs 1 to 3, The at least one processor (512) individually and / or collectively controls the wearable device (100): Determine the central axis within the first image based on the position of the left leg and the position of the right leg, Setting the first RoI based on the above central axis, Wearable device (100).

5. In any one of paragraphs 1 to 4, The above first depth value pattern is, Based on a coordinate system based on a first axis that positions pixels on the first RoI based on their distances from the user and a second axis that represents depth values ​​of the pixels on the first axis, Wearable device (100).

6. In any one of paragraphs 1 to 5, The at least one processor (512) individually and / or collectively controls the wearable device (100): Compute a first similarity between a first reference depth value pattern and the first depth value pattern among a plurality of reference depth value patterns corresponding to states of a plurality of walking terrains, Determine the first state of the walking terrain corresponding to the first RoI based on the first similarity. To do, Wearable device (100).

7. In any one of paragraphs 1 to 6, The first state of the above walking terrain is, Any of the following: flat, uphill, downhill, uphill stairs, downhill stairs, upper obstacles, or lower obstacles, Wearable device (100).

8. In any one of paragraphs 1 to 7, An auxiliary IMU (812) including a sensor for measuring the attitude of the above distance sensor. Including more, The at least one processor (512) individually and / or collectively controls the wearable device (100): Using the auxiliary IMU, the first attitude of the distance sensor is determined at the time when the first image is generated, Based on the first posture, determine whether the first depth value pattern is valid, If the first depth value pattern is determined to be valid, determining the first state of the walking terrain corresponding to the first RoI based on the first depth value pattern. Wearable device (100).

9. In any one of paragraphs 1 to 8, The at least one processor (512) individually and / or collectively controls the wearable device (100): Determine whether the above first state is identical to the current state, If the first state is different from the current state, outputting a first notification associated with the first state. Wearable device (100).

10. In any one of paragraphs 1 to 9, The at least one processor (512) individually and / or collectively controls the wearable device (100): Controlling the first driving module and the second driving module based on the first state; Wearable device (100).

11. In any one of paragraphs 1 to 10, The at least one processor (512) individually and / or collectively controls the wearable device (100): determining the user's walking pattern based on at least one of the left leg or the right leg; Controlling the first driving module and the second driving module based on the above walking pattern, Wearable device (100).

12. In any one of paragraphs 1 to 11, The at least one processor (512) individually and / or collectively controls the wearable device (100): Determining the walking speed of the user based on at least one of the left leg or the right leg, Controlling the first driving module and the second driving module based on the walking speed To do, Wearable device (100).

13. A method for determining the state of a walking terrain, performed by an electronic device (100), An operation of generating a first image by capturing a scene using a distance sensor (814); An action of identifying at least one of the user's left leg or right leg within the first image based on depth information about the first image; An operation of setting a first region of interest (RoI) within the first image based on at least one of the left leg or the right leg; An operation of generating a first depth value pattern for the first RoI based on depth information for the first RoI; and An operation for determining a first state of a walking terrain corresponding to the first RoI based on the first depth value pattern. Including, method.

14. In paragraph 13, An operation of setting a first RoI within the first image based on at least one of the left leg or the right leg, An operation of determining a central axis within the first image based on the position of the left leg and the position of the right leg; and An operation for setting the first RoI based on the above central axis. Including, method.

15. A computer-readable recording medium containing a program for performing the method of Article 13 or 14.

Images