Multimodal sensor fusion platform

Abstract

To provide a smart insole system.SOLUTION: A system may include a sensing layer provided on a bottom layer and a first surface of the bottom layer. A first sensing layer may include a first sensing element and a second sensing element. The first sensing element and the second sensing element may include one of a force sensing element, a strain sensing element, a motion sensing element and an environment sensing element. The first sensing element and the second sensing element may be sensing elements of different types. The system may further include a communication interface configured to connect the sensing layer to a host controller.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The embodiments discussed herein are directed to a multi-modal sensor fusion platform. Regarding. [Background technology]

[0002] The Internet of Things (IoT) collects and processes data, often without user input. The electronic circuits, software, sensors, actuators, and networks that enable A network of physical devices, cars, buildings, and other things that have embedded network connectivity. Typically, IoT devices are also called smart devices. Some progress has been made in developing smart insoles for use in shoes. These smart insole solutions use force sensing elements to detect force and / or The method relies primarily on mapping pressure. However, smart insoles may present various limitations.

[0003] The subject matter claimed herein is intended to solve the disadvantages or problems described above. The present invention is not limited to the embodiments that operate as described herein. Rather, this background is provided for purposes of illustration only. It is provided merely to illustrate one example technology area in which one embodiment may be practiced. do not have. [Brief description of the drawings]

[0004] Exemplary embodiments will be described and explained with additional specificity and detail through the use of the accompanying drawings. can be. [Figure 1] FIG. 1 shows an example system diagram of multimodal sensor fusion. [Diagram 2]FIG. 2 illustrates an exemplary smart shoe system configuration for sensing various characteristics associated with a human foot and / or shoe. [Diagram 3] FIG. 3 illustrates an exemplary system configuration for sensing various characteristics associated with a human foot and / or shoe. [Figure 4] FIG. 4 illustrates an exemplary computational process flow of a method for determining various functions including, but not limited to, foot force / pressure mapping and biomechanics, such as foot flexion characteristics, during a walking or running motion, etc. [Diagram 5] FIG. 5 shows an example computational process flow for determining various functions including, but not limited to, foot force / pressure mapping and biomechanics, such as foot flexion characteristics, during walking, running, turning, or jumping movements, etc. [Figure 6] FIG. 6 shows an example implementation of a smart shoe system in which a smart shoe includes one or more sensing elements. [Figure 7] FIG. 7 shows an example of a graphical user interface (GUI) for parameter monitoring such as foot / pressure mapping based on measurement data from a force sensing element. [Figure 8] FIG. 8 illustrates various motion animation displays that may be generated based at least in part on measurements obtained from one or more sensing elements described herein. [Figure 9] FIG. 9 illustrates an exemplary computational process flow for determining various functions including, but not limited to, user profile identification and visualization. [Figure 10] FIG. 10 shows an example computational process flow for determining various functions including, but not limited to, applications of smart shoes in fitness and / or physiological monitoring. [Figure 11] FIG. 11 shows an example of real-time measurement of monitoring parameters for six defined motion states. [Figure 12]FIG. 12 shows an example of real-time measurement of monitoring parameters for six defined motion states. [Figure 13] FIG. 13 shows an example of real-time measurement of monitoring parameters for six defined motion states. [Figure 14] FIG. 14 shows an example of real-time measurement of monitoring parameters for six defined motion states. [Figure 15] FIG. 15 shows an example of real-time measurement of monitoring parameters for six defined motion states. [Figure 16] FIG. 16 shows an example of real-time measurement of monitoring parameters for six defined motion states. [Figure 17] FIG. 17 shows an example of real-time measurement of monitoring parameters for six defined motion states. [Figure 18] FIG. 18 shows an example of real-time measurement of monitoring parameters for six defined motion states. [Figure 19] FIG. 19 shows an example of real-time measurement of monitoring parameters for six defined motion states. [Figure 20] FIG. 20 shows an example of real-time measurement of monitoring parameters for six defined motion states. [Figure 21] FIG. 21 shows an example of real-time measurement of monitoring parameters for six defined motion states. [Figure 22] FIG. 22 illustrates a block diagram of an example computer system for a multimodal array in accordance with at least one embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0005] Smart insoles are often used for force and / or pressure mapping of a person's foot. Conventional smart insole solutions use smart Force and / or pressure sensing elements, typically integrated within the insole Conventional smart insole solutions often rely on foot pressure mapping. For mapping, extraction of parameters including walking / running speed, and pronation / supination In the prior art, the force sensing layer is large, providing a concentrated physical sensing function. The size and shape must correspond to the size, type and character of the shoe, otherwise The force sensing element may not provide an accurate reading. The factors are temperature, which can change dramatically based on different user activities (like running). They are often sensitive to environmental conditions including temperature and humidity.

[0006] Using only force sensing elements presents various limitations to smart insoles. For example, a moving person's foot can generate many different forces. can be difficult to measure accurately in a consistent manner. It can be difficult to measure motion characteristics using only the elements. For example, During movement (due to the movement of the user's foot), the insole may experience bending and flexing. Conventional systems are unable to distinguish or separate these bending or flexing forces from other types of forces. In addition, conventional systems may not be able to measure physiological monitoring parameters. may not provide real-time physical user feedback based on May not provide intuitive real-time visualization of physiological monitoring parameters Furthermore, conventional systems do not provide physiological monitoring using user profile identification. And conventional systems may not allow tracking of sensors and It may not provide a full-duplex data communications link between host devices.

[0007] Aspects of the present disclosure include force / pressure mapping and motion measurement within smart insoles, foot System for pressure monitoring, shoe customization, and other related applications (shoe By providing a smart insole or system that can be placed inside the To address these and other shortcomings.

[0008] Embodiments of the present disclosure are further described with reference to the accompanying drawings.

[0009] FIG. 1 illustrates an exemplary multi-modal sensor fusion platform ("System"). The system 100 includes various sensors, haptic devices, controllers, and The device may include a power management device, a power supply interface, and the like.

[0010] The system 100 may include one or more strain sensing elements 105 and / or one or more force sensors. The one or more strain sensing elements 105 may include a two-dimensional or The one or more force sensing elements may include a three-dimensional strain sensing element. / May include a pressure sensor.

[0011] In at least one embodiment, the system 100 includes one or more force sensing elements. The insole may include one or more force sensing elements 105. Detect and / or measure foot force and / or pressure distribution over part or all of the surface of the ball. The insole may include one or more strain sensing elements 105. One or more strain sensing elements 105 detect and / or flex the insole. and / or may be configured to measure.

[0012] The insole may include one or more environmental sensing elements 110. The sensing element 110 is adapted to detect and / or measure environmental parameters including temperature and humidity. At least some environmental parameters may contribute to noise in the system. Environmental parameters are used as a basis for explanation to reduce noise in the system. and / or may be mathematically reduced, minimized, or ignored. The integration of more than one type of sensing element together allows for robust force and pressure mapping and This may provide a solution for motion measurement.

[0013] The system 100 includes motion vectors that can be used to extract physiological monitoring parameters. One or more motion detection devices 115 may also be included to detect movement, changes in movement, and The device may be configured to detect and / or measure changes in motion, changes in inertia, and the like. The sensing device 115 may include an accelerometer, a gyroscope, etc. Analysis of running parameters can be used to improve running performance, improve physical therapy and prevent injuries. The results of the data analysis can be quantitatively analyzed on a mobile device. The information may be presented in the form of statistical data and charts.

[0014] In at least one embodiment, the system 100 includes a physical stacking arrangement of the insoles. The insole may include a stack-up topology, which includes a top insole layer, an insert, one or more force sensing elements 105, one or more strain sensing elements 105, one or more rings a boundary sensing element, one or more motion sensing elements, and a bottom insole. An interposer is an electrical interface between sockets or connection points. For example, the insert may include one or more force sensing elements 10. 5, one or more strain sensing elements 105, one or more environmental sensing elements 110, and and / or any of the one or more motion sensing elements 115 to a host controller. It can be connected to 120.

[0015] In at least one embodiment, the system 100 includes at least two separate sensors. The two separate sensing systems may include the physical placement of the toe and toe-sensing systems. Area / zone systems may include a heel area / zone system. For example, a toe The area / zone system is sized and tailored to fit the toe area / zone of the shoe. Similarly, the heel area / zone system may be configured to include a heel area / zone of the shoe. These two separate sensing systems can be sized and configured to fit The system uses the same two separate sensing systems to detect different shoes, sizes, or types, etc. Thus, the same two separate sensing systems can be used The system is used to provide accurate force / pressure mapping and motion measurements in different shoes. The two separate sensing systems may be communicatively and / or electrically connected to each other. The two separate sensing systems can communicate with the host controller 120. and / or electrically connected to

[0016] In at least one embodiment, the system 100 includes at least two sensing rays. Each sensing layer may include a dynamic indicator within the respective sensing layer. The console may include one or more force sensing elements that may provide force / pressure detection and measurement. Each sensing layer contains one or more strain sensing elements and one or more environmental sensors. It may also include sensing elements.

[0017] In at least one embodiment, the system 100 includes a plurality of force sensing elements 105. Each force sensing element 105 may include, but is not limited to, a force / pressure range. Individually tailored for optimal dynamic force / pressure characteristics, including range, rise time, fall time, etc. Each force sensing element 105 can be customized to a specific location within the shoe, A position on the insole and / or a position on the foot can be assigned. The tracking elements 105 are configured to be positioned based on their location in the shoe, on the insole, or on the foot. The system 100 can be individually customized to measure dynamic force / pressure characteristics. may also include a plurality of strain sensing elements 105, which are used to measure dynamic insole bending / The system 100 may also include an environmental sensing element 110, which may provide bending detection. They can provide dynamic environmental parameter measurements.

[0018] Similarly, in at least one embodiment, the system 100 includes multiple motion sensing elements. Each motion sensing element 115 may include a motion sensing element 115 for detecting optimal dynamic motion characteristics. Each motion-sensing element can be individually customized to detect the motion of the foot in the shoe. A specific location, a location on the insole, and / or a location on the foot may be assigned. The motion sensing elements may be located at different positions within the shoe, on the insole, or on the foot. It can be individually customized to measure position-based motion characteristics.

[0019] In at least one embodiment, the host controller 120 controls a multi-modal HMI In at least one embodiment, the host controller 120 may include and configured to perform computational processing of dynamic force detection and measurement data from the force sensing element. The processor may include, for example, a processor that is configured to recognize a portion of the insole surface or Dynamic foot force / pressure maps are obtained across or along the bottom of the user's foot. The processor may use the dynamic strain detection and strain measurement data. The processor may also be configured to perform computational processing of the measurement data from the output elements. The processor may further include a processor for detecting the strain of the foot and measuring the strain of the foot using the measured strain data. and performing computational processing of dynamic environmental sensing data received from one or more environmental sensing elements. The processor may also be configured to use the environmental sensing data to Dynamic environmental compensation of the sensing elements and strain sensing elements may also be achieved. performing computational processing of the dynamic motion sensing data received from the motion sensing elements above; The processor may also be configured to use the motion sensing data to perform force sensing. to achieve dynamic motion compensation of the sensor element, the strain sensing element and / or the environmental sensing element. The host controller may also include an embedded host controller. The controller may include circuitry configured to receive data from the sensing element. The host controller contains a memory for storing data and a processor for performing operations. The device may include a processor for performing the above steps.

[0020] The host controller is electronically connected to the client devices via a communication link 125. In at least one embodiment, the sensor may be connected to the client device via wired communication. The communication link may be any communication between the system and any other device. Any form of wired or wireless communication capability may be provided. In some embodiments, the communication link may be wireless. By way of example and not limitation, the communications link may include a wireless Through the mechanism, LAN connection possibility, Bluetooth connection possibility, Bluetooth slot -Energy (BLE), Wi-Fi connectivity, NFC connectivity, M2M connectivity, D 2D connectivity, GSM connectivity, 3G connectivity, 4G connectivity, LTE connectivity The present invention may be configured to provide a communications function, any other suitable communications functionality, or any suitable combination thereof. The insole may include any number of communication links. Android / iOS controller and display module, Ride engine visualization, various modes (e.g. walking and running modes), 15 Various interfaces such as Bluetooth Low Energy Interface etc. It can provide 5 functions.

[0021] In at least one embodiment, the host controller 120 (e.g., a processor) Sensing elements (e.g., force sensing element 105, strain sensing element 105, environmental The processor may scan the sensing elements 110, the motion sensing elements 115, The sensing element may be periodically scanned. uses different scan rates for at least some of the sensing elements. This may provide optimal data resolution and / or power consumption benefits. For example, some areas of the user's foot may move more frequently, i.e., less than other areas of the user's foot. These areas may experience a greater rate of force or pressure change compared to the For data resolution, scans can be made more frequently. The lower regions can be scanned more sparsely in time, which reduces the power consumption of the system 100. Consumption may be reduced.

[0022] The processor receives the signal from the sensing element (and from any other sensor as described herein). The processor may perform various analyses based on the received data. For example, the processor may , a force and / or pressure map of the human foot may be generated. The force and / or pressure map may be an instantaneous snapshot of the current state of the foot. Data over time may also be included, and the map may represent averages, medians, or other values. The map can be used to determine the level of pronation (e.g., overpronation, underpronation, supination) The map can be viewed as a "heat map" which may show ranges of force or pressure in different colors. In at least one embodiment, the processor may further include a processor for processing the sensor data. The processor may transmit the information to another device (e.g., a server, a client device). on smartphones or other portable or wearable devices such as smart watches Sensor data may also be sent.

[0023] The system 100 may include a power management system that may provide and / or regulate power for the system 100. It may also include a device 130.

[0024] The system 100 includes a haptic feedback control unit that can drive haptic feedback for the system. For example, the host controller 120 may include a sensing array Receives sensor data from a sensor (e.g., any of sensors 105, 110, 115) Based on the sensor data, the host controller 120 may The haptic response (e.g., the haptic feedback that the user can feel on his / her feet via the insole) haptic feedback unit 135. The tactile response can be felt by the user through the insole and through the haptic device. Exemplary tactile responses include, but are not limited to, pressing, flicking, It may include an impact, a release, which may be short, long, or repeated. Posing can be used to encourage certain behaviors. For example, if a runner puts their heels If guarding, the haptic response notifies or alerts the user when the user is guarding the heel. The user can then adjust their running style to be more on their toes and away from their heels. It can be adjusted so that

[0025] The systems and methods described herein include shoes, insoles, smart sensing mats, Such as flooring, recreational equipment, or other gym or exercise related applications. It can be used in many applications.

[0026] FIG. 2 illustrates an exemplary smart device for sensing various characteristics of a human foot and / or shoe. 2 shows the configuration of a smart shoe system 200. The exemplary smart shoe system is the system shown in FIG. For example, the smart shoe system 200 may include some or all of the elements of the one or more force sensing elements 105, one or more strain sensing elements 105, one or more The sensor may include an environmental sensing element 110, one or more motion sensing elements 115, may be collectively referred to as FSR sensing array 205. FSR sensing array 205 includes , may be scalable and / or adjustable according to shoe size. For example, the first FSR The sensing array 205 may be adjustable for shoe sizes 6 to 9. The sensing array 205 may include any number of sensors. In one example, FSR sensing The array 205 may include 20 or more sensor nodes.

[0027] The host controller 120 may measure approximately 5 cm x 5 cm x 1 cm. The roller 120 generates a pressure map based on the readings from the FSR sensing array 205. The host controller 120 may include logic for determining the timing and / or motion control. The host controller 120 may be coupled to a communication link 125. It can also be combined with other controllers such as the NI 6111 controller and display module. This can be configured for game engine visualization. , the motion and movement detected by the FSR sensing array 205 ) can be translated into movement of an avatar of the user or a digital character.

[0028] FIG. 3 illustrates an exemplary embodiment for sensing various characteristics of a human foot and / or shoe. The left side of FIG. 3 may show a first view 305 of the system, where The sensing element may be closer to the object to be measured (e.g., the foot). A second view 350 of the stem may be shown, which includes a circuit board 355, a power source (e.g., a battery) 3 60 etc. can be coupled to the system.

[0029] The system 300 measures the forces, strains, movements, motions, and The system may include a sensing element 310 that may be configured to measure other environmental characteristics. 00 may include an insole that may be removably inserted into the shoe. The insole may be incorporated into or attached to the shoe. The insole includes a top insole layer 320 and an interposer 325. , one or more sensing elements 310 coupled to an interposer 325, and a bottom-in It may include a sole layer 330 .

[0030] The top insole layer 320 may be attached to a shoe, boot, sandal, or any other type of The top insole layer 320 may be shaped to fit within the footwear of the wearer. The bottom insole layer 330 may be formed from any of the following materials or combinations of materials: The material may be formed from any material or combination of materials. Materials include porous materials, foam materials, plaster materials, and the like. The top insole layer may be made of a plastic material, or any other natural or synthetic material. 320 may be constructed from a different material or materials than the bottom insole layer 330 . The bottom insole layer 330 may be formed from a harder material. The overall stiffness of the ear can be stiffer than the top insole layer 320. In one embodiment, the bottom insole layer 330 includes an interposer 325 and / or may provide a hard / stable base for the sensing element 310. Layer 320 may be attached to interposer 325 (e.g., by adhesive). , welding, sewing, etc.) to the interposer 325. In this embodiment, the bottom insole layer 330 is adapted to withstand repeated contact with a hard surface (e.g., concrete). The at least one may be formed from a resilient material configured to withstand impacts. In an embodiment, the bottom insole layer 330 comprises or is part of the sole of the shoe. It can be a part.

[0031] Additionally or alternatively, the interposer 325 may be configured to receive one of the external host controllers. The interposer may be connected to an external circuit board (not shown in FIG. 3) as a part of the The circuit board may include or be part of any type of circuit board. The circuit board may be made of any material. The circuit board may be rigid or flexible.

[0032] One or more sensing elements 210 may be coupled to the interposer. The sensing elements 210 may be referred to as a sensing array. The sensor comprises one or more force sensing elements, one or more strain sensing elements, one or more motion sensing elements, One or more strain sensing elements, and / or one or more environmental sensing elements. The sensing element may include one or more two-dimensional strain sensing elements. The sensing elements may be spatially distributed on the interposer. Alternatively, one or more of the sensing elements may be separate components coupled to the poser. Directly formed, etched, deposited, printed, etc. on the interposer For example, the sensing element may be printed on a flexible circuit board (i.e., flex). It can be printed.

[0033] As shown on the left side 305 of FIG. 3, the exemplary system includes two sensing arrays 3 40, 345 (e.g. top and bottom, toe area and heel area, etc.) Each of the sensing arrays 340 and 345 detects the motion in the respective sensing layer. One or more force sensing elements 31 that can provide effective insole force / pressure detection and measurement. Each of the sensing arrays 340, 345 may include one or more strain sensors. As discussed above, each sensor may include a sensing element and one or more environmental sensing elements. The singular array includes a toe area / zone array and a heel area / zone array. It can be seen.

[0034] The system 300 may also include a controller, as further described in conjunction with FIG. The controller includes circuitry configured to receive data from the sensing element 310. The controller includes a memory for storing data and a processor for executing operations. The controller may include a communications link.

[0035] The controller receives signal from the sensing element (and from other sensors, as described herein). The controller may perform various analyses based on the data received. The force and / or pressure map may be generated for the human foot. The step can be an instantaneous snapshot of the current state of the human foot. The pressure map may also include data over time, and the map may include average, median, , or other values. The map may represent pronation (e.g., overpronation, underpronation, supination, etc.). The map can be used to determine the level of force or pressure. In at least one embodiment, the The controller transmits the sensor data to other devices (e.g., servers, clients) for processing. The controller can send the sensor data to a smartphone or smart device. It may also be sent to other portable or wearable devices such as watches.

[0036] The insole may include any number of sensors. The sensors may be located in or on the insole. It is possible to detect any characteristic (such as data representing motion or the environment) in the vicinity of a cell. Any hardware or software sensor may be represented, including but not limited to: However, accelerometer, gyroscope, altimeter, Global Positioning System (GPS), and step counter meter, magnetometer, thermometer, hygrometer sensor, atmospheric pressure sensor, GPS receiver, motion, environment, or human The present invention includes any other sensor capable of detecting a condition between the sensor and the power source, or any combination thereof. Any motion detected by the sensor may be called a motion signature. The sensor may detect a specific motion of a human. The sensors may detect a variety of movement patterns that may be linked to one or more of the following: Any suitable system, apparatus, device, or routine capable of detecting or determining That is, tilt, shake, rotation, rocking, and any other motion. For example, The sensor detects the movement of the insole as it moves (e.g., walking, running). It may be possible to detect periodic movements in a circular manner that represent the individual being tracked. In an embodiment, the sensor is configured to detect or determine the location of a particular tracked individual. For example, the sensor may be a GPS receiver, a Wi-Fi signal detector, a cellular network signal detector, Bluetooth beacon detector, Internet Protocol (IP) A tracking address detector, or any other system capable of detecting or determining the location of a particular tracked individual. The location may include one or more labels or In some embodiments, the sensors may include names (e.g., home, work, gym). The sensor may be an integrated sensor that includes two or more different sensors integrated together. For example, the sensor may include: An integration combining a three-dimensional (3D) accelerometer, a 3D gyroscope, and a 3D magnetometer It may be a chemical sensor.

[0037] The insole may also include any number of activity trackers. The lacquer may be used in, but is not limited to, heart rate monitors, blood pressure monitors, thermometers, moisture sensors, etc. The insole includes sensors such as a breathing sensor, a skin electrodermal activity sensor, and a sleep sensor. Any information that can be used to detect characteristics (or data representative of characteristics) of a tracked individual. Activity Tracing may represent hardware or software sensors or devices. The searcher is used to identify characteristics of tracked individuals who use the insoles. In some embodiments, the heart rate monitor measures or determines a heart rate or an indicator of a heart rate. For example, a heart rate monitor may be configured to monitor the pulse of an individual being tracked. One or more sensors (e.g., photoresistors or The sensor may include a photodiode or the like.

[0038] In these or other embodiments, the activity tracker may include a heart rate monitor. and one or more systems, devices, or systems configured to determine a heart rate based on the detected indicators. In some embodiments, a particular tracked individual may include a device or module. The events in a person's life will be recorded based on the heart rate of that particular tracked individual, Includes heart rate sustained by a particular tracked individual during the workout, heart rate recovery time, etc. These may be obtained from one or more heart rate monitors or other activity trackers or sensors. The host controller (or an external computer) calculates the The time can be determined by the

[0039] The insole may include any number of insoles that may provide any type of tactile feedback to the user. A haptic feedback device may also be included.

[0040] The insole may include more or fewer features. For example, the insole may include: The top insole layer may not be included, and the sensing elements may each The outward facing surface of the element is substantially flush with the surface facing the interposer feet. In such an embodiment, the interposer may be disposed so as to be on a surface. The interposer may be formed from a material that may provide a degree of comfort to the foot. For example, it may be formed from a pliable material. The sensor may include one or more recesses in which the sensing element may be mounted.

[0041] As mentioned above, in at least one embodiment, the left side 305 of FIG. 1 illustrates an example system including a first sensing array. The first sensing array is a first part of a shoe. Shaped to fit in the area of ​​the A second portion of the shoe (e.g., the portion near where the user places their heel) may be made The first sensing array and the second sensing array may be shaped to fit the Each of the rays may include any number of sensing elements. As shown, the first The sensing array includes a strain sensing element, a plurality of force sensing elements, and a plurality of environmental sensing elements. As shown, the second sensing array includes a plurality of force sensing elements. The first sensing array and the second sensing array include a plurality of environmental sensing elements. The ray can independently communicate and / or electronically communicate with the host via a connector interface. At least one of the at least one In an embodiment, the first sensing array and the second sensing array are framed with a connector. through the respective connector interface, as through a flexible printed circuit, In such an embodiment, the first sensor Either the sensing array or the second sensing array is connected to the connector interface. It may be communicatively and / or electronically coupled to a host controller.

[0042] The configuration of the two sensing arrays allows for reconfiguration of the length or size of the shoe. Additionally, the force sensing element can detect higher levels of force / pressure applied by the foot. The strain sensing element (e.g., a two-dimensional strain sensing element) may be located in the region of the foot. located in an area where higher levels of bending / flexing can be applied by The environmental sensing element may be designed to be sensitive to minimal levels of force / pressure or bending / deflection. However, the influence of environmental parameters equivalent to the force sensing element and the two-dimensional strain sensing element is The device may be located in an area that is susceptible to

[0043] In one example, the system performs physical exercises with the goal of allowing the user to walk correctly after treatment. In another example, the system may be used for dynamic motion monitoring in therapy. Users can monitor their running performance based on dynamic motion monitoring both in real-time and over time. For dynamic motion monitoring in sports (e.g. athletics) that can improve performance In one embodiment, a haptic device that can provide haptic feedback may be used to , are built into the system to stimulate the right behavior.

[0044] FIG. 4 illustrates a cross-sectional view of the foot during, but not limited to, a walking or running motion. Determine various functions including foot force / pressure mapping and biomechanics such as flexion characteristics of the foot 4 shows an example computational process flow of a method 400 for detecting the force of a force sensing element. , dimensional strain sensing elements, motion sensing elements, and environmental sensing elements, or other It may be obtained by analysis of measurement data from a sensor or activity tracker. The process flow can be used to self-trigger the smart shoes. Hardware (circuits, dedicated logic, etc.), software (general-purpose computer systems or dedicated by processing logic which may include a The processing logic may be implemented in the system of FIG. 1 or in another computer system. However, other systems or combinations of systems may be included. For ease of explanation, the method described herein may be Although depicted and described as a series of acts, acts in accordance with this disclosure may be performed in various orders. may occur simultaneously with and / or in conjunction with other actions not presented and described herein. Moreover, in order to implement the method according to the disclosed subject matter, all of the illustrated In addition, those skilled in the art will appreciate that the method is interrelated via a state diagram. It is understood and appreciated that the present invention may alternatively be represented as a series of related states or events. Additionally, the methods disclosed herein may be implemented in a computing device. In order to facilitate the transfer and disclosure of such information, Such information may be stored on an article of manufacture. As used herein, the term article of manufacture means Any computer-readable device or storage medium accessible by the computer Although shown as separate blocks, Various blocks may be divided into additional blocks or into fewer blocks depending on the desired implementation. They may be combined into blocks or omitted.

[0045] The method 400 may begin at block 405 where processing logic may boot the controller. Processing logic may enter a sleep mode at block 410, which may include a 1 Hz sub-mode. At block 415, processing logic may include sampling one or more sensing elements. identifying data from the shoe, and the identified data indicating that the user is wearing the shoe. By determining whether the user is wearing the shoes or not, the profile that indicates If the user is not wearing shoes (block 415), "No"), processing logic continues to wait for a predetermined amount of time in sleep mode at block 410. can be entered.

[0046] If the user is wearing shoes (block 415: "yes"), the processing logic The block may enter activation mode at block 420, which is the 50 Hz sampling The processing logic may include processing one or more sensing elements as described herein. For example, the processing logic may obtain data from a user's foot force while the user is moving. Data and motion data may be acquired.

[0047] At block 430, processing logic identifies a user profile associated with the user. In at least one embodiment, at block 430, processing logic may To identify your profile, you may use other devices (e.g. mobile devices, wearables, smartphones, tablets, etc.) To do so, the processing logic may Controller (e.g. Android controller, iOS controller) may initiate a handshake with a user (e.g., a The processing logic may use the data obtained from the one or more sensing elements to A visualization of the data (e.g., via a graphical user interface) may be generated.

[0048] At block 435, processing logic processes data from one or more sensing elements. and the identified data indicates that the user is wearing the shoes. By determining if the data profile matches, the system detects whether the user is still wearing the shoes. If the user has not yet put on the shoes (block 43), 5 "no", processing logic enters sleep mode for a predetermined amount of time at block 410. If the user still has the shoes on (block 435: "Yes" "), processing logic initiates or initiates a handshake with the remote device at block 430. can be checked.

[0049] FIG. 5 illustrates a motion of, but not limited to, walking, running, rolling, or jumping. Various tests were performed including force / pressure mapping and biomechanics of the foot such as flex characteristics at different times during the 5 illustrates an exemplary computational process flow 500 for determining the functions of the sensing sensing elements, dimensional strain sensing elements, motion sensing elements, and environmental sensing elements, may be obtained by analysis of measurement data from other sensors or activity trackers. This approach to multi-sensor fusion allows the system to To be able to detect specific signatures for multiple types of human movement to.

[0050] As shown, force / pressure measurement data, strain measurement data, inertial (e.g., motion) measurement Data and / or environmental measurement data are used for multimodal parameter extraction, user calibration and This can be used for temperature control, setup, and environmental compensation. All sensors have some form of temperature dependence. However, the temperature dependence of the environmental measurement data is compensated for in different environmental / weather conditions. Different users have different weights and different movement signatures. User calibration enables the system to correctly detect the motion profile. These data and modeling are used for user / security identification, pressure mapping display, system User motion animation display (e.g. virtual reality, augmented reality), and / or user motion animation It can be used for displaying animation.

[0051] FIG. 6 shows a smart shoe that includes one or more sensing elements (e.g., pressure measurement nodes). 6 illustrates an example implementation of a smart shoe system 600. In one embodiment, the smart shoe includes the insole of FIG. The embedded controller in the mobile device detects one or more sensing requirements. The data received from the source may be used to generate a pressure mapping display, as shown in FIG. Thus, the pressure mapping display 600 may be displayed on a user interface of a mobile device 610. The pressure mapping display 600 is presented in a color-coded manner to indicate various levels of pressure. It may include a coded "heat map."

[0052] The controller embedded on the smart shoe or in the mobile device may include one or more A motion animation display 620 is also generated using data received from the above sensing elements. The movement animation display 620 can be achieved in the same way as a person wearing smart shoes. In at least one embodiment, the person's movements may be recorded as a motion avatar. This is effectively reflected in the avatar in the animation display 620. For example, if the user leans to the left, At that time, the insole can detect the leftward tilt and send a corresponding signal to animate the movement. A motion animation display 620 may be generated. In this example, the motion animation display 620 may include: Includes an avatar that leans left in a similar manner to a user wearing this insole(s). become.

[0053] Figure 7 shows the force distribution based on measurement data from the force sensing elements of any of the systems described. Graphical user interface for monitoring parameters such as foot / pressure mapping 7 illustrates an example of a GUI 700.

[0054] In at least one embodiment, the GUI 700 may include a pressure mapping display 705. 7, which may include an outline of one or more feet 710. The sensor may also include a graphical representation of an element of the sensor (hereinafter, a graphical sensor 715). A measured value (e.g., a force value) can be represented at each sensing element. As shown, each graphical sensor is shaped according to its respective force value. The shading may be colored, patterned, etc. As shown, each graphical sensor includes a respective force measurement. Measurements can be absolute (e.g. in units of Newtons or kg m / s2) or relative .

[0055] The GUI 700 is a physical orientation of a smart shoe currently being worn by a person. The GUI may also include other portions that may illustrate the motion animations described above. 620 may also be provided.

[0056] FIG. 8 illustrates a method for detecting a small amount of a signal from measurements obtained from one or more sensing elements as described herein. FIG. 8 illustrates various motion animation displays 800 that may be generated based, at least in part, on the Show.

[0057] FIG. 9 illustrates various features including, but not limited to, user profile identification and visualization. 9 illustrates an exemplary computational process flow 900 for determining the various functions. These functions are Sensing element, dimensional strain sensing element, motion sensing element, environmental sensing element , body sensing elements, or measurement data from other sensors or activity trackers This approach uses a wearable sensor on the body or arm. The inclusion of the 10-GHz bandpass filter enables extended multi-sensor fusion.

[0058] As shown, force / pressure measurement data, strain measurement data, inertial (e.g., motion) measurement The data, environmental measurement data, and / or body sensing data are multimodal parameters. These data can be used for data processing, user calibration and configuration, and environmental compensation. Data and modeling are used for user / security identification, pressure mapping display, shoe movement accelerometer, Animation display (e.g. for virtual reality, augmented reality) and / or user movement animation The extended multi-sensor fusion can be used for augmented reality (AR) display. It also enables more detailed animation of user movements in the

[0059] FIG. 10 illustrates a method for monitoring, including but not limited to, fitness and / or physiological monitoring. Exemplary computational processes for determining various functions, including the application of smart shoes in The process flow 1000 is illustrated. These functions include a force sensing element, a motion sensing element, Environmental sensing elements, body sensing elements, or other sensors or activity trackers This can be obtained by analysis of measurement data from

[0060] As shown, user / security identification, pressure mapping display, shoe movement alarm The animation display and / or user movement animation display may be used to display multimodal data. used for analysis, comparison with reference physiological parameters, and / or analysis of physiological conditions. Additionally, these data and modeling may be used to display physiological conditions, Resume reminders and / or warnings, goal tracking for exercise equipment, and / or remote It can be used for warning purposes.

[0061] For example, Figure 10 shows a user wearing smart shoes running on a treadmill. The functionality of the example application may be illustrated as follows: You can set the parameters of the running machine and adjust the parameters during the exercise. This is a highly manual process. In this example, the smart shoe interface The sensor is connected wirelessly to the running machine. Analyze physiological parameters, compare with reference physiological parameters, analyze physiological status, If the running machine parameters are excessive (speed is too fast, If the slope is too steep, etc., provide stress reminders / warnings to the user. Running machine performance when the user is overstressed and unable to perform manual adjustments. Automatic adjustment of parameters, if the user is injured (jerky movements, no movements), etc. The system provides several functions, such as providing remote alarm activation. It may be used in combination with other body sensors such as a heart rate monitor, blood pressure monitor, etc.

[0062] During a typical treadmill stress test, the patient is monitored using chest and arm sensors. The ECG and blood pressure measurements can be monitored, respectively. There are some inherent limitations to the use of pediatric pediatric cardiology. The movement of the patient's legs or feet is typically monitored. Therefore, the walking / running stability of the patient is not monitored. These include, but are not limited to, whether the patient's stride is consistent with the speed of the treadmill. If the patient does not match the running machine, the position of the patient on the running machine may be abnormal, or the running The patient increases their walking / running speed while the speed of the training machine increases. If the patient is unable to move faster, this could be due to physical stress or fatigue of the patient, or the medical staff accidentally putting the patient on the running machine. During some scenarios, including maneuvers involving heavy lifting, the patient may be at risk of injury due to effort or a falling patient. There are inherent safety risks.

[0063] If there was a smart shoe interface that could wirelessly connect to the running machine, Safety risks are greatly reduced by monitoring the physiological and movement status of the patient. The width can be reduced.

[0064] Figure 11-21 shows the real-time distribution of monitoring parameters for six defined motion states. These are examples of real-time measurements. Walking forward (Figure 11) Walking backwards (Figure 12) -Transition from walking to running (Figure 13) Jumping (Figure 14) Rotating (Figure 15) It is.

[0065] Three different types of sensors can be present in each shoe insole. They are, Force / pressure sensing array (11 sensing nodes, e.g. as in Figure 16) 3-axis accelerometer (illustrated in Figures 17 and 18) 3-axis gyroscope (illustrated in Figures 19 and 21) It is.

[0066] "Simplified" identifiers for the motion states are shown in each of Figures 11-21.

[0067] FIG. 22 is a schematic diagram of a multimodal array in accordance with at least one embodiment of the present disclosure. 22 shows a block diagram of an example computer system 2200 for implementing the host controller described above. The roller may be implemented as a computing system, such as exemplary computer system 2200. The computer system 2200 may be configured to perform one or more operations of the present disclosure. It can be done.

[0068] The computer system 2200 may be adapted to perform any one or more of the methods described herein. 2226, which executes one or more sets of instructions for the machine to execute. As a server or client machine in a to-server network environment, or It can act as a peer machine in a peer-to-peer (i.e., distributed) network environment. Shin is a leading provider of personal computers (PCs), tablet PCs, set-top boxes (S TB), personal digital assistants (PDAs), mobile phones, web devices, servers, Network routers, switches or bridges, or any actions taken by the machine Any program capable of executing a set of instructions (sequential or otherwise) that specify an operation. Further, although only a single machine is illustrated, the term "machine" is used. The term refers to a set of instructions 2226 that performs any one or more of the methods described herein. "Computer" shall also be construed to include any collection of machines operating independently or in concert.

[0069] The computer system 2200 includes a processor 2202 and a main memory 2204 (e.g., , read-only memory (ROM), flash memory, synchronous DRAM (SDRAM ) or dynamic random access memory (DRRAM) such as Rambus DRAM (RDRAM) AM), and a static memory 2206 (e.g., flash memory, static A static random access memory (SRAM) and a data storage device 2216 are included. communicate with each other via bus 2208.

[0070] The processor 2202 may be one or more general-purpose processors such as a microprocessor, a central processing unit, or the like. More specifically, the processor 2202 represents a complex instruction set computer. CISC microprocessor, Reduced Instruction Set Computing (RI SC microprocessor, Very Long Instruction Word (VLIW) microprocessor, or other instruction A processor that implements a set of instructions or a combination of sets of instructions. The processor 2202 may be an application specific integrated circuit (ASIC), a field programmable FPGAs, digital signal processors (DSPs), network processors The processor 2202 may be one or more dedicated processing units, such as a processor 2202. It is configured to execute instructions to perform the operations and steps described.

[0071] The computer system 2200 may be used in a local area network (LAN), an intranet A network 2218, such as a net or the Internet, that allows communication with other machines. The network interface device 2222 may further include a network interface device 2222 for providing a network interface. The interface device 2222 may include any number of physical or logical interfaces. The network interface device 2222 is a device for connecting network elements in a network. Any device, system, component (including, but not limited to, any other device, system, component or components) configured to enable or facilitate communication between It may also include a component, or a collection of components. For example, a network interface The device 2222 may be a modem, a network card (wireless or wired), an infrared communication device, an optical communication device, or the like. receiving devices, wireless communication devices (such as antennas), and / or chipsets (such as Bluetooth devices, 802.xx devices (such as metropolitan area networks (MANs)), WiFi devices, This may include, without limitation, WiMax equipment, cellular equipment, and / or others. The interface device 2222 may be connected to a network (a mobile phone network, to name a few). network, WiFi network, MAN, optical network, etc.) and / or remote device allowing data to be exchanged with any of the other devices described in this disclosure, including In at least one embodiment, the network interface device 2222 , logically distinct on a single physical component, e.g., a single physical cable or optical fiber There may be multiple communication streams passing through the signal.

[0072] The computer system 2200 includes a display device 2210 (e.g., a liquid crystal display (LCD) an alphanumeric input device 2212 (e.g., a keyboard); Solve control device 2214 (e.g., a mouse), and signal generating device 2220 (e.g., a speaker). ) may also be included.

[0073] The data storage device 2216 may include any one or more of the methods or functions described herein. A computer readable storage medium 2224 having stored thereon a set of instructions 2226 for embodying The set of instructions 2226 may also be stored in the main memory 2204 and / or in the computer. During execution by the computer system 2200, the The main memory 2204 and the processor 2202 also include computer-readable media. The set of instructions 2226 further comprises a network interface device 222 2 via network 2218.

[0074] Although the example of computer readable storage medium 2224 is shown as a single medium, The term "computer-readable storage medium" refers to a single one or more media (e.g., centralized or distributed databases and / or associated The term "computer-readable storage medium" may include a storage medium that is a , capable of storing, encoding, and carrying a set of instructions for execution by a machine The present disclosure may include any medium that causes a machine to perform any one or more of the methods of the present disclosure. The term "computer-readable storage medium" includes solid-state memory, optical This may include, but is not limited to, media and magnetic media.

[0075] Modifications, additions, or improvements may be made to computer system 2200 without departing from the scope of the present disclosure. Omissions may be made. For example, in at least one embodiment, a computer system 2200 may include any number of other components that may not be explicitly shown or described. It may also include.

[0076] As used in this disclosure, the term "module" or "component" refers to a module. specific hardware implementations configured to perform the operations of a module or component; and / or The general-purpose hardware of a computer system (e.g., a computer-readable medium) software objects that can be stored in and / or executed by a computer, a processor, etc. In at least one embodiment, the present invention may refer to a program or a software routine. The various components, modules, engines, and services described in the disclosure include: As an object or process running on a computer system (e.g., independently) Some of the systems and methods described in this disclosure may be implemented in software. Implemented in software (stored in and / or on general purpose hardware) Although generally described as a dedicated hardware implementation, or software implementation, A combination of software and dedicated hardware implementations is also possible and contemplated. A "computing entity" may be any computing system as defined above in this disclosure. Any module or combination of modules operating on a computer system It is also possible.

[0077] As used in this disclosure and particularly in the appended claims (e.g., the body of the appended claims), The terms used are "open" terms (e.g., the term "contains" means "contains"). and the term "having" can be interpreted as "including, but not limited to," and the term "having" can be interpreted as "including at least and the term "including" may be interpreted as "including, but not limited to." "may be interpreted as "a good idea" or "a good idea").

[0078] Moreover, if a specific number of recitations in an introduced claim is intended, such intention The claims are clearly set forth in the claims, and in the absence of such a statement no such intention exists. For example, as an aid to understanding, the following appended claims are provided to guide the recitation of the claims: The invention may include the use of the introductory phrases "at least one" and "one or more" to include However, use of such phrases is not intended to be construed as limiting the scope of the invention, even if the same claim may refer to "one or more" or "at least one " and an indefinite article such as "a" or "an." The introduction of a claim recitation with the indefinite article "an" is considered to be a Limit any particular claim containing a recitation to an embodiment containing only one such recitation. (e.g., "a" and / or "an" should not be construed to imply "at least "one or more"). The same is true for the use of the definite articles used.

[0079] In addition, even if a specific number of recitations in the introduced claims are explicitly recited, The reader will understand that such a description may be construed to mean at least the number recited. (For example, the unqualified statement "two statements" implies at least two (meaning one or more of A, B, and C, etc.) In cases where a practice similar to "one or more of A, B and C, etc." is used, Such an interpretation would be A only, B only, C only, A and B, A and C, B and C, or A, B and C, etc. It is intended to include.

[0080] Furthermore, in any of the specification, claims, or drawings, two or more alternative uses may be used. Any disjunctive word or phrase expressing a term may be used to express one of the terms, It can be understood to anticipate the possibility of including either or both of the terms. For example, "A The phrase "or B" may be understood to include the possibilities of "A," "B," or "A and B." .

[0081] All examples and qualified terms described in this disclosure are provided to aid the reader in the advancement of the art. The present invention is intended for educational purposes to aid the reader in understanding the invention and concepts presented by the inventor. The present invention is illustrated by way of example and is not to be construed as being limited to such specifically described examples and conditions. Although the embodiments of the present disclosure have been described in detail, it is to be understood that the present disclosure is not to be construed as limiting the scope of the present disclosure. Various changes, substitutions, and alterations can be made without departing from the scope.

Claims

1. A smart insole system, A first bottom layer, a first sensing layer disposed on a first surface of the first bottom layer, the first sensing layer comprises a first sensing element and a second sensing element; The first sensing element and the second sensing element are a force sensing element, a strain sensing element, one of a sensing element, a motion sensing element, or an environmental sensing element; The first sensing element and the second sensing element are different types of sensing elements. It is made up of elements, A first sensing layer, and a communication interface configured to couple the first sensing layer to a host controller; Face A smart insole system equipped with

2. A bonding layer is bonded to the sensing layer on a surface of the sensing layer opposite to the bottom layer. a top layer formed on the outer surface of the The top layer comprises a first material and the bottom layer comprises a second material. The smart insole system according to claim 1 .

3. The first material is more flexible than the second material. The smart insole system according to claim 2.

4. a first insole portion including the first bottom layer and the first sensing layer; and A second insole portion including a second bottom layer and a second sensing layer. Further equipped with The second sensing layer is coupled to the host controller. The smart insole system according to claim 1 .

5. A connector configured to physically connect the first insole portion and the second insole portion. Nectar Further equipped The smart insole system according to claim 4.

6. The first insole portion and the second insole portion are configured to fit within a shoe. R The smart insole system according to claim 5.

7. When the smart insole system is placed in a shoe, it fits within the shoe. The lengths of the first insole part and the second insole part are adjustable so that The connector is adjustable between at least two shoe sizes. The smart insole system according to claim 6.

8. The first sensing layer and the second sensing layer are generally at least two Contains 0 sensors The smart insole system according to claim 1 .

9. Providing a haptic sensation based on the data generated by the first sensing layer. A haptic feedback unit configured to Further equipped The smart insole system according to claim 1 .

10. A processor receives data from a sensing element coupled to the wearable foot device. receiving The wearable foot device comprises: A first bottom layer, a first sensing layer disposed on a first surface of the first bottom layer, The first sensing layer includes a first sensing element and a second sensing element. 、 The first sensing element and the second sensing element are a force sensing element, a strain sensing element, one of a light sensing element, a motion sensing element, or an environmental sensing element; The first sensing element and the second sensing element are different types of sensing elements. It is made up of elements A first sensing layer, and a communication interface configured to couple the first sensing layer to a host controller; Taface receiving data; calculating one or more motion data parameters based on said data; determining a live graphical display based on said motion data parameters; and providing said live graphical display via a graphical user interface; and A method for providing the above.

11. The live graphical display is configured to move based on the motion data parameters. Includes the avatar The method of claim 10.

12. The live graphical display indicates whether the user is wearing the wearable foot device. Includes indicators of status, Calculating one or more motion data parameters based on the data includes: determining whether the wearable foot device is being worn by the wearer based on the one or more movement data parameters; including making judgments based on the Determining a live graphical display based on the motion data parameters includes: The appearance of the indicator to indicate that the user is wearing the wearable foot device. Including deciding to adjust The method of claim 11.

13. Calculating one or more motion data parameters based on the data includes: is walking, running, jumping, or spinning This includes determining that The method of claim 11.

14. providing said live graphical display via a graphical user interface; This means at least one of walking, running, jumping, or spinning. This includes presenting an avatar The method of claim 12.

15. The live graphical display includes a two-dimensional pressure map. The method of claim 11.

16. The live graphical display includes a three-dimensional pressure map. The method of claim 11.

17. Identifying environmental parameters that contribute noise to the data; and Adjusting the data to account for the environmental parameters to reduce the noise in the data. The method further comprises: The method of claim 11.

18. a bottom layer having a groove disposed on a first surface thereof; a first sensing layer disposed on a second surface of the bottom layer, the first sensing layer comprises a first sensing element and a second sensing element; The first sensing element and the second sensing element are a force sensing element, a strain sensing element, one of a sensing element, a motion sensing element, or an environmental sensing element; The first sensing element and the second sensing element are different types of sensing elements. It is made up of elements, A first sensing layer, and a communication interface configured to couple the first sensing layer to a host controller; Face A smart shoe system comprising:

19. a second sensing layer disposed on the second surface of the bottom layer. The smart shoe system according to claim 18.

20. A controller, and At least one of the first sensing layer and the second sensing layer a haptic feedback device configured to provide a haptic sensation based on the data generated by the Q Unit Further equipped The smart shoe system according to claim 19.

Images