Multimodal Sensor Fusion Platform

Abstract

To provide a smart insole system.SOLUTION: A system may include a sensing layer provided on a bottom layer and a first surface of the bottom layer. A first sensing layer may include a first sensing element and a second sensing element. The first sensing element and the second sensing element may include one of a force sensing element, a strain sensing element, a motion sensing element and an environment sensing element. The first sensing element and the second sensing element may be sensing elements of different types. The system may further include a communication interface configured to connect the sensing layer to a host controller.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The embodiments discussed herein are based on a multimodal sensor fusion platform. Regarding. [Background technology]

[0002] The Internet of Things (IoT) collects and processes data, often without user input. The electronic circuits, software, sensors, actuators, and networks that enable A network of physical devices, cars, buildings, and other things that have embedded network connectivity. Typically, IoT devices are also called smart devices. Some progress has been made in developing smart insoles for use in shoes. These smart insole solutions utilize force sensing elements to detect force and / or It relies primarily on pressure mapping. However, smart insoles may present various limitations.

[0003] The subject matter claimed herein solves the disadvantages or is intended to be used only in circumstances such as those described above. It is not intended to be limiting to working embodiments. Rather, this background is provided for at least the purposes described herein. It is provided merely to illustrate one example technology area in which one embodiment may be practiced. do not have. [Brief explanation of the drawings]

[0004] Example embodiments will be described and explained with additional specificity and detail through the use of the accompanying drawings. can be. [Figure 1] FIG. 1 shows an example system diagram of multimodal sensor fusion. [Figure 2]FIG. 2 illustrates an exemplary smart shoe system configuration for sensing various characteristics related to a human foot and / or shoe. [Figure 3] FIG. 3 illustrates an exemplary system configuration for sensing various characteristics related to a human foot and / or shoe. [Figure 4] FIG. 4 shows an exemplary computational process flow of a method for determining various functions including, but not limited to, foot force / pressure mapping and biomechanics, such as foot flexion characteristics, during walking or running movements, etc. [Figure 5] FIG. 5 shows an exemplary computational process flow for determining various functions including, but not limited to, foot force / pressure mapping and biomechanics, such as foot flexion characteristics, during walking, running, turning, or jumping movements, etc. [Figure 6] FIG. 6 shows an example implementation of a smart shoe system in which the smart shoe includes one or more sensing elements. [Figure 7] FIG. 7 shows an example of a graphical user interface (GUI) for parameter monitoring such as foot / pressure mapping based on measurement data from a force sensing element. [Figure 8] FIG. 8 illustrates various motion animation displays that may be generated based at least in part on measurements obtained from one or more sensing elements described herein. [Figure 9] FIG. 9 illustrates an exemplary computational process flow for determining various functions including, but not limited to, user profile identification and visualization. [Figure 10] FIG. 10 shows an example computational process flow for determining various functions including, but not limited to, applications of smart shoes in fitness and / or physiological monitoring. [Figure 11] FIG. 11 shows an example of real-time measurement of monitoring parameters for six defined motion states. [Figure 12]FIG. 12 shows an example of real-time measurement of monitoring parameters for six defined motion states. [Figure 13] FIG. 13 shows an example of real-time measurement of monitoring parameters for six defined motion states. [Figure 14] FIG. 14 shows an example of real-time measurement of monitoring parameters for six defined motion states. [Figure 15] FIG. 15 shows an example of real-time measurement of monitoring parameters for six defined motion states. [Figure 16] FIG. 16 shows an example of real-time measurement of monitoring parameters for six defined motion states. [Figure 17] FIG. 17 shows an example of real-time measurement of monitoring parameters for six defined motion states. [Figure 18] FIG. 18 shows an example of real-time measurement of monitoring parameters for six defined motion states. [Figure 19] FIG. 19 shows an example of real-time measurement of monitoring parameters for six defined motion states. [Figure 20] FIG. 20 shows an example of real-time measurement of monitoring parameters for six defined motion states. [Figure 21] FIG. 21 shows an example of real-time measurement of monitoring parameters for six defined motion states. [Figure 22] FIG. 22 shows a block diagram of an example computer system for a multimodal array in accordance with at least one embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0005] Smart insoles are often used for force and / or pressure mapping of a person's foot. Conventional smart insole solutions use smart insoles as a force sensing layer. Force and / or pressure measurements using force sensing elements, typically incorporated within the insole. Traditional smart insole solutions often rely on foot pressure mapping. For mapping, extraction of parameters including walking / running speed and pronation / supination In conventional technology, the force sensing layer is large and provides a concentrated physical sensing function. The size and shape must correspond to the size, type and character of the shoe, otherwise The force sensing element may not provide an accurate reading. The factors are temperature, which can change dramatically based on different user activities (like running). They are often sensitive to environmental conditions, including temperature and humidity.

[0006] Using only force sensing elements presents various limitations to smart insoles. For example, a person's foot in motion can generate many different forces, and these forces can be difficult to measure accurately in a consistent manner. It can be difficult to measure motion characteristics using only the elements. For example, external forces (e.g. During movement (due to the movement of the user's foot), the insole may experience bending and flexing. Conventional systems are unable to distinguish or separate these bending or flexing forces from other types of forces. Furthermore, conventional systems may not be able to measure physiological monitoring parameters. may not provide real-time physical user feedback based on May not provide intuitive real-time visualization of physiological monitoring parameters Furthermore, conventional systems do not provide physiological monitoring using user profile identification. and conventional systems may not allow tracking of sensors and It may not provide a duplex data communication link between host devices.

[0007] Aspects of the present disclosure include force / pressure mapping and motion measurement within smart insoles, System for pressure monitoring, shoe customization, and other related applications (shoe by providing a smart insole or system that can be placed inside the to address these and other shortcomings.

[0008] Embodiments of the present disclosure will be further described with reference to the accompanying drawings.

[0009] FIG. 1 illustrates an exemplary multimodal sensor fusion platform ("system"). The system 100 includes various sensors, haptic devices, controllers, and It may include a power management device, an interface, and the like.

[0010] The system 100 may include one or more strain sensing elements 105 and / or one or more force sensors. The one or more strain sensing elements 105 may be two-dimensional or The one or more force sensing elements may include a three-dimensional strain sensing element. / May include a pressure sensor.

[0011] In at least one embodiment, the system 100 includes one or more force sensing elements. The insole may include one or more force sensing elements 105. Detect and / or measure foot force and / or pressure distribution over part or all of the surface of the ball The insole may include one or more strain sensing elements 105. One or more strain sensing elements 105 detect and / or flex the insole. and / or may be configured to measure

[0012] The insole may include one or more environmental sensing elements 110. The sensing element 110 is adapted to sense and / or measure environmental parameters including temperature and humidity. At least some environmental parameters may contribute to noise in the system. Environmental parameters are used as a basis for explanation to reduce noise in the system. and / or can be mathematically reduced, minimized, or ignored. The integration of more than one type of sensing element together allows for robust force and pressure mapping and This may provide a solution for motion measurement.

[0013] The system 100 includes a motion sensor that can be used to extract physiological monitoring parameters. One or more motion detection devices 115 may also be included to detect movement, changes in movement, and The device may be configured to detect and / or measure changes in motion, changes in inertia, etc. The sensing device 115 may include an accelerometer, a gyroscope, etc. Analysis of running parameters can be used to improve running performance, improve physical therapy and prevent injuries. The results of the data analysis can be analyzed quantitatively on a mobile device. This can be presented in the form of statistical data and charts.

[0014] In at least one embodiment, the system 100 includes a physical stacking arrangement of the insoles. The insole may include a stack-up topology, which includes a top insole layer, an insert, and a one or more force sensing elements 105, one or more strain sensing elements 105, one or more rings a boundary sensing element 110, one or more motion sensing elements 115, and a bottom insole An interposer is an electrical interface between sockets or connection points. For example, the insert may include one or more force sensing elements 10. 5, one or more strain sensing elements 105, one or more environmental sensing elements 110, and and / or any of one or more of the motion sensing elements 115 to the host controller 120.

[0015] In at least one embodiment, the system 100 includes at least two separate sensors. The two separate sensing systems may include the physical placement of the toe Area / zone systems may include a heel area / zone system, e.g., a toe area / zone system. The area / zone system is sized and designed to fit the toe area / zone of the shoe. Similarly, the heel area / zone system may be configured to These two separate sensing systems can be sized and configured to fit within the The system uses the same two separate sensing systems to detect different shoes, sizes, or types, etc. Therefore, the same two separate sensing systems can be used The system is used to provide accurate force / pressure mapping and motion measurements in different shoes. The two separate sensing systems may be communicatively and / or electrically connected to each other. The two separate sensing systems can communicate with the host controller 120. and / or electrically connected to

[0016] In at least one embodiment, the system 100 includes at least two sensing rays. Each sensing layer may include a dynamic indicator within the sensing layer. The console may include one or more force sensing elements that may provide force / pressure detection and measurement. Each sensing layer contains one or more strain sensing elements and one or more environmental sensors. It may also include a sensing element.

[0017] In at least one embodiment, the system 100 includes a plurality of force sensing elements 105. Each force sensing element 105 may include, but is not limited to, a force / pressure range. Individually configured for optimal dynamic force / pressure characteristics, including range, rise time, fall time, etc. Each force sensing element 105 can be customized to a specific location within the shoe, A position on the insole and / or a position on the foot can be assigned. The tracking elements 105 are configured based on their location in the shoe, on the insole, or on the foot. The system 100 can be individually customized to measure dynamic force / pressure characteristics. may also include a plurality of strain sensing elements 105, which may be used to measure dynamic insole bending / The system 100 may also include an environmental sensing element 110, which may provide bending detection. They can provide dynamic environmental parameter measurements.

[0018] Similarly, in at least one embodiment, the system 100 includes multiple motion sensing elements. Each motion sensing element 115 may detect optimal dynamic motion characteristics. Each motion sensing element can be individually customized to detect the movement of the foot in the shoe. A specific location, a location on the insole, and / or a location on the foot may be assigned. The motion sensing elements may be located at their respective positions within the shoe, on the insole, or on the foot. It can be individually customized to measure position-based motion characteristics.

[0019] In at least one embodiment, the host controller 120 controls a multimodal HMI. In at least one embodiment, the host controller 120 may include configured to perform computational processing of dynamic force detection and measurement data from the force sensing element. The processor may include, for example, a processor configured to process a portion of the insole surface or Dynamic foot force / pressure maps are obtained across or along the bottom of the user's foot. The processor may use dynamic strain detection and measurement data. The processor may also be configured to perform computational processing of measurement data from the output elements. The processor may use the dynamic strain detection and measurement data to determine the flexion characteristics of the foot. and performing computational processing of dynamic environmental sensing data received from one or more environmental sensing elements. The processor may also be configured to use the environmental sensing data to Dynamic environmental compensation of the sensing and strain sensing elements may also be achieved. performing computational processing of dynamic motion sensing data received from the motion sensing elements above; The processor may also be configured to use the motion sensing data to perform force sensing. to achieve dynamic motion compensation of the sensor element, the strain sensing element and / or the environmental sensing element. The host controller may also include an embedded host controller. The remote controller may include circuitry configured to receive data from the sensing element. The host controller has a memory for storing data and a processor for performing operations. The device may include a processor for performing the following steps:

[0020] The host controller is electronically connected to the client devices via a communication link 125. In at least one embodiment, the sensor may be connected to the client device via wired communication. The communication link may be any communication link between the system and any other device. Any form of wired or wireless communication capability may be provided. In some embodiments, the communication link may be wireless. By way of example and not limitation, the communication link may include a radio frequency (RF) antenna. Through the mechanism, LAN connection possibility, Bluetooth connection possibility, Bluetooth slot - Energy (BLE), Wi-Fi connectivity, NFC connectivity, M2M connectivity, D 2D connectivity, GSM connectivity, 3G connectivity, 4G connectivity, LTE connectivity The present invention may be configured to provide a communication function, any other suitable communication function, or any suitable combination thereof. The insole may include any number of communication links. Lloyd(R) / iOS(R) controller and display module, Rumble engine visualization, various modes (e.g. walking and running modes), 15. Various interfaces such as Bluetooth low energy interface, etc. It can provide five functions.

[0021] In at least one embodiment, the host controller 120 (e.g., a processor) Sensing elements (e.g., force sensing elements 105, strain sensing elements 105, environmental The processor may scan the sensing elements 110, the motion sensing elements 115, etc. The sensing element may be periodically scanned. uses different scan rates for at least some of the sensing elements. This may provide optimal data resolution and / or power consumption benefits. For example, some areas of the user's foot may move more frequently, i.e., less frequently than other areas of the user's foot. These areas may experience a greater rate of force or pressure change compared to the For data resolution, scans can be taken more frequently. Lower regions can be scanned more sparsely in time, which reduces the power consumption of the system 100. Consumption may be reduced.

[0022] The processor receives the signal from the sensing element (and from any other sensor as described herein). The processor may perform various analyses based on the received data. For example, , a force and / or pressure map of the human foot may be generated. The force and / or pressure map may be an instantaneous snapshot of the current state of the foot. The map may also include data over time, and the map may represent mean, median, or other values. The map can be used to determine the level of pronation (e.g., overpronation, underpronation, supination) The map can be viewed as a "heat map" which can show ranges of force or pressure with different colors. In at least one embodiment, the processor may include a processor for processing the sensor data. The processor may send the data to other devices (e.g., a server, a client device). on smartphones or other portable or wearable devices such as smartwatches Sensor data may also be sent.

[0023] The system 100 may include a power management system that may provide and / or regulate power for the system 100. It may also include a device 130.

[0024] The system 100 includes a haptic feedback system that can drive haptic feedback for the system. For example, the host controller 120 may include a sensing array receive sensor data from the sensor (e.g., any of sensors 105, 110, 115) Based on the sensor data, the host controller 120 may The tactile response (e.g., the tactile feedback through the insole that the user can feel on their feet) haptic feedback unit 135. The tactile response can be felt by the user through the insole and via the haptic device. Exemplary tactile responses include, but are not limited to, pressing, flicking, It may include impacts, releases, which may be short, long, or repeated. Posing can be used to encourage specific behaviors. For example, if a runner puts their heels If guarding, the haptic response notifies or reminds the user when they are guarding their heel. The user can then shift their running style towards the toes and away from the heels. It can be adjusted so that

[0025] The systems and methods described herein include shoes, insoles, smart sensing mats, such as flooring, recreational equipment, or other gym or exercise related applications. It can be used in many applications.

[0026] FIG. 2 illustrates an exemplary smart device for sensing various characteristics of a human foot and / or shoe. 1 shows the configuration of a smart shoe system 200. The exemplary smart shoe system is the system shown in FIG. For example, the smart shoe system 200 may include some or all of the elements of the one or more force sensing elements 105, one or more strain sensing elements 105, one or more The environmental sensing element 110 may include one or more motion sensing elements 115. may be collectively referred to as the FSR sensing array 205. The FSR sensing array 205 , may be scalable and / or adjustable depending on shoe size. For example, the first FSR The sensing array 205 may be adjustable for shoe sizes 6 to 9. The sensing array 205 may include any number of sensors. In one example, FSR sensing The array 205 may include 20 or more sensor nodes.

[0027] The host controller 120 may measure approximately 5 cm x 5 cm x 1 cm. The roller 120 generates a pressure map based on the readings from the FSR sensing array 205. The host controller 120 may include logic for determining the timing and / or motion control. may be coupled to a communication link 125. The host controller 120 may It may also be combined with other controllers such as the (Registered Trademark) controller and display module. This can be configured for game engine visualization. , motion and movement detected by the FSR sensing array 205 ) can be translated into movement of the user or a digital character's avatar.

[0028] FIG. 3 illustrates an exemplary embodiment for sensing various characteristics of a human foot and / or shoe. The left side of Figure 3 may show a first view 305 of the system, where The sensing element may be closer to the object being measured (e.g., the foot). A second view 350 of the stem may be shown, which includes a circuit board 355, a power source (e.g., a battery) 3 60, etc. can be coupled to the system.

[0029] The system 300 measures the forces, strains, movements, motions, and The system may include a sensing element 310 that may be configured to measure other environmental characteristics. 00 may include an insole that may be removably inserted into the shoe. The insole may be incorporated into or attached to the shoe. The insole includes a top insole layer 320 and an interposer 325. , one or more sensing elements 310 coupled to an interposer 325, and a bottom-in It may include a sole layer 330.

[0030] The top insole layer 320 may be used in shoes, boots, sandals, or any other type of The top insole layer 320 may be shaped to fit within the footwear. The bottom insole layer 330 may be formed from any of the following materials or combinations of materials: The material may be formed from any material or combination of materials. Materials include porous materials, foam materials, plastic materials, and the like. The top insole layer may be made of a plastic material, or any other natural or synthetic material. 320 may be constructed from a different material or materials than the bottom insole layer 330 . The bottom insole layer 330 may be formed from a harder material. The overall stiffness of the ear can be stiffer than the top insole layer 320. In one embodiment, the bottom insole layer 330 includes an interposer 325 and / or may provide a hard / stable base for the sensing element 310. Layer 320 may be attached to interposer 325 (e.g., by adhesive). , welding, sewing, etc.) to the interposer 325. In this embodiment, the bottom insole layer 330 is adapted to withstand repeated contact with a hard surface (e.g., concrete). The at least one may be formed from a resilient material configured to withstand impacts. In an embodiment, the bottom insole layer 330 comprises or is part of the sole of the shoe. It can be a part.

[0031] Additionally or alternatively, the interposer 325 may be configured to interface with one of the external host controllers. The interposer can be connected to an external circuit board (not shown in FIG. 3) as a part. The circuit board may include or be part of any type of circuit board. The circuit board may be rigid or flexible.

[0032] One or more sensing elements 210 may be coupled to the interposer. The sensing elements 210 may be referred to as a sensing array. consists of one or more force sensing elements, one or more strain sensing elements, one or more motion sensing elements, One or more strain sensing elements may be included, and / or one or more environmental sensing elements. The sensing element may include one or more two-dimensional strain sensing elements. The sensing elements may be spatially distributed on the interposer. Alternatively, one or more of the sensing elements may be separate components coupled to the poser. Directly formed, etched, deposited, printed, etc. on the interposer For example, the sensing element may be printed on a flexible circuit board (i.e., flex). It can be printed.

[0033] As shown on the left side 305 of FIG. 3, an exemplary system includes two sensing arrays 3 40, 345 (e.g., top and bottom, toe area and heel area, etc.) Each sensing array 340, 345 detects the motion within the respective sensing layer. One or more force sensing elements 31 that can provide effective insole force / pressure detection and measurement. Each sensing array 340, 345 may include one or more strain sensors. The sensor may also include a sensing element and one or more environmental sensing elements. The single array includes a toe area / zone array and a heel area / zone array. It can be seen.

[0034] The system 300 may also include a controller, as further described in conjunction with FIG. The controller includes circuitry configured to receive data from the sensing element 310. The controller includes a memory for storing data and a processor for executing operations. The controller may include a communication link.

[0035] The controller receives information from the sensing element (and, as described herein, from other sensors). The controller may perform various analyses based on the data received. may generate a force, movement, and / or pressure map of the human foot. The step can be an instantaneous snapshot of the current state of the human foot. The pressure map may also include time-varying data, and the map may include mean, median, The map may represent pronation (e.g., overpronation, underpronation, supination, etc.) or other values. The map can be used to determine the level of force or pressure. In at least one embodiment, the The controller sends the sensor data to other devices (e.g., servers, clients) for processing. The controller can send the sensor data to a smartphone or smart device. It may also be sent to other portable or wearable devices such as watches.

[0036] The insole may include any number of sensors. The sensors may be located in or on the insole. It is possible to detect any characteristic of the vicinity of a cell (such as data representing motion or the environment) It can represent any hardware or software sensor, including but not limited to: Not included are accelerometer, gyroscope, altimeter, Global Positioning System (GPS), and step count. sensors, magnetometers, thermometers, hygrometer sensors, atmospheric pressure sensors, GPS receivers, motion, environment, or people This includes any other sensor capable of detecting a condition between the sensor and the power source, or any combination thereof. Any movement detected by the sensor can be called a movement signature. The sensors may detect various movement patterns that may be linked to one or more of the following: Any suitable system, apparatus, device, or routine capable of detecting or determining That is, tilt, shake, rotation, rocking, and any other motion. For example, This sensor allows the insole to track your progress (e.g., walking, running). It may be possible to detect periodic movements in a circular manner, representing the individual being tracked. In an embodiment, the sensor is configured to detect or determine the location of a particular tracked individual. For example, the sensor may be a GPS receiver, a Wi-Fi signal detector, a cellular network Signal detector, Bluetooth beacon detector, Internet Protocol (IP) a location detector, or any other system capable of detecting or determining the location of a particular tracked individual. The location may include one or more labels or In some embodiments, the sensors may be clustered together. The sensor may be an integrated sensor that includes two or more different sensors integrated together. For example, the sensor may include: An integration combining a three-dimensional (3D) accelerometer, a 3D gyroscope, and a 3D magnetometer It can be a chemical sensor.

[0037] The insole may also include any number of activity trackers. The lacquer may be used in devices such as, but not limited to, heart rate monitors, blood pressure monitors, thermometers, moisture sensors, etc. The insoles include sensors such as a breathing sensor, an electrodermal activity sensor, and a sleep sensor. Any information that can be used to detect characteristics (or data representative of characteristics) of the tracked individual. It may represent a hardware or software sensor or device. The searcher is used to identify characteristics of the tracked individual using the insole. In some embodiments, the heart rate monitor measures or determines a heart rate or an indicator of a heart rate. For example, a heart rate monitor may be configured to monitor the pulse of an individual being tracked. One or more sensors (e.g., photoresistors or The sensor may include a photodiode or other device.

[0038] In these or other embodiments, the activity tracker may include a heart rate monitor. , one or more systems, devices configured to determine a heart rate based on the detected indicators In some embodiments, a particular tracked individual may include a device or module. Events in the life of a particular tracked individual will affect their heart rate, This includes the heart rate maintained by a particular tracked individual during the workout, heart rate recovery time, etc. These may be obtained from one or more heart rate monitors or other activity trackers or sensors. by the host controller (or externally calculated data) based on data received from the sensor. can be determined by the device.

[0039] The insoles may include any number of insoles that may provide any type of tactile feedback to the user. The device may also include a haptic feedback device.

[0040] The insole may include more or fewer features. For example, the insole may include: The top insole layer may not be included, and the sensing elements may be The outward-facing surface of the element is substantially flush with the surface facing the interposer feet. In such an embodiment, the interposer may be positioned such that it can be positioned on a surface. The interposer may be formed from a material that may provide a degree of comfort to the foot. For example, it may be formed from a flexible material. The sensor may include one or more recesses in which the sensing element may be mounted.

[0041] As mentioned above, in at least one embodiment, the left side 305 of FIG. 3 includes two sensors. 1 illustrates an exemplary system including a first sensing array. The shape is designed to fit in the area (e.g., near where the user places their toes). a second portion of the shoe (e.g., the portion near where the user places their heel) The first sensing array and the second sensing array can be shaped to fit the Each of the rays may include any number of sensing elements. The sensing array includes a strain sensing element, a plurality of force sensing elements, and a plurality of environmental sensing elements. As shown, the second sensing array includes a plurality of force-sensing elements. The first sensing array and the second sensing array include a plurality of environmental sensing elements. The ray may be independently communicatively and / or electronically connected to the host via a connector interface. At least one In an embodiment, the first sensing array and the second sensing array are framed with connectors. through the respective connector interface, as through a flexible printed circuit, In such an embodiment, the first sensor Either the sensing array or the second sensing array is connected to the It may be communicatively and / or electronically coupled to a host controller.

[0042] The configuration of the two sensing arrays allows for reconfiguration of the shoe length or size. Additionally, the force sensing element may be adapted to accommodate higher levels of force / pressure applied by the foot. The strain sensing element (e.g., a two-dimensional strain sensing element) may be located in the foot area. located in areas where higher levels of bending / flexing can be applied by The environmental sensing element may be designed to be sensitive to minimal levels of force / pressure or bending / deflection. However, the influence of environmental parameters equivalent to the force sensing element and the two-dimensional strain sensing element is The device may be located in an area that is susceptible to

[0043] In one example, the system performs physical exercises with the goal of enabling the user to walk correctly after treatment. In another example, the system may be used for dynamic motion monitoring in therapy. Users can track their running performance based on dynamic movement monitoring both in real time and over time. For dynamic movement monitoring in sports (e.g., athletics) that can improve performance In one embodiment, a haptic device that can provide haptic feedback may be used to , built into the system to stimulate correct behavior.

[0044] FIG. 4 illustrates the foot during, but not limited to, walking or running movements. Determine various functions including foot force / pressure mapping and biomechanics such as flexion characteristics of the foot 4 shows an example computational flow of a method 400 for detecting the force of a force sensing element. , dimensional strain sensing elements, motion sensing elements, and environmental sensing elements, or other It may be obtained by analyzing measurement data from a sensor or activity tracker. The process flow can be used to self-trigger the smart shoes. Hardware (circuits, dedicated logic, etc.), software (general-purpose computer systems or dedicated by processing logic that may include a combination of both (such as running on a consumer device) The processing logic may be implemented by the system of FIG. 1 or other computer system. However, other systems or combinations of systems may be included. For ease of explanation, the method described herein may be Although the acts are shown and described as a series of acts, the acts according to this disclosure may be performed in various orders. may occur in conjunction with and / or at the same time as other actions not presented and described herein. Furthermore, to implement the method according to the disclosed subject matter, all of the illustrated Additionally, those skilled in the art will appreciate that the method is interrelated via a state diagram. It is understood and appreciated that the Additionally, the methods disclosed herein may be implemented in a computing device. To facilitate the transfer and storage of As used herein, the term article of manufacture means , computer accessible from any computer-readable device or storage medium Although shown as separate blocks, the Various blocks may be divided into additional blocks and fewer blocks depending on the desired implementation. They may be combined into blocks or omitted.

[0045] Method 400 may begin at block 405 where processing logic may boot the controller. Processing logic may enter a sleep mode at block 410, which may include a 1 Hz sub-threshold. At block 415, processing logic may include sampling one or more sensing elements. identifying data from the shoe, and the identified data indicating that the user is wearing the shoe; By determining whether the user is wearing shoes, If the user is not wearing shoes (block 415), "No"), processing logic continues in sleep mode for a predetermined length of time at block 410. It can be included.

[0046] If the user is wearing shoes (block 415, "yes"), the processing logic The block may enter activation mode at block 420, which is the 50 Hz sampling The processing logic may include processing one or more sensing elements as described herein. For example, the processing logic may acquire data from the force of the user's feet while the user is moving. Data and motion data may be acquired.

[0047] At block 430, processing logic identifies a user profile associated with the user. In at least one embodiment, at block 430, processing logic Other devices (e.g., mobile devices, wearables, smartphones) can be used to identify your profile. To do so, the processing logic may Controller (e.g., Android controller, iOS controller) may initiate a handshake with the The processing logic may use the data obtained from the one or more sensing elements to A visualization of the data (e.g., via a graphical user interface) may be generated.

[0048] At block 435, processing logic analyzes data from one or more sensing elements. and the identified data indicates that the user is wearing the shoes. By determining if the data profile matches the user's shoes, If the user has not yet put on shoes (block 43), 5 ("No"), processing logic enters sleep mode for a predetermined length of time at block 410. If the user still has shoes on (block 435 "Yes" "), processing logic initiates or initiates a handshake with the remote device at block 430. can be checked.

[0049] FIG. 5 illustrates a variety of motions, including but not limited to walking, running, turning, or jumping. Various analyses were performed, including foot force / pressure mapping and biomechanics, such as flexion characteristics, during 5 illustrates an exemplary computational process flow 500 for determining various functions. These functions are used in sensing sensing elements, dimensional strain sensing elements, motion sensing elements, and environmental sensing elements, may be obtained by analysis of measurement data from other sensors or activity trackers. This approach to multi-sensor fusion allows the system to To be able to detect specific signatures for multiple types of human movement To do so.

[0050] As shown, force / pressure measurement data, strain measurement data, and inertial (e.g., motion) measurement Data, and / or environmental measurement data are used for multimodal parameter extraction, user calibration and This can be used for temperature setting and environmental compensation. All sensors have some form of temperature dependence. However, the temperature dependence of the environmental measurement data is compensated for in different environmental / weather conditions. Different users have different weights and different movement signatures. User calibration allows the system to correctly detect the motion profile. These data and modeling are used for user / security identification, pressure mapping display, system User motion animation displays (e.g., virtual reality, augmented reality), and / or user motion animation It can be used for displaying animation.

[0051] Figure 6 shows a smart shoe that includes one or more sensing elements (e.g., pressure measurement nodes). 6 illustrates an example implementation of a smart shoe system 600. In one embodiment, the smart shoe includes the insole of FIG. The embedded controller within the mobile device controls one or more sensing elements. The data received from the source may be used to generate a pressure mapping display, as illustrated in FIG. Thus, the pressure mapping display 600 may be displayed on the user interface of the mobile device 610. The pressure mapping display 600 is presented in color, showing different levels of pressure. It may include a coded "heat map."

[0052] The controller embedded on the smart shoe or in the mobile device may be one or more A motion animation display 620 is also generated using data received from the above sensing elements. The movement animation display 620 can be achieved in the same way as a person wearing smart shoes. In at least one embodiment, the person's movements may be recorded as a motion avatar. This is effectively reflected in the avatar in the animation display 620. For example, if the user leans to the left, When the insole detects a leftward tilt, it sends a corresponding signal to create a movement animation. A display 620 may be generated. In this example, the motion animation display 620 may include: Including an avatar that leans left in a similar manner to a user wearing this insole(s). becomes.

[0053] Figure 7 is based on measurement data from the force sensing element of any of the systems described. Graphical user interface for monitoring parameters such as foot / pressure mapping 7 illustrates an example of a GUI 700.

[0054] In at least one embodiment, the GUI 700 may include a pressure mapping display 705. This may include an outline of one or more feet 710. The GUI 700 may include The sensor may also include a graphical representation of the element (hereinafter, graphical sensor 715). A measured value (e.g., a force value) can be represented at each sensing element. As shown, each graphical sensor is shaped according to its respective force value. The shading may be colored, patterned, etc. As shown, each graphical sensor contains a respective force measurement. Measurements can be absolute (e.g., in units of Newtons or kg m / s²) or relative .

[0055] GUI700 is a physical orientation of smart shoes currently being worn by a person. The GUI may also include other portions that may illustrate the movement animations, as described above. 620 may also be provided.

[0056] FIG. 8 illustrates a method for detecting a small change in the measurements obtained from one or more sensing elements as described herein. 8 illustrates various motion animation displays 800 that may be generated based at least in part on the Show.

[0057] FIG. 9 illustrates various features including, but not limited to, user profile identification and visualization. 9 illustrates an exemplary computational process flow 900 for determining various functions. These functions are Sensing element, dimensional strain sensing element, motion sensing element, environmental sensing element , body sensing elements, or measurement data from other sensors or activity trackers This approach involves using a wearable sensor on the body or arm. The inclusion of the sensor enables enhanced multi-sensor fusion.

[0058] As shown, force / pressure measurement data, strain measurement data, and inertial (e.g., motion) measurement The data, environmental measurement data, and / or body sensing data are multimodal parameters. These data can be used for data processing, user calibration and configuration, and environmental compensation. Data and modeling are used for user / security identification, pressure mapping display, shoe movement analysis, Animation display (e.g., for virtual reality, augmented reality), and / or user movement animation Augmented multi-sensor fusion can be used for augmented reality It also enables more detailed animation of user movements in the

[0059] FIG. 10 illustrates a method for monitoring, including but not limited to, fitness and / or physiological monitoring. Exemplary computational processes for determining various functions, including applications in smart shoes The process flow 1000 is illustrated. These functions include a force sensing element, a motion sensing element, Environmental sensing elements, body sensing elements, or other sensors or activity trackers This can be obtained by analysis of measurement data from

[0060] As shown, user / security identification, pressure mapping display, shoe movement alarm The animation display and / or user movement animation display may be multimodal data. used for analysis, comparison with reference physiological parameters, and / or analysis of physiological status Additionally, these data and modeling may be used to display physiological status, Resume reminders and / or warnings, goal tracking for exercise equipment, and / or remote It can be used for warning purposes.

[0061] For example, Figure 10 shows a user wearing smart shoes running on a treadmill. The functionality of the example application can be illustrated by the following example: You can set the parameters of the running machine and adjust the parameters during exercise. This is a highly manual process. In this example, the smart shoe interface The sensor is connected wirelessly to the running machine. Parameter data analysis, comparison with reference physiological parameters, analysis of physiological status, physiological If the running machine parameters are excessive (speed too fast, If the slope is too steep, provide stress reminders / warnings to the user. Treadmill performance when the user is overstressed and unable to perform manual adjustments Automatic adjustment of parameters, if the user is injured (jerky movements, no movement), etc. The system provides several functions, such as providing remote alarm activation. It may be used in combination with other body sensors such as a heart rate monitor, blood pressure monitor, etc.

[0062] During a typical treadmill stress test, the patient is monitored using chest and arm sensors. ECG and blood pressure measurements can be monitored, respectively. There are some inherent limitations to the use of MRI: the patient's leg or foot movements are typically monitored. Therefore, the patient's walking / running stability is not monitored. These include, but are not limited to, the patient's stride speed and If the patient's position on the treadmill is abnormal, the treadmill The patient increases their walking / running speed while the speed of the training machine increases. If the patient is unable to move faster, this may be due to physical stress or fatigue of the patient, or the medical professional accidentally putting the patient on the treadmill. During some scenarios, including maneuvers involving heavy loads, efforts and the risk of a falling patient may be significant. There are inherent safety risks.

[0063] If there was a smart shoe interface that could wirelessly connect to the running machine's interface, Safety risks are greatly reduced by monitoring the patient's physiological and movement status. The width can be reduced.

[0064] Figure 11-21 shows the real-time distribution of monitoring parameters for six defined motion states. Examples of real-time measurements are shown below. Walking forward (Figure 11) Walking backwards (Figure 12) Transition from walking to running (Figure 13) Jumping (Figure 14) Rotating (Figure 15) is.

[0065] Three different types of sensors can be placed in each shoe insole, They are, Force / pressure sensing array (11 sensing nodes, e.g., as shown in Figure 16) 3-axis accelerometer (illustrated in Figures 17 and 18) 3-axis gyroscope (illustrated in Figures 19 and 21) is.

[0066] The "simplified" identifiers for the motion states are shown in each of Figures 11-21.

[0067] FIG. 22 is a diagram relating to a multimodal array in accordance with at least one embodiment of the present disclosure. 22 shows a block diagram of an example computer system 2200 for implementing the host controller described above. The controller may be implemented as a computing system, such as exemplary computer system 2200. The computer system 2200 may be configured to perform one or more operations of the present disclosure. It can be done.

[0068] The computer system 2200 may perform any one or more of the methods described herein. Executes a set of one or more instructions 2226 that the machine executes. as a server or client machine in a client-server network environment, or It can act as a peer machine in a peer-to-peer (i.e., distributed) network environment. Shin is a leading provider of personal computers (PCs), tablet PCs, set-top boxes (S TB), personal digital assistants (PDAs), mobile phones, web devices, servers, Network routers, switches, or bridges, or any actions taken by the machine Any program that can execute a set of instructions (sequential or otherwise) that specifies an operation. Furthermore, although only a single machine is illustrated, the term "machine" is used. The term "instruction" refers to a set of instructions 2226 that performs any one or more of the methods described herein. "machines" shall also be construed to include any collection of machines operating independently or in concert.

[0069] The computer system 2200 includes a processor 2202 and a main memory 2204 (e.g., , read-only memory (ROM), flash memory, synchronous DRAM (SDRAM ) or dynamic random access memory (DRRAM) such as Rambus DRAM (RDRAM) A static memory 2206 (e.g., flash memory, static memory, etc.) Random Access Memory (SRAM) and data storage device 2216, communicate with each other via bus 2208.

[0070] The processor 2202 may be one or more general-purpose processors such as a microprocessor, a central processing unit, or the like. More specifically, the processor 2202 represents a complex instruction set computer. CISC microprocessor, Reduced Instruction Set Computing (RI SC microprocessor, Very Long Instruction Word (VLIW) microprocessor, or other instruction A processor that implements an instruction set or a combination of instruction sets. The processor 2202 may be an application specific integrated circuit (ASIC), a field programmable FPGAs, digital signal processors (DSPs), network processors The processor 2202 may also be one or more dedicated processing units such as a processor or the like. It is configured to execute instructions that perform the actions and steps described.

[0071] The computer system 2200 is a network system that can be used in a local area network (LAN), an intranet, Communication with other machines over a network 2218, such as a net or the Internet The network interface device 2222 may further include a network interface device 2222 for providing a network interface. The interface device 2222 may include any number of physical or logical interfaces. The network interface device 2222 is a device for connecting network elements within a network. Any device, system, component (or components) configured to enable or facilitate communication between It may also include a component, or a collection of components, such as a network interface. The device 2222 may be a modem, a network card (wireless or wired), an infrared communication device, an optical communication device, or the like. receiving devices, wireless communication devices (such as antennas), and / or chipsets (such as Bluetooth devices, 802.xx devices (such as metropolitan area networks (MANs)), WiFi devices, This may include, without limitation, WiMax equipment, cellular equipment, and / or other networks. The interface device 2222 may be connected to a network (a mobile phone network, to name a few). network, WiFi network, MAN, optical network, etc.) and / or remote device allowing data to be exchanged with any of the other devices described in this disclosure, including In at least one embodiment, the network interface device 2222 , logically distinct on a single physical component, e.g., a single physical cable or optical fiber There can be multiple communication streams passing through the signal.

[0072] The computer system 2200 includes a display device 2210 (e.g., a liquid crystal display (LCD) an alphanumeric input device 2212 (e.g., keyboard); a signal generator 2220 (e.g., a speaker); ) may also be included.

[0073] Data storage 2216 may include any one or more of the methods or functions described herein. a computer-readable storage medium 2224 having stored thereon a set of instructions 2226 embodying The set of instructions 2226 may also be stored in main memory 2204 and / or in the computer. 2202 during execution by the computer system 2200. The main memory 2204 and the processor 2202 also include computer-readable media. The set of instructions 2226 further comprises a network interface device 222 2 and can be transmitted or received over network 2218.

[0074] Although examples of computer readable storage medium 2224 are shown as a single medium, The term "computer-readable storage medium" refers to a single medium that stores a set of instructions 2226. one or multiple media (e.g., centralized or distributed databases and / or associated The term "computer-readable storage medium" may include a computer-readable storage medium (e.g., a cache and a server). , capable of storing, encoding, and carrying a set of instructions for execution by a machine Any medium that causes a machine to perform any one or more of the methods of this disclosure may be included. The term "computer-readable storage medium" includes solid-state memory, optical This may include, but is not limited to, optical media and magnetic media.

[0075] Modifications, additions, or modifications may be made to computer system 2200 without departing from the scope of this disclosure. Omissions may be made. For example, in at least one embodiment, a computer system 2200 may include any number of other components that may not be explicitly shown or described. It may also include:

[0076] As used in this disclosure, the terms "module" or "component" refer to a module. specific hardware implementations configured to perform the operations of a module or component; and / or refers to the general-purpose hardware of a computer system (e.g., computer-readable media) software objects that can be stored on and / or executed by a computer, a processor, etc. In at least one embodiment, this may refer to an object or software routine. The various components, modules, engines, and services described in the disclosure As an object or process running on a computer system (e.g., independently) Some of the systems and methods described in this disclosure may be implemented in software. Implemented in software (stored in and / or on general-purpose hardware) Although generally described as a dedicated hardware implementation, or software implementation, A combination of software and dedicated hardware implementations is also possible and contemplated. A "computing entity" may be any computing system as defined earlier in this disclosure. and any module or combination of modules operating on a computer system. It is also possible.

[0077] As used in this disclosure and particularly in the appended claims (e.g., the body of the appended claims), Terms used in this document are "open" terms (e.g., the term "contains" is used in conjunction with "contains"). "including, but not limited to," and the term "having" can be interpreted as "including at least and the term "including" may be interpreted as "including, but not limited to." "It can be interpreted as 'it's okay', etc.)

[0078] Furthermore, if a specific number of recitations in the introduced claims is intended, such intention The claims must clearly state that the invention is intended to be invented, and the absence of such a statement indicates that no such intention exists. For example, as an aid to understanding, the following appended claims are intended to guide the recitation of the claims: This may include the use of the introductory phrases "at least one" and "one or more" to include However, the use of such phrases does not necessarily imply a conflict of interest, even if the same claim may refer to "one or more" or "at least one" " and an indefinite article such as "a" or "an." The introduction of a claim by the indefinite article "an" is considered to be a Limit any particular claim containing a recitation to an embodiment containing only one such recitation. (e.g., "a" and / or "an" means "at least" (This may be interpreted to mean "at least one" or "one or more"). The same is true for the use of the definite article used.

[0079] In addition, even if a specific number of claims is explicitly recited, those skilled in the art The reader understands that such a description may be construed to mean at least the number described. (For example, the unqualified statement "two statements" implies at least two (This means a statement or two or more statements.) Furthermore, "one of A, B and C, etc." or "A In cases where a convention similar to "one or more of B and C, etc." is used, Such an interpretation would be A only, B only, C only, A and B, A and C, B and C, or A, B and C, etc. It is intended to include.

[0080] Furthermore, in either the specification, claims, or drawings, two or more alternative uses Any disjunctive word or phrase expressing a term may be used to express one of the terms, It can be understood that the term "A" contemplates the possibility of including either or both of the terms. The phrase "or B" may be understood to include the possibilities of "A," "B," or "A and B." .

[0081] All examples and qualified language described in this disclosure are provided to aid the reader in advancing the art. It is intended for educational purposes to help readers understand the inventions and concepts brought forth by the inventors. The present invention is not intended to be limited to the specifically described examples and conditions. Although the embodiments of the present disclosure have been described in detail, it is to be understood that the spirit and scope of the present disclosure are not to be construed as limiting the scope of the present disclosure. Numerous changes, substitutions, and alterations can be made without departing from the scope.

Claims

1. A first insole comprising a first sensing layer, the first sensing layer includes a plurality of first force sensors; the plurality of first force sensors are configured to generate first force data from a scan of the plurality of first force sensors; one or more force sensors of the plurality of first force sensors in a first region of the first insole are scanned at a first rate; one or more force sensors in a second region of the first insole among the plurality of first force sensors are scanned at a second rate different from the first rate; the first rate of scanning is selected based on a location of the first region of the first insole; the second rate of scanning is selected based on a location of the second region of the first insole; one or more characteristics of two or more of the plurality of first force sensors are customized based on positions of two or more of the plurality of first force sensors in the first insole, such that one of the plurality of first force sensors has a different characteristic from another of the plurality of first force sensors. A first insole; a communication interface configured to couple the first sensing layer to a host controller; An insole system comprising:

2. The first insole comprises: a top layer coupled to the first sensing layer; a bottom layer coupled to the first sensing layer; Including, the top layer comprises a first material and the bottom layer comprises a second material; The insole system of claim 1 .

3. the first material is more flexible than the second material; The insole system of claim 2 .

4. the first insole further includes a first acceleration sensor configured to generate first acceleration data; the host controller is configured to determine a type of activity of a user of the insole system based on a correlation between the first force data and the first acceleration data. The insole system of claim 1 .

5. the first sensing layer further includes an environmental sensor; The insole system of claim 1 .

6. acquiring, by a processor, data from a first sensing layer of a wearable foot device including a plurality of force sensors; Obtaining the data includes: scanning one or more force sensors of the plurality of force sensors in a first region of the wearable foot device at a first rate; scanning one or more force sensors in a second region of the wearable foot device from the plurality of force sensors at a second rate different from the first rate; Including, the first rate of scanning is selected based on a position of the first region of the wearable foot device; the second rate of scanning is selected based on a position of the second region of the wearable foot device; one or more characteristics of two or more of the plurality of force sensors are customized based on positions of two or more of the plurality of force sensors on the first sensing layer, such that one of the plurality of force sensors has a different characteristic than another of the plurality of force sensors. Obtaining data and analyzing the data; and determining, by the processor, a type of activity of a user of the wearable foot device based on the data; A method comprising:

7. determining a live graphical display based on said activity; providing said live graphical display via a graphical user interface; The method of claim 6 further comprising:

8. the activity is at least one of walking, running, jumping, or spinning; The method of claim 6.

9. identifying environmental parameters from environmental sensors that contribute to noise in the data; adjusting the data in response to the environmental parameters to reduce the noise in the data; The method of claim 6 further comprising:

10. The method comprising: acquiring, by the processor, first acceleration data from an acceleration sensor included in the first sensing layer; acquiring, by the processor, second data from a second insole of a second wearable foot device used by the user; The second data is second force data from a second force sensor in the second insole; and second acceleration data from a second acceleration sensor in the second insole; Including, acquiring second data; further comprising the type of activity of the user is determined based on a correlation between the data, the first acceleration data, the second force data, and the second acceleration data. The method of claim 6.

Images