Systems, methods and devices for monitoring gait

Abstract

Described embodiments relate to systems, methods and devices for monitoring gait One embodiment relates to a method for assessing a subject's gait, the method comprising: accessing image data captured by an imaging sensor located on a mobility aid of the subject, identifying at least one foot of the subject in the image data, and determining a relative position and orientation of the at least one foot with respect to the mobility aid based on a position of the imaging sensor with respect to the mobility aid. Another embodiment relates to a device comprising a mobility aid for providing support to a subject, and an imaging sensor secured to the mobility aid and positioned to capturing image data of at least one foot of the subject, wherein the imaging sensor is configured to communicate the image data to a processing device to identifying at least one foot of a subject in the image data and determining a relative position and orientation of the at least one foot with respect to the mobility aid based on a position of the imaging sensor with respect to the mobility aid.

Description

[0001] "Systems, methods and devices for monitoring gait"

[0002] Technical Field

[0003] Described embodiments relate to systems, methods and devices for monitoring measures of gait. In particular, described embodiments relate to systems, methods and devices for monitoring measures of gait of a subject with a mobility aid.

[0004] Background

[0005] Monitoring the gait of a subject can be useful in many contexts. Data gathered from such monitoring can be used to provide training or therapy to the subject, to optimise the size or configuration of any mobility aids used by the subject, to track the progress of any rehabilitation or physiotherapy being undertaken by the subj ect, or to assess the subj ect’ s risk of falls, among other uses.

[0006] Traditionally, such monitoring has been performed in clinical environments. In some cases, one or more sensors may be placed on the body of the subject or on the subject’s clothing, to allow for their gait to be tracked and monitored. In some cases, an image capture system may be set up within the clinical environment to capture the movement of the subject as the subject performs movements under the instruction of a clinician. However, there are a number of limitations to these previous methods of tracking gait within a clinical settings.

[0007] It is desired to address or ameliorate one or more shortcomings or disadvantages associated with prior systems, methods and devices for monitoring gait, or to at least provide a useful alternative thereto.

[0008] Throughout this specification the word "comprise", or variations such as "comprises" or "comprising", will be understood to imply the inclusion of a stated element, integer or step, or group of elements, integers or steps, but not the exclusion of any other element, integer or step, or group of elements, integers or steps.

[0009] Any discussion of documents, acts, materials, devices, articles or the like which has been included in the present specification is not to be taken as an admission that any or all of these matters form part of the prior art base or were common general knowledge in the field relevant to the present disclosure as it existed before the priority date of each of the appended claims.

[0010] Summary

[0011] Some embodiments relate to a method for assessing a subject’s gait, the method comprising: accessing image data captured by an imaging sensor located on a mobility aid of the subject; identifying at least one foot of the subject in the image data; and determining a relative position and orientation of the at least one foot with respect to the mobility aid based on a position of the imaging sensor with respect to the mobility aid.

[0012] Some embodiments further comprise determining an absolute position and orientation of the at least one foot with respect to an environment based on a position of the imaging sensor with respect to the environment.

[0013] Some embodiments further comprise fusing the image data with additional sensor data, wherein determining the relative position and orientation of the at least one foot with respect to the mobility aid is based on both the position of the foot in the image data and the additional sensor data.

[0014] According to some embodiments, the additional sensor data comprises motion sensor data.

[0015] In some embodiments, the additional sensor data comprises accelerometer data.

[0016] In some embodiments, the additional sensor data comprises gyroscopic data.

[0017] According to some embodiments, the additional sensor data comprises further imaging sensor data.

[0018] In some embodiments, the further imaging sensor data comprises depth image data.

[0019] In some embodiments, the further imaging sensor data comprises colour image data. According to some embodiments, the further imaging sensor data is generated by a second imaging sensor located on a second mobility aid of the subject.

[0020] Some embodiments further comprise deriving at least one gait parameter of the subject based on the determined relative position of the foot.

[0021] According to some embodiments, the at least one gait parameter is derived using stored relative position data together with the determined relative position of the foot.

[0022] In some embodiments, deriving the at least one gait parameter comprises calculating at least one surface contact time point at which the foot contacted a contact surface.

[0023] According to some embodiments, the gait parameter comprises at least one of a step size, step width, step height or step length.

[0024] According to some embodiments, the gait parameter comprises at least one of a step frequency, gait speed; single or dual stance time, gait asymmetry and / or stance asymmetry.

[0025] Some embodiments further comprise updating a gait model based on at least one gait parameter.

[0026] Some embodiments further comprise generating a predicted next position of the at least one foot based on the gait model.

[0027] Some embodiments further comprise determining an orientation of the foot.

[0028] In some embodiments, determining the position or orientation of the foot comprises processing the image data using an image processing technique.

[0029] In some embodiments, determining the position or orientation of the foot comprises processing the image data using a foot localisation algorithm.

[0030] In some embodiments, the foot localisation algorithm may comprise: determining an approximate location of the foot by generating a region of interest in the image data; identifying a principal axis of the foot using a computer vision method; identifying features of the foot based on at least the geometry of the foot; projecting the features of the foot onto the image data to determine the location of the features in three-dimensional space; and defining the position and orientation of the foot using the location of the features.

[0031] In some embodiments, the computer vision method may be thresholding and / or polygon approximation.

[0032] According to some embodiments, the image processing technique uses a trained machine learning model.

[0033] In some embodiments, the image data comprises colour image data.

[0034] In some embodiments, the image data comprises depth image data.

[0035] According to some embodiments, the mobility aid is a non-wheeled mobility aid.

[0036] Some embodiments relate to a device comprising: a mobility aid for providing support to a subject; and an imaging sensor secured to the mobility aid and positioned to capturing image data of at least one foot of the subject; wherein the imaging sensor is configured to communicate the image data to a processing device to identifying at least one foot of a subject in the image data and determining a relative position and orientation of the at least one foot with respect to the mobility aid based on a position of the imaging sensor with respect to the mobility aid.

[0037] Some embodiments relate to a system comprising: mobility aid device; and a computing device for performing a method to identifying at least one foot of a subject in image data and determining a relative position and orientation of the at least one foot with respect to the mobility aid based on a position of an imaging device with respect to the mobility aid. Brief Description of Drawings

[0038] Various ones of the appended drawings merely illustrate example embodiments of the present disclosure and cannot be considered as limiting its scope.

[0039] Figure 1 is a diagram illustrating a system of monitoring gait according to some embodiments;

[0040] Figure 2 is a block diagram illustrating aspects of the system of Figure 1 in further detail.

[0041] Figure 3 A illustrates a subject using a first embodiment of a device of Figure 1;

[0042] Figure 3B illustrates a support polygon constructed based on the scenario illustrated in Figure 3 A;

[0043] Figure 4A illustrates a subject using a second embodiment of a device of Figure 1;

[0044] Figure 4B illustrates a support polygon constructed based on the scenario illustrated in Figure 4 A;

[0045] Figure 5A illustrates a subject using a third embodiment of a device of Figure 1;

[0046] Figure 5B illustrates a support polygon constructed based on the scenario illustrated in Figure 5 A;

[0047] Figure 6 shows a graph illustrating data availability based on data captured by the system of Figure 5A;

[0048] Figure 7A is a process flow diagram of a method of monitoring measures of gait according to some embodiments;

[0049] Figure 7B is a process flow diagram of a method of determining the position and orientation of a foot within an input image, according to some embodiments; and Figures 8A to 8E illustrate a number of example images captured by an imaging sensor positioned on a mobility aid, according to some embodiments.

[0050] Description of Embodiments

[0051] Described embodiments relate to systems, methods and devices for monitoring measures of gait. In particular, described embodiments relate to systems, methods and devices for monitoring measures of gait of a subject with a mobility aid. Monitoring measures of gait of a subject can provide important information that can assist with improving or maintaining the mobility of the subject. For example, such measures can be used to determine whether the subject would benefit from training, therapy, or other interventions, as well as determining the type of training, therapy or intervention to provide. The measures can be used to assess the subject’s risk of falls, and assist in determining how that risk can be minimised. Such measures may also provide data that can be used to better calibrate the size or configuration of any mobility aids used by the subject. Where the subject is undergoing rehabilitation or physical therapy, monitoring measures of the subject’s gait can be helpful in tracking the progress of the rehabilitation or therapy.

[0052] Monitoring gait is often done by tracking parts of the subject’s body relative to a surface the subject is traversing. For example, laboratory-based motion capture systems may be used to track the position of the subject’s body by using a set of fixed cameras and markers placed on the body of the subject. The movement of the markers with respect to the motion capture system can be tracked and used to determine measures of gait. However, such a system requires specialised infrastructure, making it infeasible in community settings. Furthermore, the subject may consciously or sub-consciously change their gait within the laboratory setting, such that the measurements may not be a good representation of the subject’s natural gait.

[0053] Another commonly used method to track measures of gait is to have the subject wear sensors on their body. These may include such as Inertial Measurement Units (IMUs), which may be placed on one or more limb segments of the subject. However, such a system is still unsuitable for everyday use, particularly due to the number of sensors and the calibration process required. Such a system is therefore also better suited to a laboratory or clinical environment, and may not be well suited to capturing the subject’s natural gait in the subject’s day to day activities.

[0054] Described embodiments provide systems, methods and devices for monitoring measures of a subject’s gait that do not require a fixed motion capture system, and do not require the subject to wear tracking markers or sensors on their body or clothing. Instead, the motion and gait monitoring is performed using a motion capture system mounted on a mobility aid. The system can detect and track ground contact points, such as the position of a subject’s feet with respect to each other and with respect to the mobility aid. This allows for various measures relating to balance and gait to be derived. By reducing the amount of specialised clothing or additional equipment required, the motion and gait monitoring system can be used during the subject’s everyday activities, allowing for measures of their natural gait to be captured and used for the beneficial applications noted above, among others. The measures can be used to actively monitor the subject’s natural capabilities, such as while the subject ages or recovers from injury, to improve the subject’s balance during gait or to recover from a measured loss of balance. Furthermore, some embodiments use methods for dealing with intermittencies in captured data, so that measures of gait can be adequately monitored even when the subject is using a non- wheeled mobility aid.

[0055] Figure 1 shows an example system 100 for monitoring measures of gait of a subject. Measures of gait may include measures of balance. The measures of gait and balance may be determined based on identifying interactions between a surface that is being traversed and one or more feet and mobility aids of a subject. In system 100, sensors to identify such interactions are integrated into a mobility aid.

[0056] System 100 comprises a mobility aid 110 having a sensor module 120. Sensor module 120 is in communication with a processing device 140 over a network 130. Sensor module 120 and / or processing device 140 are further in communication with a display device 150 via network 130.

[0057] Figure 1 shows just one arrangement of system 100. In some embodiments, processing device 140 and / or display device 150 may be located on mobility aid 110, and may be integral with sensor module 120 in some cases. In some embodiments, display device 150 may be integral with processing device 140. System 100 may comprise more than one mobility aid 110, more than one processing device 140, and / or more than one display device 150.

[0058] Mobility aid 110 comprises a handle 111, a body 112 and a base 113. In the illustrated embodiment, mobility aid 110 is in the form of a walking stick. However, mobility aid 110 may comprise one or more walking sticks, crutches, canes, walking frames or other mobility devices. Mobility aid 110 may be used to assist a subject with legged locomotion. In some embodiments, mobility aid 110 may be used to assist a subject with balance. According to some embodiments, mobility aid 110 may be configured to support at least a portion of the subject’s weight. Handle 111 may be configured to contact a hand or arm of the subject, to allow a subject to place their weight through handle 111. Handle 111 may comprise a gripping portion in some embodiments. In some embodiments, mobility aid 110 may comprise more than one handle. For example, where mobility aid 110 is a walking frame, mobility aid 110 may comprise two handles 111.

[0059] Body 112 may extend between handle 111 and base 113. In some embodiments, body 112 may comprise a substantially elongate support member. In some embodiments, body 112 may comprise a frame.

[0060] Base 113 may be configured to contact a surface that the subject is traversing for at least a portion of the time the subject is in locomotion. In some embodiments, base 113 may comprise at least one wheel, and base 113 may be configured to contact the surface that the subject is traversing for substantially the whole time that the subject is in locomotion. For example, base 113 may comprise four wheels where mobility aid 110 is a wheeled walking frame. In some embodiments, base 113 may comprise a non-wheeled terminal structure such as a foot, ferrule or heel, which may be configured to contact the surface that the subject is traversing for only part of the time that the subject is in locomotion. In such embodiments, the mobility aid may be configured to be picked up and relocated periodically or intermittently as the subject traverses, making contact with the surface at a new location each time. For example, base 113 may comprise four feet where mobility aid 110 is a non-wheeled walking frame. Base 113 may comprise three or more feet where mobility aid 110 is a tetrapod walking stick. Base 113 may comprise one foot where mobility aid 110 is a walking stick, cane or crutch.

[0061] Sensor module 120 may be affixed to or integrated with mobility aid 110 to monitor measures of the gait of a subject as the subject uses mobility aid 110. According to some embodiments, sensor module 120 may be affixed to mobility aid via a mount. The mount may be configured to attach a sensor module 120 to mobility aid 110. Depending on the type of mobility aid 110, for example, walking sticks, crutches, and walking frames, the mount may have different configurations. The mount comprises one or more mounting brackets, constructed to securely attach sensor module 120 to the respective mobility aid 110. The mounting bracket may be configured to accommodate different angles required for optimal functionality of the sensor module, as users may use mobility aids differently. In some embodiments, the mount may include a clamping mechanism configured to attach the mount to a range of different mobility aids. The clamping mechanism locks the mount in position to allow the sensor module 120 to remains securely attached to the mobility aid 110, even when the user is in motion. In some embodiments, the clamping mechanism may be an adjustable clamping mechanism that can be tightened around a part of the mobility aid. For example, the adjustable clamping mechanism may be attached around the shaft of a walking stick. The clamping mechanism may include a locking mechanism configured to securely fastens the mount to the mobility aid. This locking mechanism may include a screw or lever that can be tightened to apply pressure to the clamping mechanism, ensuring a firm and substantially immovable connection of the mount to the mobility aid 110. Once the clamping mechanism is locked in place, it holds the mount and the attached sensor module 120 at the desired angle, substantially mitigating any unwanted movement or vibration. In some embodiments, inner surfaces of the clamping mechanism may be lined with a non-slip material, such as rubber or silicone, to increase friction and prevents the clamping mechanism from slipping or rotating around the part of the mobility aid to which it is attached.

[0062] In some embodiments, the mount further includes an integrated device holder, configured to securely hold an imaging device that forms part of the sensor module, such as a mobile phone, tablet, camera, and the like. For example, in some embodiments, the integrated device holder is a spring-loaded phone mount that securely grips a phone, acting as a sensor module, and holding it in place with respect to the mobility aid 110 even during vigorous movement. The combination of the clamping mechanism and the integrated device holder that form the mount ensures that the sensor module 120 remains stable and oriented correctly, providing a reliable platform for the sensor module 120 to function effectively.

[0063] For walking sticks or crutches, the mount may be configured such that, the sensor module 120, such as a phone, is to be angled between 10 and 20 degrees towards the user's opposite foot. This positioning ensures that the sensor module is as high as possible on the walking stick, providing a substantially optimal field of view for the sensor module 120. In the case of walking frames, the angle at which the sensor module 120 is mounted can vary significantly depending on the specific mounting location on the frame. The mount for the walking frame may be configured similarly to the mount for the walking stick. In some embodiments, the mount for the walking frame may be configured similarly to the mount for the walking stick but may include at least two clamping mechanisms attachable to the walking frame to provide additional stability and hold the sensor module 120 more securely. For example, in some embodiments, the mount for a walking frame may include a first clamping mechanism and a second clamping mechanism to provide additional stability. This second clamping mechanism works in conjunction with the first clamping mechanism to distribute the load and reduce the risk of the sensor module 120 shifting or becoming loose when mounted to the mobility aid 110.

[0064] According to some embodiments, sensor module 120 may be integral with mobility aid 110. Sensor module 120 may comprise one or more sensors, as described in further detail below with reference to Figure 2, and may be configured to capture and optionally store sensor data. The sensor data may be communicated to processing device 140 for processing, and / or to display device 150 for display, via network 130. Sensor module 120, network 130, processing device 140 and display device 150 are described in further detail below with reference to Figure 2.

[0065] Figure 2 shows an arrangement of system components, including hardware and software of systems that may be used to perform the presently disclosed methods. It would be readily understood by the person skilled in the art that the system of Figure 2 is simply one embodiment of a number of potential embodiments that would be suitable for performing the present methods.

[0066] Figure 2 shows a block diagram illustrating the software and hardware components of system 100 in further detail. Specifically, the software and hardware component of sensor module 120, network 130, processing device 140 and display device 150 are shown.

[0067] Sensor module 120 comprises components configured to capture, store, and communicate sensor data to processing device 140 and / or display device 150 via network 130. In some embodiments, sensor module 120 may be a computing device such as a tablet, smart phone, or special purpose computing device. Sensor module 120 may comprise a processor 121 configured to read and execute program code. Processor 121 may include one or more data processors for executing instructions, and may include one or more of a microprocessor, microcontroller-based platform, a suitable integrated circuit, and one or more application-specific integrated circuits (ASICs). Sensor module 120 may further comprise at least one memory 123. Memory 123 may include one or more memory storage locations which may include volatile and nonvolatile memory, and may be in the form of ROM, RAM, flash or other memory types. Memory 123 may also comprise system memory, such as a BIOS, in some embodiments.

[0068] Memory 123 is arranged to be accessible to processor 121, and to store data that can be read and written to by processor 121. Memory 123 may store sensor data 124, for example. Memory 123 may also contain program code 125 that is executable by processor 121, to cause processor 121 to perform various functions. Processor 121 executing program code 125 may be caused to receive sensor data from sensors 160, store the received data to sensor data 124, read sensor data 124 and / or send the sensor data to an external device via communications module 122, for example.

[0069] Sensor module 120 may further comprise a communications module 122, to facilitate communication between user sensor module 120 and other remote or external devices. Communications module 122 may allow for wired or wireless communication between sensor module 120 and external devices, and may use Wi-Fi, USB, Bluetooth, or other communications protocols. According to some embodiments, communications module 122 may facilitate communication between user sensor module 120 and processing device 140 and / or display device 150 via a network 130, for example.

[0070] Sensor module 120 further comprises one or more sensors 160. These may include imaging sensor 161, motion sensor 162, pressure sensor 163 and / or positioning sensor 164, for example.

[0071] Imaging sensor 161 may comprise at least one camera. According to some embodiments, the camera may capture colour data, such as RGB data, for example. In some embodiments, the camera may additionally or alternatively capture depth data. In some embodiments, imaging sensor 161 may comprise a Realsense D435i (Intel Corporation) camera. Imaging sensor 161 may be configured to capture images at a frame rate of around 30 frames per second, in some embodiments. In some embodiments, colour and depth images may be captured at around 1280x720 and 848x480 resolution, respectively.

[0072] Imaging sensor 161 may be positioned in or on mobility aid 110 in a position that allows capture of image data relating to the position of at least one foot of a subject using mobility aid 110 while the subject is standing, walking, or performing other legged locomotion. According to some embodiments, in order to increase the field of view of imaging sensor 161, imaging sensor 161 may be positioned such that it is located toward the top half of mobility aid 110 when mobility aid 110 is in use. According to some embodiments, imaging sensor 161 may be located near a handle 111 of mobility aid 110. According to some embodiments, the position of imaging sensor 161 on mobility aid 110 may be selected taking into account the gait style, grip style and / or height of a subject using mobility aid 110.

[0073] Motion sensor 162 may comprise at least one inertial sensor configured to measure the acceleration and / or angular velocity of the sensor module 120 as the subject moves mobility aid 110. In some embodiments, motion sensor 162 may comprise one or more accelerometers and / or gyroscopes. Motion sensor 162 may comprise an inertial measurement unit (IMU), for example. Motion sensor 162 may be configured to capture accelerometer data at around 250Hz, and gyroscope data at around 400Hz, in some embodiments. In some embodiments, motion sensor 162 may comprise a Realsense D435i (Intel Corporation) camera with integrated accelerometer and gyroscope. According to some embodiments, in order to capture the full range of motion of mobility aid 110, motion sensor 162 may be positioned such that it is located toward the lower half of mobility aid 110 when mobility aid 110 is in use. According to some embodiments, imaging sensor 161 may be located near a base 113 of mobility aid 110.

[0074] Pressure sensor 163 may be configured to measure the pressure applied by a subject through mobility aid 110 when the subject uses mobility aid 110.

[0075] Positioning sensor 164 may be configured to determine the position of sensor module 120, and may comprise at least one of a global positioning system (GPS) module or a magnetometer, in some embodiments.

[0076] Sensor module 120 may further comprise user input and output peripherals (not shown). These may include one or more of a display screen, touch screen display, speaker, or microphone, for example. User input and output peripherals may be used to receive data and instructions from a user, and to communicate information to a user.

[0077] According to some embodiments, sensor module 120 may be free of any processor 121 and / or memory 123. Instead, sensors 160 may be configured to automatically communicate sensor data to an external device such as processing device 140, which may be via communications module 122. This may be done in real time or on a periodic basis. In some embodiments, sensors 160 may be configured to automatically send sensor data to memory 123 for storage in sensor data 124. The data may synchronously or asynchronously be retrieved by processing device 140 for processing in real time or at a later time.

[0078] Network 130 may comprise one or more local area networks or wide area networks that facilitate communication between elements of system 400. For example, according to some embodiments, network 130 may be the internet. However, network 130 may comprise at least a portion of any one or more networks having one or more nodes that transmit, receive, forward, generate, buffer, store, route, switch, process, or a combination thereof, etc. one or more messages, packets, signals, some combination thereof, or so forth. Network 130 may include, for example, one or more of: a wireless network, a wired network, an internet, an intranet, a public network, a packet- switched network, a circuit- switched network, an ad hoc network, an infrastructure network, a public-switched telephone network (PSTN), a cable network, a cellular network, a satellite network, a fibre-optic network, or some combination thereof.

[0079] Processing device 140 may be a device in communication with sensor module 120 and configured to perform data processing functions. Processing device 140 may be located on mobility aid 110 in some embodiments. In some embodiments, processing device 140 may be integral with sensor module 120. For example, processing device 140 and sensor module 120 may both form part of a single device located on mobility aid 110, which may be a smart phone in some embodiments. In some embodiments, processing device 140 may be located externally to and / or remotely from mobility aid 110 and / or sensor module 120.

[0080] Processing device 140 may be a computing device such as a personal computer, laptop computer, desktop computer, tablet, or smart phone, for example. Processing device

[0081] 140 comprises a processor 141 configured to read and execute program code. Processor

[0082] 141 may include one or more data processors for executing instructions, and may include one or more of a microprocessor, microcontroller-based platform, a suitable integrated circuit, and one or more application-specific integrated circuits (ASICs). Processing device 140 further comprises at least one memory 143. Memory 143 may include one or more memory storage locations which may include volatile and nonvolatile memory, and may be in the form of ROM, RAM, flash or other memory types. Memory 143 may also comprise system memory, such as a BIOS, in some embodiments.

[0083] Memory 143 is arranged to be accessible to processor 141, and to store data 144 that can be read from and written to by processor 141. For example, data 144 may include sensor data 145 as received from sensor module 120, gait model data 146 relating to a gait model generated based on sensor data 145, and position data 147 relating to calculated positions of ground contact points being monitored, such as the feet of a subject.

[0084] The gait model characterised by gait model data 146 may be a parameterised model that allows the trajectory of gait to be modelled over time. According to some embodiments, the gait model may be an average gait model derived from a group of subjects. In some embodiments, the gait model may be specific to an individual. In some embodiments, an average gait model for a population may be used initially, and the model may be updated as data relating to an individual subject’s gait becomes available. In some embodiments, the gait model may be a mathematically derived model of gait. The gait model may be configured to predict the position and / or orientation of one or more feet of a subject with respect to a real- world frame (such as a ground plane or other contact surface) for a given point in time.

[0085] In some embodiments, the gait model may be characterised by one or more gait parameters, such as step height, step length and / or step width, for example. In some embodiments, the gait model may use the gait parameters to derive one or more curves through space along which the ground contact points, such as the feet of the subject, are expected to move. In some embodiments, the gait model may comprise individual models for different stages of gait, such as the toe off stage, the swing stage and the ground contact stage.

[0086] Memory 143 may also contain program code 170 that is executable by processor 141, to cause processor 141 to perform various functions.

[0087] Program code 170 may include a calibration module 171. Executing calibration module 171 may cause processor 141 to perform a calibration process with respect to one or more sensors 160. For example, processor 141 may be caused to calibrate imaging sensor 161 to allow imaging sensor 161 to be used to identify the relative position and orientation of imaged objects.

[0088] Program code 170 may further include prediction module 172. Executing prediction module 172 may cause processor 141 to predict a future position of a ground contact point, such as the foot of a subject, based on a model of the gait of that subject, using the stored gait model data 146.

[0089] Program code 170 may also include a foot identification module 173. Foot identification module may also be referred to as foot detection module. Processor 141 executing foot identification module 173 may be caused to process sensor data 145 and / or gait model data 146 to identify the position of one or more feet of a subject using mobility aid 110 with respect to the position of the mobility aid 110. According to some embodiments, foot identification module 173 may comprise an image processing module configured to process image data and output information based on the image processed. This could include an image processing module configured to identify specific 2 dimensional or 3 dimensional shapes from image data, which could include colour and / or depth data. According to some embodiments, the image processing module of foot identification module 173 may comprise a machine learning model that has been trained to detect feet based on image data. For example, a machine learning model may be trained on a set of images with labelled regions of interest. In some embodiments, the training data may include images labelled with data indicative of foot orientation. Given an image, the machine learning model may be configured to identify any feet in the image, and determine the orientation of each identified foot with respect to the imaging device that the image was captured with.

[0090] The image processing module of foot identification module 173 may include a foot localisation module. Processor 141 executing foot identification module 173 may be caused to process an input image on which the foot localisation algorithm is to be applied. The input image may be a depth image or colour image. In some embodiments, if a colour image is used, it may be converted to greyscale to simplify further processing. In some embodiments, an object detection algorithm is applied to the input image to determine the location and / or position of the foot. In some embodiments, the object detection algorithm may include, but is not limited to, YOLO, SSD, RetinaNet, Faster R-CNN, Mask R-CNN, cascade R-CNN, Grounding DINO, SAM, HOG, and / or the Viola-Jones algorithm. In some embodiments, the Create ML Object Detection may be used as the object detection model, with additional training on a custom dataset. The position of the foot may be determined within a threshold range of approximation or may be determined with a predetermined error margin. In some embodiments, the foot localisation algorithm may generate a region of interest (ROI) from the object detection model. The region of interest may define a region of the input image within which the foot is located. In some embodiments, the region of interest may include all or part of the foot. In some embodiments, the region of interest may include all or part of the foot and may include part of the surrounding environment. The region of interest may be defined by a bounding box. In some embodiments, the input image may be cropped to the region of interest. Within the region of interest, computer vision techniques may be used to identify the principal axis of the foot. In some cases, the principal axis of the foot may also refer to the “direction” of the foot or the “orientation” of the foot.

[0091] In some embodiments, the computer vision techniques used may include polygon approximation. For example, contour detection methods such as the findContours function or Canny edge detection followed by contour finding, may be used to identify the outline of the foot within the region of interest. The contour may then be approximated to a polygon. In some embodiments, the polygon approximation may be performed using techniques like the Douglas-Peucker algorithm or convex hull. In some embodiments, the image processing module of foot identification module 173 takes the region of interest and applies an edge detection algorithm to identify the contours of the foot within the region of interest. The edge detection algorithm may be, for example, Canny edge detection. Then, the image processing module may apply a contour detection algorithm to extract the outline of the foot from the image. The largest contour, which may be assumed to be at least one foot, is then simplified using a polygon approximation algorithm. The polygon approximation algorithm may reduce the number of points in the contour while maintaining the overall shape. For example, the polygon approximation algorithm may include the Ramer-Douglas-Peucker algorithm. The vertices of the approximated polygon are then analysed using Principal Component Analysis (PCA). PCA is a statistical method that transforms the data to a new coordinate system where the greatest variance by any projection of the data lies on the first coordinate, which may be referred to as the principal component. The first or principal component derived from PCA indicates the orientation of the foot, which can be visualised by drawing a line along this vector on the image.

[0092] In some embodiments, the computer vision techniques used may include thresholding. For example, thresholding techniques such as Otsu’s method, adaptive thresholding, or simple binary thresholding, may be applied to the region of interest to segment the foot from the background. In some embodiments, the image processing module of foot identification module 173 takes the region of interest and converts it to grayscale. In some embodiments, the image processing module may apply a Gaussian blur to reduce noise in the region of interest. The image processing module then applies thresholding to convert the grayscale region of interest into a binary image, thereby separating the foot (foreground) from the background. In some embodiments, the thresholding technique may separate pixels into two classes, foreground and background. This may be done by finding a single intensity threshold that minimises the intra-class variance or equivalently maximises the inter-class variance. For example, the thresholding technique used may be Otsue’s method. Contour detection algorithms may then be applied to identify the outline of the foot in the binary image, with the largest contour being selected to represent the foot. In some embodiments, a contour of approximate expected dimensions and shape of a foot or shoe may be used to identify the foot in the binary image. In some embodiments, the contour detection algorithm may include OpenCV’s findContours function. The coordinates of the points making up the largest contour are then extracted, and Principal Component Analysis (PCA) is performed on these points to identify the principal axes of the foot shape. The first or principal component derived from PCA indicates the orientation of the foot, which can be visualised by drawing a line along this vector on the image.

[0093] In general, feet and shoes have similar shapes across different individuals which makes the principal axis a meaningful indication of direction and / or orientation. In some embodiments, the principal axis of the foot may be identified based on the above discussed thresholding or polygon approximation techniques in combination with the shape of the foot.

[0094] The image processing module is then configured to identify features of the foot based on the geometry of the foot. In some embodiments, the shape of the foot is analysed by the image processing module to identify key features such as, but not limited to, the toe, heel and / or sides of the foot. In some embodiments, the determined principal axis of the foot in the region of interest may be used to identify features of the foot. For example, geometric properties of the foot and the principal axis may be used to locate these features, for example, with the toe being the point farthest along the principal axis and the sides being the points at the widest part perpendicular to the principal axis.

[0095] In some embodiments, features of the foot may be identified without knowledge of the principal axis, but using additional information that indicates the orientation of the foot. For example, features of the foot may be identified using an object detection algorithm, for example, to identify the toes of the foot. In further embodiments, the features of the foot may be identified using additional image analysis techniques such as, but not limited to, segmentation algorithms, local feature detection, depth information analysis, and the like.

[0096] These identified geometric features may be projected back to the depth image to determine the location of these features in 3D space. The depth image may be used to obtain the depth values at the identified feature points, and these 2D coordinates are converted to 3D coordinates using the depth values. The position of the foot may be determined by calculating the centroid of the foot features in 3D space, while the orientation may be defined using the principal axis and the 3D coordinates of the features. The location of these identified features in 3D space may be then used to define the position and / or orientation of the foot identified in input image. As such, the foot identification module thereby provides the position and / or orientation of the foot in 3D space.

[0097] This approach enables the use of object detection algorithms while reducing the computational cost of classic computer vision techniques due to the reduced region of interest in the input image. The method applied by the foot identification module 173 may also be used to localise other body parts, such as the shin.

[0098] The foot identification module 173 may also be configured to apply a ground detection algorithm to provide an additional “virtual measurement” to more accurately determine the position and / or orientation of the foot. The virtual measurement provided by the ground detection algorithm may be particularly beneficial for systems involving walking sticks or crutches, which vary in height from the ground. Additionally, the virtual measurement can be constructed to report the height of the foot from the ground. This allows the system to estimate whether the foot is in the air or on the ground, indicating whether it is likely to be moving or stationary.

[0099] The ground detection algorithm may utilise a Random Sample Consensus (RANSAC) algorithm to detect the ground plane in an image, for example, the ground. The Random Sample Consensus (RANSAC) algorithm may be used to identify the ground plane from depth image data of the input image. In some embodiments, for example in the context of determining the height of a foot from the ground, RANSAC is applied by iteratively selecting random subsets of the data points and fitting a plane to these points. The algorithm then evaluates how well this plane fits the entire dataset by counting the number of inliers, which are points that lie within a certain distance from the plane. The process is repeated multiple times, and the plane with the highest number of inliers is considered the best representation of the ground. Once the ground plane is determined or established, the algorithm can calculate the perpendicular distance from the foot to this plane, effectively determining the foot's height from the ground. This measurement is beneficial for applications involving walking aids, as it helps in assessing whether the foot is in contact with the ground or in the air when determining the position and / or orientation of the foot in the input image.

[0100] The ground detection algorithm allows for the calculation of the distance between the camera and the ground in the direction perpendicular to the ground, which is referred to as “camera height”. Assuming the ground is horizontal, this is equivalent to the height. This measurement does not utilise any prior knowledge of the system's height and the camera height should remain constant for a fixed height mobility aid, such as a wheeled walker. However, the camera height will change with non-fixed height mobility aids such as crutches, canes, and non- wheeled walkers as the user lifts and tilts these walking aids during use, resulting in a varying camera height within each image. Further, the ground detection algorithm also allows for the calculation of the distance between the foot and the ground, which is referred to as “foot height”. Calculating the foot height includes identifying the foot's location, which can then be used to calculate the distance from the ground. This measurement can also be used to calculate virtual absolute velocity measurements by noting that the foot's absolute velocity will be zero when the distance between the foot and the ground is zero.

[0101] In some embodiments, the foot identification module 173 may also be configured to use optical flow as an additional virtual measurement to estimate movement of the foot compared to the camera, and movement of the camera compared to the ground. This may assist in more accurately determining the position and / or orientation of the foot in the input image. Optical flow may provide an additional virtual measurement by analysing two consecutive images and measuring the displacement of each pixel between these images. This displacement data can then be projected into the depth image to estimate the three-dimensional movement of the foot relative to the camera, and / or the movement of the camera relative to the ground. By providing this additional virtual measurement, the optical flow enhances the ability of the foot identification module 173 to accurately track and analyse the position and movement of the foot and camera.

[0102] The ground detection and optical flow virtual measurements may be used to calculate the position and orientation of the foot in the input image. That is, the foot identification module 173 may be configured to identify foot characteristics as well as environmental characteristics (such as the height of the foot from the ground), and then determine the relative and / or absolute position and velocity data of the foot. As used herein, “relative” may refer to measurements (real or virtual) that relate the position and / or velocity of the feet to that of the camera and / or other sensors. For example, a relative measurement could be the distance, direction, and orientation of a foot with respect to the camera. “Absolute” may refer to measurements that relate the position and / or velocity of a foot or camera to the environment, such as the ground. In some embodiments, an example of an absolute measurement may be the distance of a foot to the ground.

[0103] Program code 170 may also comprise gait modelling module 174. Processor 141 executing gait modelling module 174 may be caused to process sensor data 145 and / or existing gait model data 146 to generate or improve a gait model relating to the gait of a subject using mobility device 110.

[0104] Program code 170 may also comprise position calculation module 175. Processor 141 executing position calculation module 175 may be caused to process sensor data 145 and / or existing gait model data 146 to calculate the positions of ground contact points being monitored, such as the feet of a subject, over time.

[0105] The functions performed by processor 141 executing modules 171 to 175 are described in further detail below with reference to Figure 7 A and method 700.

[0106] In some embodiments, program code 170 may include additional applications that are not illustrated in Figure 2, such as an operating system application, which may be a mobile operating system if processing device 140 is a mobile device, a desktop operating system if processing device 140 is a desktop device, or an alternative operating system.

[0107] Processing device 140 may further comprise a communications module 122, to facilitate communication between processing device 140 and other remote or external devices. Communications module 122 may allow for wired or wireless communication between processing device 140 and external devices, and may use Wi-Fi, USB, Bluetooth, or other communications protocols. According to some embodiments, communications module 122 may facilitate communication between processing device 140 and sensor module 120 and / or display device 150 via network 130, for example.

[0108] Processing device 140 may further comprise user input and output peripherals (not shown). These may include one or more of a display screen, touch screen display, mouse, keyboard, speaker, microphone, and camera, for example. User input and output peripherals may be used to receive data and instructions from a user, and to communicate information to a user.

[0109] Display device 150 may be a device in communication with processing device 140 and / or sensor module 120 and configured to perform data display functions. Display device 150 may be a device operated by a clinician or health practitioner, in some embodiments, and may provide information to the clinician or health practitioner about the gait or balance of the subject using mobility aid 110. In some embodiments, display device 150 may be a device operated by the subject. Display device 150 may be located on mobility aid 110 in some embodiments. In some embodiments, display device 150 may be integral with sensor module 120. For example, display device 150 and sensor module 120 may both form part of a single device located on mobility aid 110, which may be a smart phone in some embodiments. In some embodiments, processing device 140 may be located externally to and / or remotely from mobility aid 110 and / or sensor module 120. In some embodiments, display device 150 may be integral with processing device 140. For example, display device 150 and processing device 140 may both form part of a single device which may be a personal computer in some embodiments. In some embodiments, display device 150 may be located externally to and / or remotely from processing device 140.

[0110] Display device 150 may be a computing device such as a personal computer, laptop computer, desktop computer, tablet, or smart phone, for example. Display device 150 comprises a processor 151 configured to read and execute program code. Processor 151 may include one or more data processors for executing instructions, and may include one or more of a microprocessor, microcontroller-based platform, a suitable integrated circuit, and one or more application- specific integrated circuits (ASICs).

[0111] Display device 150 further comprises at least one memory 153. Memory 153 may include one or more memory storage locations which may include volatile and nonvolatile memory, and may be in the form of ROM, RAM, flash or other memory types. Memory 153 may also comprise system memory, such as a BIOS, in some embodiments.

[0112] Memory 153 is arranged to be accessible to processor 151, and to store data that can be read and written to by processor 151. Memory 153 may also contain program code that is executable by processor 151, to cause processor 151 to perform various functions. For example, the program code may include a data display application for causing data relating to measures of the gait and / or balance of a subject to be displayed. According to some embodiments, the data display application may be a web browser application (such as Chrome, Safari, Internet Explorer, Opera, or any other alternative web browser application) which may be configured to access web pages that provide data display functionality via an appropriate uniform resource locator (URL).

[0113] Program code may also include an operating system application, which may be a mobile operating system if display device 150 is a mobile device, a desktop operating system if display device is a desktop device, or an alternative operating system.

[0114] Display device 150 may further comprise user input and output peripherals. These may include one or more of a display screen 154, touch screen display, mouse, keyboard, speaker, microphone, and camera, for example. User input and output peripherals may be used to receive data and instructions from a user, and to communicate information to a user. For example, display 154 may be used to display data relating to measures of the gait and / or balance of a subject.

[0115] Display device 150 may further comprise a communications module 152, to facilitate communication between display device 150 and other remote or external devices. Communications module 122 may allow for wired or wireless communication between display device 150 and external devices, and may use Wi-Fi, USB, Bluetooth, or other communications protocols. According to some embodiments, communications module 122 may facilitate communication between display device 150 and processing device 140 and / or sensor module 120 via a network 130, for example.

[0116] System 100 as illustrated in Figures 1 and 2 may operate to determine measures of gait, which may include measures of balance. This may be done by first detecting and tracking ground contact points, such as the positions of the feet of the subject, relative to the position of mobility aid 110. These ground contact points may be used to determine measures of balance, such as by estimating a support polygon which may be used to assess balance and stability during gait. In some embodiments, communications module 142 of system 100 is configured to transmit instructions from processing device 140 to communications module 122 of sensor module 120 to remotely control starting and / or stopping the capture of information from sensors 160 to perform determination of gait measurement. In some embodiments, the remote control of performing determination of gait measurement may utilise a Bluetooth trigger, which enables the initiation and termination of gait analysis remotely.

[0117] Figures 3A to 5B show example systems illustrating how a subject may use mobility aid 110, and how the positions of ground contact points may be captured by imaging sensor 161 of sensor module 120.

[0118] Figure 3A illustrates a system 300 comprising a subject 310 using a mobility aid 110A. Subject 310 has a right foot 311 and a left foot 312, which each act as ground contact points, making intermittent contact with a surface over which subject 310 traverses.

[0119] Mobility aid 110A is an example of a particular embodiment of the mobility device 110 of Figures 1 and 2. Specifically, mobility aid 110A is a walking stick. Subject 310 is holding mobility aid 110A in the subject’s right hand, and using mobility aid 110A to assist them with support and balance as they walk.

[0120] Mobility aid 110A comprises sensor module 120 including an imaging sensor 161. Imaging sensor 161 has a field of view 320. The field of view 320 moves as subject 310 moves mobility aid 110A. Mobility aid 110A comprises a base 113 which is configured to intermittently make contact with a surface being traversed, to act as a further ground contact point. As the subject moves mobility aid 120A and their feet 311 and 312, their feet 311 and 312 may move in and out of the field of view 320.

[0121] Figure 3B illustrates a diagram 350 that shows the data captured by the field of view 320 of imaging sensor 161 which may be used to determine the position of the contact points between system 300 and a surface over which subject 310 is traversing, where the contact points may include one or more of feet 311, 312 and base 113. These contact points may also be referred to as ground contact points. In the illustrated example, field of view 320 captures base 113 and part of the left foot 312 but does not capture the right foot 311. According to some embodiments, imaging sensor 161 may be positioned in a way such that base 113 is not in a field of view 320. As the relative position of base 113 with respect to imaging sensor 161 is known based on the known position of imaging sensor 161 on mobility aid 110A, and as imaging sensor 161 has been calibrated based on its position on mobility aid 11 A, the relative position of visible foot 312 can be determined with respect to the position of mobility aid 110A and base 113.

[0122] Figure 4A illustrates a system 400 comprising a subject 410 using a mobility aid 110B. Subject 410 has a right foot 411 and a left foot 412, which each act as ground contact points, making intermittent contact with a surface over which subject 410 traverses.

[0123] Mobility aid HOB is an example of a particular embodiment of the mobility device 110 of Figures 1 and 2. Specifically, mobility aid 110A is a non-wheeled walking frame. Subject 410 is holding mobility aid 110A in both hands, and using mobility aid 110B to assist them with support and balance as they walk.

[0124] Mobility aid HOB comprises sensor module 120 including an imaging sensor 161. Imaging sensor 161 has a field of view 420. The field of view 420 moves as subject 410 moves mobility aid HOB. Mobility aid 110A comprises four bases 113-1 to 113-4, which are configured to intermittently make contact with a surface being traversed, to act as further ground contact points. As the subject moves mobility aid 120B and their feet 411 and 412, their feet 411 and 412 may move in and out of the field of view 420.

[0125] Figure 4B illustrates a diagram 450 that shows the data captured by the field of view 420 of imaging sensor 161 which may be used to determine the position of the contact points between system 400 and a surface over which subject 410 is traversing, where the contact points may include one or more of feet 411, 412 and bases 113-1 to 113-4. These contact points may also be referred to as ground contact points. In the illustrated example, field of view 420 captures bases 113-1 and 113-2, and a portion of each of the right foot 411 and the left foot 412. According to some embodiments, imaging sensor 161 may be positioned in a way such that bases 113 are not in a field of view 420. as the relative position of bases 113 with respect to imaging sensor 161 are known based on the known position of imaging sensor 161 on mobility aid 110B, and as imaging sensor 161 has been calibrated based on its position on mobility aid 1 IB, the relative position of visible feet 411 and 412 can be determined with respect to the position of mobility aid HOB and bases 113.

[0126] Figure 5A illustrates a system 500 comprising a subject 510 using mobility aids HOC. Subject 510 has a right foot 511 and a left foot 512, which each act as ground contact points, making intermittent contact with a surface over which subject 510 traverses. Mobility aids 110C are an example of a particular embodiment of the mobility device 110 of Figures 1 and 2. Specifically, mobility aids HOC are crutches. Subject 510 is holding one mobility aid 110C in each arm, and using mobility aids 110C to assist them with support and balance as they walk. Specifically, subject 510 holds a first mobility aid 110C-R in their right arm and a second mobility aid 110C-L in their left arm.

[0127] Mobility aids HOC each comprise a sensor module 120 including an imaging sensor 161. Specifically, mobility aid 110C-R has a sensor module 120-R having an imaging sensor 161-R, and mobility aid 110C-L has a sensor module 120-L having an imaging sensor 161-L. Each imaging sensor 161 has a field of view 520. Specifically, imaging sensor 161-R has a field of view 520-R, and imaging sensor 161-L has a field of view 520-L. Each mobility aid HOC comprises a base 113 which is configured to intermittently make contact with a surface being traversed, to act as a further ground contact point. Specifically, mobility device 110C-R comprises a base 113-R and mobility device 110C-L comprises a base 113-L. The fields of view 520 move as subject 510 moves mobility aids HOC. As the subject moves mobility aids 120C and their feet 511 and 512, their feet 511 and 512 may each move in and out of each field of view 520.

[0128] Figure 5B illustrates a diagram 550 that shows the data captured by the fields of view

[0129] 520 of imaging sensors 161 which may be used to determine the position of the contact points between system 500 and a surface over which subject 510 is traversing, where the contact points may include one or more of feet 411, 412 and bases 113-R, 113-L. These contact points may also be referred to as ground contact points. The data captured by each field of view 520-R, 520-L may be combined to determine the positions of each contact point. In the illustrated example, field of view 520-L captures base 113-L and a portion of the left foot 512. Field of view 520-R captures base 113-R and a portion of the left foot 512. Neither of the fields of view 520 capture right foot 511. According to some embodiments, imaging sensor 161 may be positioned in a way such that bases 113 are not in a field of view 520 corresponding to the mobility aid 110C to which they belong, but may be captured by the field of view of the opposing imaging sensor 161. For example, filed of view 520-R may not include base 113-R, but may intermittently capture base 113-L. As the relative positions of bases 113 with respect to imaging sensors 161 are known based on the known position of each imaging sensor 161 on each mobility aid HOC, and as imaging sensors 161 have been calibrated based on their position on mobility aid 110C, the relative position of visible foot 512 can be determined with respect to the position of mobility aids 110C-R and 110C-L, and bases 113-R and 113-L.

[0130] As illustrated by Figures 3 A to 5B, having imaging sensor 161 located on a mobility aid 110 means that the data captured will only intermittently include either one of the feet of the subject. This is particularly true where mobility aid 110 is a non-wheeled mobility aid, being a mobility aid that must be picked up and repositioned rather than wheeled along a surface. This is further illustrated in Figures 8A to 8E.

[0131] Figures 8A to 8E illustrate a number of example images that might be captured by an imaging sensor 161 positioned on a mobility aid 110. Specifically, Figures 8A to 8E show example images captured by an imaging sensor 161 positioned on a mobility aid 110 in the form of a walking stick, such as mobility aid 110A as illustrated in Figure 3 A. Figures 8A to 8E show that portions of either or both of the feet of a user might be captured by imaging sensor 161 at any given point in time. Figure 8A shows an image 800. A surface 805, being a floor that is being traversed, forms the background of image 800. Mobility aid 110 having a base 113 is visible in the lower part of image 800. A right leg 810-R is visible extending across image 800, and a portion of a foot 815-R is visible at the right edge of image 800. Left leg 810-L and left foot 815-L are not visible in image 800, which is due to the position of mobility aid 110 with respect to legs 810 and feet 815. Left leg 810-L and left foot 815- L are either out of shot or obscured by the position of right leg 810-R.

[0132] Figure 8B shows an image 820. The surface 805, being the floor that is being traversed, forms the background of image 820. Mobility aid 110 having base 113 is visible in the lower part of image 820. A small portion of right leg 810-R is visible at a left edge of image 820, and right foot 815-R is visible in the left side of the image. Left leg 810-L and left foot 815-L are also visible in image 820.

[0133] Figure 8C shows an image 840. The surface 805, being the floor that is being traversed, forms the background of image 840. Mobility aid 110 having base 113 is visible in the lower part of image 840. Right leg 810-R is visible extending across image 840, and a small portion of a foot 815-R is visible at the right edge of image 840. A small portion of left leg 810-L is also visible in the upper left corner of image 840, but left foot 815-L is not visible in image 840 due to the position of mobility aid 110 with respect to legs 810 and feet 815. Left foot 815-L is obscured by the position of right leg 810-R.

[0134] Figure 8D shows an image 860. The surface 805, being the floor that is being traversed, forms the background of image 860. Mobility aid 110 having base 113 is visible in the lower part of image 860. A small portion of left leg 810-L is visible at a left edge of image 820, and left foot 815-L is visible in the left side of the image. Right leg 810-R and right foot 815-R are not visible in image 860, due to the position of mobility aid 110 with respect to legs 810 and feet 815. Right leg 810-R and right foot 815-R are out of shot of image 860. Figure 8E shows an image 880. The surface 805, being the floor that is being traversed, forms the background of image 880. Mobility aid 110 having base 113 is visible in the lower part of image 880. A portion of right leg 810-R is visible at a left edge of image 820, and right foot 815-R is visible towards the centre of the image. Left leg 810-L and left foot 815-L are also visible in image 880.

[0135] Figures 8A to 8E show that imaging sensor data relating to the position of any one foot may be intermittent. This may be due to the movement of the imaging sensor, and / or the movement of the feet. At any point in time, one or both feet may be out of the field of view of the imaging device, or may be obscured or occluded by another object. A leg of a subject that is closer to the mobility aid 110 during use may appear in the imaging sensor data more often, and may sometimes occlude capture of the opposing leg.

[0136] Figure 6 shows a graph 600 illustrating data captured by system 500 as shown in Figure 5A that further shows the intermittency of the captured foot location data.

[0137] Graph 600 shows four data steams captured over time, being streams 610, 620, 630 and 640. The data shown in graph 600 is data captured by a system as shown in Figure 5A, where a subject 510 is using two mobility aids HOC in the form of crutches. Data stream 610 shows the periods of time in which the imaging sensor 161-L located on the left mobility aid 110C-L could detect a left foot 512 of the subject 510. Data stream 620 shows the periods of time in which the imaging sensor 161-R located on the left mobility aid 110C-R could detect a right foot 511 of the subject 510. Data stream 630 shows the periods of time in which the imaging sensor 161 located on the right mobility aid 110C could detect a left foot 512 of the subject 510. Data stream 640 shows the periods of time in which the imaging sensor 161 located on the right mobility aid 110C could detect a right foot 511 of the subject 510.

[0138] As shown in graph 600, each foot is only detected by each sensor for a portion of time. The imaging sensor 161 located on the left mobility aid 110C could detect a left foot 512 of the subject 510 around 64% of the time, but could detect the right foot 511 of the subject 510 only 31% of the time. The imaging sensor 161 located on the right mobility aid 110C could detect a right foot 511 of the subject 51072% of the time, could detect the left foot 512 of the subject 510 only 23% of the time. If only a one sided mobility aid 110 was used, such as the mobility aid 110A shown in Figure 3A in the form of a walking stick, it is clear that the data would be even more incomplete.

[0139] Figure 7A is a process flow diagram of a method 700 of monitoring measures of gait according to some embodiments. Method 700 may be carried out by the systems as illustrated in any one or more of Figures 1, 2, 3 A, 4A or 5 A. While the method steps are described as being performed by specific components of the system, such as by processor 141 of processing device 140, the method steps may be carried out by any appropriate parts of the system. For example, some or all of the method steps may be carried out by one or more of processor 121 or processor 151.

[0140] At step 705, processor 141 executing calibration module 171 is caused to perform a calibration process to calibrate sensors 160 of sensor module 120. According to some embodiments, this may comprise performing a calibration process on imaging sensor 161. Processor 141 may receive image data captured by imaging sensor of an object of a known size and at a known distance from imaging sensor 161. For example, a user may be instructed to print a QR code or calibration image at a specific size and to place this on the ground in a field of view of imaging sensor 161, with mobility aid 110 positioned in an orientation that corresponds to a normal orientation of mobility aid 110 during use. Where the position of imaging sensor 161 is movable with respect to mobility aid 110, details of its position may also be provided to processor 141. Processor 141 may use the received calibration data to calibrate imaging sensor 161, such that the position and orientation of objects appearing in the field of view of imaging sensor 161 can be determined relative to the position of mobility aid 110 and imaging sensor 161.

[0141] At step 710, processor 141 receives or accesses sensor data. According to some embodiments, the sensor data may be received from sensor module 120, and processor 141 may be caused to store the sensor data locally as sensor data 145. In some embodiments, processor 141 may access sensor data from stored sensor data 145. The sensor data may include one or more of image data generated by imaging sensor 161, motion data generated by motion sensor 162, pressure data generated by pressure sensor 163, or position data generated by positioning sensor 164. At step 715, where the sensor data comprises image data generated by imaging sensor 161, processor 141 may be caused to identify a contact surface such as a ground plane in the image data. The contact surface may correspond to a surface with which the ground contact points make contact with during locomotion. For example, the contact surface may comprise or correspond to a floor or ground surface, which may be a substantially planar surface, or a non-planar surface such as stairs, in some cases. The contact surface may correspond to a surface on which a subject is traversing or will traverse, such as surface 805 as illustrated in Figures 8A to 8E.

[0142] According to some embodiments, the contact surface may be identified using depth image data captured by imaging sensor 161. In some embodiments, the depth data may be parametrised to provide an indication of whether ground contact points, such as the feet of the subject, are in contact with the surface. In some alternative embodiments, the contact surface may be identified through colour image data captured by imaging sensor 161 by identifying areas of the data that are stationary, using two images captured by imaging sensor 161 at two timepoints and comparing these to identify areas that are not moving. Then movement of the mobility aid 110 can be taken into account using sensors such as motion sensor 162.

[0143] Step 715 may be optional in some embodiments.

[0144] At step 720, processor 141 executing prediction module 172 is caused to predict a next position of a foot of a subject using mobility aid 110. The prediction may be based on a gait model of the subject as stored in gait model data 146 along with the last recorded position of the foot as stored in position data 147. Processor 141 may be caused to use gait model data 146 to predict the position of at least one foot of the subject at a point in time based on the previously calculated position of the at least one foot at a previous point in time by modifying the previously calculated position in line with a movement predicted by the gait model described by the gait characteristics stored in gait model data 146. According to some embodiments, the prediction may be made with respect to a future period of time during which sensor data from sensors 160 is expected to be received. The prediction may comprise a predicted position of one or more ground contact points, such as one or both feet of the subject and / or one or more mobility aids 110. At step 725, processor 141 executing foot detection module 173 processes any received or accessed image data to determine whether a foot is visible in the image data. In some embodiments, the image data may include depth data. Processor 141 executing foot detection module 173 may use an image processing technique to identify specific 2 dimensional or 3 dimensional foot shapes from the image data. According to some embodiments, processor 141 executing foot detection module 173 may use a machine learning model trained on labelled data to identify any feet visible in the image data.

[0145] If no foot is detected in the image data by processor 141 executing foot detection module 173, then processor 141 proceeds to execute step 740 of method 700, as described below.

[0146] If processor 141 executing foot detection module 173 does detect a foot in the image data, then processor 141 proceeds to execute step 730. Where more than one foot is detected in the image data, processor 141 may perform steps 730 to 745 with respect to each foot.

[0147] At step 730, processor 141 executing foot detection module 173 is caused to identify characteristics of the detected foot and characteristics of the environment around the detected foot. According to some embodiments, this may include determining whether the foot belongs to the subject, by comparing the identified foot to previously identified feet in recent image data. According to some embodiments, processor 141 executing foot detection module 173 is caused to identify whether the foot is a left or right foot of the subject. In some embodiments, this may be done by using an image processing technique or a machine learning model trained on labelled data to identify and differentiate right and left feet. In some embodiments, this may be done based on the relative size and position of the detected foot relative to other objects in the image, such as another foot.

[0148] Processor 141 executing foot detection module 173 may further be caused to determine an orientation of the foot. This may be done by using an image processing technique or a machine learning model trained on labelled data to identify the position of the foot. According to some embodiments, the machine learning model may be trained on depth image data, and may be provided with captured depth image data to determine the orientation.

[0149] In some embodiments, processor 141 executing foot detection module may be caused to determine the position and orientation of the foot using a foot localisation method. Figure 7B is a process flow diagram of a method 760 of determining the position and orientation of a foot within an input image, according to some embodiments. Method 760 may be carried out by the systems as illustrated in any one or more of Figures 1, 2, 3 A, 4A or 5A. While the method steps are described as being performed by specific components of the system, such as by processor 141 of processing device 140, the method steps may be carried out by any appropriate parts of the system. For example, some or all of the method steps may be carried out by one or more of processor 121 or processor 151. The method 760 may be performed by the processor 141 executing foot identification module 173 to determine the position and / or orientation of the foot in an input image. Method 760 may be performed as part of 730, 735, 740 and 745 of method 700.

[0150] At step 765, processor 141 executing foot detection module applies an object detection algorithm to identify the approximate position of a foot within an input image. The input image may be received as part of sensor data received at 710 of method 700. At step 770, a region of interest is generated based on the approximate position of the foot identified in step 765. Within the region of interest, at step 775, a principal axis of the foot is identified. The principal axis of the foot may be identified using one or more computer vision techniques, including, but not limited to, thresholding and / or polygon approximation. The principal axis of the foot provides an indication of the direction of the foot.

[0151] At step 780, features of the foot are identified based on the geometry of the foot. In some embodiments, the features of the foot may be identified using the principal axis identified at step 775. For example, the shape of the foot may be analysed in combination with the principal axis to identify key features of the foot such as, but not limited to, the toe, heel, and / or sides of the foot. In some embodiments, the features of the foot may be identified using other information related to the orientation of the foot. For example, the features of the foot may be identified using image analysis methods, such as, but not limited to segmentation algorithms, local feature detection, object detection algorithms, and depth analysis.

[0152] At 785, the identified features of 780 are projected back to the input image, or the depth image, to determine the location of these features in 3D space. At step 790, the location of these features in 3D space is used to define the position and orientation of the foot.

[0153] This, at step 790, the position and orientation values of the foot are output.

[0154] Referring back to Figure 7 A, at step 735, processor 141 executing foot detection module 173 is caused to calculate the estimated relative position of the detected foot with respect to the mobility aid 110 and the contact surface. As the relative position of the imaging sensor 161 to the mobility device 110 is known and the imaging sensor 161 has been calibrated as described above with reference to step 705, the relative position of the identified foot from the imaging sensor 161 and therefore the mobility aid 110 can be calculated. In some embodiments, image data captured by imaging device 161 may also capture portions of mobility aid 110, which may be identified and used to more accurately determine the relative positions of the identified foot with respect to mobility aid 110. The estimated relative position of the identified foot with respect to the identified contact surface may also be derived. In some embodiments, step 735 may also calculate the estimated absolute position of the detected foot with respect to the mobility aid 110 and the contact surface. That is, foot identification module 173 is caused to calculate the estimated absolute position of the detected foot with respect to the environment, such as the ground or contact surface. This may be calculated using a ground detection algorithm, as described herein, to calculate the height of the foot above the ground. The camera height is calculated and used to determine the absolute position of the identified foot from the imaging sensor 161 and therefore the environment. In some embodiments, image data captured by imaging device 161 may also capture portions of the environment, such as contact surface, which may be identified and used to more accurately determine the absolute positions of the identified foot with respect to the environment. As such step 735, processor 141 executing foot detection module 173 may be caused to determine relative position data and absolute position data of the foot and environmental characteristics.

[0155] In some embodiments, identifying foot and environmental characteristics at 730 and determining relative and absolute position data at 735 in method 700 may also include determining virtual measurements to more accurately determine the position and orientation data of the foot. In some embodiments, virtual measurements may be used to supplement the foot localisation method 760 to more accurately determine the position and orientation data of the foot. Virtual measurements may include, for example, use of a ground detection algorithm or optical flow to determine distances between the foot, camera and / or contact surface, such as the ground. According to some embodiments, at step 735 processor 141 may also compare the estimated positions against the predicted positions as determined at step 720. If the predicted positions are significantly different from the estimated positions, processor 141 may use the predicted positions instead of the estimated positions until further data can be collected to confirm the correctness of the estimates.

[0156] At step 740, where mobility aid 110 comprises multiple sensors 160 or where multiple measurements of data have been received from sensors 160 or derived from the sensor data, processor 141 executing position calculation module 175 is caused to combine measurements from multiple sources to allow processor 141 to better determine the actual positions of the tracked ground contact points. This may include combining estimated position data as determined at step 735, data from motion sensor 162, data from pressure sensor 163 and / or data from positioning sensor 164, for example. Where multiple imaging sensors 161 are being used, estimated position data as derived from each imaging sensor 161 may be combined. This may include combining estimated position data from colour and depth imaging sensors 161, in some embodiments.

[0157] According to some embodiments, processor 141 may use a Manifold Extended Kalman Filtering (MEKF) technique to fuse these measurements received and / or derived from sensors 160 to determine the position of the ground contact points.

[0158] The MEKF-based technique combines the various measurements to optimise the predicted position of the ground contact points. The technique enables the application of the Extended Kalman Filter to systems with multiple bodies. Furthermore, the technique allows for the estimation of rotational and translational positions of multiple bodies, such as the feet of a subject and the mobility aid 110, whilst ensuring appropriate representation of rotation.

[0159] MEKF techniques are not usually applied to systems with intermittent data. As described above, particularly with reference to Figure 6 and 8A to 8E, the described systems and devices are subject to receiving intermittent data. To mitigate this problem, processor 141 performing a MEKF technique to combine measurements may be configured to identify when particular data is not available at a given timestep, and to remove the requirement of that data in the MEKF algorithm, whilst including any available data. In other words, any sensor date that is intermittent can be excluded from the update step of the filtering technique for timesteps where the data is not available. At step 745, processor 141 executing position calculation module 175 is caused to output the position values for the ground contact points as determined at step 740. This outputting step may comprise storing the values to memory, such as to position data 147 of memory 143, and / or sending the values to an external computing device such as display device 150. Each set of position values may correspond to a specific time or timestep, and may be stored and / or communicated in association with that time or timestep. According to some embodiments, each set of position values may be timestamped with a time at which they were recorded.

[0160] At step 750, processor 141 executing gait modelling module 174 is optionally caused to derive one or more gait parameters, metrics or measures based on the determined position values as stored in position data 147. According to some embodiments, one or more sets of position data over time may be used to derive gait parameters.

[0161] According to some embodiments, deriving gait parameters may comprise first determining the times at which the ground contact points were in contact with the contact surface as identified in step 715, or in other words when they were touching the ground. These times may be referred to as the surface contact time points. According to some embodiments, this may be determined by identifying when the vertical component of the positions of the ground contact points are equal to the vertical position of the ground, which may be when the z-component of their positions is 0. In some embodiments, the times at which the ground contact points were in contact with the contact surface may be determine using pressure sensors located on the feet of the subject and / or on the mobility aid 110. A contact with the contact surface may be determined if the pressure reading is greater than a threshold value, such as 0, for example.

[0162] In some embodiments, surface contact time points may be determined by processor 141 analysing the trajectories of the ground contact points and identifying times when the ground contact points stop moving downwards in the vertical direction.

[0163] According to some embodiments, gait parameters that may be determined may include an average, maximum, minimum or individual value for one or more of a step size, step width, step frequency, and / or gait speed; single or dual stance time, or asymmetries in any of the above metrics, such as differences in the parameters between the right and left foot, which may be a gait asymmetry or stance asymmetry, for example. These parameters may be determined by analysing position data 147 and plotting the movement of each ground contact point through space, in some embodiments. In some embodiments, measurement of the movement between instances of contact with the contact surface may be determined. For example, a step length may be calculated as the distance travelled by a foot in between contact with the contact surface.

[0164] At step 755, processor 141 executing gait modelling module 174 may then be caused to update the gait model data 146 based on the position values determined at step 745 and / or the gait measures determined at step 750. In some embodiments, the gait model data may be updated based on derived average gait parameters such as step height, length and / or width. In some embodiments, the gait model may be updated by modifying the model parameters in a way that would minimise the error between the calculated and predicted measurements.

[0165] Method 700 then iterates, by continuing on from step 710, when new sensor data is received at a new timestep.

[0166] According to some embodiments, processor 141 may also be caused to use the stored sensor data 145 for other purposes, such as detection and classification of the environment, such as the type of surface being traversed by the subject.

[0167] The parameters, metrics or measures derived by processor 141 may be used for a number of purposes. The subject may be provided training to better use mobility aid 110. The subject’s fall risk may be determined, and appropriate interventions may be put in place. The size or configuration of mobility aid 110 may be modified or optimised. A physical therapist may be provided the data to assist the subject with gait training. The data could be used to assess whether the patient’s gait is improving or deteriorating over time. The data may also be used to assist in the control of electromechanically actuated devices, such as exoskeletons, or spinal cord stimulation. Such assistive devices are currently very limited in their application, but their robustness and usability may be improved by integrating appropriate measurements of balance into their control strategies.

[0168] In some embodiments, the system 100 may further include a client application that provides mechanisms for setting up and utilising the system 100. The client application may form part of, or be in communication with, processing device 140. The client application may be configured to facilitate the setup and use of the system 100, particularly the sensor module 120, as well as to control capture and analyse data for gait analysis. The client application may be configured to provide instructions on how to set up the system 100, such as guidance on attaching the sensor module 120 to the mobility aid 110, angling the sensor module 120 correctly, configuring the device settings, and ensuring that the system is ready for use. In some embodiments, the client application may be configured to capture data to use for gait analysis. For example, the client application may utilise the sensors 160 to monitor the movements of a foot with respect to the mobility aid 110. The client application may be configured to calculate the movement of at least one foot using the foot identification module 173 described herein and / or using a configurable version of the algorithm that may be adapted to different scenarios depending on the mobility aid, sensor module and the like. In some embodiments, the client application is configured to calculates one or more metrics related to the user’s gait. These metrics may include speed, step length, step width, and / or other parameters that provide insights into the user's gait and mobility. In some embodiments, the client application may be configured to track and report changes in these metrics over time. The application may be configured to maintain a historical record of the user's gait data, and / or generate reports that highlight trends, patterns and improvements over time.

[0169] It will be appreciated by persons skilled in the art that numerous variations and / or modifications may be made to the above-described embodiments, without departing from the broad general scope of the present disclosure. The present embodiments are, therefore, to be considered in all respects as illustrative and not restrictive.

Claims

CLAIMS:

1. A method for assessing a subject’s gait, the method comprising: accessing image data captured by an imaging sensor located on a mobility aid of the subject; identifying at least one foot of the subject in the image data; and determining a relative position and orientation of the at least one foot with respect to the mobility aid based on a position of the imaging sensor with respect to the mobility aid.

2. The method of claim 1, further comprising determining an absolute position and orientation of the at least one foot with respect to an environment based on a position of the imaging sensor with respect to the environment.

3. The method of claim 2, further comprising fusing the image data with additional sensor data, wherein determining the relative position and orientation of the at least one foot with respect to the mobility aid is based on both the position of the foot in the image data and the additional sensor data.

4. The method of claim 3, wherein the additional sensor data comprises motion sensor data.

5. The method of claim 3 or claim 4, wherein the additional sensor data comprises accelerometer data.

6. The method of any one of claims 3 to 5, wherein the additional sensor data comprises gyroscopic data.

7. The method of any one of claims 3 to 6, wherein the additional sensor data comprises further imaging sensor data.

8. The method of claim 7, wherein the further imaging sensor data comprises depth image data.

9. The method of claim 7 or claim 8, wherein the further imaging sensor data comprises colour image data.

10. The method of any one of claims 7 to 9, wherein the further imaging sensor data is generated by a second imaging sensor located on a second mobility aid of the subject.

11. The method of any one of claims 1 to 10, further comprising deriving at least one gait parameter of the subject based on the determined relative position of the foot.

12. The method of claim 11, wherein the at least one gait parameter is derived using stored relative position data together with the determined relative position of the foot.

13. The method of claim 11 or claim 12, wherein deriving the at least one gait parameter comprises calculating at least one surface contact time point at which the foot contacted a contact surface.

14. The method of any one of claims 11 to 13, wherein the gait parameter comprises at least one of a step size, step width, step height or step length.

15. The method of any one of claims 11 to 14, wherein the gait parameter comprises at least one of a step frequency, gait speed; single or dual stance time, gait asymmetry and / or stance asymmetry.

16. The method of any one of claims 11 to 15, further comprising updating a gait model based on at least one gait parameter.

17. The method of claim 16, further comprising generating a predicted next position of the at least one foot based on the gait model.

18. The method of any one of claims 1 to 17, wherein determining the position or orientation of the foot comprises processing the image data using afoot localisation algorithm.

19. The method of claim 18, wherein the foot localisation algorithm comprises: determining an approximate location of the foot by generating a region of interest in the image data; identifying a principal axis of the foot using a computer vision method; identifying features of the foot based on at least the geometry of the foot;projecting the features of the foot onto the image data to determine the location of the features in three-dimensional space; and defining the position and orientation of the foot using the location of the features.

20. The method according to claim 19, wherein the computer vision method is thresholding and / or polygon approximation.

21. The method of any one of claims 1 to 20, wherein the image data comprises colour image data.

22. The method of any one of claims 1 to 21, wherein the image data comprises depth image data.

23. The method of any one of claims 1 to 22, wherein the mobility aid is a nonwheeled mobility aid.

24. A device comprising: a mobility aid for providing support to a subject; and an imaging sensor secured to the mobility aid and positioned to capturing image data of at least one foot of the subject; wherein the imaging sensor is configured to communicate the image data to a processing device to identifying at least one foot of a subject in the image data and determining a relative position and orientation of the at least one foot with respect to the mobility aid based on a position of the imaging sensor with respect to the mobility aid.

25. A system comprising: mobility aid device; and a computing device for performing a method to identifying at least one foot of a subject in image data and determining a relative position and orientation of the at least one foot with respect to the mobility aid based on a position of an imaging device with respect to the mobility aid.

Images