Scanning device with imaging and pressure sensing functionality

Abstract

Embodiments of the present disclosure relate to a scanning device comprising a support base, a pressure panel disposed on an upper surface of the support base, and a plurality of cameras distributed about an outer perimeter of the support base.

Description

[0001] Related applications are cross-referenced

[0002] This application claims priority to U.S. non-provisional patent application 17 / 143878, filed January 7, 2021; U.S. provisional patent application 63 / 108067, filed October 30, 2020; and U.S. provisional patent application 62 / 986502, filed March 6, 2020, the entirety of which are incorporated herein by reference. Technical Field

[0003] This disclosure relates to the field of custom orthopedic devices, and more specifically, to scanning equipment related to the manufacture of custom orthopedic devices or the recommendation of orthopedic devices. Background Technology

[0004] Foot problems and the associated costs of foot care are significant in the United States and elsewhere. If a foot problem interferes with a particular activity, it can result in lost hours of work. Foot problems can be caused by medical conditions, work conditions requiring standing or walking, physical activity, and more. Therefore, foot problems stem from medical conditions, work activities, or leisure activities.

[0005] Foot orthotics is the field related to the design, manufacture, fitting, and modification of footwear, orthotics, and foot devices designed to help alleviate foot pain or disability. The goal of foot orthotics is to provide protection and comfort to consumers / patients, primarily through the development of orthotic devices that reduce pressure in areas of maximum impact. Recently, additive manufacturing techniques have been used to create custom orthotics or insoles, replacing traditional subtractive manufacturing techniques and injection molding. Technologies such as pressure sensing or imaging have been used to calculate three-dimensional (3D) models of the foot, which serve as the basis for generating custom orthotics or recommended pre-fabricated orthotics through additive manufacturing. Summary of the Invention

[0006] The following abstract provides a brief summary of various aspects of this disclosure to provide a basic understanding of these aspects. This abstract is not a comprehensive overview of the disclosure. Its purpose is neither to identify key or essential elements of the disclosure, nor to describe any scope of any particular embodiment of the disclosure or any scope of the claims. Its purpose is to present some concepts of the disclosure in a simplified form as a prelude to the more detailed description that follows.

[0007] Some aspects of the present disclosure relate to scanning devices having imaging and / or pressure sensing functionality. In one aspect, a scanning device includes a support base including a generally planar upper surface and a lower surface, a pressure panel disposed on the upper surface of the support base, a plurality of cameras distributed about an outer perimeter of the support base and oriented substantially toward a center of the pressure panel, and a data processing device operatively coupled to the pressure panel and each of the plurality of cameras. In some embodiments, the processing device is configured to activate and receive data generated by the pressure panel and each of the plurality of cameras.

[0008] In some embodiments, the plurality of cameras distributed about the outer perimeter of the support base are equidistant from the center of the pressure panel.

[0009] In some embodiments, the outer perimeter of the support base is a circular perimeter. In some embodiments, a total number of the plurality of cameras is four. In some embodiments, the four cameras are unevenly distributed about the circular perimeter. In some embodiments, a first distance between a first camera and a second camera is equal to a second distance between a third camera and a fourth camera. In some embodiments, a third distance between the first camera and the third camera is less than a fourth distance between the second camera and the fourth camera. In some embodiments, the positions of the cameras define a walk path through a central region of the scanning device.

[0010] In some embodiments, at least one of the plurality of cameras includes a depth sensor configured to capture depth data during which its respective camera captures images.

[0011] In some embodiments, the pressure panel includes a plurality of pressure sensors arranged in a planar configuration. In some embodiments, each of the plurality of pressure sensors is configured to generate a signal representative of plantar pressure when a personal foot is in contact with the pressure panel when the pressure panel is activated, the signals collectively defining a two-dimensional pressure map of the personal foot.

[0012] In some embodiments, the processing device is configured to generate a three- dimensional reconstruction of the personal foot based on data captured by the pressure panel and each of the plurality of cameras when the personal foot is in contact with the pressure panel.

[0013] In some embodiments, the processing device is configured to transmit data generated by the pressure panel and each of the plurality of cameras to a processing server to generate a three-dimensional reconstruction of the personal foot and / or data describing an orthotic device customized according to the individual’s anatomy.

[0014] In some embodiments, the processing device is configured to capture dynamic gait data of the individual's foot as the individual's foot steps onto and / or off of the pressure panel.

[0015] In another aspect, a method includes: capturing a two-dimensional pressure map of an individual's foot as the individual stands on a pressure panel; capturing images of the individual's foot by a plurality of cameras arranged around the pressure panel; and computing a three-dimensional reconstruction of the individual's foot based on the two-dimensional pressure map and the captured images.

[0016] In some embodiments, the method further includes: capturing a series of two-dimensional pressure maps of the individual's foot as the individual's foot steps onto and / or off of the pressure panel.

[0017] In some embodiments, the method further includes: generating data describing an orthotic device based on the three-dimensional reconstruction of the individual's foot; and sending the data describing the orthotic device to a manufacturing device to manufacture the orthotic device.

[0018] In some embodiments, computing the three-dimensional reconstruction of the individual's foot includes: evolving a surface representing the individual's foot using depth data from the captured images by minimizing a fitting score using partial differential equations (PDEs); and applying a penalty to the fitting score based on a mismatch between the two-dimensional pressure map boundary and a boundary of the surface.

[0019] In another aspect, a scanning device includes: a support base including a generally planar upper surface and a lower surface; a plurality of cameras distributed around an outer perimeter of the support base, and a processing device operatively coupled to each of the plurality of cameras. In some embodiments, the plurality of cameras are configured to rotate around the outer perimeter while remaining oriented toward a center of the support base. In some embodiments, the processing device is configured to activate and receive data generated by each of the plurality of cameras.

[0020] In some embodiments, the processing device is configured to cause the plurality of cameras to capture images while rotating around the outer perimeter of the support base.

[0021] In some embodiments, the scanning device further includes: a pressure panel disposed on the upper surface of the support base. In some embodiments, the processing device is operatively coupled to the pressure panel and configured to activate and receive data generated by the pressure panel.

[0022] In another aspect, the scanning device of any of the above-described scanning devices is adapted to perform any of the above-described methods. BRIEF DESCRIPTION OF DRAWINGS

[0023] For a more complete understanding of the present disclosure, reference is now made to the following descriptions taken in connection with the accompanying drawings in which like numerals represent like elements. These drawings should not be construed as limiting the present disclosure, but are intended to be merely examples.

[0024] Figure 1 An exemplary system architecture is shown in accordance with embodiments of the present disclosure.

[0025] Figure 2A A perspective view of an exemplary scanning device is shown in accordance with embodiments of the present disclosure.

[0026] Figure 2B A side view of an exemplary scanning device is shown in accordance with embodiments of the present disclosure.

[0027] Figure 2C A top view of an exemplary scanning device is shown in accordance with embodiments of the present disclosure.

[0028] Figure 3A An exemplary contour plot of plantar pressure presented by a user interface is shown in accordance with embodiments of the present disclosure.

[0029] Figure 3B An exemplary contour plot of plantar pressure obtained from dynamic gait analysis measurements is shown in accordance with embodiments of the present disclosure.

[0030] Figure 4A A top surface of a 3D model of a person's foot modeled using an evolutionary process that utilizes depth images is shown in accordance with embodiments of the present disclosure.

[0031] Figure 4B A bottom surface of a 3D model of a person's foot modeled using an evolutionary process that incorporates plantar pressure and depth images is shown in accordance with embodiments of the present disclosure.

[0032] Figure 5 is a flowchart showing a method of scanning a person's single foot or both feet in accordance with embodiments of the present disclosure.

[0033] Figure 6 is a block diagram showing an exemplary computer system used in accordance with embodiments of the present disclosure. DETAILED DESCRIPTION

[0034] Embodiments of a scanning device are described herein that are capable of capturing a two-dimensional pressure map of a person's foot using a pressure panel while simultaneously capturing images of the user's foot from different angles using multiple cameras. The scanning device is also capable of performing dynamic gait analysis by capturing a series of pressure maps of the plantar pressure as the person's foot steps onto and / or off of the scanning device. In at least one embodiment, the scanning device or a separate device performs a 3D reconstruction of the person's foot based on the pressure maps (representing the plantar surface) and images captured at different angles (representing a top view, front view, side view, and back view of the foot).

[0035] In at least one embodiment, the cameras are evenly distributed around the perimeter of the scanning device. In at least one embodiment, the cameras are unevenly distributed. For example, in an embodiment that uses only four cameras, the cameras can be arranged to define the four corners of a rectangle while being oriented toward the center of the pressure panel (i.e., toward the person's foot).

[0036] In at least one embodiment, the scanning device can perform gait analysis by capturing pressure data of the user's foot at intervals of, for example, 5-10 seconds as the user enters, traverses, and / or walks out of the scanning device. The data can be processed to generate a video showing the change in plantar pressure over time.

[0037] Certain embodiments of the present disclosure also relate to a method of generating a 3D surface representing a foot using geometric partial differential equations. The method can advantageously compute a 3D model using depth images obtained from the cameras and plantar pressure maps, thereby accounting for the plantar surface that the cameras cannot see.

[0038] In the following description, reference is made to the analysis of a person's foot for the manufacture of orthotic devices. It should be understood that the embodiments described herein are not limited to use in any one particular application and that modifications can be made to the disclosed embodiments without departing from the spirit and scope of the present disclosure. Although the present disclosure has been described in the context of foot orthotics, those of ordinary skill in the art will recognize that the usefulness of the present disclosure is not so limited and that the present disclosure can be beneficially implemented in the manufacture of orthotic devices for other body parts as well. Furthermore, the embodiments described herein are not limited to measurements that require both imaging and pressure sensing. Embodiments that use only imaging or only pressure sensing are also contemplated.

[0039] System Architecture

[0040] An example implementation of an embodiment of the present disclosure is now described. Figure 1 An example system architecture 100 according to an embodiment of the present disclosure is shown. The system architecture 100 includes a scanning device 200, a data processing server 120, a client device 130, and a data store 140, each of which are communicatively coupled by a network 105. One or more of the devices in the system architecture 100 can use the techniques described herein with respect toFigure 6 The described general-purpose computer system 600 is implemented to perform the functions of the system architecture 100. The devices of the system architecture 100 are merely illustrative and it should be understood that other scanning devices, user devices, data processing servers, data stores, and networks can exist.

[0041] In one embodiment, the network 105 can include a public network (e.g., the Internet), a private network (e.g., a local area network (LAN) or wide area network (WAN)), a wired network (e.g., Ethernet network), a wireless network (e.g., an 802.11 network or a Wi-Fi network), a cellular network (e.g., a Long Term Evolution (LTE) network), routers, hubs, switches, server computers, and / or a combination thereof. Although the network 105 is described as a single network, the network 105 can include one or more networks that operate independently of, or in cooperation with, each other. The network 105 can utilize one or more protocols of one or more devices with which it communicates.

[0042] In one embodiment, the scanning device 200 includes a support base including a generally planar upper and lower surface, a pressure panel disposed on the upper surface of the support base, and a plurality of cameras distributed about the outer perimeter of the support base and oriented substantially toward the center of the pressure panel. In at least one embodiment, the scanning device 200 further includes an on-board processing device operatively coupled to the pressure panel and each of the plurality of cameras. The processing device can be configured to activate and receive data generated by the pressure panel and each of the plurality of cameras. The scanning device 200 is described in greater detail with reference to Figures 2A-2C

[0043] In one embodiment, the data processing server 120 can include one or more computing devices (e.g., rack-mounted servers, router computers, server computers, personal computers, mainframes, laptop computers, tablet computers, desktop computers, etc.), data stores (e.g., hard disks, memory, databases), networks, software components, and / or hardware components from which digital content can be retrieved. In at least one embodiment, the data processing server 120 can be a server used by the scanning device 200, such as for processing generated personal anatomical scan data. In at least one embodiment, additional data processing servers can exist. In at least one embodiment, the data processing server 120 utilizes the modeling component 122 to generate and reconstruct 3D model data from data received from the scanning device 200, the functions of which are described in greater detail with reference to Figure 5

[0044] ​​In one embodiment, client device 130 can include a computing device, such as a personal computer (PC), a laptop computer, a mobile phone, a smart phone, a tablet computer, a netbook computer, etc. A single user can be associated with (e.g., own and / or operate) client device 130. As used herein, a “user” can be represented as a single individual. However, other embodiments of the present disclosure include a “user” that is an entity controlled by a group of users and / or automated sources. For example, a group of individual users that collectively are a community in a corporate or government organization can be considered a “user.” In at least one embodiment, the user is the individual that is the subject of a scan by scanning device 200. In at least one embodiment, the user is an operator, technician, or physician that is using scanning device 200 to perform or assist in a scan of another person.

[0045] Client device 130 can use one or more local data stores, which can be internal or external to the device, and each of which can include one or more short-term memory (e.g., random access memory), cache, drive (e.g., hard drive), flash drive, database system, or other type of component or device capable of storing data. A local data store can also include multiple storage components (e.g., multiple drives or multiple databases), which can also span multiple computing devices (e.g., multiple server computers). In at least one embodiment, a local data store can be used for data backup or archival purposes.

[0046] Client device 130 can implement a user interface 132 that can allow client device 130 to send / receive information to / from other client devices, scanning device 200, data processing server 120, and data store 140. User interface 132 can be a graphical user interface (GUI). For example, user interface 132 can be a web browser interface that can access, retrieve, display, and / or navigate content (e.g., web pages, such as hypertext markup language (HTML) pages) provided by data processing server 120. In one embodiment, user interface 132 can be a standalone application (e.g., a mobile “app,” etc.) that allows a user to send / receive information to / from other client devices, scanning device 200, data processing server 120, and data store 140 using client device 130.

[0047] In one embodiment, data store 140 can include one or more short-term memory (e.g., random access memory), a cache, a drive (e.g., a hard drive), a flash drive, a database system, or another type of component or device capable of storing data. Data store 140 can also include multiple storage components (e.g., multiple drives or multiple databases), which can also span multiple computing devices (e.g., multiple server computers). In at least one embodiment, data store 140 can be cloud-based. One or more devices of system architecture 100 can utilize their own storage and / or data store 140 to store public and private data, and data store 140 can be configured to provide secure storage for private data. Such private data can include, for example, data describing individuals scanned using scanning device 200, including names, contact information, biometric data, and scan data. In at least one embodiment, data store 140 can be used for data backup or archival purposes.

[0048] Although each of scanning device 200, data processing server 120, client device 130, and data store 140 are described in Figure 1 the singular, different components, these components can be implemented together in a single device, or networked in different combinations of multiple different devices operating together. In at least one embodiment, some or all of the functionality of data processing server 120 and / or data store 140 can be performed by scanning device 200, client device 130, or other devices. In one example embodiment, client device 130 can be proximate to or integrated with scanning device 200, for example as part of a scanning kiosk. In these embodiments, client device 130 can implement the functionality of modeling component 122, or can utilize data processing server 120 to implement some or all of the functionality of modeling component 112.

[0049] Scanning device embodiments

[0050] Figures 2A-2CVarious views of an exemplary scanning device 200 according to embodiments of the present disclosure are shown. The scanning device 200 includes a support base 202, a pressure panel 204, and a plurality of cameras 206 distributed around the support base 202. Each camera 206 may be configured to capture high-definition images (e.g., a single image or movie), and in at least one embodiment may include an infrared sensor for capturing depth data. In at least one embodiment, one or more of the cameras 206 may be stereo depth cameras. The scanning device 200 may have one or more onboard processing devices operatively coupled to the cameras 206 and the pressure panel 204, and may send activation signals to the respective components and control the timing of capturing, collecting signals, and sending signals to one or more external devices (e.g., data processing server 120, client device 130, etc.) for processing.

[0051] In at least one embodiment, one or more of the cameras 206 are housed within or mechanically coupled to a respective support arm 208. Each of the cameras 206 is mechanically coupled to or integrally formed with a support base via a support arm 208 (which is generally an L-shaped rigid member). In at least one embodiment, one or more of the support arms 208 are fixed in place, thereby placing the camera 206 in a fixed, immovable position. This facilitates optimization of the angle and distance for capturing foot images. In at least one embodiment, the position of each camera 206 can be adjusted along the circumference of the support base 202. For example, one or more of the support arms 208 may extend radially from the support base 202 and / or may rotate about the central axis of the support base 202 (e.g., slidably coupled to a track below the support base 202) and be adjusted to a specific azimuth angle. In at least one embodiment, one or more of the support arms 208 may be telescopic to adjust the vertical position of their respective cameras 206 relative to the support base 202.

[0052] In at least one embodiment, such as Figure 2C As shown, the position of camera 206 can define a travel path 214 through the support base 202. Figure 2C In the top view, the leftmost and rightmost support arms 208 can be horizontally separated by a distance (e.g., 24-36 inches) to allow an individual to walk onto the support base 202 and pressure panel 204 to enter the scanning device for static scanning or to perform dynamic gait analysis. For example, to perform a static scan of both feet, the user can... Figure 2C The bottom of the foot enters the scanning device, and its feet / body are rotated approximately 90 degrees. In at least one embodiment, camera 206 may be further separated to define additional walking paths (e.g., walking paths orthogonal to walking path 214).

[0053] In at least one embodiment, cameras 206 can be configured to rotate around the outer perimeter of support base 202 to perform image capture at different angles relative to a user’s single foot or both feet. Scanning device 200 can include a motorized coupling mechanism that allows support arms 208 to travel along a fixed track, or each of support arms 208 can be coupled to a motorized track. One or more cameras can be controlled to capture foot images at different angles as cameras 206 traverse the track. In at least one embodiment, fewer cameras 206 are used than all of the cameras shown, such as two or three cameras.

[0054] In at least one embodiment, pressure plate 204 includes a plurality of pressure sensors arranged in a planar configuration (e.g., in rows, in columns, or in other configurations) adapted to generate multicolor foot pressure readings. In at least one embodiment, pressure plate can be a Pedar® Pressure Plate (Aetrex Worldwide, Inc.) or a variant thereof that uses over 3700 pressure sensors, each sensor occupying an area of 0.25 cm 2 A method of generating a custom insole for footwear using information obtained from a personal foot pressure map is described in U.S. Patent 7493230, the entire disclosure of which is hereby incorporated by reference. Pressure measurements and capturing personal foot images and processing captured data can be performed using functionality similar to that described in U.S. Patents 9402567, 10417772, 10463257, and 10492712, the entire disclosures of which are hereby incorporated by reference.

[0055] In at least one embodiment, support base 202 includes a power button / power indicator light 210 for activating scanning device 200. In at least one embodiment, support base includes a panel 212 that can include a power input port and one or more ports for establishing a hardwired connection with a client device (e.g., client device 130) or a data processing server (e.g., data processing server 120). In at least one embodiment, scanning device 200 can be communicatively coupled with a client device or a data processing server through a wireless connection.

[0056] ​In one embodiment, an example process for performing a scan using the scanning device 200 includes first performing a static scan of an individual's single foot or both feet. For example, the individual can be instructed (e.g., via a display screen operatively coupled to the scanning device 200 or an intermediary device such as the client device 130 implementing the user interface 132) to step onto the pressure panel 204 with a single foot or both feet. In embodiments where the user steps onto the pressure panel 204 with one foot, upon completion of the scan of the first foot, the individual is instructed to place the other foot alone on the pressure panel 204. In at least one embodiment, the static scan includes measuring plantar pressure of the individual's single foot or both feet via the pressure panel 204 and capturing images and / or depth data of the individual's single foot or both feet via the camera 206. Upon completion of the static scan, the individual can be instructed to step out of the scanning device 200 and a dynamic gait analysis is performed by measuring changes in plantar pressure over time during the individual's movement. In at least one embodiment, the user can be instructed to enter and exit the scanning device 200, enter the scanning device 200 and remain stationary, or walk out of the scanning device 200 from a stationary position. In at least one embodiment, the dynamic gait analysis is performed prior to the static scan.

[0057] Figure 3A and 3B An example static or interactive display screen is shown that can be displayed by a display device operatively coupled to the scanning device 200 during and after a scan. For example, the client device 130 can utilize the user interface 132 to present the display screen. Figure 3A A measurement of plantar pressure is shown in the form of a contour plot 300 to display areas of low to high plantar pressure captured during a static scan of an individual's both feet via the pressure panel. Figure 3B A contour plot 350 generated from a dynamic gait analysis measurement is shown. The contour plot 350 can correspond to video frames generated from the measurement, each frame corresponding to a point in time at which plantar pressure data was measured. Overlaid on each frame are paths 352 and 354 that track the maximum pressure measured over time during the individual's step. In at least one embodiment, the user interface 132 provides a playback option 360 to view the video.

[0058] Geometric partial differential equations for evolving and fitting surfaces

[0059] In at least one embodiment, prior to the final meshing step, the 3D shape of each foot is reconstructed in the form of a smooth (non-triangulated) 3D surface. In at least one embodiment, the reconstructed generated data can be used for rendering and visualization purposes. In at least one embodiment, the smooth 3D surface is implicitly represented as a zero-level surface of a 3D scalar level-set function, which can be discretely sampled on a uniform Cartesian 3D grid. The 3D grid can represent a rectangular volume within a 3D space, within which the biped is positioned during image capture. Depth images can be captured from cameras (e.g., cameras 206), all of which are positioned outside the rectangular volume while pointing towards the volume itself. The spacing between grid samples (voxels) can be chosen to be, for example, 3 millimeters (or another suitable spacing), but the zero-level surface representing the reconstructed foot can be computed and extracted at sub-voxel resolution via trilinear interpolation between neighboring voxels.

[0060] In at least one embodiment, a rough initial voxelized estimate of the foot in the form of a solid volume composed of grid voxels (3D binary mask) is obtained using a depth carving method, e.g., by intersecting the depth hulls computed from each of the four calibrated depth cameras. The initial estimate of the model can be computed quickly, but, in at least one embodiment, the volume estimate of the model is biased low and not smooth due to the nature of the general depth carving method. Once the initial carved estimate is obtained, it can be converted to an isosurface representation by, for example, applying a signed distance transform to the binary mask. The generated 3D signed distance function can serve as a starting scalar level-set function that evolves according to a geometric partial differential equation (PDE) that discretizes to match the uniform 3D grid structure.

[0061] In at least one embodiment, the PDE evolution is computed to most effectively reduce a fitting score that minimizes the average depth between each surface point visible to a given camera and the depth value taken at that point by the same camera. The fitting score can also include a smoothing term that can penalize noisy, jagged, or otherwise non-smooth structures. The tradeoff between depth fidelity and smoothness of the reconstruction can be adjusted to obtain the desired smoothness along different parts of the reconstructed foot. In at least one embodiment, the evolution continues until the final shape of the foot converges to an optimal combination of smoothness and average depth mismatch for all cameras. The evolution can be computed prior to discretization according to a “gradient descent” PDE (described in detail below), while its final discrete implementation can take the form of an explicit update of the discretized PDE via finite differences. Figure 4A A 3D representation of an upper portion of a personal foot is shown after surface evolution and fitting based on depth images of the foot.

[0062] An exemplary variational model for PDE-based foot reconstruction from depth images will now be described in more detail. The variational model utilizes the back-projection between each depth image, the estimated foreground surface S (the individual’s single foot or both feet) and the estimated background plane B (the support base 202 or pressure panel 204). The camera projection of a 3D point X into the corresponding two-dimensional (2D) pixel at location u in the i-th camera can be denoted as p. i The mapping is unique in the forward direction (from point X, 3D to 2D), but not well-defined in the backward direction (from u i , 2D to 3D) because there is a complete 3D ray passing through the camera center and the associated image pixel u i (can be viewed as a 3D point inside the focal plane of the i-th camera). Backtracking along this ray from the camera until it intersects with the foreground surface S or the background plane B, the first intersection can be defined as the unique 3D back-projection point X associated with the camera image pixel u i . The camera projections in the forward and backward directions can be denoted as u i = p i (X) and X = p i -1 (u i ).

[0063] In at least one embodiment, given the estimated foreground surface S and background plane B, the estimated depth image value d i for each camera pixel u This can be achieved by measuring the depth between each back-projected 3D point X and the camera center according to the following equation:

[0064]

[0065] In at least one embodiment, a weighted residual depth error for each camera can be formulated, weighted by a function This error is achieved by penalizing the difference between the measured depth image value d i and the back-projected depth value according to the following equation:

[0066]

[0067] where h > 0 is chosen to increase for positive parameters and decrease for negative parameters according to the following formula:

[0068]

[0069] In this way, the penalty is always increased as the depth discrepancy absolute value grows (whether the discrepancy represents an overestimation or an underestimation).

[0070] In at least one embodiment, the continuous limit of the weighted residual depth error is in integral form as the camera image resolution increases, which allows a resolution-independent cost function to be expressed as:

[0071]

[0072] In at least one embodiment, the continuum cost function (representing the fitting score) can be defined by summing over each depth image and adding a surface smoothness function that penalizes the surface area of the reconstructed surface S according to the following equation:

[0073]

[0074] where λ > 0 is a weighting factor that can be tuned to balance the required smoothness and achieve the desired consistency between measured depth values d i and reconstructed depth values One important advantage of this approach is the ability to naturally integrate the depth discrepancies and smoothness into a single continuous error function, which helps to avoid various complications associated with traditional point cloud stitching methods for 3D surface reconstruction.

[0075] In at least one embodiment, a continuous PDE (e.g., a gradient descent PDE) can be constructed to describe the evolution of the surface S, thereby minimizing the integrated cost function e as quickly as possible. In at least one embodiment, the gradient descent PDE is computed as follows:

[0076]

[0077] where κ represents the unit normal to the surface, χ i represents the visibility indicator for camera i (0 or 1), X i = (X i , Y i , Z i ) represents a 3D point in the camera reference coordinate system (as opposed to the inertial coordinate system), b i represents the background plane depth relative to camera i, and N represents the outward unit normal to the surface.

[0078] The gradient descent PDE is discretized using finite differences according to the chosen 3D grid resolution of the reconstructed surface and can be iteratively evolved through a fixed number of explicit forward Euler update steps (applied to an initial voxelized depth carving estimate of the foot surface). This allows for refinement of the sub-voxel estimates of the surface before final meshing and visualization.

[0079] Another advantage of the iterative PDE approach (beyond the direct control of smoothness already discussed above) is that it can also directly integrate additional information about the plantar surface not visible in any of the four depth cameras, through the pressure map images obtained via sensors underneath the board (e.g., pressure plate 204) stepped on by the foot. In at least one embodiment, by choosing the bottom of the 3D Cartesian grid to correspond to the top of the board, the reconstructed foot surface can remain open (in the form of a hole) along the higher pressure portion of the 3D grid boundary, corresponding to the location of direct contact between the board and the foot. This hole boundary can then be compared to the boundary of the 2D region shown in the measured pressure map. The mismatch between these two boundaries can then be added to the surface fitting score, which in turn changes the behavior of the computed gradient descent PDEs, now balancing not only the average depth and surface smoothness, but also the matching pressure boundary shape of the plantar surface. One significant advantage of the PDE approach over other point cloud or finite element methods is the ability to mix all three of these factors without artifacts in the final fitted 3D surface.

[0080] Figure 5 is a flowchart illustrating a method 500 of scanning a single foot or a double foot of a person, according to embodiments of the present disclosure. The method 500 can be performed by processing logic that comprises hardware (e.g., circuitry, dedicated logic, programmable logic, microcode, etc.), software (e.g., instructions run on a processing device to simulate the hardware), or a combination thereof. In at least one embodiment, the method 500 is performed by a processing device of a data processing server 120 implementing a modeling component 122 that sends signals to a scanning device 200 to manage data capture. In at least one embodiment, portions or all of the functionality of the modeling component 122 is distributed between the scanning device 200, the data processing server 120, and / or the client device 130.

[0081] The method 500 begins at block 510, where the processing device captures (e.g., directly through the scanning device 200 or through the scanning device 100 under control of the data processing server 120) a two-dimensional pressure map of a single foot (or double foot) of a person as the person stands on a pressure plate (e.g., pressure plate 204).

[0082] At block 520, the processing device captures images of the single foot (or double foot) of the person through a plurality of cameras (e.g., cameras 206) arranged around the pressure plate. In at least one embodiment, the cameras are depth cameras that capture images containing depth data. In some implementations, the processing device captures a series of two-dimensional pressure maps of the individual’s foot as the individual steps on and / or off the pressure plate (e.g., dynamic gait analysis measurements).

[0083] At block 530, the processing device computes a three-dimensional reconstruction of the individual's foot based on the two-dimensional pressure map and the captured image (e.g., as described herein with respect to gradient descent PDE and associated fit score). In at least one embodiment, the processing device utilizes the PDE to evolve a surface representative of the individual's foot using depth data from the captured image by minimizing a fit score, and applies a penalty to the fit score based on a mismatch between the two-dimensional pressure map boundary and the surface boundary.

[0084] In at least one embodiment, the processing device generates data describing an orthotic device based on the three-dimensional reconstruction of the individual's foot, e.g., by generating a shape that matches the plantar surface represented by the three-dimensional reconstruction. In at least one embodiment, the processing device generates a recommendation for a pre-made orthotic device based on various features represented by or derivable from the three-dimensional reconstruction (e.g., a size of the shoe, a height of an arch, a width of a heel, or other features as would be appreciated by one of ordinary skill in the art). In at least one embodiment, the processing device sends the data describing the orthotic device to a manufacturing device to manufacture the orthotic device.

[0085] Example computer system embodiments

[0086] Figure 6 A schematic diagram of a machine in the example form of a computer system 600 is shown, in which a set of instructions (for example, software) can be executed to cause the machine to perform any one or more of the methodologies discussed herein. In alternative embodiments, the machine can be connected (e.g., networked) to other machines in a LAN, an intranet, an extranet, or the Internet. The machine can operate in the capacity of a server or a client machine in client-server network environments, or it can act as a peer machine in peer-to-peer (or distributed) network environments. The machine can be a personal computer (PC), a tablet computer, a set-top box (STB), a personal digital assistant (PDA), a cellular telephone, a web appliance, a server, a network router, switch or bridge, or any machine capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by that machine. Further, while only a single machine is illustrated, the term "machine" shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein. Portions or all of the components of the computer system 600 can be used by or illustrative of at least some of the devices of the system architecture 100, such as the scanning device 200, the data processing server 120, the client devices 130, and the data store 140.

[0087] The example computer system 600 includes a processing device (processor) 602, a main memory 604 (e.g., read-only memory (ROM), flash memory, dynamic random access memory (DRAM) such as synchronous DRAM (SDRAM) or Rambus DRAM (RDRAM), etc.), a static memory 606 (e.g., flash memory, static random access memory (SRAM), etc.), and a data storage device 620, which communicate with each other via a bus 610.

[0088] Processor 602 represents one or more general-purpose processing devices such as a microprocessor, central processing unit, or the like. More particularly, the processor 602 can be a complex instruction set computing (CISC) microprocessor, reduced instruction set computing (RISC) microprocessor, very long instruction word (VLIW) microprocessor, or a processor implementing other instruction sets or processors implementing a combination of instruction sets. The processor 602 can also be one or more special-purpose processing devices such as an ASIC, a field programmable gate array (FPGA), a digital signal processor (DSP), network processor, or the like. The processor 602 is configured to execute the instructions 626 for performing the operations and steps discussed herein, such as the operations associated with the modeling component 122.

[0089] The computer system 600 can further include a network interface device 608. The computer system 600 also can include a video display unit 612 (e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT), or a touch screen), an alphanumeric input device 614 (e.g., a keyboard), a cursor control device 616 (e.g., a mouse), and / or a signal generation device 622 (e.g., a speaker).

[0090] The power device 618 can monitor the power level of a battery used to power the computer system 600 or one or more components thereof. The power device 618 can provide one or more interfaces to provide an indication of the power level, a window of time remaining before the computer system 600 or one or more components thereof is shut down, a rate of power consumption, an indication of whether the computer system is using an external power source or a battery power source, and other power-related information. In at least one embodiment, the indications related to the power device 618 can be accessed remotely (e.g., a remote backup management module can be accessed via a network connection). In at least one embodiment, the battery used by the power device 618 can be an uninterruptible power supply (UPS) that is local or remote to the computer system 600. In these embodiments, the power device 618 can provide information regarding the power level of the UPS.

[0091] The data storage device 620 can include a computer-readable storage medium 624 on which is stored one or more sets of instructions 626 (e.g., software) embodying any one or more of the methodologies or functions described herein. The instructions 626 can also reside, completely or at least partially, within the main memory 604 and / or within the processor 602 during execution thereof by the computer system 600, the main memory 604 and the processor 602 also constituting computer- readable storage media. The instructions 626 can further be transmitted or received over a network 630 (e.g., the network 105) via the network interface device 608.

[0092] In one embodiment, the instructions 626 include instructions to operate or process data generated by the scanning device 200, as described throughout this disclosure. While the computer-readable storage medium 624 is shown in an example embodiment to be a single medium, the

[0093] In the description above, numerous specific details are set forth. However, it is understood that embodiments of the disclosure can be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form rather than in detail in order to avoid obscuring the disclosure.

[0094] Some portions of this detailed description are presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is here, and generally, conceived to be a self- consistent sequence of steps leading to a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.

[0095] It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise as apparent from the above discussion, it is appreciated that throughout the description, discussions utilizing terms such as "configuring" "receiving" "transforming" "causing" "streaming" "applying" "masking" "displaying" "retrieving" "transmitting" "computing" "generating" "adding" "subtracting" "multiplying" "dividing" "selecting" "resolving" "optimizing" "calibrating" "detecting" "storing" "executing" "analyzing" "determining" "enabling" "identifying" "modifying" "converting" "aggregating" "extracting" "running" "scheduling" "processing" "capturing" "evolving" "fitting" or the like, refer to the action and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical quantities (e.g., electronic

[0096] The present disclosure also relates to an apparatus, device, or system for performing the operations herein. This apparatus, device, or system can be specially constructed for the required purposes, and include a general purpose computer selectively activated or reconfigured by a computer program stored in the computer. Such a computer program can be stored in a computer or machine readable storage medium, such as, but not limited to, any type of disk including floppy disks, optical disks, CD-ROMs and magnetic-optical disks, read-only memories (ROMs), random access memories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, or any type of media suitable for storing electronic instructions.

[0097] The word “example” or “exemplary” is used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as “example” or “exemplary” is not necessarily to be construed as preferred or advantageous over other aspects or designs. Rather, use of the words “example” or “exemplary” is intended to present concepts in a concrete fashion. As used in this application, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or.” That is, unless specified otherwise, or as is clear from the context, the phrase “X includes A or B” is intended to mean any of the natural inclusive permutations. That is, if X includes A; X includes B; or X includes both A and B, then “X includes A or B” is satisfied under any of the foregoing instances. In addition, the use of the term “a” and “an” to refer to a singular entity is taken to mean “one or more” or “at least one” unless otherwise indicated by context. Reference throughout this specification to “certain embodiments,” “an embodiment,” “one embodiment,” or “at least one embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of the phrases “in certain embodiments,” “in an embodiment,” “in one embodiment,” or “in at least one embodiment” or the like in various places throughout this specification are not necessarily referring to the same embodiment. Furthermore, the particular features, structures, or characteristics can be combined in any suitable manner in one or more embodiments.

[0098] The scope of the disclosure is not limited to the specific embodiments described herein. In fact, other various embodiments and modifications of the disclosure, apart from those described herein, will be apparent from the description and drawings to those skilled in the art. Therefore, such other embodiments and modifications are intended to fall within the scope of the disclosure. Furthermore, although the disclosure is described in the context of particular embodiments for particular applications, those skilled in the art will recognize that its usefulness is not limited thereto and the disclosure can be beneficially implemented in any number of environments for any number of purposes. Accordingly, the claims set forth below are to be construed in accordance with the full scope of the disclosure as described herein and with the full scope of equivalents to which such claims are entitled.

Claims

1. A scanning device comprising: a support base comprising a generally planar upper surface and a lower surface; a pressure panel disposed on the upper surface of the support base; a plurality of cameras distributed around an outer perimeter of the support base and oriented substantially toward a center of the pressure panel; and a processing device operatively coupled to the pressure panel and each of the plurality of cameras, wherein the processing device is configured to activate and receive data generated by the pressure panel and each of the plurality of cameras when a person's foot is positioned on the pressure panel, wherein the processing device computes a three-dimensional reconstruction of the person's foot in the form of a three-dimensional surface, the three-dimensional reconstruction comprising an upper portion and a lower portion, wherein the upper portion is computed based on images captured by the plurality of cameras, and wherein the lower portion is computed based on a two-dimensional pressure map captured by the pressure panel to account for plantar surfaces that are not visible to the plurality of cameras, wherein computing the three-dimensional reconstruction of the person's foot comprises: evolving a surface representative of the person's foot using depth data from the captured images with partial differential equations (PDEs) by minimizing a fitting score; and applying a penalty to the fitting score based on a mismatch between boundaries of the two-dimensional pressure map and boundaries of the surface.

2. The scanning device of claim 1, wherein the plurality of cameras disposed around the outer perimeter of the support base are equidistant from the center of the pressure panel.

3. The scanning device of claim 1, wherein the outer perimeter of the support base is a circular perimeter.

4. The scanning device of claim 3, wherein a total number of the plurality of cameras is four.

5. The scanning device of claim 4, wherein the four cameras are unevenly distributed around the circular perimeter.

6. The scanning device of claim 5, wherein a first distance between a first camera and a second camera is equal to a second distance between a third camera and a fourth camera, and wherein a third distance between the first camera and the third camera is less than a fourth distance between the second camera and the fourth camera.

7. The scanning device of claim 3, wherein positions of the cameras define a walk path through a central region of the scanning device.

8. The scanning device of any one of claims 1-7, wherein at least one of the plurality of cameras comprises a depth sensor configured to capture depth data during which its respective camera captures images.

9. The scanning device of any one of claims 1-7, wherein the pressure panel comprises a plurality of pressure sensors arranged in a planar configuration.

10. The scanning device of claim 9, wherein each of the plurality of pressure sensors is configured to generate a signal representative of plantar pressure when a person's foot is in contact with the pressure panel when the pressure panel is activated, the signals collectively defining a two-dimensional pressure map of the person's foot. ​ 11. The scanning device of any one of claims 1-7, wherein the processing device is configured to generate a three-dimensional reconstruction of the individual’s foot based on data captured by the pressure panel and each of the plurality of cameras while the individual’s foot is in contact with the pressure panel.

12. The scanning device of any one of claims 1-7, wherein the processing device is configured to transmit data generated by the pressure panel and each of the plurality of cameras to a processing server to generate a three-dimensional reconstruction of the individual’s foot and / or data describing an orthotic device customized according to the individual’s anatomy.

13. The scanning device of any one of claims 1-7, wherein the processing device is configured to capture dynamic gait data of an individual’s foot as the individual’s foot steps onto and / or off of the pressure panel.

14. A method comprising: capturing a two-dimensional pressure map of an individual’s foot while the individual is standing on a pressure panel; capturing images of the individual’s foot by a plurality of cameras arranged around the pressure panel; and computing a three-dimensional reconstruction of the individual’s foot in the form of a three-dimensional surface based on the two-dimensional pressure map and the captured images, the three-dimensional reconstruction comprising an upper portion and a lower portion, wherein the upper portion is computed based on images captured by the plurality of cameras, and wherein the lower portion is computed based on the two-dimensional pressure map to account for plantar regions that are not visible to the plurality of cameras, wherein computing the three-dimensional reconstruction of the individual’s foot comprises: evolving a surface representative of the individual’s foot using depth data from the captured images with partial differential equations (PDEs) by minimizing a fitting score; and applying a penalty to the fitting score based on a mismatch between a boundary of the two-dimensional pressure map and a boundary of the surface.

15. The method of claim 14, further comprising: capturing a series of two-dimensional pressure maps of the individual’s foot as the individual’s foot steps onto and / or off of the pressure panel.

16. The method of claim 14 or claim 15, further comprising: generating data describing an orthotic device based on the three-dimensional reconstruction of the individual’s foot; and transmitting the data describing the orthotic device to a manufacturing device to manufacture the orthotic device.

17. A scanning device comprising: a support base comprising a generally planar upper surface and a lower surface; a plurality of cameras distributed around an outer perimeter of the support base and mechanically coupled to the support base, wherein the plurality of cameras are configured to rotate around the outer perimeter while remaining oriented toward a center of the support base; a pressure panel disposed on the upper surface of the support base; and ​ ​ a processing device operatively coupled to each of the plurality of cameras, wherein the processing device is configured to activate and receive data generated by each of the plurality of cameras when a person's foot is positioned on the pressure panel, the processing device is operatively coupled to the pressure panel and configured to activate and receive data generated by the pressure panel, the processing device is further configured to compute a three-dimensional reconstruction of the person's foot in the form of a three-dimensional surface, the three-dimensional reconstruction comprising an upper portion and a lower portion, wherein the upper portion is computed based on images captured by the plurality of cameras, and wherein the lower portion is computed based on a two-dimensional pressure map captured by the pressure panel to account for the sole of the foot that is not visible to the plurality of cameras, wherein computing the three-dimensional reconstruction of the person's foot comprises: evolving a surface representative of the person's foot using depth data from the captured images with partial differential equations (PDEs) by minimizing a fitting score; and applying a penalty to the fitting score based on a mismatch between a boundary of the two-dimensional pressure map and a boundary of the surface.

18. The scanning device of claim 17, wherein the processing device is configured to cause the plurality of cameras to capture images while rotating around the outer perimeter of the support base.

Images