Machine learning for generating three-dimensional meshes based on images

Abstract

Techniques are provided for improved machine learning. A set of two-dimensional images of a user is accessed. A three-dimensional mesh depicting the head of the user is generated based on processing the set of two-dimensional images using a machine learning model, where the three-dimensional mesh is scaled to a size of the head of the user. The three-dimensional mesh is modified to remove one or more facial expressions. A set of facial measurements is determined based on the modified three-dimensional mesh, and a user interface is selected for the user based on the set of facial measurements.

Description

[0001] Cross-reference to related applications This application claims the benefit and priority of U.S. Provisional Patent Application No. 63 / 565,412, filed March 14, 2024, and U.S. Provisional Patent Application No. 63 / 688,394, filed August 29, 2024, the entire contents of which are incorporated herein by reference. Background Technology

[0002] The embodiments of this disclosure generally relate to computer vision and machine learning. More specifically, the embodiments relate to using machine learning to generate 3D meshes based on image data.

[0003] In a wide variety of medical (and non-medical) settings, accurate facial or head measurements are relied upon to drive user decisions and choices. For example, in many cases, the specific dimensions of an individual user's face are required to help design, construct, and / or select an appropriate mask that will comfortably and completely fit the user's face. For instance, continuous positive airway pressure (CPAP) machines typically use a mask or nasal mask to provide a constant and stable air pressure to the user during sleep. However, for the system to function properly (and to improve user experience and health), it is important that the mask fits correctly (e.g., comfortably and without leakage around the face).

[0004] In some standard systems, users can travel to a physical environment (e.g., the office of a healthcare provider or mask distributor) to try on various masks. However, this requires the user's in-person presence, which is not always feasible due to factors such as the user's and / or provider's location or remoteness, travel ability, and the user's available time. Furthermore, users are limited to the available (pre-configured) mask sizes and often have to manually try on multiple masks to find the correct fit, which can be problematic when sterility is required.

[0005] In some standard systems, attempts have been made to measure or estimate a user's facial dimensions to drive mask selection or design. For example, some methods involve users measuring their own faces, such as using a ruler, coin, or other objects of known size. However, such methods have proven inaccurate and frustrating for users, leading to adverse outcomes. For instance, obtaining the wrong mask based on inaccurate measurements may result in the mask failing to perform its intended purpose, which could then affect the user's condition, treatment, and outcomes.

[0006] Additionally, some methods have attempted to improve manual measurements by using specialized equipment or systems, such as cameras configured to capture 3D data, including depth. These specialized devices are complex, expensive, and often unavailable. Furthermore, the fact that specialized equipment typically still requires the user to travel to the device's location (or physically bring the device to the user's location) significantly limits their use.

[0007] Improved methods and techniques are needed for determining facial measurements. Summary of the Invention

[0008] According to one embodiment presented in this disclosure, a method is provided. The method includes: accessing a set of two-dimensional images of a user; generating a three-dimensional mesh depicting the user's head based on processing the set of two-dimensional images using a first machine learning model, wherein the three-dimensional mesh is scaled to the size of the user's head; modifying the three-dimensional mesh to remove one or more facial expressions; determining a set of facial measurements based on the modified three-dimensional mesh; and selecting a user interface for the user based on the set of facial measurements.

[0009] Other embodiments provide: a processing system configured to perform the methods mentioned above and those described herein; a non-transitory computer-readable medium including instructions that, when executed by one or more processors of the processing system, cause the processing system to perform the methods mentioned above and those described herein; a computer program product embodied on a computer-readable storage medium and including code for performing the methods mentioned above and those further described herein; and a processing system including components for performing the methods mentioned above and those further described herein. Attached Figure Description

[0010] To gain a more detailed understanding of the features described above in this disclosure, a more specific description of the disclosure summarized above can be obtained by referring to the embodiments, some of which are illustrated in the accompanying drawings. However, it should be noted that the drawings illustrate exemplary embodiments only and should not be construed as limiting their scope; other equally effective embodiments are permissible.

[0011] Figure 1 An example workflow for generating a mesh and selecting a user interface according to some embodiments of this disclosure is described.

[0012] Figure 2 An example workflow for facilitating image data collection for improved mesh generation, according to some embodiments of this disclosure, is described.

[0013] Figure 3 An example workflow for improved interface selection based on a generated mesh, according to some embodiments of this disclosure, is described.

[0014] Figure 4 This is a flowchart depicting a method for generating a 3D mesh and selecting interface components using a machine learning model, according to some embodiments of the present disclosure.

[0015] Figure 5 This is a flowchart depicting a method for collecting and evaluating image data according to some embodiments of the present disclosure.

[0016] Figure 6A and Figure 6B A flowchart illustrating a method for collecting user image data according to some embodiments of the present disclosure is shown.

[0017] Figure 7 An example user interface for collecting user image data according to some embodiments of this disclosure is depicted.

[0018] Figure 8 This is a flowchart depicting a method for improved interface selection according to some embodiments of the present disclosure.

[0019] Figure 9 This is a flowchart depicting a method for selecting a user interface using machine learning, according to some embodiments of the present disclosure.

[0020] Figure 10 Example computing devices configured to perform various embodiments of the present disclosure are depicted.

[0021] For ease of understanding, the same reference numerals have been used where possible to identify common elements in the figures. It is conceivable that elements and features of one embodiment may be advantageously incorporated into other embodiments without further description. Detailed Implementation

[0022] Embodiments of this disclosure provide apparatus, methods, processing systems, and computer-readable media for selecting machine learning (ML)-based interfaces based on three-dimensional meshes.

[0023] In some embodiments, a measurement system is provided. The measurement system can be configured to use machine learning to evaluate a set of images captured via an imaging sensor (e.g., a webcam) to generate a three-dimensional mesh corresponding to a user's face depicted in the images. In some embodiments, such a mesh can be used to generate accurate biometric measurements. Advantageously, the system can use two-dimensional images to generate the three-dimensional mesh and does not require reliance on a complex three-dimensional imaging system. This allows the system to be implemented using a variety of widely available imaging sensors, including webcams, mobile device cameras, etc. In some embodiments, a deep learning model is trained end-to-end to generate a three-dimensional mesh with an appropriate size or proportion relative to the user's face. That is, the mesh can be scaled according to the size of the user's face, thereby ensuring that measurements acquired based on the mesh (e.g., nose size, mouth position, etc.) are accurate.

[0024] In some embodiments, the user can be instructed to move body parts (such as their face) within a certain range of motion while the imaging sensor captures images of the body parts from different angles. A trained machine learning model can then be used to evaluate the images to ensure they are satisfactory (e.g., to ensure the user's ears are visible in one or more images), thereby ensuring the generation of an accurate grid.

[0025] In some embodiments, the measurement system can use the generated mesh to perform various operations. In some embodiments, the measurement system can alter or deform the mesh to remove facial expressions (if any) depicted in the image. For example, if a user smiles, raises an eyebrow, or makes another expression, the system can modify the mesh to remove such expressions. A wide variety of facial expressions can lead to inaccurate facial measurements (e.g., inaccurate prediction of nostril size due to skin deformation when a user smiles). Therefore, by manipulating the 3D mesh to remove such expressions, the measurement system ensures that the captured measurement data is highly accurate.

[0026] In some embodiments, once a 3D facial mesh has been generated and appropriately processed (e.g., to remove facial expressions), the mesh can be used to estimate, compute, calculate, or otherwise determine a set of facial measurements of the user. The specific measurements collected may vary depending on the specific implementation and the task. For example, in some embodiments, the measurement system generates facial measurements to facilitate the selection and / or fitting of one or more devices or components designed to be worn on the user's head or face, such as a user interface (e.g., a mask) for respiratory therapy (e.g., CPAP).

[0027] For example, in some embodiments, the measurement system may determine measurements such as face height, nose width, nose depth, nostril size, etc. In various embodiments, the specific facial measurements determined and used may vary depending on the specific task (e.g., to allow determination of the appropriate size of a catheter (e.g., an over-the-top tube mask), the appropriate size of a headband, the nostril size of a pillow-type mask, etc.). As discussed in more detail below, these measurements can be used to select, design, customize, or otherwise retrieve facial devices or user interfaces (such as a properly fitting mask for the user) to ensure functionality, comfort, and stability.

[0028] Example workflow for generating a mesh and selecting a user interface Figure 1 An example workflow 100 for generating a mesh and selecting a user interface according to some embodiments of the present disclosure is described.

[0029] In the illustrated example, measurement system 110 accesses a set of images 105 and generates or selects an interface 150 (e.g., a recommendation or selection of interface 150) based on the images 105. As used herein, “accessing” data can generally include receiving, retrieving, requesting, obtaining, collecting, capturing, measuring data, or otherwise obtaining access to data. For example, in some embodiments, images 105 may be captured via one or more imaging sensors (e.g., a webcam or a camera on a smartphone) and may be transmitted to measurement system 110 via one or more communication links. In some embodiments, measurement system 110 is implemented as a cloud-based service that evaluates user images 105 to generate interface 150. For example, a user may use an application to capture images 105 on their local device (e.g., a user’s laptop or mobile phone) and then upload images 105 to measurement system 110 for evaluation. Typically, measurement system 110 may be implemented using hardware, software, or a combination of hardware and software. Furthermore, although illustrated as a discrete system for clarity of concept, in some embodiments, the operation of the measurement system 110 may be combined or distributed across any number and type of devices and systems.

[0030] In some embodiments, image 105 typically corresponds to a two-dimensional image depicting a user's head and / or face. In some embodiments, image 105 depicts the user from multiple angles or orientations. For example, image 105 may include a frontal image (e.g., captured when the user's face is directly facing the imaging sensor, such that the image depicts the user's face from the front), one or more side or lateral images (e.g., captured when the user turns their face toward the left and / or right of the imaging sensor, such that the image depicts the side of the user's face and / or the user's ears), a bottom image (e.g., captured when the user is looking upward relative to the imaging sensor, such that the image depicts the user's chin, neck, and / or nostrils), and / or a top image (e.g., captured when the user is looking downward relative to the imaging sensor, such that the image depicts the top of the user's head).

[0031] Although not depicted in the illustrated example, in some respects, the measurement system 110 may receive various other data, such as metadata associated with one or more images 105 (e.g., characteristics of the camera that captured the image 105, such as field of view (FOV) or focal length).

[0032] In the illustrated workflow 100, image 105 is accessed by image component 115. Image component 115 typically facilitates the collection and evaluation of image 105. The specific operations performed by image component 115 may vary depending on the specific implementation. For example, in some embodiments, image component 115 may perform various preprocessing operations, such as enhancing contrast, reducing noise, resizing the image, cropping the image, performing color correction on the image, performing feature extraction, etc. In some embodiments, image component 115 may evaluate one or more images of 105 to determine whether image 105 is suitable for mesh generation. For example, image component 115 may use one or more machine learning models to detect the presence (or absence) of various facial features or markers (such as a user's ears in a side view) useful in mesh generation. In some embodiments, if such markers are not visible, image component 115 may request additional images of 105 to improve the measurement process.

[0033] As illustrated, image component 115 provides image data 120 to mesh component 125. Image data 120 may correspond to or include image 105 itself, and / or may correspond to image 105 after preprocessing operations such as noise reduction or resizing have been applied. In some embodiments, image data 120 corresponds to or includes the result of feature extraction. That is, image data 120 may include a feature map generated for image 105.

[0034] Mesh component 125 processes image data 120 to generate mesh 130. In some embodiments, mesh component 125 uses one or more machine learning models to generate mesh 130. For example, mesh component 125 may use a deep learning model (e.g., a convolutional neural network) to generate mesh 130. In some embodiments, mesh component 125 uses a machine learning model to fit a statistical shape model representing a statistically average face to image data 120, thereby causing mesh 130 to depict or correspond to a user's face.

[0035] In some embodiments, the mesh component 125 uses a camera model to scale the mesh 130. For example, in some embodiments, the mesh component 125 may (e.g., based on metadata associated with image 105 and / or by processing the image itself) determine the field of view (FOV) of the camera used to capture image 105. In some embodiments, the perceived size of various facial markers may change as they move closer to or further away from the camera. In some embodiments, based on the perceived size changes of the markers (e.g., the user's head, or more subtle markers such as eyes or ears) the mesh component 125 may use the camera model to determine or infer the FOV of the camera and / or the distance between the camera and the markers. In some embodiments, using this information, the mesh component 125 may determine the proportions of a face or features within a face. For example, in determining the distance between the user's nose and the camera... Millimeter distance and the camera's FOV is After that, the grid component 125 can determine the actual size of the user's nose (e.g., in millimeters).

[0036] In some embodiments, the mesh component 125 may use a deep learning model to generate or scale a properly scaled mesh 130 instead of a camera model. For example, the model may be trained based on a facial paradigm to generate the mesh 130 in a way that inherently understands facial proportions without using a separate camera model (e.g., without explicitly determining or evaluating the camera's field of view). In some embodiments, the measurement system 110 (or another system) may use a relatively dense paradigm (such as images and corresponding meshes, or dense coordinates of facial markers) to train the model.

[0037] In some embodiments, some or all of the training data includes synthetic data. For example, various computer programs can be used to generate accurate 3D meshes or models of synthetic heads and / or faces (e.g., models of people that are not real individuals or users, but are still realistic). In some embodiments, measurement system 110 (or another system) can render images depicting the modeled head from various angles (e.g., by placing virtual cameras at various locations around the head in virtual space). These images can be used as training inputs to the model, paired with part or all of the mesh itself (e.g., data points in 3D space, such as various markers defining the face) that are used as the target output. In some embodiments, in addition to or instead of using synthetic data to train the model, the training data may include real data (e.g., real images of the user, and highly accurate 3D data points). For example, the user may voluntarily use a scanning device capable of capturing image data and 3D positioning data of their face.

[0038] However, such scanning devices are typically bulky, expensive, and difficult to use. Furthermore, scanning users' actual faces to train models can raise various privacy concerns. Experiments show that robust mesh generation can be provided during runtime even when training models entirely with synthetic data.

[0039] In the illustrated workflow 100, mesh 130 is a three-dimensional mesh that represents at least a portion of the user's head and face. For example, mesh 130 may depict the user's face, a portion of the user's neck, and / or a portion of the user's head (e.g., including the ears). Measurement component 135 accesses mesh 130. In the illustrated example, measurement component 135 generates a set of measurements 140 based on mesh 130.

[0040] In some embodiments, the measurement component 135 may apply one or more preprocessing operations before generating the measurement 140. For example, the measurement component 135 may alter or deform the mesh 130 to remove the user's facial expressions (if any), thereby obtaining a mesh that reflects the user's face with a neutral expression.

[0041] In embodiments, as discussed above, the specific measurements captured by the measuring component 135 may vary depending on the specific implementation and task. For example, in some aspects, the measuring component 135 may evaluate the grid 130 to determine features such as the user's nasal width, nasal height, and / or nasal depth. Such measurements may be useful for selecting or providing facial devices such as masks that cover the user's nose (e.g., respiratory therapy masks). As in some embodiments, for similar reasons, the measuring component 135 may measure the height and / or width of the user's mouth, and / or the positioning of the mouth relative to the nose.

[0042] In some embodiments, the measuring component 135 may determine the overall size of the user's head (e.g., the circumference of the user's head), which may be a useful indicator for determining the size of the catheter and / or headband (e.g., to select a catheter large enough to comfortably wrap around the user's head without being too large for discomfort, and / or to select a headband that will fit comfortably). As used herein, "headband" refers to a strap, band, or other component used to secure the user interface to the user's nose, mouth, or both. As used herein, "catheter" refers to a conduit that connects the user interface (e.g., a CPAP mask) to a respiratory therapy device (e.g., a flow generator) and provides airflow from the flow generator to the user via that interface. In some aspects, to facilitate the determination of catheter size, the measuring component 135 may determine the length of the catheter path along the user's face (e.g., along the path the catheter is designed to be placed, such as from the nose and / or mouth and upwards over each ear).

[0043] In some embodiments, to determine the catheter size, the measuring component 135 may construct or mark multiple points on the user's face and / or head (as reflected in the grid 130), at which the catheter should be placed during use. For example, the measuring component 135 may mark a point below the nose (e.g., in the middle of the user's philtrum), one or more points on each cheek (e.g., at a defined location relative to the nostrils or another facial feature (e.g., the temporal process of the zygomatic bone), a point between the ear and eye (e.g., at the midline between the user's left ear and left eye and at the midline between the user's right ear and right eye), a point on the top of the user's head (e.g., the highest point), etc. In some aspects, the measuring component 135 may similarly mark or place intermediate points between the aforementioned markers on the surface of the grid. The measuring component 135 may then connect the points to a spline (placed on the surface of the grid) and measure the length of the spline. In some embodiments, the length of the spline may then be used to determine or infer the appropriate catheter size, as discussed below.

[0044] In some embodiments, to determine the headband size, the measuring component 135 may construct or mark multiple points on the user's face and / or head (as reflected in the grid 130) in a manner similar to the conduit splines discussed above (where the specific points may differ). For example, the measuring component 135 may mark points located where the various straps or other components of the headband will fit against the user's head. The measuring component 135 may then construct splines connecting these points, as discussed above. In some aspects, the measuring component 135 may similarly generate multiple such splines along the grid surface (e.g., for each section or strap of the headband) to measure the length of each headband spline. In some embodiments, the lengths of these headband splines can then be used to determine or infer an appropriate headband size, as discussed below.

[0045] In some embodiments, the measurement component 135 may determine the size of a user's nostrils based on a grid 130. For example, the measurement component 135 may use four parameters defining an ellipse to characterize the user's nostrils: the major and minor axes of the ellipse; the degree of rotation of the nostril / ellipse relative to a fixed orientation (e.g., relative to a plane of the face); and the distance between the nostril / ellipse and the centerline of the grid 130 (e.g., the centerline of the user's face). While some of the examples discussed use ellipses to define nostril measurements, in some embodiments, the measurement component 135 may use various polygons with any number of sides to define the shape of the nostrils.

[0046] In the illustrated example, measurement 140 (also referred to as facial measurement in some embodiments) is accessed by selection component 145. Selection component 145 evaluates measurement 140 to select or generate interface 150. In some embodiments, selection component 145 may use one or more thresholds or mappings to evaluate one or more measurements in measurement 140 to select various components of interface 150. For example, based on the user's nose size and / or shape, selection component 145 may evaluate a predefined mapping indicating which interfaces will fit best or most comfortably. For example, selection component 145 may determine that a first nose-only mask may be too small to fit comfortably, a second nose-only mask will be too large (e.g., causing air leakage), and / or a third nose-only mask will fit well and comfortably. As another example, based on measurement 140, selection component 145 may determine that the user should use a specific type or model of full-face mask (e.g., a mouth-nose mask).

[0047] For example, in determining catheter size, selection component 145 may use rule-based or threshold-based methods to evaluate catheter spline length (discussed above) (e.g., selecting catheter size based on the range in which the spline length falls), and / or may use a machine learning model (e.g., a trained classifier) ​​to process the spline length to select catheter size. Similarly, in determining headband size, selection component 145 may use rule-based or threshold-based methods to evaluate headband spline length (discussed above) (e.g., selecting headband size based on the range in which the spline length falls), and / or may use a machine learning model (e.g., a trained classifier) ​​to process the spline length to select headband size.

[0048] In some embodiments, selection component 145 may evaluate some or all of the measurements 140 to select a nose pillow size for a user. A nose pillow is typically a soft insert that partially fits into the user's nostrils to provide airflow through the nostrils (unlike a nose mask which fits the nose, and a full-face mask which fits the nose and mouth). In some embodiments, selection component 145 uses a classifier machine learning model to select a pillow size based on nostril measurements. For example, the classifier may process measurements such as the major and minor axes of the nostrils, rotation, and / or distance to the center line to generate a classification indicating which size pillow will best fit the user. In some embodiments, the classifier may be a relatively small or simple machine learning model. Measurement system 110 (or another system) may use labeled examples to train the nostril classifier. For example, training data may include nostril measurements of one or more users (as discussed above), where the label for each training sample indicates the pillow size the user finds most comfortable (or otherwise best for results, such as minimal air leakage).

[0049] Typically, the generated or selected interface 150 may include a selection of various interface components, as discussed above. For example, interface 150 may indicate a recommended user interface style or design (e.g., nose only, full face, or nasal pillow type), a recommended interface type or size (e.g., from a set of alternative options), a determination of a recommended cannula size, a determination of a recommended headband size, a recommended pillow size (for nasal pillow type masks), etc.

[0050] In some embodiments, the measurement system 110 may delete the user image 105 and / or grid 130 after processing to protect user privacy. For example, in some embodiments, the measurement system 110 may delete the image 105 and image data 120 once the grid 130 is generated. Furthermore, the measurement system 110 may delete the grid 130 once the measurement 140 is generated. Additionally, in some embodiments, the measurement system 110 may delete the measurement 140 once the interface 150 is generated.

[0051] In some embodiments, the measurement system 110 may provide the selected interface 150 to the user depicted in image 105. In some embodiments, the interface 150 may be directed to another user (such as the healthcare provider of the depicted user) who may facilitate the ordering and / or delivery of the directed equipment. In these ways, the measurement system 110 may use machine learning to generate accurate three-dimensional meshes and then evaluate these meshes to select or recommend equipment for respiratory therapy in a highly refined manner. This can improve the outcomes achieved by the user (e.g., improve treatment progress) while reducing or eliminating negative outcomes (e.g., discomfort caused by poor mask fit, severe air leakage, etc.).

[0052] Example workflow for facilitating image data collection for improved mesh generation Figure 2 An example workflow 200 for facilitating image data collection for improved mesh generation, according to some embodiments of this disclosure, is described. In some embodiments, workflow 200 may be provided by a measurement system (such as...) Figure 1 The measurement system 110) is executed.

[0053] As illustrated, the image component 115 of the measurement system includes an evaluation component 205 and a preprocessing component 210. Although depicted as discrete components for clarity of concept, in some embodiments, the operation of the depicted components (and other components not illustrated) may be combined or distributed across any number of components. In the depicted workflow 200, the image component 115 provides one or more instructions 215 to the user equipment 220. Instructions 215 may typically include any information used to facilitate the collection of user images.

[0054] For example, instruction 215 may include text instructions, image instructions, video instructions, audio instructions, etc. In some embodiments, instruction 215 may instruct the user on how to position themselves (e.g., by overlaying an ellipse or a person's head or face onto a live view from the camera of user device 220). For example, instruction 215 may instruct the user to look at the camera, turn their head to either side of the camera, look up, etc. For example, in some embodiments, instruction 215 may include drawing or overlaying a box and / or two ellipses (one for each nostril) on the image from the device's camera, and text instructing the user to position their nostrils within the box and / or ellipses and to capture an image when their nostrils are properly positioned. Such instruction 215 enables improved image capture.

[0055] In some aspects, instruction 215 may include requesting the user to perform breathing exercises to determine how smoothly the user can breathe through their nose. For example, instruction 215 may use user device 220 to output an animation via a display and request the user to breathe in sync with the animation (through their nose). The specific content of the animation may vary depending on the implementation. For example, the animation may include one or more circles (or other shapes) expanding and contracting, requesting the user to inhale as the shape expands and exhale as the shape contracts. After one or more such breathing cycles, instruction 215 may request the user to indicate whether they were able to breathe comfortably during the exercise (or to rate their comfort level).

[0056] Typically, any number and type of communication links can be used to provide instruction 215. For example, if the imaging component 115 is operating in a cloud deployment, instruction 215 may be sent to user equipment 220 via one or more wired and / or wireless networks (including the Internet). User equipment 220 typically represents any computing device that a user can use to capture and / or provide image 105 to the imaging component 115. For example, user equipment 220 may correspond to a laptop, desktop computer, smartphone, tablet, etc. In some embodiments, user equipment 220 includes one or more imaging sensors (e.g., cameras) integrated into the device or available as an external device (e.g., a plug-in webcam).

[0057] Although not depicted in the illustrated examples, in some embodiments, the image component 115 (or another component of the measurement system 110) may provide one or more questions or surveys via user equipment 220 to help guide the interface selection process. For example, in some embodiments, the user may be asked whether they have used any other interface within a defined time period (e.g., the past thirty days), and if so, may be asked to provide further information such as the model or type of the previous interface, the model or type of their current interface, and / or the reason for their switch (e.g., because they could not get a good seal, because the previous interface was uncomfortable, because they had facial indentations or irritation, because they felt claustrophobic using the old interface, because of leaks and / or because they were breathing through their mouth, because the mask could not stay in place, etc.). In some embodiments, the system may similarly ask the user to indicate whether they initiated the switch (rather than, for example, because their healthcare provider recommended the switch). Such information may be useful for recommending new interfaces to the user (e.g., for selecting a full-face mask, pillow mask, or nasal mask based on the user's responses and / or previous interface usage). For example, if a user indicates claustrophobia while using a full-face mask, the measurement system 110 may suggest a nasal connector or a pillow connector.

[0058] In some embodiments, the image component 115 (or another component of the measurement system 110) can provide questions related to whether a user is breathing through their mouth or otherwise has difficulty breathing through their nose. For example, the user may be asked if they have experienced various common problems (e.g., dry mouth, nasal congestion, or nasal irritation). Alternatively, the system may ask the user the following: whether the user has noticed any leaks in their current interface (if they have been participating in treatment); whether the user breathes through their mouth while using treatment; whether the user finds themselves breathing through their mouth when exerting themselves (e.g., climbing stairs); whether the user finds it easier to breathe through their mouth or nose when asked to take deep breaths; and whether the user suffers from any condition that makes it difficult to breathe through their nose (such as the common cold, chronic sinusitis, chronic allergies, deviated septum, etc.).

[0059] In some embodiments, such questions (for determining whether a user tends to breathe through their mouth) can be useful in allowing the measurement system to select a good interface recommendation, as discussed above. For example, in addition to recommending a specific size or model, the measurement system may recommend a specific type based on user response (e.g., suggesting a full-face mask for users who have difficulty breathing through their nose or otherwise tend to breathe through their mouth).

[0060] As illustrated, user equipment 220 sends one or more images 105 to image component 115. In the illustrated workflow 200, preprocessing component 210 may first perform one or more preprocessing operations on image 105. For example, as discussed above, preprocessing component 210 may resize image 105 to a standard or default size and / or perform various operations (such as contrast enhancement and noise reduction) to improve the machine learning process.

[0061] In the illustrated example, evaluation component 205 may evaluate image 105 (or preprocessed image data generated by preprocessing component 210) to determine whether image 105 is acceptable. For example, evaluation component 205 may use various machine learning models to detect whether the user's face is depicted in image 105, whether the lighting is adequate, etc. In some embodiments, evaluation component 205 uses a machine learning model trained to identify or detect whether ears are depicted in image 105. For example, evaluation component 205 may process image 105 corresponding to when the user turns left and / or right to determine whether the user's ears are visible. Such labeling can be useful for improving mesh generation because it enables more accurate shape and size determination of the model's head, which can improve headband size determination. In some embodiments, if evaluation component 205 determines that one or more images 105 are unacceptable, image component 115 may transmit a new set of instructions 215 to the user, asking them to try again (e.g., take another side view image, but with their hair pulled back).

[0062] In the illustrated workflow 200, the process can be repeated any number of times until an acceptable set of images 105 is obtained.

[0063] Although depicted as being performed by the image component 115 of the measurement system, in some respects, some or all of the operations discussed may be performed locally on the user device 220. For example, the ear detection machine learning model may be a lightweight classifier that can be executed by the user device 220 to detect ear visibility locally, allowing the user to immediately capture another image when needed. This reduces the network bandwidth consumed by the process (e.g., reducing the number of images 105 transmitted over the network) and lowers the computational overhead of the measurement system.

[0064] Although not depicted in the illustrated workflow 200, in some embodiments, once an acceptable set of images 105 has been generated, the image component 115 may provide the images 105 (or the resulting image data, such as feature maps) to one or more other components of the measurement system (e.g., Figure 1 The mesh component 125), as discussed above.

[0065] Example workflow for improved interface selection based on the generated mesh Figure 3 An example workflow 300 for improved interface selection based on a generated mesh, according to some embodiments of this disclosure, is described. In some embodiments, workflow 300 may be provided by a measurement system (such as...) Figure 1 The measurement system 110) is executed.

[0066] As illustrated, the selection component 145 of the measurement system includes a mapping component 305 and a classifier component 310. Although depicted as discrete components for clarity, in some embodiments, the operation of the depicted components (and other components not illustrated) may be combined or distributed across any number of components. In the depicted workflow 300, the selection component 145 accesses a set of measurements 140 (e.g., measurements generated by measurement components such as…) Figure 1 The measurement component 135) is based on a three-dimensional facial mesh depicting the user (such as... Figure 1 (using a grid of 130) to generate facial measurements).

[0067] As discussed above, measurement 140 typically includes one or more measurements indicating the size, shape, and / or location of one or more facial markers in three-dimensional space (such as the relative size, shape, and / or position of the user's eyes, nose, mouth, ears, nostrils, etc.).

[0068] In the illustrated example, selection component 145 also accesses a set of maps 315. In some embodiments, maps 315 typically indicate a prior size determination for one or more components of the user interface. That is, maps 315 may indicate, for one or more components (e.g., catheter size, interface type or model, headband size, etc.), the measurement range in which the component is designed and / or the component will fit appropriately. For example, map 315 may indicate that a first nasal-only mask is optimal for a user with a first set of nasal measurements, while a second nasal-only mask is better for a user with a second set of nasal measurements. In some embodiments, maps 315 may be defined or provided by the designer or manufacturer of the therapeutic component, and / or may be determined based on user interaction (e.g., surveying users to determine which component they prefer).

[0069] In some embodiments, mapping component 305 may use mapping 315 to evaluate some or all of the measurements 140 to select an appropriate interface component. For example, as discussed above, mapping component 305 may select one or more alternatives that correspond to measurement 140, such as one or more interfaces, one or more catheters, one or more headband sizes, etc.

[0070] In the illustrated example, classifier component 310 may similarly evaluate some or all of these measurements to select an appropriate interface component. In some embodiments, as discussed above, classifier component 310 may use one or more machine learning models to process the measurement data in order to select a component. For example, classifier component 310 may use a classifier model to process nostril measurements (e.g., the major and minor axes of the ellipse corresponding to the nostril, the rotation of the nostril, and / or the distance between the nostril and the center of the nose) to select a pillow size for the user.

[0071] As illustrated, selection component 145 generates interface 150 based on measurement 140. While a single interface 150 (e.g., a single set of components) is depicted for clarity of concept, in some embodiments, selection component 145 may generate a set of alternatives. For example, selection component 145 may generate a first interface 150 of a nose-only style (e.g., recommending a specific interface, catheter, and headband if the user wants to use a nose mask), a second interface 150 of a full-face style (e.g., recommending a specific interface, catheter, and headband if the user wants to use a full-face mask), and / or a third interface 150 of a pillow style (e.g., recommending a specific interface, catheter, headband, and pillow size if the user wants to use a pillow-style mask).

[0072] Similarly, in some embodiments, selection component 145 may indicate alternative options within the same style or type of face mask. For example, suppose mapping 315 includes overlapping measurement ranges of one or more components. In some embodiments, if the user's measurement 140 is within the overlapping area, interface 150 may indicate that any of these alternative options may be suitable.

[0073] In these ways, the selection of component 145 can provide users with significantly improved interface options, thereby improving treatment outcomes and comfort.

[0074] Example methods for generating 3D meshes using machine learning models and selecting interface components. picture Figure 4 This is a flowchart depicting a method 400 for generating a 3D mesh using a machine learning model and selecting interface components according to some embodiments of the present disclosure. In some embodiments, method 400 may be provided by a measurement system (such as...) Figure 1 The measurement system 110 and / or the above reference Figures 2 to 3The measurement system under discussion is executed.

[0075] At box 405, the measurement system accesses a set of images (e.g., Figure 1 and / or Figure 2 Image 105). In some embodiments, as discussed above, the image typically depicts the user's head and / or face. In some embodiments, the image is provided to enable the measurement system to generate a three-dimensional mesh corresponding to the user's face, thereby allowing the measurement system (or another system) to capture highly accurate measurements related to the size, shape, location, and / or orientation of various facial features or markers such as the nose, nostrils, eyes, mouth, ears, etc. In some aspects, as discussed above, accessing the images includes: evaluating the images to confirm that they meet defined acceptance criteria, such as minimum size, minimum resolution, minimum illumination, and / or contrast. References below Figure 5 A sample method for accessing images will be discussed in more detail.

[0076] At box 410, the measurement system generates a mesh (e.g., based on processing the accessed image (or the resulting image data) using one or more machine learning models (e.g., deep learning models). Figure 1 (Grid 130). For example, as discussed above, the measurement system (or another system) may use training samples to train a machine learning model, each sample comprising: one or more images (depicting the corresponding user) as input; and a corresponding set of 3D data points for a set of markers (or a grid of the corresponding user's face) on the user's face, which is used as a target or label. In some embodiments, as discussed above, some or all of the training samples may include synthetic data (e.g., synthetic or artificial facial grids used as labels, and rendered images of grids used as input).

[0077] Typically, the specific operations used to train a machine learning model can vary depending on the specific architecture. For example, in some embodiments, the measurement system (or other training system) processes images of training samples as input to a model (e.g., a deep learning convolutional neural network) to generate a mesh. The mesh can then be compared with labels (e.g., an actual mesh in 3D space or other data points) to generate a loss. The loss can typically be formulated using various methods, such as surface-to-surface loss, point-to-point loss, surface normal loss, Laplacian regularization loss, etc. In some embodiments, the model's parameters can then be updated based on the loss (e.g., using backpropagation). In this way, the model learns to generate a more accurate output mesh based on the input images. In embodiments, the model can be trained using individual samples (e.g., using stochastic gradient descent) and / or batch samples (e.g., using batch gradient descent) in any number of iterations or rounds.

[0078] In some embodiments, once trained (e.g., once the model reaches the desired level of accuracy, once no additional training samples are available, once a defined number of training iterations or a defined amount of computational resources have been spent, etc.), the model can be used to generate runtime meshes based on input user images.

[0079] In some embodiments, as discussed above, the mesh generated by the machine learning model has been scaled to the size of the user's head and / or facial features. That is, because the model can be trained end-to-end using relatively dense labels (e.g., dense point clouds and / or meshes), it can intrinsically learn to predict facial proportions without separately predicting how the camera affects perceived size (e.g., based on FOV and / or distance). For example, in some embodiments, the machine learning model learns to preserve or reconstruct a person's identity (e.g., facial shape) regardless of any changes in angle, background, camera FOV, camera distance, etc. (in a manner similar to how some facial recognition models work). In this way, the generated mesh can be intrinsically scaled correctly according to this pattern.

[0080] In some embodiments, if the mesh is not inherently scaled, the measurement system can use such a camera model to scale the output mesh. For example, the measurement system can use the camera model and / or the camera's FOV (if known) to predict the appropriate size of the mesh (or features within the mesh). For example, objects farther from the camera are perceived as smaller relative to objects closer to the camera. Therefore, the measurement system can evaluate the changes in the perceived size of one or more facial markers (e.g., a user's ears, nose, mouth, etc.) across the image to predict the camera's FOV, the distance to the markers, and / or the actual size of the features. This allows the measurement system to accurately scale the mesh. In some embodiments, using a camera model may refer to using a perspective projection technique that projects the mesh from world space to camera space. If the camera's parameters (e.g., FOV) are known, one or more projected keypoints can be compared to the locations of reference ground truth keypoints (on the image) to determine the appropriate scaling of the mesh.

[0081] At box 415, the measurement system removes facial expressions present in the mesh (if any). For example, the measurement system may deform the mesh to place the face in a neutral position (e.g., to remove expressions such as smiling, opening the mouth, raising eyebrows, etc.). In some embodiments, to remove facial expressions, a statistical model including a shape kernel (e.g., indicating the statistical average head shape) and one or more expression kernels (e.g., indicating various facial expressions) may be used. The kernels typically correspond to a statistical model generated using principal component analysis (PCA) on a facial dataset. For example, for a shape model, the kernel can be generated by performing PCA on a neutral expression dataset. For an expression model, the kernel can be generated by similarly performing PCA on a dataset of various expressions. In some embodiments, the expression kernel can be used to remove any facial expressions present in the mesh (so that the mesh depicts or corresponds to a neutral expression). As discussed above, removing facial expressions can improve measurement accuracy compared to obtaining measurements from a mesh depicting one or more facial expressions. Furthermore, by dynamically removing expressions from the mesh itself, the measurement system avoids the need to request additional images from the user (e.g., asking the user to take another picture without smiling). This improves the user experience and reduces the time consumed in the measurement process.

[0082] At box 420, the measurement system generates one or more facial measurements based on a grid (e.g., Figure 1 and / or Figure 3 Measurement 140). For example, as discussed above, the measurement system may generate measurements reflecting features such as: the size and shape of the nose, the size and shape of the mouth, the position of the mouth relative to the nose, the size, shape and position of the nostrils, the length of the duct path or other circumferential measurements around the user's head (e.g., for determining the size of the duct and / or the size of the headband), etc.

[0083] At box 425, the measurement system selects one or more interface components for the user based on facial measurements. For example, as discussed above, the measurement system may select one or more alternatives best suited to the user for each category of components (e.g., one or more catheters, one or more mask types and / or models, etc.) based on the determined measurements. Although not depicted in the illustrated examples, in some embodiments, the measurement system may further select interface components based at least in part on the user's response to survey questions, as discussed above. For example, if the user reports claustrophobia, the measurement system may select a nasal interface or a pillow-style interface. As another example, if the user reports difficulty breathing through their nose and / or a tendency to breathe through their mouth, the measurement system may select a full-face style interface. Furthermore, the measurement system may ask the user to perform nasal breathing exercises (e.g., synchronizing their breathing with an animation) and then ask the user to report how smoothly they could breathe (through their nose) during the exercise. The mask style may be selected (at least in part) based on the user's response to the exercise. See below for reference. Figure 8 This section discusses a sample method for selecting interface components in more detail.

[0084] In these ways, using method 400, the measurement system is able to use machine learning to generate a highly accurate three-dimensional mesh based on two-dimensional images, and then collect highly detailed facial measurements based on the mesh. This significantly improves measurement accuracy, thereby increasing the reliability of selecting appropriate interface components. As discussed above, these improved selections then enable improved respiratory therapy, such as by increasing comfort (which can increase the acceptance or usage of treatment), reducing air leaks or other negative issues, and reducing the difficulty or hassle of determining which equipment to choose (which may increase the number of patients deciding to begin treatment due to lower entry barriers). This can significantly improve outcomes for a wide variety of users receiving respiratory therapy.

[0085] Example methods for collecting and evaluating image data Figure 5 This is a flowchart depicting a method 500 for collecting and evaluating image data according to some embodiments of the present disclosure. In some embodiments, method 500 may be provided by a measurement system (such as...) Figure 1 The measurement system 110 and / or the above reference Figures 2 to 4 The method 500 is performed by the measurement system discussed. In some embodiments, part or all of the method 500 may be performed by other systems, such as by a user device for capturing images (e.g., Figure 2 The method 500 is executed locally on user equipment 220. In some embodiments, the method 500 is... Figure 4 Box 405 provides additional details.

[0086] At box 505, the measurement system provides user instructions to the user (e.g., Figure 2 (Instruction 215). Typically, as discussed above, instructions can take many forms, such as text instructions, image instructions, video instructions, audio instructions, etc. Instructions usually indicate which images are needed, such as by illustrating acceptable example images, instructing the user how to angle their head relative to the camera, etc. In some embodiments, the measurement system may provide instructions independently for each image. For example, the measurement system may provide instructions for a first image in the desired set, capture that image, and then provide instructions for the next image. In other embodiments, the measurement system may provide instructions for all images at once.

[0087] In some embodiments, as discussed above, the measurement system may similarly provide the user with one or more questions or surveys (e.g., to infer or determine whether they tend to breathe through their mouth, or to identify any previous interfaces that the user has stopped using). Such information can be useful for providing improved interface selection, as discussed above.

[0088] At box 510, the measurement system receives one or more user images from the user equipment (e.g., Figures 1 to 2 Image 105). In some embodiments, as discussed above, the measurement system receives one or more individual images (e.g., the user equipment may capture one or more images, such as when the user instructs them to be ready). In some embodiments, the measurement system receives video clips (e.g., a stream or sequence of frames) rather than a set of images. For example, the user equipment may record video as the user follows instructions, thereby allowing the measurement system to select the best image.

[0089] At box 515, the measurement system evaluates the received user image to determine whether the image meets one or more predefined acceptance criteria. The specific criteria used may vary depending on the specific implementation. For example, in some embodiments, the measurement system may determine whether the image has sufficiently high resolution, sufficient contrast or sharpness, appropriate lighting, etc.

[0090] In some embodiments, as discussed above, the measurement system may evaluate an image to confirm whether the user is properly following instructions. For example, the measurement system may use one or more computer vision models to process the image, which are trained to identify the presence of one or more markers or features, such as ears, eyes, mouth, nose, etc. For example, for a side view image, the measurement system may use an ear detection model to confirm whether the user's ears are visible.

[0091] Typically, the specific operations used to train a machine learning model can vary depending on the specific architecture. For example, in some embodiments, a measurement system (or other training system) may process images of training samples as input to the model to generate a binary output (or a set of binary outputs) indicating the presence of one or more labels or features. The output can then be compared with labels (e.g., whether each label is actually present) to generate a loss. Depending on the specific implementation, the loss can typically be formulated using various formulas, such as cross-entropy loss. In some embodiments, the parameters of the detection model can then be updated based on the loss (e.g., using backpropagation). In this way, the model learns to predict the presence of one or more labels or facial features in the provided images. In embodiments, the model can be trained using individual samples (e.g., using stochastic gradient descent) and / or batch samples (e.g., using batch gradient descent) in any number of iterations or rounds.

[0092] At box 520, the measurement system (based on evaluation) determines whether the criteria are met. If not, method 500 returns to box 505. In some embodiments, the measurement system provides additional instructions at box 505 based on specific criteria that were not met. For example, the measurement system may specifically indicate poor lighting, the user being too far from the camera, an incorrect angle, or that the user should ensure their ears are visible.

[0093] If the measurement system determines at box 520 that the criteria are met, method 500 proceeds to box 525, where the measurement system determines whether one or more additional images that have not yet been provided exist in the desired set of images. If yes, method 500 returns to box 505. If no, method 500 proceeds to box 530. While the illustrated method 500 depicts a sequential process (e.g., iteratively receiving and evaluating each image in turn) for conceptual clarity, in some embodiments, the measurement system may receive and / or evaluate some or all of these images in parallel. For example, in some embodiments, the measurement system receives video of a user moving their head to each designated location (e.g., forward, left, right, up, and down) and may extract appropriate images from that video sequence.

[0094] At box 530, the measurement system may optionally apply one or more preprocessing operations to the image. As discussed above, preprocessing operations may typically include any operations used to facilitate or improve the machine learning process. For example, the measurement system may adjust the contrast and / or brightness of the image, resize the image, crop the image, etc. In some embodiments, as discussed above, the measurement system may extract one or more features from the image (e.g., process the image using a feature extraction machine learning model to generate one or more feature maps), as discussed above.

[0095] Example methods for collecting user image data Figure 6A and Figure 6B A flowchart illustrating a method 600 for collecting user image data according to some embodiments of the present disclosure is shown. Specifically, Figure 6A Method 600A is described, while Figure 6B Method 600B is described, wherein methods 600A and 600B together form method 600. In some embodiments, method 600 may be provided by a measurement system (such as...) Figure 1 The measurement system 110 and / or the above reference Figures 2 to 5 The measurement system under discussion is used for execution. In some embodiments, method 800 is... Figure 4 Box 405 and / or Figure 5 Method 500 provides additional details.

[0096] Now for reference Figure 6A At frame 602, the measurement system outputs an instruction indicating that the user's head should be within a defined viewfinder. Typically, the measurement system can use various techniques (such as via a graphical user interface (GUI), via audio, etc.) to output this instruction. For example, the measurement system can output text instructions via a GUI, output natural language speech audio indicating the instruction, etc. In some aspects, a "viewfinder" refers to a defined portion or area of ​​the visual field of an imaging sensor (e.g., a smartphone camera) used to capture an image of the user. In some aspects, the measurement system can output real-time (or near-real-time) images captured by the imaging sensor via a display. For example, the measurement system can output the captured images via the user's smartphone GUI, thereby instructing the user to position themselves and / or the camera so that their head is within the viewfinder indicated on the GUI.

[0097] In some respects, before instructing the user to ensure their head is within the frame, the measurement system may perform various preprocessing operations, such as using machine learning to evaluate one or more captured images to determine whether a single face is detected anywhere in the image (e.g., rather than multiple faces or no face). In some respects, method 600 is performed when images are captured continuously or periodically for evaluation using various criteria discussed in more detail below.

[0098] At box 604, the measurement system determines whether the user's head is within the desired frame of the captured image. In some aspects, the presence of the user's head within the desired frame or location in the image may be referred to as a "frame criterion." If not, method 600 returns to box 602 to prompt the user to place their head within the frame. If the user's head is within the frame, method 600 continues to box 606, where the measurement system determines whether the captured image or image stream meets one or more brightness criteria. For example, the measurement system may determine whether the image is too bright (e.g., overexposed by direct sunlight) and / or too dark (e.g., obscured by shadows or a dark room). In some aspects, the desired brightness level may be referred to as a "brightness criterion."

[0099] If the brightness criterion is not met, method 600 proceeds to box 608, where the measurement system indicates the brightness criterion. For example, the measurement system may output an indication that the image is too dark or too bright (as the case may be), and suggest or instruct the user to move to an area with better lighting, avoid shadows, avoid bright direct light, etc. Then, method 600 returns to box 606.

[0100] If the measurement system determines at box 606 that the brightness criterion is met, method 600 continues to box 610, where the measurement system determines whether the user's head is depicted as horizontal relative to the imaging sensor in the image. For example, the measurement system may determine whether the imaging sensor is horizontal with the user's face (or at a defined angle relative to the horizontal plane) (such that the image depicts the head from the front), or below the user's face (such that the image depicts the head from below), and so on. In some respects, this may be referred to as the "level criterion".

[0101] If the measurement system determines that the user's head is not level with the camera, method 600 proceeds to box 612, where the measurement system indicates a leveling criterion. For example, the measurement system may output an instruction that the user should position themselves and / or the imaging sensor so that the camera is level with the user's face. Method 600 then returns to box 608.

[0102] If the measurement system determines at box 608 that the level criterion is met, method 600 continues to box 614, where the measurement system determines whether the user's head is within a defined distance relative to the imaging sensor. For example, the measurement system may determine whether the estimated distance between the head and the imaging sensor is above a minimum threshold and / or below a maximum threshold (e.g., based on the relative size of the user's head in the captured image). In some respects, this may be referred to as a "distance criterion".

[0103] If the measurement system determines that the user's head is too far or too close to the camera, method 600 proceeds to box 616, where the measurement system indicates a distance criterion. For example, the measurement system may output an instruction that the user should move themselves and / or the imaging sensor closer together or further apart so that the user's face fills the indicated framing in the image. Method 600 then returns to box 614.

[0104] If the measurement system determines at box 614 that the distance criterion is met, method 600 continues to box 618, where the measurement system captures one or more frontal images of the user's face. In some aspects, if the measurement system continuously or repeatedly captures images (e.g., to evaluate framing criteria, brightness criteria, level criteria, and / or distance criteria), then “capturing” a frontal image at box 618 may correspond to saving, selecting, storing, or otherwise retaining or marking one or more images (e.g., images captured just before and / or after determining that box 614 is met) for subsequent evaluation (e.g., to predict facial measurements and / or select a user interface).

[0105] In some aspects, the measurement system may output an indication that a frontal image has been captured (e.g., via camera shutter sound, flashing white or green light on the screen, etc.). In other aspects, the measurement system may not provide such an indication (e.g., proceeding directly to box 620, making it possible for the user not to know which images are being evaluated for the measurement process).

[0106] At box 620, the measurement system instructs the user to turn their head to the side (e.g., to the right or left relative to the camera). For example, the measurement system may instruct the user to slowly turn their head to the left or right. At box 622, the measurement system determines whether at least one of the user's ears is visible in the most recently captured image (e.g., whether the full outline of the ear is visible, or whether the ear is completely or partially obscured by hair, a hat, etc.). In some respects, this may be referred to as the "ear criterion".

[0107] If the measurement system determines that the ear criterion is not met, method 600 proceeds to box 624, where the measurement system indicates the ear criterion. For example, the measurement system may output an indication that the user's ear should be visible (e.g., on the side of the head opposite the side the user is turning towards), and the measurement system may instruct or require the user to remove any obstructions. Method 600 then returns to box 620.

[0108] If the measurement system determines that the ear is visible at box 622, then method 600 continues via box 626 to box 628. Figure 6B(As depicted in the image). At box 628, the measurement system determines whether the user's head has rotated relative to the imaging sensor to at least a threshold angle (as depicted in the recently captured image). In some respects, this is referred to as the "threshold angle criterion". If the measurement system determines that the angle criterion is not met (e.g., the user's head has rotated too little or too much to the side), method 600 proceeds to box 630.

[0109] At box 630, the measurement system indicates an angle threshold. For example, the measurement system may output an indication that the user should turn their head further (e.g., away from the camera) or reduce the degree of rotation (e.g., turn back towards the camera). Method 600 then returns to box 628.

[0110] If the measurement system determines at box 628 that a threshold angle criterion is met, method 600 proceeds to box 632, where the measurement system captures one or more side images of the user's face. In some aspects, if the measurement system continuously or repeatedly captures images (e.g., to evaluate framing criteria, brightness criteria, horizontal criteria, distance criteria, ear criteria, angle criteria, etc.), then “capturing” the side images at box 632 may correspond to saving, selecting, storing, or otherwise retaining or tagging one or more images (e.g., images captured just before and / or after determining that box 628 is met) for subsequent evaluation (e.g., to predict facial measurements and / or select a user interface).

[0111] In some aspects, the measurement system may output indications that a side image has been captured (e.g., via a camera shutter sound, a flashing white or green light on the screen, an audio prompt indicating that the user has fully turned their head, etc.). In other aspects, the measurement system may not provide such indications (e.g., proceeding directly to box 634, making it possible for the user not to know which images are being evaluated for the measurement process).

[0112] At box 634, the measurement system determines whether there is at least one additional side or angle for which it needs to capture images for analysis. For example, if the user turns their head to the left, the measurement system determines that it still needs to capture images of the user turning their head to the right (and vice versa). If the measurement system has not yet captured an image from the other side of the user's head, method 600 returns via box 636. Figure 6A Box 620. In some respects, if the measurement system relies on images from one side (e.g., the left or right side) rather than both sides, box 634 can be bypassed.

[0113] If at box 634 the measurement system determines that it has captured images from both sides of the user's head (or from one side if only one side is needed), then method 600 proceeds to box 638. At box 638, the measurement system instructs the user to tilt their head (e.g., tilt it upwards relative to the camera).

[0114] At box 640, the measurement system determines whether the user's head is tilted relative to the imaging sensor at at least a threshold angle (as depicted in the recently captured image). In some respects, this is referred to as the "tilt criterion" or simply the "threshold angle criterion," as discussed above. If the measurement system determines that the angle criterion or tilt criterion is not met (e.g., the user's head is not turned or tilted upwards by an insufficient amount, or by an excessive amount), method 600 proceeds to box 642.

[0115] At box 642, the measurement system indicates an angle threshold or tilt threshold. For example, the measurement system may output an indication that the user should turn their head further upward (e.g., away from the camera) or reduce the upward rotation (e.g., closer to the camera). Method 600 then returns to box 638.

[0116] If the measurement system determines at box 640 that a threshold tilt criterion is met, method 600 proceeds to box 644, where the measurement system captures one or more tilted images of the user's face. In some aspects, if the measurement system continuously or repeatedly captures images (e.g., to evaluate framing criteria, brightness criteria, horizontal criteria, distance criteria, ear criteria, angle criteria, etc.), then “capturing” a tilted image at box 644 may correspond to saving, selecting, storing, or otherwise retaining or marking one or more images (e.g., images captured just before and / or after determining that box 640 is met) for subsequent evaluation (e.g., to predict facial measurements and / or select a user interface).

[0117] Typically, using method 600, the measurement system can capture any number of images for processing to predict or estimate the size or measurement of a user's face. In some respects, each instruction or instruction provided during method 600 can typically be provided in various modalities, including text output via a display, audio (e.g., spoken) output via one or more speakers, tactile output, etc.

[0118] Furthermore, while the illustrated examples depict a series of evaluations, in some respects, some or all of these evaluations are performed in parallel, either partially or entirely. In some respects, the measurement system captures images continuously or periodically while performing method 600, such that each evaluation is performed on one or more of the most recently captured images. In some respects, capturing or recording images for further evaluation (e.g., at boxes 618, 632, and 644) may correspond to saving images for future use, while the remaining images captured during method 600 may be discarded.

[0119] In some aspects, determining whether one or more of the indicated criteria are met may include: evaluating the criteria over a series of images (e.g., at least five consecutive images) over a period of time (e.g., whether the criteria are met after five seconds), and so on. In some aspects, determining whether a criterion is met may be performed based on a single image (e.g., allowing method 600 to proceed quickly to the next step), while determining whether a criterion is not met may be performed based on a set of multiple images and / or the minimum amount of time required to determine the criteria that are not met.

[0120] In some aspects, the measurement system can “capture” multiple images at each location (e.g., a series of images from a frontal view, from each side, and from an oblique angle). In other aspects, the measurement system can then select the best image for downstream processing. For example, from each image pool or each set of images (e.g., for each angle), the measurement system can select the image with the least blur, the best lighting, etc.

[0121] Example workflow for collecting user image data Figure 7 An example workflow 700 for collecting user image data according to some embodiments of this disclosure is described. In some embodiments, workflow 700 may be provided by a measurement system (such as...) Figure 1 The measurement system 110 and / or the above reference Figure 2 See Figure 6. In some embodiments, workflow 700 is used to facilitate the capture of user images, as discussed above.

[0122] In some aspects, workflow 700 is implemented by displaying a user interface 705 (e.g., a GUI) on a display such as a smartphone, laptop, desktop computer, etc. In some aspects, workflow 700 is used to guide the user to turn or tilt their head to a desired angle, as discussed above. Specifically, in the illustrated embodiments, user interface 705A may include a viewfinder 710 and one or more orientation indicators 715. The viewfinder 710 may generally correspond to any visual indication of where the user should position their head in the captured image.

[0123] For example, as discussed above, the measurement system may continuously capture images (e.g., video or frame sequences) and display them on a user interface 705A, where a viewfinder 710 may be overlaid on the stream and / or may obscure some or all of these images. For example, the background of the user interface 705A may be opaque, and the viewfinder 710 may be a window (e.g., an ellipse or other shape) showing a portion of the captured image. The user may be instructed to align their head with the viewfinder 710.

[0124] Typically, the direction indicator 715 may correspond to any visual indication of which direction the user should turn or tilt their head. For example, the illustrated user interface 705A depicts the direction indicator 715 pointing to the left from the viewfinder 710 (e.g., instructing the user to turn their head to the left). In other examples, the direction indicator 715 may be depicted pointing to the right from the viewfinder 710 (e.g., instructing the user to turn their head to the right), and / or pointing upwards from the viewfinder 710 (e.g., instructing the user to tilt their head upwards).

[0125] Although the illustrated example depicts the arrow as a direction indicator 715, the direction indicator 715 can generally take any form (including non-visual indicators, such as audio recordings indicating direction). Furthermore, as illustrated in user interface 705A, a threshold indicator 720 is depicted. The threshold indicator 720 can be used to visually indicate the extent to which a user should turn their head in the indicated direction. For example, as illustrated in user interface 705B, a portion 725 of the direction indicator 715 is shaded (indicated by dotted lines) to indicate the amount by which the user has turned their head relative to the camera. That is, the measurement system can monitor the angle of the user's head relative to the camera and can update the portion 725 to visually indicate that angle. For example, when the user is facing the camera directly, the portion 725 may start at a first size (e.g., small or invisible). As the angle of the user's head relative to the camera increases, the measurement system can expand the portion 725 to indicate the amount by which the user's head has turned relative to the facing angle.

[0126] As illustrated in user interface 705B, the user has turned their head slightly to the left, but not enough to satisfy the angle criterion (indicated by threshold indicator 720). In some respects, the interface updates in real-time (or near real-time) based on the determined angle of the user's head. For example, the measurement system may dynamically change the size of section 725 to reflect the current angle.

[0127] Typically, part 725 can be implemented using any suitable technique. For example, the measuring system may make the color of part 725 different from the color of the direction indicator 715 (e.g., where the indicator is white and part 725 is filled with a color such as green). In some respects, in addition to or instead of visually indicating the angle (via part 725), the measuring system may output other indications, such as audio indications (e.g., giving the current estimated angle, instructing the user to turn a little further, etc.).

[0128] In user interface 705C, the user has turned their head beyond a minimum threshold (indicated by threshold indicator 720), as indicated by the shaded portion 725 of direction indicator 715. In some respects, as discussed above, the measurement system may capture one or more images at that angle for subsequent evaluation. In the illustrated example, the background of user interface 705C has been updated to reflect or indicate that the angle criterion has been met. For example, the measurement system may change the background color (e.g., to green) to flash one or more colors or graphics, etc. Such indicators, which are likely more visible in the user's peripheral vision (when their head is turned to the side) than smaller indicators such as portion 725, help the user recognize that an image has been captured and that they can turn back toward the camera. In some respects, in addition to or instead of changing the background of the interface, the measurement system may use other indicators such as audio output (e.g., camera shutter sound or verbal cues that an image has been captured) to assist the user.

[0129] Example methods for improved interface selection Figure 8 This is a flowchart depicting a method 800 for improved interface selection according to some embodiments of the present disclosure. In some embodiments, method 800 may be provided by a measurement system (such as...) Figure 1 The measurement system 110 and / or the above reference Figures 2 to 7 The measurement system under discussion is used for execution. In some embodiments, method 800 is... Figure 4 Box 425 provides additional details.

[0130] At box 805, the measurement system selects a pillow size for the user by processing one or more nostril measurements using a trained machine learning model. For example, as discussed above, the measurement system (or another system) may train a classifier model to categorize nostril measurements into pillow sizes. In some embodiments, as discussed above, nostril measurements may include parameters such as the major and minor axes of the ellipse corresponding to the nostril, the rotation of the ellipse or nostril, and the distance between the ellipse or nostril and the centerline of the user's nose. Using such user-specific measurements and machine learning allows for a more comfortable and accurate pillow fit (and is also easier and more hygienic compared to guesswork and checking methods).

[0131] Typically, the specific operations used to train a machine learning model for a classifier can vary depending on the specific architecture. For example, in some embodiments, the measurement system (or other training system) may process nostril measurements of training samples as input to the model to generate a classification (e.g., selecting a pillow size). The classification can then be compared to a label (e.g., based on the user's nostrils being an actual pillow size appropriate and / or comfortable for the user) to generate a loss. The loss can typically be formulated using various formulas, such as cross-entropy loss. In some embodiments, the model's parameters can then be updated based on the loss (e.g., using backpropagation). In this way, the model learns to generate a more accurate pillow size classification based on the input measurements. In embodiments, the model can be trained using individual samples (e.g., using stochastic gradient descent) and / or batch samples (e.g., using batch gradient descent) in any number of iterations or rounds.

[0132] At box 810, the measurement system selects the cannula size based on one or more head measurements (e.g., a measurement of the length of the cannula path extending upwards from the user's nose and / or mouth, across the cheekbone, and over the user's ear). For example, the measurement system may select a dynamic cannula size based on the measurements (e.g., selecting a cannula with the same length as the cannula path), or it may select a cannula size from a set of predefined alternative cannula sizes (e.g., using a defined mapping between facial measurements and cannula sizes, such as...). Figure 3 (Mapping 315). Using this user-specific measurement can greatly improve catheter size determination compared to more general or less accurate methods.

[0133] At box 815, the measurement system selects the headband size based on one or more head measurements (e.g., a measurement of the occipital-frontal circumference of the user's head). In some embodiments, to facilitate or improve the determination of the headband size, the measurement system may fit a statistical shape model of the human head to a grid, making it possible to estimate the head circumference (even without imaging the back of the user's head). For example, the measurement system may select a dynamic headband size based on measurements (e.g., indicating the use of a headband size equal to the head circumference), or it may select one of a set of predefined alternative headband sizes (e.g., using a defined mapping between facial measurements and headband sizes, such as...). Figure 3 (Mapping 315). Using this user-specific measurement allows for a significant improvement in headband size determination compared to more general or less accurate methods.

[0134] At box 820, the measurement system selects a user interface (e.g., a nose mask, full-face mask, and / or nose pillow mask) based on one or more head or facial measurements (e.g., measurements of the user's nose and / or mouth). For example, the measurement system may select one of a set of predefined alternative interfaces (e.g., using a predefined mapping between facial measurements and interfaces, such as...). Figure 3 (Mapping 315). Using these user-specific measurements can significantly improve interface fit compared to more general or less accurate methods.

[0135] In some embodiments, as discussed above, the measurement system may select from multiple interface alternatives. For example, the measurement system may select one interface for each type (e.g., a nasal interface, a nasal pillow interface, and a full-face interface), or it may select multiple alternatives within each type category. In some embodiments, the specific category (or categories) to which the measurement system generates the selection may depend on user input. For example, a user may specify that they want a recommended nasal mask. In some embodiments, the measurement system may generate the selection based on predicted user preferences or fit (e.g., based on facial measurements). For example, the measurement system may determine or infer, based on facial measurements, that a particular type or category of interface would likely be most comfortable for the user.

[0136] In some embodiments, as discussed above, the measurement system may select interface components based at least in part on user responses to various questions or surveys. For example, the measurement system may select the interface type based on responses related to the user's previous interface usage (e.g., if they have tried a pillow-type interface but disliked it), based on the user's comfort level with various types of interfaces, based on whether the user prefers to breathe through their mouth or their nose, etc. As another example, the measurement system may select the interface type based on the user's response to breathing exercises (e.g., in which the user is asked to breathe through their nose in sync with an animation), as discussed above. For example, the measurement system may select a full-face interface type for users who have difficulty breathing through their nose, and a nose-type and / or pillow-type for users who report feeling claustrophobic while wearing a full-face mask.

[0137] Typically, the specific components selected by the measurement system can vary depending on the specific task and implementation. Examples illustrated (e.g., pillow size, catheter size, headband size, and interface type) are depicted for clarity. However, in various embodiments, the measurement system may select additional components (not illustrated) for the user, or may select a subgroup of the illustrated components for the user.

[0138] Example methods for using machine learning to select a user interface. Figure 9This is a flowchart depicting a method 900 for selecting a user interface using machine learning according to some embodiments of the present disclosure. In some embodiments, method 900 may be provided by a measurement system (such as...) Figure 1 The measurement system 110 and / or the above reference Figures 2 to 8 The measurement system under discussion is executed.

[0139] At box 905, access a set of two-dimensional images of the user (e.g., Figures 1 to 2 Image 105).

[0140] At box 910, a 3D mesh depicting the user's head is generated based on processing a set of 2D images using a first machine learning model (e.g., Figure 1 A grid of 130 (where the 3D grid is scaled to the size of the user's head).

[0141] At box 915, modify the 3D mesh to remove one or more facial expressions.

[0142] At box 920, a set of facial measurements is determined based on the modified 3D mesh (e.g., Figure 1 and / or Figure 3 Measurement 140).

[0143] At box 925, the user interface is selected for the user based on a set of facial measurements (e.g., ...). Figure 1 and / or Figure 3 Interface 150).

[0144] Example computing device for mesh generation and interface selection Figure 10 Example computing device 1000 configured to perform various embodiments of the present disclosure is depicted. Although depicted as a physical device, in embodiments, computing device 1000 may be implemented using virtual devices and / or across multiple devices (e.g., in a cloud environment). In one embodiment, computing device 1000 corresponds to a measurement system, such as... Figure 1 The measurement system 110 and / or the above reference Figures 2 to 9 The measurement system under discussion.

[0145] As illustrated, computing device 1000 includes CPU 1005, memory 1010, network interface 1025, and one or more I / O interfaces 1020. In the illustrated embodiment, CPU 1005 retrieves and executes programming instructions stored in memory 1010, and stores and retrieves application data residing in one or more repositories (not depicted). CPU 1005 typically represents a single CPU and / or GPU, multiple CPUs and / or GPUs, a single CPU and / or GPU with multiple processing cores, etc. Memory 1010 is typically included to represent random access memory. In some embodiments, computing device 1000 may include a storage device (not depicted) that can be any combination of disk drives, flash-based storage devices, etc., and may include fixed and / or removable storage devices such as fixed disk drives, removable memory cards, caches, optical storage devices, network-attached storage devices (NAS), or storage area networks (SANs).

[0146] In some embodiments, I / O devices 1035 (such as a keyboard, monitor, etc.) are connected via I / O interface 1020. Furthermore, via network interface 1025, computing device 1000 can be communicatively coupled to one or more other devices and components (e.g., via a network, which may include the Internet, a local area network, etc.). As illustrated, CPU 1005, memory 1010, network interface 1025, and I / O interface 1020 are communicatively coupled via one or more buses 1030.

[0147] In the illustrated embodiment, memory 1010 includes an image component 1050, a grid component 1055, a measurement component 1060, and a selection component 1065, which perform one or more of the embodiments discussed above. Although depicted as discrete components for clarity of concept, in the embodiments, the operation of the depicted components (and other components not illustrated) may be combined or distributed across any number of components. Furthermore, although depicted as software residing in memory 1010, in the embodiments, the operation of the depicted components (and other components not illustrated) may be implemented using hardware, software, or a combination of hardware and software.

[0148] In some embodiments, the image component 1050 (which may correspond to...) Figures 1 to 2 Image component 115) can be used to access, evaluate, and / or preprocess images (e.g., Figures 1 to 2 Image 105), as discussed above. For example, image component 1050 may send or output instructions on how to capture images, preprocess images, and / or evaluate images to confirm whether they meet acceptance criteria (e.g., whether an ear is visible in a side view).

[0149] In some embodiments, the mesh component 1055 (which may correspond to...) Figure 1 The mesh component 125 can be used to generate three-dimensional meshes based on two-dimensional images (e.g., Figure 1 The mesh 130), as discussed above. For example, the mesh component 1055 can process images using one or more deep learning models (or other machine learning models) trained on image data of the user's face and corresponding point clouds (or other 3D data, such as mesh data).

[0150] In some embodiments, the measuring component 1060 (which may correspond to...) Figure 1 The measurement component 135 can be used to generate facial measurements (e.g., Figure 1 and / or Figure 3 Measurement 140), as discussed above. For example, measurement component 1060 can collect measurements that define the size, shape, position, and / or rotation of various facial markers or features such as mouth, nose, ears, nostrils, etc.

[0151] In some embodiments, component 1065 (which may correspond to) is selected. Figure 1 and / or Figure 3 The selection component 145 can be used to generate or select interface components (e.g., Figure 1 and / or Figure 3 Interface 150), as discussed above. For example, component selection 1065 may use mapping (such as mapping 1070) to identify appropriate components based on facial measurements, and / or may use one or more auxiliary machine learning models to process one or more of these measurements (e.g., nostril parameters) to predict or identify the best interface component.

[0152] In the illustrated example, memory 1010 also includes a mapping 1070 for one or more machine learning models and model parameters 1075. In some embodiments, mapping 1070 (which may correspond to...) Figure 3 Mapping 315 typically includes mappings that, for one or more interface components, indicate that the component is an acceptable or suitable set of facial measurements (e.g., measurement range). Alternatively, mapping 1070 may indicate an acceptable or suitable set of interface components for one or more facial measurement ranges. Model parameters 1075 may typically include parameters for any number of models, such as mesh generation models (e.g., deep learning patterns used by mesh component 1055 to generate facial meshes), marker or feature detection models (e.g., ear detection models used by image component 1050 to determine whether a side view image is acceptable), component classifier models (e.g., the nostril pillow classifier model discussed above used by selection component 1065 to select the pillow size for determination), etc.

[0153] Although the mapping 1070 and model parameters 1075 are depicted as residing in memory 1010 for conceptual clarity, they may be stored in any suitable location (including one or more local repositories) or in one or more remote systems different from computing device 1000.

[0154] Additional considerations The foregoing description is provided to enable any person skilled in the art to practice the various embodiments described herein. The examples discussed herein do not limit the scope, applicability, or embodiments set forth in the claims. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments. For example, changes may be made to the function and arrangement of the elements discussed without departing from the scope of this disclosure. Various processes or components may be appropriately omitted, substituted, or added to the various examples. For example, the order of execution of the described methods may differ from the order described, and various steps may be added, omitted, or combined. Furthermore, features described with respect to some examples may be combined in some other examples. For example, an apparatus or method may be implemented using any number of the embodiments set forth herein. Furthermore, the scope of this disclosure is intended to cover such apparatuses or methods practiced using other structures, functionalities, or structures and functionalities besides the various embodiments set forth herein. It should be understood that any embodiment of this disclosure may be embodied by one or more elements of the claims.

[0155] As used herein, the term "exemplary" means "as an example, instance, or illustration." Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.

[0156] As used herein, the phrase “at least one of” in a list of items refers to any combination of these items, including a single member. For example, “at least one of a, b, or c” is intended to cover a, b, c, ab, ac, bc, and abc, as well as any combination with multiple identical elements (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbc, cc, and ccc, or any other order of a, b, and c).

[0157] As used herein, the term "determine" encompasses a wide variety of actions. For example, "determine" can include calculation, computation, processing, derivation, investigation, searching (e.g., looking in a table, database, or other data structure), ascertaining, etc. Furthermore, "determine" can include receiving (e.g., receiving information), accessing (e.g., accessing data in memory), etc. Additionally, "determine" can include resolving, selecting, picking, establishing, etc.

[0158] The methods disclosed herein include one or more steps or actions for implementing the method. The method steps and / or actions are interchangeable without departing from the scope of the claims. In other words, unless a specific order of steps or actions is specified, the order and / or use of a particular step and / or action may be modified without departing from the scope of the claims. Furthermore, the various operations of the methods described above can be performed by any suitable means capable of performing the corresponding functions. Such means may include various hardware and / or software components and / or modules, including but not limited to circuits, application-specific integrated circuits (ASICs), or processors. Typically, where operations illustrated in the figures are present, these operations may have corresponding means plus functional components with similar numbering.

[0159] Embodiments of the present invention can be provided to end users through cloud computing infrastructure. Cloud computing generally refers to the provision of scalable computing resources as a service over a network. More formally, cloud computing can be defined as a computing capability that provides an abstraction between computing resources and their underlying technical architecture (e.g., servers, storage devices, networks), thereby enabling convenient, on-demand network access to a shared pool of configurable computing resources that can be rapidly provisioned and released with minimal management effort or service provider interaction. Therefore, cloud computing allows users to access virtual computing resources (e.g., storage devices, data, applications, and even complete virtualized computing systems) in the “cloud” without considering the underlying physical systems (or the location of these systems) used to provide the computing resources.

[0160] Typically, cloud computing resources are provided to users on a pay-as-you-go basis, where users are charged only for the computing resources actually used (e.g., the amount of storage space consumed or the number of virtualized systems instantiated by the user). Users can access any resource residing in the cloud anytime, anywhere via the Internet. In the context of this invention, users can access applications or systems available in the cloud (e.g., Figure 1 The measurement system (110) or related data. For example, the measurement system may run on a computing system in the cloud and train / use machine learning models to generate facial meshes and select interface components. In such cases, the measurement system may maintain the models in the cloud and use these models to drive improved interface recommendations. Doing so allows users to access this information from any computing system attached to a network connected to the cloud (e.g., the internet).

[0161] The following claims are not intended to limit themselves to the embodiments shown herein, but should be given the full scope consistent with the language of the claims. Within the claims, unless specifically stated otherwise, elements referred to in the singular are not intended to mean “one and only one”, but rather “one or more”. Unless specifically stated otherwise, the term “some” means one or more. Claim elements shall not be interpreted in accordance with 35 USC §112(f) unless an element of a claim is expressly described using the phrase “component for…”, or, in the case of a method claim, using the phrase “step for…”. All structural and functional equivalents of the elements of the various embodiments described throughout this disclosure that are known to or subsequently become known to those skilled in the art are expressly incorporated herein by reference and are intended to be covered by the claims. Furthermore, nothing disclosed herein is intended for public disclosure, whether or not such disclosure is expressly stated in the claims.

[0162] Example Terms Examples of implementation methods are described in the following numbered clauses: Clause 1: A method comprising: accessing a set of two-dimensional images of a user; generating a three-dimensional mesh depicting the user's head based on processing the set of two-dimensional images using a first machine learning model, wherein the three-dimensional mesh is scaled to the size of the user's head; modifying the three-dimensional mesh to remove one or more facial expressions; determining a set of facial measurements based on the modified three-dimensional mesh; and selecting a user interface for the user based on the set of facial measurements.

[0163] Clause 2: The method described in Clause 1, wherein selecting the user interface includes: generating a recommended pillow size for the user interface based on a set of nostril measurements from the set of facial measurements.

[0164] Clause 3: The method according to Clause 2, wherein the set of nostril measurements defines at least a first ellipse and includes at least one of the following: (i) the major axis, (ii) the minor axis, (iii) the degree of rotation, or (iv) the distance of the first ellipse from the center of the user's nose.

[0165] Clause 4: The method according to any one of Clauses 2 to 3, wherein generating the recommended pillow size comprises: using a second machine learning model to process the set of nostril measurements.

[0166] Clause 5: The method according to any one of Clauses 1 to 4, wherein selecting the user interface includes: generating a recommended catheter size for the user interface based on the set of facial measurements.

[0167] Clause 6: The method according to any one of Clauses 1 to 5, wherein selecting the user interface includes: generating a recommended headband size for the user interface based on the set of facial measurements.

[0168] Clause 7: The method described in Clause 6, wherein generating the recommended headband size comprises: fitting a statistical shape model of the human head to the three-dimensional mesh.

[0169] Clause 8: The method according to any one of Clauses 1 to 7, wherein, prior to generating the three-dimensional mesh, a second machine learning model is used to process at least one of the set of two-dimensional images to detect the presence of the user's ear in the at least one two-dimensional image.

[0170] Clause 9: The method according to any one of Clauses 1 to 8, wherein the first machine learning model is trained based on a set of training images, the set of training images depicting a corresponding set of three-dimensional data points of the training user and the head of the training user.

[0171] Clause 10: The method described in Clause 9, wherein the first machine learning model does not use a camera model to generate the 3D mesh.

[0172] Clause 11: The method according to any one of Clauses 1 to 10, wherein the set of two-dimensional images includes an image depicting the left side of the user's head, an image depicting the right side of the user's head, an image depicting the front of the user's head, and an image depicting the bottom of the user's head.

[0173] Clause 12: The method according to any one of Clauses 1 to 11, the method further comprising: after selecting the user interface, deleting the set of two-dimensional images, the three-dimensional mesh and the set of facial measurements.

[0174] Clause 13: The method according to any one of Clauses 1 to 12, the method further comprising: providing the user with one or more requests for information, wherein the one or more requests for information require the user to indicate whether they are experiencing difficulty breathing through their nose; receiving one or more responses from the user to the one or more requests; and further selecting the user interface based on the one or more responses.

[0175] Clause 14: The method according to any one of Clauses 1 to 13, wherein accessing the set of two-dimensional images comprises: capturing an image depicting the user; in response to determining that the first image satisfies at least one of the following: (i) a framing criterion, (ii) a brightness criterion, (iii) a horizontal criterion, or (iv) a distance criterion, outputting an instruction to the user to rotate the user's head; capturing a second image depicting the user; and in response to determining that the second image satisfies a threshold angle criterion, adding the second image to the set of two-dimensional images.

[0176] Clause 15: The method according to Clause 14, wherein accessing the set of two-dimensional images further comprises: capturing a third image depicting the user before capturing the second image; and in response to determining that the third image does not satisfy the threshold angle criterion, instructing the user to further rotate the user's head.

[0177] Clause 16: The method according to any one of Clauses 14 to 15, wherein accessing the set of two-dimensional images further includes, before capturing the second image: capturing a third image depicting the user; and in response to determining that the third image does not depict at least one ear of the user, instructing the user to make the user's ear visible.

[0178] Clause 17: The method according to any one of Clauses 14 to 16, wherein accessing the set of two-dimensional images further includes, before capturing the second image: monitoring the angle of the user's head rotation; and outputting a visual indication of the angle relative to the threshold angle criterion.

[0179] Clause 18: The method according to any one of Clauses 14 to 17, the method further comprising: in response to determining that the second image satisfies the threshold angle criterion, outputting a visual indication of satisfying the threshold angle criterion.

[0180] Clause 19: The method according to any one of Clauses 14 to 18, the method further comprising: in response to determining that the second image satisfies the threshold angle criterion, outputting an audio indication of satisfying the threshold angle criterion.

[0181] Clause 20: A system comprising: a memory including computer-executable instructions; and one or more processors configured to execute the computer-executable instructions and cause the processing system to perform the method according to any one of Clauses 1 to 19.

[0182] Clause 21: A system comprising components for performing the method according to any one of Clauses 1 to 19.

[0183] Clause 22: A non-transitory computer-readable medium comprising computer-executable instructions that, when executed by one or more processors of a processing system, cause the processing system to perform the method according to any one of Clauses 1 to 19.

[0184] Clause 23: A computer program product embodied on a computer-readable storage medium, the computer-readable storage medium including code for performing the method according to any one of Clauses 1 to 19.

Claims

1. A method, the method comprising: Access a set of two-dimensional images for the user; A three-dimensional mesh depicting the user's head is generated by processing the set of two-dimensional images using a first machine learning model, wherein the three-dimensional mesh is scaled to the size of the user's head; Modify the 3D mesh to remove one or more facial expressions; A set of facial measurements is determined based on a modified 3D mesh; as well as The user interface is selected for the user based on the set of facial measurements.

2. The method according to claim 1, wherein selecting the user interface comprises: The recommended pillow size is generated for the user interface based on a set of nostril measurements from the set of facial measurements.

3. The method of claim 2, wherein the set of nostril measurements defines at least a first ellipse and includes at least one of the following: (i) a major axis, (ii) a minor axis, (iii) a degree of rotation, or (iv) the distance of the first ellipse from the center of the user's nose.

4. The method of claim 2, wherein generating the recommended pillow size comprises: A second machine learning model was used to process the set of nostril measurements.

5. The method of claim 1, wherein selecting the user interface comprises: The recommended catheter size is generated for the user interface based on the set of facial measurements.

6. The method of claim 1, wherein selecting the user interface comprises: The recommended headband size is generated for the user interface based on the set of facial measurements.

7. The method of claim 6, wherein generating the recommended headband size comprises: The statistical shape model of the human head is fitted onto the three-dimensional mesh.

8. The method of claim 7, wherein, prior to generating the three-dimensional mesh, a second machine learning model is used to process at least one of the set of two-dimensional images to detect the presence of the user's ear in the at least one two-dimensional image.

9. The method of claim 1, wherein the first machine learning model is trained based on a set of training images, the set of training images depicting a set of corresponding three-dimensional data points of the training user and the head of the training user.

10. The method of claim 9, wherein the first machine learning model does not use a camera model to generate the 3D mesh.

11. The method of claim 1, wherein the set of two-dimensional images includes an image depicting the left side of the user's head, an image depicting the right side of the user's head, an image depicting the front of the user's head, and an image depicting the bottom of the user's head.

12. The method according to claim 1, further comprising: After selecting the user interface, delete the set of two-dimensional images, the three-dimensional mesh, and the set of facial measurements.

13. The method according to claim 1, further comprising: Provide the user with one or more requests for information, wherein the one or more requests for information require the user to indicate whether they are experiencing difficulty breathing through their nose; Receive one or more responses to the one or more requests from the user; as well as The user interface is further selected based on the one or more responses.

14. The method of claim 1, wherein accessing the set of two-dimensional images comprises: Capture an image depicting the user; In response to determining that the first image satisfies at least one of the following: (i) a framing criterion, (ii) a brightness criterion, (iii) a horizontal criterion, or (iv) a distance criterion, an instruction to rotate the user's head is output to the user; Capture a second image depicting the user; as well as In response to determining that the second image satisfies the threshold angle criterion, the second image is added to the set of two-dimensional images.

15. The method of claim 14, wherein accessing the set of two-dimensional images further includes, prior to capturing the second image: Capture a third image depicting the user; and In response to determining that the third image does not meet the threshold angle criterion, the user is instructed to further rotate the user's head.

16. The method of claim 14, wherein accessing the set of two-dimensional images further includes, prior to capturing the second image: Capture a third image depicting the user; and In response to determining that the third image does not depict at least one of the user's ears, the user is instructed to make the user's ear visible.

17. The method of claim 14, wherein accessing the set of two-dimensional images further includes, prior to capturing the second image: Monitor the angle of the user's head rotation; and The visual indication of the angle is output relative to the threshold angle criterion.

18. The method according to claim 17, further comprising: In response to determining that the second image satisfies the threshold angle criterion, a visual indication of satisfying the threshold angle criterion is output.

19. The method according to claim 18, further comprising: In response to determining that the second image satisfies the threshold angle criterion, an audio indication of satisfying the threshold angle criterion is output.

Images