System and method for adaptively generating facial device selection based on visually determined anatomical dimension data.

Abstract

The present invention provides a computing apparatus and method for adaptively generating facial device selections based on visually determined anatomical dimension data. [Solution] The computing device includes a processor and memory. The memory receives image data representing the user's face and includes processor-executable instructions for determining anatomical dimension data related to the nose-mouth region of the user's face based on the received image data. The processor-executable instructions generate one or more facial device recommendations based on the recommended model and anatomical dimension data, and provide one or more facial device recommendations for display in the user interface.

Description

Technical Field

[0001] This claims the benefit of U.S. Provisional Patent Application No. 63 / 166,723, filed Mar. 26, 2021, the entire contents of which are hereby incorporated by reference.

[0002] Embodiments of the present disclosure relate to the field of facial devices, and more particularly to adaptively generating facial device selections.

Background Art

[0003] For certain functions, a facial device or garment may be positioned adjacent to a user's nasal cavity and / or oral cavity. For example, a facial mask may be positioned adjacent to a user's nasal cavity and / or oral cavity to provide a physical barrier between the user's environment and the user's respiratory system. In another example, a facial mask of a continuous positive airway pressure (CPAP) machine may be positioned adjacent to a user's nasal cavity (e.g., over the user's nose) and / or the user's oral cavity (e.g., over the user's mouth) as an interface for providing air pressure to the user's nose and / or mouth while the user is sleeping. Other examples of facial devices are also contemplated. In some scenarios, the effectiveness of a facial device may be correlated with how well the facial device fits the user's face.

Summary of the Invention

[0004] In one embodiment, the Disclosure provides a computing device for adaptively generating facial device selections based on visually determined anatomical dimension data. The device may include a processor and memory coupled to the processor. The memory may store processor executable instructions that, when executed, receive image data representing a user's face, determine anatomical dimension data related to the nose-mouth region of the user's face based on the received image data, generate one or more facial device recommendations based on a recommendation model defined based on a Bayesian-based scoring metric and the anatomical dimension data, and provide the one or more facial device recommendations for display in a user interface.

[0005] In another embodiment, the Disclosure provides a method for adaptively generating facial device selections based on visually determined anatomical dimension data. The method may include receiving image data representing a user's face, determining anatomical dimension data related to the nose-mouth region of the user's face based on the received image data, generating one or more facial device recommendations based on a recommendation model defined based on a Bayesian-based scoring metric and the anatomical dimension data, and providing the one or more facial device recommendations for display in a user interface.

[0006] In another embodiment, one or more non-temporary computer-readable media store machine-interpretable instructions, which, when executed by a processor, can cause the processor to perform one or more of the methods described herein.

[0007] In various further embodiments, the disclosure provides corresponding systems and devices, as well as logical structures such as machine-executable coded instruction sets for implementing such systems, devices, and methods.

[0008] In this regard, before describing at least one embodiment in detail, it should be understood that such embodiments are not limited to the configuration details and component arrangements described or shown in the drawings below. Furthermore, it should be understood that the expressions and terms used herein are for illustrative purposes only and should not be considered limiting.

[0009] Numerous further features relating to the embodiments described herein, and combinations thereof, will become apparent to those skilled in the art after reading this disclosure. [Brief explanation of the drawing]

[0010] Embodiments are shown as examples in the drawings. It should be clearly understood that the description and drawings are for illustrative purposes only and are intended to aid understanding.

[0011] Herein, embodiments will be described simply as examples with reference to the attached drawings, including the following. [Figure 1] This document shows a system platform in accordance with embodiments of this disclosure. [Figure 2] Figure 1 shows a block diagram illustrating the operation of the system platform. [Figure 3] Figures 3 to 5 show sample programming code according to embodiments of the present disclosure. [Figure 4] Figures 3 to 5 show sample programming code according to embodiments of the present disclosure. [Figure 5] Figures 3 to 5 show sample programming code according to embodiments of the present disclosure. [Figure 6] A sample object inheritance table according to an embodiment of this disclosure is shown. [Figure 7] Figures 7 to 9 show 3D point cloud rendering according to embodiments of the present disclosure. [Figure 8]Figures 7 to 9 show 3D point cloud rendering according to embodiments of the present disclosure. [Figure 9] Figures 7 to 9 show 3D point cloud rendering according to embodiments of the present disclosure. [Figure 10A] Figures 10A and 10B illustrate the identification of the contour of a user's face according to an embodiment of the present disclosure. [Figure 10B] Figures 10A and 10B illustrate the identification of the contour of a user's face according to an embodiment of the present disclosure. [Figure 11] Sample programming pseudocode according to embodiments of this disclosure is shown. [Figure 12A] Figures 12A and 12B show examples of visual data representing the topology of a face according to embodiments of the present disclosure. [Figure 12B] Figures 12A and 12B show examples of visual data representing the topology of a face according to embodiments of the present disclosure. [Figure 13A] Figures 13A and 13B show enlarged views of a 3D mesh model according to an embodiment of the present disclosure. [Figure 13B] Figures 13A and 13B show enlarged views of a 3D mesh model according to an embodiment of the present disclosure. [Figure 14] This invention provides a method for estimating head posture based on image data, according to an embodiment of this disclosure. [Figure 15A] Figures 15A, 15B, 16A, 16B, 17A, 17B, 18A, 18B, 19 to 21, 22A, and 22B show images based on 3D point cloud data related to facial features, according to embodiments of the present disclosure. [Figure 15B] Figures 15A, 15B, 16A, 16B, 17A, 17B, 18A, 18B, 19 to 21, 22A, and 22B show images based on 3D point cloud data related to facial features, according to embodiments of the present disclosure. [Figure 16A]Figures 15A, 15B, 16A, 16B, 17A, 17B, 18A, 18B, 19 to 21, 22A, and 22B show images based on 3D point cloud data related to facial features according to embodiments of the present disclosure. [Figure 16B] Figures 15A, 15B, 16A, 16B, 17A, 17B, 18A, 18B, 19 to 21, 22A, and 22B show images based on 3D point cloud data related to facial features according to embodiments of the present disclosure. [Figure 17A] Figures 15A, 15B, 16A, 16B, 17A, 17B, 18A, 18B, 19 to 21, 22A, and 22B show images based on 3D point cloud data related to facial features according to embodiments of the present disclosure. [Figure 17B] Figures 15A, 15B, 16A, 16B, 17A, 17B, 18A, 18B, 19 to 21, 22A, and 22B show images based on 3D point cloud data related to facial features according to embodiments of the present disclosure. [Figure 18A] Figures 15A, 15B, 16A, 16B, 17A, 17B, 18A, 18B, 19 to 21, 22A, and 22B show images based on 3D point cloud data related to facial features according to embodiments of the present disclosure. [Figure 18B] Figures 15A, 15B, 16A, 16B, 17A, 17B, 18A, 18B, 19 to 21, 22A, and 22B show images based on 3D point cloud data related to facial features according to embodiments of the present disclosure. [Figure 19] Figures 15A, 15B, 16A, 16B, 17A, 17B, 18A, 18B, 19 to 21, 22A, and 22B show images based on 3D point cloud data related to facial features according to embodiments of the present disclosure. [Figure 20]Figures 15A, 15B, 16A, 16B, 17A, 17B, 18A, 18B, 19 to 21, 22A, and 22B show images based on 3D point cloud data related to facial features, according to embodiments of the present disclosure. [Figure 21] Figures 15A, 15B, 16A, 16B, 17A, 17B, 18A, 18B, 19 to 21, 22A, and 22B show images based on 3D point cloud data related to facial features, according to embodiments of the present disclosure. [Figure 22A] Figures 15A, 15B, 16A, 16B, 17A, 17B, 18A, 18B, 19 to 21, 22A, and 22B show images based on 3D point cloud data related to facial features, according to embodiments of the present disclosure. [Figure 22B] Figures 15A, 15B, 16A, 16B, 17A, 17B, 18A, 18B, 19 to 21, 22A, and 22B show images based on 3D point cloud data related to facial features, according to embodiments of the present disclosure. [Figure 23] This invention provides a method for determining anatomical dimensional data of facial features according to embodiments of this disclosure. [Figure 24A] Figures 24A, 24B, 25A, 25B, 26, 27A, and 27B show RGB images for determining data representing facial features according to embodiments of the present disclosure. [Figure 24B] Figures 24A, 24B, 25A, 25B, 26, 27A, and 27B show RGB images for determining data representing facial features according to embodiments of the present disclosure. [Figure 25A] Figures 24A, 24B, 25A, 25B, 26, 27A, and 27B show RGB images for determining data representing facial features according to embodiments of the present disclosure. [Figure 25B]Figures 24A, 24B, 25A, 25B, 26, 27A, and 27B show RGB images for determining data representing facial features according to embodiments of the present disclosure. [Figure 26] Figures 24A, 24B, 25A, 25B, 26, 27A, and 27B show RGB images for determining data representing facial features according to embodiments of the present disclosure. [Figure 27A] Figures 24A, 24B, 25A, 25B, 26, 27A, and 27B show RGB images for determining data representing facial features according to embodiments of the present disclosure. [Figure 27B] Figures 24A, 24B, 25A, 25B, 26, 27A, and 27B show RGB images for determining data representing facial features according to embodiments of the present disclosure. [Figure 28] This document describes a method for determining the recommendation of one or more facial devices according to embodiments of the present disclosure. [Figure 29] The sample boundary ranges related to the size of the facial device according to embodiments of this disclosure are illustrated. [Figure 30] A flowchart illustrating a method for updating boundary ranges related to generating facial device recommendations, according to embodiments of this disclosure, is shown. [Figure 31] Figures 31 to 33 show a user interface for providing facial device recommendations according to embodiments of the present disclosure. [Figure 32] Figures 31 to 33 show a user interface for providing facial device recommendations according to embodiments of the present disclosure. [Figure 33] Figures 31 to 33 show a user interface for providing facial device recommendations according to embodiments of the present disclosure. [Figure 34] This is an example of a table showing the scores assigned to each facial device type based on patient attributes and clinical patient data. [Figure 35]This is an example of a user interface for providing ranked facial device recommendations according to embodiments of the present disclosure. [Modes for carrying out the invention]

[0012] A system and method for adaptively generating facial device selection based on visually determined anatomical dimensional data is described herein. For ease of explanation, embodiments of this disclosure may be described based on examples relating to facial devices associated with positive airway pressure (PAP) devices. PAP devices may include Continuous Positive Airway Pressure (CPAP), bilevel or BiPAP machines, non-invasive ventilators (NIVs), and adaptive servo ventilators (ASVs). A PAP machine may be configured to provide a flow of air pressure to the user's nose or mouth while the user is sleeping. Such a flow of air pressure to the user's nose or mouth may help keep the user's airway open and thereby assist normal breathing. Examples of such facial devices can serve as an interface between the PAP device and the user's respiratory system, and a combination of fit and comfort contributes to the optimal operation of such facial devices.

[0013] In some scenarios, several different types or manufacturers of PAP devices may exist. Each manufacturer's PAP device may be associated with a facial device that has unique characteristics regarding the delivery of air pressure to the user's nose or mouth. Since the fit and comfort of the facial device for a particular user may be an important consideration related to the use of a PAP device, a one-size-fits-all mask device may not exist. It may be beneficial to provide a system and method for dynamically generating facial device recommendations for each user.

[0014] While this disclosure describes examples related to facial devices accompanying PAP devices, systems and methods for dynamically providing recommendations for other types of facial devices may also be considered. For example, embodiments of the systems and methods described herein may, among other examples, be for dynamically providing recommendations for respiratory mask device fittings (e.g., N95 mask fittings), headgear device fittings, or similar devices.

[0015] Referring to Figure 1, a system platform 100 according to an embodiment of the present disclosure is shown. The system platform 100 may include a plurality of computing devices that send and receive data messages to and from other computing devices via a network 150.

[0016] In some embodiments, the system platform 100 may be configured to dynamically generate facial device recommendations based on one or more quantitative or qualitative data. In some embodiments, the quantitative data may include acquired image data related to a prospective user of a facial device. In some scenarios, the image data may represent anatomical dimension data related to the user. For example, the anatomical dimension data may provide, among other examples, the distance between the user's nostrils, the width or length of the user's nose, or the width of the user's face. As described in this disclosure, the system platform 100 may include an application for dynamically determining the quantitative data based on the acquired image data.

[0017] In some embodiments, quantitative data may include data related to facial device specifications retrieved from a client device or service provider device. For example, facial device specifications may include PAP device tuning parameters obtained via one or more service provider devices.

[0018] In some scenarios, a substantially accurate fit may not result in optimal comfort and operation for all users. In some embodiments, qualitative data may include data related to user feedback data associated with a facial device category or anatomical data category. For example, not all facial device users with an internostril distance of approximately 0.5 centimeters may be a suitable fit for a given facial device. Therefore, in some embodiments, the system platform 100 may dynamically generate facial device recommendations based on either quantitative or qualitative data, or a combination thereof.

[0019] The system platform 100 may include multiple computing devices, such as a server 110, a web interface platform 112, one or more client devices 120, or one or more service provider devices 130.

[0020] One or more client devices 120 may be associated with their respective facial device users. For example, a mask device user may be a PAP device user. In order to provide facial device users with a facial device of the optimal size in terms of function and comfort, among other attributes, it may be beneficial to generate recommendations for facial device users. These recommendations may be qualitative or quantitative data related to the facial device user, previous facial device recommendations related to users with substantially similar anatomical features to that facial device user, or a subset of facial devices recommended based on facial device configuration specifications provided by a service provider user (e.g., clinicians, technicians, among other examples). For example, qualitative or quantitative data may include measurement data related to the user's facial features, user demographic data, clinical data, PAP machine device configuration data, data representing previous user satisfaction, user preference data (e.g., data representing a given user's previous satisfaction), or data representing dimensions related to the facial device.

[0021] In some embodiments, one or more client devices 120 may be configured to receive user data that provides mask device recommendations via a user interface. For example, user data may include, among other examples, user demographic data, user preference data, and PAP treatment prescription data.

[0022] In some embodiments, one or more client devices 120 may be configured to capture image data related to the mask device user via an image capture device. One or more client devices 120 may perform operations to extract anatomical feature data based on the image data. One or more client devices 120 may transmit the extracted anatomical feature data and user data to the server 110 so that operations for mask device recommendation may be performed.

[0023] One or more service provider devices 130 may be associated with a clinician or technician user. In this example, the clinician or technician user may include a healthcare professional engaged in providing medical care or therapy to a facial device user. One or more service provider devices 130 may be configured to receive clinician data about the facial device user or treatment plans associated with the facial device user via a user interface. In some embodiments, one or more service provider devices 130 may receive patient management data, including desired facial device specification data.

[0024] In some embodiments, the server 110 may receive user data or clinician data via the network 150 as input for generating mask device recommendations based on facial device recommendation models. The server 110 may include a machine learning architecture for performing operations for generating one or more facial device recommendation models, and for generating facial device recommendations based on a combination of datasets representing qualitative and quantitative data. In some embodiments, the server 110 may be based on Amazon® Relational Database Services (RDS) accessible via an application-programmable interface.

[0025] In some embodiments, the server 110 may include memory comprising one or a combination of computer memories, such as random access memory, read-only memory, electro-optical memory, magneto-optical memory, erasable programmable read-only memory, electrically erasable programmable read-only memory, ferroelectric random access memory, or similar. In some embodiments, the memory may be a storage medium, such as a hard disk drive, solid-state drive, optical drive, or other type of memory.

[0026] The memory may contain or store a database structure. The database structure may store data records received from one or more client devices 120 or service provider devices 130. In some embodiments, the database structure may contain a large dataset representing previous mask device recommendation results.

[0027] In some embodiments, datasets may be managed based on the characteristics of a PostgreSQL 12 database. PostgreSQL can be an open-source object-relational database system that uses or extends the SQL language in combination with capabilities for storing and scaling large dataset operations. PostgreSQL can manage internal security on a role-by-role basis. In some examples, roles may be associated with users or groups (e.g., roles that other roles may be members of). In some scenarios, permissions may be granted or revoked at the database column level, and may allow / prevent the creation of new objects at the database structure, schema, or table level.

[0028] In some embodiments, the server 110 may send or receive data to and from other computing devices via the interface platform 112. For example, the interface platform 112 may be a Django-based platform. The Django platform may include security mechanisms configured to counter potential data security threats related to the sending or receiving of data between multiple computing devices. In some other examples, the interface platform 112 may be based on PHP scripts to implement mechanisms for countering potential data security threats.

[0029] In some embodiments, the interface platform 112 may be based on features such as cross-site scripting protection, cross-site request forgery protection, SQL injection protection, clickjacking protection, or secure password hashing, among other examples.

[0030] Network 150 may include wired or wireless wide area networks (WANs), local area networks (LANs), combinations thereof, or other networks for carrying communication signals. In some embodiments, network communication may be based on HTTP POST requests or TCP connections. Other network communication operations or protocols may also be contemplated. In some embodiments, Network 150 may include other networks, including the Internet, Ethernet®, legacy telephone service lines, public switched telephone networks, integrated services digital networks, digital subscriber lines, coaxial cables, optical fibers, satellite, mobile, wireless, SS7 signaling networks, fixed lines, local area networks, wide area networks, or combinations of one or more of these networks.

[0031] In some embodiments, one or more client devices 120 may store and run a facial device application. Embodiments of the facial device application may be configured for a mobile device operating system such as iOS® or Android®.

[0032] In some cases, native iOS® applications may be based on the SwiftUI framework. In some cases, native Android® applications may be based on Java® and OpenCV / MLKit. In some cases, one or more applications may be developed based on React JS, i.e., a JavaScript® library package for generating user interfaces or user interface components.

[0033] In some embodiments, one or more client devices 120 may include an image capture device and may be configured to capture image data of a facial device user. As described in this disclosure, embodiments of the facial device recommendation application may generate facial device recommendations based on anatomical dimensions (e.g., dimensions of the user's face) determined from the captured image data.

[0034] In some embodiments, one or more servers 110 may store datasets for generating and training facial device recommendation models. In some embodiments, the facial device recommendation model may generate facial device recommendations based on input datasets related to facial device users. For example, the input dataset may include anatomical dimension data. In some embodiments, the anatomical dimension data may include, among other examples, data representing face height, nose height, nose width, nose depth, nostril height, or nostril width. In some examples, the input dataset may include user demographic data, user preference data, or other user inputs received from a client device 120. In some embodiments, the input dataset may include CPAP therapy prescription data or clinician-provided data received from a service provider device 130.

[0035] In some embodiments, the server 110 may generate a recommendation model based on a deep learning neural network trained on a big dataset.

[0036] In some embodiments, it may be beneficial to reduce the amount of personally identifiable information (PII) transmitted over the network 150. PII may include user data or user images related to the facial device user. In some embodiments, the operation of capturing image data and generating anatomical dimension data based on the captured image data may be performed locally in one or more client devices 120. One or more client devices 120 may then send aggregated data or a subset of mask device user data to the server 110 to generate mask device recommendations. One or more client devices 120 do not need to store any of the image data, 3D models based on the image data, or anatomical dimensions related to the facial device user in persistent memory.

[0037] In some embodiments, the client device 120 or service provider device 130 may not be able to directly access the dataset or data record stored in the server 110. The client device 120 or service provider device 130 can send data retrieval and data modification requests to the server 110, and the server 110 may be configured to validate such requests and generate a response thereto. Such implementations may be based on the assumption that a “front-end” device, such as the client device 120, may be an untrusted or unprotected device. Establishing rules to govern the flow of data requests can improve or reduce behaviors that could compromise the data security or data integrity of the system platform 100.

[0038] In some embodiments, the client device 120 or service provider device 130 does not need to directly transmit or access data stored in the server 110. Rather, the server 110 can be configured to receive data requests via an application-programmable interface and to generate or modify database structures or datasets stored in memory.

[0039] In some embodiments, data messages transmitted between computing devices may be transmitted via network 150 and interface platform 112. In some embodiments, the transmitted data messages may be based on the HTTPS (TLS version 1.2) protocol.

[0040] In some embodiments, the interface platform 112 may be a web interface platform based on an Nginx 1.19 reverse proxy server. In some embodiments, the interface platform 112 may include a cloud computing platform including a Django 2.2 web application on Amazon EC2 (HTML, CSS, Javascript®). In some embodiments, the interface platform 112 may include an authentication process based on Django session authentication. In some embodiments, the web interface platform 112 may include a Representation State Transfer (REST) ​​Application Programmable Interface (API) based on the Django REST Framework 3.10.

[0041] In some embodiments, data messages transmitted over network 150 may be based on the HTTPS protocol, which has the capability to encrypt data messages. The protocol may be transport layer security (TLS version 1.2). The protocol may secure communications based on an asymmetric public key infrastructure. Data packets transmitted under the HTTPS protocol may appear as meaningless data to malicious or unintentional users, even if intercepted.

[0042] In some embodiments, server 110 may be based on a Django 2.2 server. In some embodiments, server 110 may include a database structure based on a PostgreSQL 12 database on Amazon RDS. In some scenarios, Amazon RDS may encrypt the database structure based on managed encryption keys using AWS Key Management Service (KMS). Data stored quiescently in the underlying storage on a database instance running with Amazon RDS encryption may be encrypted, as well as automated backups, read replicas, or database snapshots. RDS encryption may be based on the industry-standard AES-256 encryption algorithm for encrypting data.

[0043] It should be understood that the above examples are illustrative, and alternative mechanisms may be conceived.

[0044] Facial device recommendations, primarily based on the anatomical dimensions of the facial device user, may be suitable for addressing a combination of user fit, preferences, or qualitative characteristics that can contribute to the optimal operation and fit of the facial device. In some scenarios, optimal facial device function or user comfort may be based on a combination of both quantitative data (e.g., physical dimensions) and qualitative data (e.g., user preferences related to material stiffness, how the mask sits on the user's face, among other examples). In some examples, the qualitative data for generating facial device recommendations may include user demographic data (e.g., age, sex, ethnicity, geographical location, or similar) or clinical data (e.g., the user's tendency to be sensitive to or experience nasal congestion, seasonal allergies, claustrophobia, skin hypersensitivity, or similar). In some embodiments, the qualitative data may represent the user's preferred sleeping position. In some embodiments, quantitative data for generating facial device recommendations may include, among other examples, the physical dimensions of the user's facial features, the target PAP machine, or facial device settings (e.g., minimum or maximum pressure settings). As described herein, a server 110, one or more client devices 120, or one or more service provider devices 130 may include a machine learning architecture for generating and improving facial device recommendations based on a dataset representing a combination of previous mask device recommendations and user feedback.

[0045] Referring to Figure 2, a block diagram 200 is shown illustrating the operation of the system platform 100 of Figure 1 according to an embodiment of the present disclosure. The operation may be performed on one or more of the computing devices of the system platform 100. For example, computing device 220 may include a mobile device running a mobile operating system such as iOS® or Android®. In some embodiments, one or more computing devices 220 may be the client device 120 or service provider device 130 shown in Figure 1. In some embodiments, server 110 may include secure storage that provides a database 280 (e.g., the MaskFit AR® database).

[0046] In some embodiments, one or more computing devices 220 may be configured to receive input data from facial device users, service provider users (e.g., clinicians or technicians providing medical or therapeutic services), or other users who provide data relating to generating facial device recommendations for facial device users.

[0047] In a scenario where the computing device 220 is associated with a facial device user, the computing device 220 may capture image data (e.g., a photograph) of the facial device user via an image capture device. The image data may be a photograph that shows, among other features, anatomical features such as the user's nose, mouth, chin, cheeks, forehead, and eyes.

[0048] In some embodiments, the computing device 220 can perform image processing operations 240 to determine anatomical dimension data related to the facial device user. For example, based on captured image data, the computing device 220 may determine, among other dimensions, face height, nose height, nose width, nose depth, nostril height, or nostril width. In some embodiments, the image processing operations 240 may be based on one or a combination of two-dimensional image data and three-dimensional point cloud data for identifying the anatomical dimension data.

[0049] In some embodiments, the computing device 220 may transmit determined anatomical dimension data related to the facial device user to the server 110 (Figure 1). The server 110 may include a facial device application that includes operations for generating facial device recommendations 250 (e.g., one or more mask selections) based on a trained facial device recommendation model and user input received from one or more computing devices 220. In some embodiments, the facial device recommendations 250 may be based on a dataset stored in a database 280 and anatomical dimension data determined by image processing operations 240.

[0050] Database 280 may include datasets storing features and attributes related to multiple facial devices. These features and attributes may include facial device type, material type, dimensions, or similar. For example, facial devices may be classified based on type, such as full facial devices, nasal fittings, nasal pillow devices, or similar. Each facial device type may be further subdivided, for example, standard, "sub-nasal," etc. In some examples, each facial device type may be further subdivided as either an overhead fitting device or a non-overhead fitting device.

[0051] In some embodiments, the database 280 may include data representing previous user feedback data related to a plurality of facial devices. User feedback may include quantitative (e.g., user star ratings) or qualitative (e.g., text-based comments) data related to previous facial device recommendations for users with specific facial dimensions, preferences, or similar characteristics.

[0052] In some scenarios, providing a facial device to a user based solely on matching the physical dimensions of the given facial device to the corresponding anatomical dimensions of the user's face may not necessarily lead to optimal facial device fit, comfort, and operation. For example, some facial device users of PAP devices may prefer certain types of facial device materials, such as silicone, plastic, or a combination thereof. In some other cases, some facial device users may prefer a facial device that can cover a larger or smaller portion of their face. In some other cases, some facial device users may prefer a facial device with a tighter or looser fit.

[0053] In some scenarios, despite having substantially similar anatomical dimensions, mask device users may have varying facial contours or facial muscle stiffness, which can lead to different experiences regarding fit, comfort, or facial device performance. Therefore, in some embodiments, the server 110 may be configured to generate facial device recommendations based on machine learning behavior or recommendation models. In some embodiments, the recommendation model may be based on a dataset representing previous facial device user data, previously generated facial device recommendations, previous facial device user ratings, or similar.

[0054] In some embodiments, the server 110 may perform the operation of a facial device application to generate a facial device recommendation 250 and then send the facial device recommendation 250 to a computing device 220 for display in the user interface. The facial device recommendation 250 may include a subset of multiple facial devices that may be optimal for the facial device user. In some embodiments, the facial device recommendation 250 may be a sorted list including combinations of multiple facial devices or subtypes.

[0055] In some embodiments, the facial device recommendation 250 may be based on user input received by the computing device 220. Such user input may include, among other examples, data representing patient profiles, treatment requirements provided by a clinician, and user preferences. In some embodiments, the facial device recommendation 250 may include Bayesian-based operation so that the recommendation is based on data representing previously received user feedback from the target user, or in combination with other facial device users with similar user profiles or similar anatomical facial structures.

[0056] In some embodiments, the server 110 can perform operations for centralized patient management 260 and can also receive data from one or more computing devices 220 or from an interactive user interface 230. Such data may be stored in a database 280 (e.g., the MaskFit AR® database).

[0057] For example, user input may include facial device user profile data, treatment prescription data, user messages, or other data related to patient management operations 260. In some embodiments, the computing device 220 may receive facial device user input to create a user profile or to communicate with the clinician user via messaging operations. In some embodiments, the computing device 220 may receive image data via an image capture device to determine anatomical dimension data related to the facial device user. In some embodiments, the computing device 220 may receive data related to responses to a facial device user questionnaire.

[0058] In some embodiments, a computing device 220 associated with a clinician device user may be configured to display data from the facial device user and / or receive data representing edits to the facial device user's data via a display interface.

[0059] In some embodiments, a computing device 220 associated with a clinician device user may be configured to display data of the facial device user and / or receive data representing edits to the facial device user's data via a display interface. In some embodiments, the centralized patient management 260 may include, among other examples, operations for adding existing patient data for storage on the server 110, adding new patient data for storage on the server 110, retrieving data representing patient information based on questionnaires, and saving and modifying patient / user data. In some embodiments, operations associated with the centralized patient management 260 may include operations for providing telemedicine (e.g., video or audio) between the patient user and the clinician user (e.g., similar to a telemedicine platform). In some embodiments, operations associated with the centralized patient management 260 may include operations for accessing patient data and transmitting preliminary patient assessments to a computing device associated with the patient user. Other operations for the centralized patient management 260 may also be contemplated.

[0060] In some embodiments, the system platform 100 may be configured to provide an interactive user interface 230, such as a web portal, on a computing device. In some embodiments, the interactive user interface 230 may be provided on any internet-accessible computing device. In some embodiments, the interactive user interface 230 may be configured to receive input from a treatment administrator, clinician, or similar. In some embodiments, the interactive user interface 230 may be configured to receive patient profile data, provide messaging functionality between users of the system platform 100, and receive a facial device user list, among other examples.

[0061] In some embodiments, the server 110 can perform operations for centralized facial device application management 270. For example, the server 110 can receive a facial device user list, data representing facial device parameters, or similar data, and can perform user management and application configuration operations 270.

[0062] The approach to providing facial device recommendations may be based at least partially on the anatomical structure of the facial device user. In some scenarios, clinicians may provide treatment to patients remotely. For example, a clinician providing treatment for sleep apnea may provide a consultation to a patient regarding a PAP device. Providing multiple sample facial devices to a facial device user (e.g., the patient) for testing can be time-consuming and resource-intensive (e.g., requiring numerous physical samples or proper hygiene / cleaning). Providing facial device recommendations based on image data representing the facial device user may be beneficial.

[0063] The operation for generating anatomical dimension data related to the face of a facial device user may include an operation for face recognition. Identifying facial features may be based on identifying multiple feature landmarks or multiple reference points. Since determining sufficient feature landmarks based on image data representing 2D data may be difficult in some scenarios, it may be beneficial to provide an image processing operation based on 3D point cloud data based on the received image data.

[0064] The client device 120 (Figure 1) may be configured to receive one or more images of the facial device user via an image capture device. In some embodiments, the client device 120 may be configured to determine the anatomical dimensions of the facial device user based on point cloud data associated with the received image data.

[0065] In some embodiments, the point cloud can be a set of data points in three-dimensional space that can function as reference / anchor points for facial features. The point cloud may include data points generated based on features that represent depth or perception through positioning in space and time. In some embodiments, the client device 120 may include an accelerometer device, which can perform actions based on accelerometer data to track the movement of the facial device user in space.

[0066] For the sake of clarity, examples of operation in the Apple iOS® operating system may be provided. Some examples of augmented reality (AR) operations (e.g., RealityKit framework, ARKit framework, SceneKit framework) may not be able to provide point cloud processing. For operations that may require improved data accuracy or 3D geometry reconstruction, the aforementioned examples of AR operations may not be able to utilize a direct point cloud rendering model based on image data captured from an imaging device (e.g., a TrueDepth® device) that provides a combination of visual data and real-time depth data. For example, the face tracking model in the ARKit framework may include .line and .fill rendering modes, but may not be able to perform operations based on depth data. To address some of the challenges suggested above, it may be beneficial to provide a system and method that includes a 3D image data rendering engine.

[0067] As described herein, one or more client devices 120 may include a facial device application that includes operations for determining anatomical dimensional data based on image data of a facial device user. In some embodiments, the facial device application may include operations of the Metal® framework. Metal® may be a hardware-accelerated 3D graphics and compute shader application programming interface based on the Apple iOS® platform. Metal may include OpenGL or OpenCL-like functionality. The Metal API may include operations for rendering 3D graphics and performing data parallel computation based on a graphics processor.

[0068] In some embodiments, one or more client devices 120 may acquire image data representing the anatomical features of the facial device user via an image capture device. One or more client devices 120 may operate a 3D rendering engine to determine anatomical dimensional data related to the facial device user.

[0069] In some embodiments, a three-dimensional (3D) rendering engine disclosed as JMetalRenderer may be an abstraction around a graphical processing unit. The 3D rendering engine may be a programming class for providing devices and command buffers. The 3D rendering engine may generate a single scene object into which content can be populated. In some embodiments, the 3D rendering engine may instantiate multiple other objects to generate anatomical dimensional data.

[0070] Referring to Figure 3, sample programming code for a 3D rendering engine according to an embodiment of the present disclosure is shown.

[0071] In some scenarios, a scene may contain multiple different root nodes to which related content may be added. In some embodiments, a 3D rendering engine may be based on at least four type nodes: a system node, a root node, a depth node, and a face node. In addition to storing node instance examples, the operation of the scene programming class may store node instances and clear colors. In some scenarios, a color may be used as a background for each new frame. In some scenarios, a 3D rendering engine may store a camera in the scene, and using this camera, the scene can be viewed from multiple locations, as if the camera were operating in three-dimensional space.

[0072] Referring to Figure 4, sample programming code for the JScene class according to an embodiment of this disclosure is shown.

[0073] In some embodiments of a 3D rendering engine, a node may determine where in the world a mesh object should be rendered based on rotation, scaling, or translational transformations. For example, a mesh may define a model in local space, and a node may include data representing an action to retrieve local coordinates and map those coordinates to a location in 3D space. In some embodiments, the 3D rendering engine may include a hierarchical structure with child nodes. In some embodiments, the node does not need to be associated with a mesh, so the mesh value may be optional.

[0074] In some embodiments, a programming class disclosed as a JNode class can represent an operation for converting 3D local data into a point in real-world 3D space. A node may include position, orientation, or scale properties that can provide movement of its content in real-world 3D space. In some embodiments, a JNode class may include optional mesh properties that can describe the 3D data associated with the node. A node may include other child nodes that provide a scene hierarchy for the conversion.

[0075] In some embodiments, the JDepthNode programming class can be a child of the JNode object, which can be used to directly render depth data as points in a Metal shader. For illustrative purposes, see Figure 5, which shows sample programming code for the JDepthNode programming class.

[0076] In some embodiments, the 3D rendering engine includes further programming classes for rendering objects. For example, a JMesh programming class may include 3D vertex data used to render an object. A JMesh programming class may include references to materials used to render the model. A JMaterial programming class may define how the model is presented on the display screen (e.g., whether the object should be rendered as a solid color or textured). In some embodiments, a JMaterial programming class may be an abstraction of a vertex or fragment shader setup.

[0077] In some embodiments, the JTexture abstraction class may include MTLTexture and MTLSamplerState objects and may provide helper methods for loading new textures.

[0078] In some scenarios, a 3D rendering engine may include a camera for traversing the scene being captured and for setting properties such as a field of view to control, for example, how the captured scene can be zoomed in or zoomed out. In some embodiments of the 3D rendering engine, perspective projection features may be provided. In some embodiments, orthographic projection features may be provided by creating a camera protocol that may include perspective and orthographic camera features. In some embodiments, the camera may provide view matrix and projection matrix properties, which can be recalculated based on any changes to the camera properties. Such matrices may be provided to a vertex shader having a uniform buffer.

[0079] In some embodiments, the 3D rendering engine may include JMesh vertex data in combination with a mechanism for importing other models. In some scenarios, the 3D model format in real-world 3D space may include .obj, .ply, .usd, or similar. In some embodiments, a Model I / O Programming Interface developed by Apple® may be incorporated for importing and exporting data based on multiple formats.

[0080] In some embodiments, the model I / O programming class may include an MDLAsset programming class that includes a mechanism for loading external data into the model I / O data structure. In scenarios where the MDLAsset programming class is instantiated, the MetalKit programming class can create a MetalKit mesh, an MTKMesh. The MTKMesh may include operations that access an MTLBuffer instance used to render the model.

[0081] Referring to Figure 6, a sample object inheritance table for the custom 3D rendering engine disclosed herein, in accordance with an embodiment of the present disclosure, is shown.

[0082] Referring to Figures 7 to 9, 3D point cloud rendering based on the operation of the 3D rendering engine described above, according to embodiments of the present disclosure, is shown. Figures 7 to 9 show example figures representing test results of custom operation of the 3D rendering engine for 3D point cloud data. In some embodiments, image and depth data acquired from an image capture device (e.g., a TrueDepth® camera) may be processed to provide image data for determining anatomical dimensional data relating to facial features.

[0083] In some scenarios, one or more client devices 120 may capture image data representing the field of view of an image capture device. iOS®-based augmented reality programming interfaces may not provide behavior for identifying and displaying a user's face (for example, the ARKit® framework in iOS® may not include such functionality). It may be beneficial to perform an action that generates a coarse-grained capture of the user's face in substantially real time.

[0084] Referring to Figures 10A and 10B, an embodiment of the present disclosure is shown, which involves the identification of a user's face contour in a captured image and a 3D point cloud representation of the identified contour.

[0085] In some embodiments, the facial device application may include operations to detect the contours of the user's face in substantially real time and to display the detected contours of the user's face on a display interface. For example, in Figure 10A, the facial device application may show a front view of the user's face by enclosing it with a polygonal boundary.

[0086] In some embodiments, the facial device application may include actions for displaying 3D point cloud data related to the user's facial features. For example, in Figure 10B, the facial device application may include visual markers that identify data representing contours or depths related to the user's facial features.

[0087] In some embodiments, a facial device application may include operations for detecting the user's face based on captured image data. In some embodiments, a programming class disclosed as ARFaceGeometry may provide 3D geometric data or topological data related to the user's face. In some embodiments, data markers 1050 that can be combined to represent or depict the contours of the user's face are selectable and can be displayed in the user interface. In some embodiments, data points 1050 shown in 3D space based on the ARFaceGeometry programming class can be converted to 2D pixel points for display on the user interface.

[0088] In some embodiments, a facial device application may include operations for determining a facial boundary marker 1055 based on 2D pixel points provided on a user interface. As an example, Figure 11 shows sample programming pseudocode for determining a facial boundary marker 1055 according to one embodiment of the present disclosure.

[0089] In some embodiments, a facial device application may include operations based on the ARKit programming class (developed by Apple®). In scenarios where ARKit operations determine that a unique face has been identified based on acquired image data, the ARSession programming class may provide an ARFaceAnchor object. The ARFaceAnchor may contain data representing the face's pose, topology, or expression. Furthermore, geometry properties may represent an ARFaceGeometry object representing detailed topology data representing the user's face. Figures 12A and 12B show examples of visual data representing the topology of a detected user's face.

[0090] In some embodiments, the ARFaceGeometry programming class may provide topology data representing the user's face in the form of a 3D mesh diagram. Topology data in the form of a 3D mesh format may be suitable for rendering by multiple third-party technologies or for export as 3D digital data.

[0091] In some embodiments, a facial device application may include actions to determine face geometry from an ARFaceAnchor object during a face-tracking AR session. During a face-tracking AR session, the face model may determine the dimensions, shape, or current expression of the detected face. In some embodiments, the ARFaceAnchor object can be used to generate face mesh data based on stored shape factors, thereby providing a detailed description of the current facial expression.

[0092] In one example of an AR session, a facial device application can utilize a model as a basis for overlaying content based on the contours of the user's face, such as for applying virtual makeup or tattoos. In some embodiments, the face model may generate occlusion geometric objects to mask virtual content behind the 3D shape of the user's face.

[0093] In some embodiments, a facial device application may determine facial features and generate data using point indices of a general face tracking model. In some scenarios, vertex indices of the ARFaceGeometry programming class may be useful for generating facial geometry data. Figures 13A and 13B show enlarged views of a 3D mesh model representing a user's face according to embodiments of this disclosure. These enlarged views show virtual vertex indices.

[0094] In some embodiments, a facial device application may include operations to detect facial features or related geometric data based on the generated vertex index. As an example, the ARFaceGeometry programming class described herein includes 1220 vertices based on the ARKit framework. Referring again to Figure 10B, the user's facial contour recognition shows the vertex index based on the operations of the ARFaceGeometry programming class. For example, a facial device application may include operations to generate automated 3D sensing measurement data based on the ARFaceGeometry generic face tracking model of the entire face or nose structure.

[0095] In some embodiments, a facial device application may include actions to determine head pose based on 3D point cloud data that dynamically or substantially in real time represents the user's head. Actions for head pose estimation may be performed as preprocessing actions for facial feature recognition. In some embodiments, actions for head pose estimation may include detecting the nasal tip based on the convex hull points of the 3D point data.

[0096] Referring to Figure 14, a method 1400 for estimating head pose based on image data, according to an embodiment of the present disclosure, is shown. Method 1400 may be executed by the processor of one or more computing devices described in the present disclosure. In some embodiments, processor-readable instructions may be stored in memory and associated with a facial device application of one or more client devices 120 (Figure 1). Method 1400 may include operations such as data retrieval, data manipulation, data storage, or similar operations, and may also include other computer-executable operations.

[0097] In operation 1402, the processor may generate a convex hull based on a 3D point cloud associated with the user's facial image data. For illustrative purposes, Figure 15A shows a 3D point cloud associated with the user's facial image data. Figure 15B shows a convex hull generated based on the 3D point cloud shown in Figure 15A.

[0098] In operation 1404, the processor may filter the nasal tip point 1510, schematically shown in Figure 15B, based on the minimum z-axis value in the current 3D Euclidean space. For example, in some scenarios, the tip of the convex hull may correspond to the nasal tip point of a typical human face. After constructing the convex hull, the processor may identify a data point with the lowest z-axis coordinate (for example, the z-axis may represent depth) and determine that the identified data point represents the tip of the nose.

[0099] After identifying the nasal tip, the processor may, in operation 1406, generate a polygonal volume or sphere whose center is located at the nasal tip. In some embodiments, the center may be the centroid of the nasal tip data point. The processor may generate a visual indicator to identify the polygonal or spherical shape 1610. In some scenarios, the polygonal or spherical shape may be other shapes depending on the anatomical shape of the user's face.

[0100] In operation 1408, the processor may estimate the surface normal 1620 of the filtered face base region point 1610 as shown in Figure 16B. For example, Figure 16B shows the estimated face XY plane.

[0101] In step 1410, the processor may align the estimated surface normal to the Z-axis, as shown in Figure 17A. In particular, Figure 17A shows a front view aligned to the Z-axis.

[0102] In operation 1412, the processor may filter out the alae points 1730 (Figure 17B) by estimating the normal vector of the data points along the Z axis. In particular, Figure 17B shows the estimated set of alae points.

[0103] In operation 1414, the processor may extract supranasal points and estimate the orientation boundary boxes of the extracted supranasal points. Figure 18A shows the extracted supranasal points 1810, and Figure 18B shows the orientation boundary boxes of the extracted supranasal points.

[0104] In operation 1416, the processor may rotate the normal direction of the maximum boundary length of the estimated orientation boundary box around the Y-axis. Figure 19 shows the aligned 3D face point cloud data. In operation 1416, the processor may rotate the data representing the user's face to obtain a better orientation. For example, if the user's face is tilted to the left or right when acquiring image data representing the user's face, operation 1416 may include rotating the user's head to compensate for the tilted position.

[0105] In some scenarios, it may be beneficial to dynamically or substantially in real time determine the user's facial feature parameters based on image data representing the user's face. In some embodiments, an aligned 3D point cloud representing the face may be associated with the operation for determining the user's facial feature parameters.

[0106] As an example, with reference to Figure 20, a 3D point cloud of a user's face is shown illustrating the detection of a nasal depth parameter according to an embodiment of the present disclosure. Figure 20 shows a front view plane in the XY plane of 3D Euclidean space. In some embodiments, determining the feature parameter may be based on the nasal tip point 2010 extracted from the convex hull points of the raw scanned face points.

[0107] For example, to determine the user's nose depth parameter, a facial device application may include actions to position the spherical region center at the centroid of the nasal tip. The facial device application may also include actions to estimate the face base plane from the spherical region center. Furthermore, the processor may detect the distance between the nasal tip and the face base plane to determine the nose depth.

[0108] In another example, a facial device application may determine the width of a nasal point based on a 3D point cloud of the user's face by estimating the z-axis normal of a candidate nasal tip point 2110. Referring to Figure 21, a 3D point cloud of the user's face showing candidate nasal tip points 2110 is shown. The processor may filter out 3D points 2120 representing the user's nose. Based on the filtered 3D points 2120 representing the user's nose, the processor may determine an estimate of the width of the nasal point.

[0109] In another example, a facial device application may determine the length of the nose based on the face base plane. Referring to Figures 22A and 22B, the detected nasal surface and the determined nose length according to an embodiment of the present disclosure are shown, respectively.

[0110] Figure 22A shows the detected bounding box 2210 of the nasal supraplanar point. The nasal supraplanar point can be filtered based on filtered 3D points 2120 (Figure 21) representing the user's nose by estimating the normal in the z-axis. The recognized bounding box of the nasal supraplanar point can be extended to the y-axis to generate candidate points. Furthermore, the processor can estimate the expected nasal root point 2220, parallel to the xy-plane, by comparing its normal with the angle between the z-axis and the nasal root point. For example, Figure 22B shows an example of the nasal root point 2220. Based on at least the above, the length of the nose can be calculated within the face base plane.

[0111] Referring to Figure 23, a method 2300 for determining anatomical dimensional data of a user's facial features, according to an embodiment of the present disclosure, is shown. Method 2300 may be executed by the processor of one or more computing devices described in the present disclosure. In some embodiments, processor-readable instructions may be stored in memory and associated with a facial device application of one or more client devices 120 (Figure 1). Method 2300 may include operations such as data retrieval, data manipulation, data storage, or similar operations, and may also include other computer-executable operations.

[0112] As a non-limiting example, Method 2300 may include an action for determining the width or height of a user's nostrils. Method 2300 may be based on depth data associated with one or more image frames captured by a TrueDepth® camera. Method 2300 may be based on detecting transitions in pixel color values ​​in the RGB images associated with each ARFrame. Thus, embodiments of the action may automatically determine the width or height of the nostrils in substantially real time.

[0113] In some embodiments, determining anatomical dimension data related to a user's nostrils may be based on operations related to 2D single-frame image data. Such operations may include detecting transitions in pixel color values ​​in an RGB image. In some embodiments, the operations for determining anatomical dimension data may be based on point cloud data, which may be generated based on depth data acquired from an image capture device (e.g., a TrueDepth® camera or similar).

[0114] In operation 2302, the processor may identify the nose or nasal bridge based on the acquired image data. In some embodiments, the processor may include the operation of a vision framework for identifying the user's nose or nasal bridge. For example, Figure 24A shows landmark 2410 representing a nasal bridge feature. Figure 24B shows landmark 2420 representing a nasal feature.

[0115] In operation 2304, the processor may determine the effective evaluation area of ​​the nostrils. In some embodiments, determining the effective evaluation area may be based on either or a combination of landmarks 2410 representing nasal bridge features or landmarks 2420 representing nasal features.

[0116] Figure 25A shows the effective evaluation area 2550 determined based on a combination of landmark 2410 representing nasal ridge features and landmark 2420 representing nasal features. Figure 25B shows the effective evaluation area 2550 determined without showing the aforementioned landmarks.

[0117] In operation 2306, the processor may determine the contour of at least one nostril based on pixel RGB value transitions, based on the diagonals of each quadrilateral boundary. For example, Figure 26 shows a diagonal marking segment 2660 that may correspond to a user's nostrils. In Figure 26, the pixel RGB values ​​may transition as they transition “inward” from the sides of each quadrilateral boundary. In some embodiments, the processor may determine one or more contour points 2662 of the left or right nostril.

[0118] In some embodiments, the processor may determine the “maximum” RGB value based on boundary points of the determined effective evaluation area 2550. The processor may determine the “minimum” RGB value based on an identified central sub-area of ​​the effective evaluation area 2550. In some embodiments, the threshold for identifying contours or boundaries may be based on a determined RGB value range across the “maximum” and “minimum” RGB values. In some embodiments, the processor may include an operation based on the average pixel RGB value of adjacent regions of a given pixel RGB value.

[0119] In operation 2308, the processor may adjust the determined nostril height and width based on the offset and rotation angles to provide nostril measurement parameters. For example, as shown in Figures 27A and 27B, the initial green nostril height and width lines may be optimized with the offset and rotation angles to obtain optimal measurement results. In some embodiments, the intersection of the nostril height line and the width line may be identified as the center point of the rotation or offset operation. In some embodiments, calculations of the amoeba numerical method may be applied to determine the maximum height or maximum width value. In some embodiments, calculations of a nonlinear optimization numerical algorithm may determine the optimal target values ​​for the rotation angle and offset distance.

[0120] Referring to Figure 28, a method 2800 for determining one or more facial device recommendations according to embodiments of the present disclosure is shown. Method 2800 may be executed by the processor of one or more computing devices described in the present disclosure. In some embodiments, processor-readable instructions may be stored in memory and associated with facial device applications of one or more client devices 120 (Figure 1). Method 2800 may include operations such as data retrieval, data manipulation, data storage, or similar operations, and may also include other computer-executable operations.

[0121] In some embodiments, the operation of Method 2800 may include an operation to determine whether the target facial device can fit the user's face for optimal operation or comfort. In some embodiments, the operation to determine whether the target facial device can fit the user's face may be based on facial device dimensions, such as upper boundary dimensions or lower boundary dimensions, and anatomical dimensional data representing the user's face. In some embodiments, Method 2800 may be for providing one or more recommendations for one or more facial device types, such as a full-face device, a nose device, or a nose pillow device.

[0122] In operation 2802, the processor may acquire facial measurement data related to the user's face. In some embodiments, the facial measurement data may be determined based on the operational embodiments described herein. For example, the processor may perform an operation to determine anatomical dimension data related to the size of the nostril opening. In another example, the processor may perform an operation to determine the width or length of the user's nose.

[0123] In some embodiments, determining facial measurement data may be based on one or a combination of 2D image data, such as image data captured based on a TrueDepth® camera, or 3D point cloud image data.

[0124] In operation 2804, the processor may determine that the acquired facial measurement data may fall within the upper boundary (AUB) and lower boundary (ALB) of a particular facial device. In some embodiments, each facial device may be associated with dimensional data suitable for a user with a specific range of anatomical dimensional data (e.g., a user with a specific nose length, nostril opening size, or similar).

[0125] In operation 2804, the processor may perform an operation to determine whether the acquired facial measurement data (related to a specific user) is located in the AUB and ALB of multiple facial devices.

[0126] In response to determining that the acquired facial measurement data is located within the AUB and ALB of a specific facial device, the processor may, in operation 2806, add that specific facial device to the facial device recommendation list.

[0127] In some embodiments, once a list containing one or more facial device recommendations is determined, the processor may perform an operation to sort the list of facial device recommendations based on quantitative or qualitative data (e.g., user reviews, star ratings, among other examples). Once the one or more facial device recommendations are sorted, the processor may perform a scoring operation to score the facial devices in order to identify the facial devices with the top scores for presentation in the user interface.

[0128] In operation 2808, the processor may perform an operation to sort or rank a list of facial devices based on patient information, the device used, and / or patient-specific clinical data. In some embodiments, patient information may include one or more of the following: age, sex, body mass index (BMI), ethnicity, or similar. In some embodiments, the device used may include one or more of the following: CPAP, BiPAP, ASV, and various pressure settings, including minimum and maximum pressure settings. In some embodiments, patient-specific clinical data may include one or more of the following: deviated septum, nasal congestion, seasonal allergies, sleeping position (e.g., prone, supine, side-lying, restless), facial hair (e.g., mustache, beard), skin hypersensitivity, claustrophobia, and similar.

[0129] In some embodiments, scores may be assigned to each facial device type that may perform better for different clinical parameters. An example of a table of scores assigned to various facial device types for specific characteristics is provided in Figure 34. As shown, the scores range from 0 to 3, but it should be understood that any suitable scoring scale or scoring system may be used, and the table in Figure 34 is merely an example.

[0130] As shown in Figure 34, when the patient has claustrophobia, the nasal pillow mask receives the highest recommendation score (score 3), followed by the nasal mask (score 2), and then the full-face mask (score 1). Similarly, when the device pressure setting is high (e.g., higher than 10 cmH2O), the nasal mask is more likely to provide the best seal (score 3), followed by the full-face mask (score 2), and then the nasal pillow mask (score 1). As another example, when the patient has a mustache, the nasal pillow mask receives the highest score (score 3), followed by the full-face mask (score 2), and then the nasal mask (score 1).

[0131] In some embodiments, an overall score can be obtained for each facial device type. In some embodiments, the overall score may be the sum of the scores for each attribute. In some embodiments, the overall score may also be a weighted sum of the scores for each attribute, and the weights may be selected to emphasize or disemphasize the importance of patient quality or attribute. In some embodiments, the scoring system can be used to rank the list of facial devices generated in 2806. Figure 35 is an example of a user interface that provides the user with a list of recommended facial devices.

[0132] In some embodiments, when ranking masks, the patient's history and / or mask preferences may be taken into consideration. For example, a patient may have previously used a particular type of facial device and subsequently had successful treatment with it. For instance, if the scoring system provided herein ranks a nasal pillow mask higher than a nasal mask on the recommendation list, but the patient has previously used a nasal mask successfully, the patient may feel more comfortable using the same type of facial device they are already familiar with (rather than introducing them to a new type of mask). In some embodiments, the scoring system may include an indication of whether the patient has previously used a particular type of facial device, and if so, may include the patient's evaluation of that facial device type based on satisfaction and comfort. If a previously used facial device type is highly rated, the ranking system may push that facial device type higher in the ranking list of facial devices.

[0133] In some embodiments, the AUB / ALB ranges can overlap between facial device sizes. For illustrative purposes, with reference to Figure 29, boundary ranges related to size (e.g., small, medium, large) for an example facial device according to embodiments of this disclosure are shown.

[0134] In some embodiments, a facial device application can define overlapping size boundary values. For example, size boundary values ​​(e.g., S1, S2, M1, etc.) may advance upward or downward based on feedback data received from previous users of each facial device. In some embodiments, one or more computing devices described herein may include a machine learning architecture for generating a facial device recommendation model. Such a facial device recommendation model may dynamically update to adjust the size boundary range or values ​​based on feedback data received from previous users of the facial device.

[0135] In some scenarios, a facial device application may include actions to update specific size boundaries as the scoring or star rating of masks changes over time. Referring to Figure 30, a flowchart of a method 3000 for updating boundary ranges related to generating facial device recommendations, according to embodiments of the present disclosure, is shown. For example, ranking / data manipulation when star ratings change: as masks are ranked, their ranking may change. ALB and AUB may be for the sizes of selected and rated masks.

[0136] Method 3000 may be executed by the processor of one or more computing devices described in this disclosure. In some embodiments, processor-readable instructions may be stored in memory and associated with a facial device application of one or more client devices 120 (Figure 1). Method 3000 may include operations such as data retrieval, data manipulation, data storage, or similar operations, and may also include other computer-executable operations.

[0137] In operation 3002, the processor may determine whether AUB - 3% < measured data input (DI) <= AUB + 3%, or ALB - 3% <= measured data input < ALB + 3%. In the present disclosure, 3% is provided for ease of explanation, but the + / -% value may be larger or smaller based on machine learning operations to improve the recommended model.

[0138] If the processor determines yes in operation 3002, then in operation 3004, the processor may determine whether (AUB + DI) / 2 >= ALB ラージサイズ or (ALB + DI) / 2 <= AUB スモールサイズ

[0139] If the processor determines yes in operation 3004, the processor may replace ALB or AUB with a new average value in operation 3006.

[0140] In operation 3006, the processor may sort the display list by weighted ratings (WR). In some embodiments, the weighted ratings may be the sum of Sn values calculated based on Bayesian analysis (see the following relationship). The processor may include an operation of showing masks with a specific star rating (SR = 0) at the end of the list in alphabetical order.

[0141] In embodiments where the processor determines "no" in operation 3002 or operation 3004, the processor may perform operation 300 to sort the display list by WR. Further, the processor may show facial devices with SR = 0 at the end of the list in alphabetical order.

[0142] ​In some embodiments, the facial device application described herein may obtain user input including an evaluation of the facial device. In some embodiments, the facial device evaluation may be an evaluation related to a new mask or an evaluation of an existing mask. In some embodiments, the processor may determine or update the facial device rank based on the following relationships.

[0143] Each k Assume there are K possible ratings, each indexed by k, which is worth a point. In the case of a “star” rating system, s k =k (for example, 1 point, 2 points, etc.). Given an item, n for k. k Assume that the total rating of N ratings has been received. The facial device application is based on the criteria:

number

[0144] Based on the above-described standard embodiment, the facial device application may include an operation for displaying a list of facial device masks that can be ranked based on an "S number".

[0145] In some embodiments, the scoring of each facial device may be based on a weight or weighted rating representing the likelihood of optimal fit for a given user. In some scenarios, the weight may be associated with a score or other metric for generating facial device recommendations to the user.

[0146] In some embodiments, a computing device may generate a user interface for display on one or more computing devices (Figure 1) according to embodiments of the present disclosure. One or more computing devices may perform automated 3D sensing measurements based on a general tracking model for analyzing the user's face for a full-face / nose facial device.

[0147] Figures 31 to 33 show user interfaces for display on one or more computing devices (of Figure 1) according to embodiments of the present disclosure. The user interface may include a facial device recommendation list or details associated with each facial device. In some embodiments, the user interface may include a user interface for displaying anatomical dimension data based on the image processing operations described in the present disclosure. In some embodiments, the user interface may be associated with an operation for dynamically determining anatomical dimension data related to generating recommendations for nasal pillow facial devices.

[0148] In some embodiments described herein, a computing device may determine anatomical dimensional data related to a user's facial features. In some other embodiments, a computing device may determine dimensional data for a facial device or other nose-mouth device based on image data representing it. For example, a computing device may be configured to receive image data (e.g., a photograph) of a nose-mouth device and to determine the physical dimensions of the nose-mouth device. Thus, a computing device may generate a set of dimensional data related to multiple nose-mouth devices based on at least a combination of (i) device manufacturer specifications and (ii) dimensional data determined based on image data.

[0149] The terms “connected” or “joined” can include both direct joining (two elements joined together are in contact with each other) and indirect joining (at least one additional element is located between the two elements).

[0150] While embodiments have been described in detail, it should be understood that various modifications, substitutions, and alterations can be made herein without departing from the scope. Furthermore, the scope of this disclosure is not intended to be limited to specific embodiments of the processes, machines, manufactures, composition of materials, means, methods, and steps described herein.

[0151] As will be readily apparent to those skilled in the art from this disclosure, currently existing or subsequently developed processes, machines, manufactures, compositions of materials, means, methods, or steps may be used to perform substantially the same functions or achieve substantially the same results as the corresponding embodiments described herein. Accordingly, the appended claims are intended to include within their scope such processes, machines, manufactures, compositions of materials, means, methods, or steps.

[0152] This description provides numerous embodiments of the subject matter of the invention. While each embodiment represents a single combination of the inventive elements, the subject matter is considered to encompass all possible combinations of the disclosed elements. Thus, if one embodiment has elements A, B, and C, and a second embodiment has elements B and D, the subject matter is considered to encompass other remaining combinations of A, B, C, or D, even if not explicitly disclosed.

[0153] Embodiments of the apparatus, systems, and methods described herein may be implemented in combination of both hardware and software. These embodiments may be implemented on a programmable computer, each computer comprising at least one processor, a data storage system (including volatile memory, non-volatile memory, other data storage elements, or a combination thereof), and at least one communication interface.

[0154] The program code is applied to the input data to execute the functions described herein and generate output information. The output information is provided to one or more output devices. In some embodiments, the communication interface may be a network communication interface. In embodiments in which multiple elements are combined, the communication interface may be a software communication interface, such as one for inter-process communication. In yet other embodiments, there may be combinations of communication interfaces implemented as hardware, software, and a combination thereof.

[0155] Throughout the above description, numerous references are made to servers, services, interfaces, portals, platforms, or other systems formed from computing devices. It should be understood that the use of such terms is considered to represent one or more computing devices having at least one processor configured to execute software instructions stored in a computer-readable, tangible, non-temporary medium. For example, a server may include one or more computers operating as a web server, database server, or other type of computer server to perform the described roles, responsibilities, or functions.

[0156] The technical solutions of the embodiments may take the form of a software product. The software product may be stored on a non-volatile or non-temporary storage medium, which may be a compact disc read-only memory (CD-ROM), a USB flash disk, or a removable hard disk. The software product includes a number of instructions that enable a computer device (personal computer, server, or network device) to perform the method provided by the embodiments.

[0157] The embodiments described herein are implemented by physical computer hardware, including computing devices, servers, receivers, transmitters, processors, memory, displays, and networks. The embodiments described herein provide useful physical machines and specially configured computer hardware configurations.

[0158] To ensure clarity, the examples described and illustrated above are intended to be illustrative only.

Claims

1. A computing device for adaptively generating facial device selections based on visually determined anatomical dimension data, Processor and A memory coupled to the aforementioned processor, which stores processor-executable instructions, and when the processor-executable instructions are executed, Receive image data representing the user's face, Based on the received image data, anatomical dimension data related to the user's facial region is determined. Based on the recommended model and the anatomical dimension data, one or more facial device recommendations are generated, and generating such one or more facial device recommendations includes determining that the anatomical dimension data falls within the upper and lower limits of the expected facial device specifications. To provide recommendations for one or more facial devices for display in the user interface, The processor is configured such as memory and A computing device having the following features.

2. The aforementioned memory, when executed, The system receives target therapy data, which includes at least one of user preferences or target facial device specification data, via the user interface. The processor includes processor-executable instructions that constitute the processor, The one or more facial device recommendations generated are based on the target therapy data combined with the anatomical dimension data. The computing device according to claim 1.

3. The computing device according to claim 1, wherein the one or more facial device recommendations include at least one of a full facial / nasal mask or a nasal pillow mask associated with a PAP device.

4. The computing device according to claim 1, wherein the received image data includes a combination of RGB image data and infrared depth data.

5. The computing device according to claim 1, wherein at least one boundary specification of a given facial device size overlaps with another boundary specification of an adjacent facial device size.

6. The computing device according to claim 1, wherein determining anatomical dimension data related to the region of the user's face is based on at least one of three-dimensional (3D) point cloud data or two-dimensional image data.

7. The computing device according to claim 1, wherein determining anatomical dimension data includes determining nostril opening dimensions based on transitions of RGB image data values ​​exceeding one or more thresholds.

8. The computing device according to claim 7, wherein the anatomical dimension data to be determined includes at least one of nostril opening size, nasal width, nasal depth, nasal length, or facial width.

9. The computing device according to claim 1, wherein the memory includes processor-executable instructions that, when executed, configure the processor to estimate head pose based on 3D point cloud data.

10. A computing device for adaptively generating facial device selections based on visually determined anatomical dimension data, Processor and A memory coupled to the processor and storing processor-executable instructions, wherein when the processor-executable instructions are executed by the processor, Receive image data representing the user's face, Based on the received image data, anatomical dimension data related to the user's facial region is determined. Based on the recommended model and the anatomical dimension data, one or more facial device recommendations are generated. To provide recommendations for one or more facial devices for display in the user interface, The processor is configured such as memory and It has, The aforementioned memory, when executed, Based on the received image data, facial features are identified. The effective evaluation area of ​​the aforementioned facial features is determined, and the aforementioned facial features include the nostrils. Based on the diagonals of each quadrilateral boundary marker, the contour of at least one nostril is determined based on the pixel RGB value transitions. Based on at least one of offset or rotation angle, the determined nostril height and width are adjusted to provide nostril measurement parameters. The processor includes, Computing device.

11. A computing device for adaptively generating facial device selections based on visually determined anatomical dimension data, Processor and A memory coupled to the processor and storing processor-executable instructions, wherein when the processor-executable instructions are executed by the processor, Receive image data representing the user's face, Based on the received image data, anatomical dimension data related to the user's facial region is determined. Based on the recommended model and the anatomical dimension data, one or more facial device recommendations are generated. To provide recommendations for one or more facial devices for display in the user interface, The processor is configured such as memory and It has, The aforementioned memory, when executed, A convex hull is generated based on the 3D point cloud associated with the user's facial image data. The nasal tip is filtered based on the minimum z-axis value in 3D Euclidean space. A sphere is generated with the aforementioned nasal tip as the center, The surface normal of the filtered face base region is generated, Align the estimated surface normal with the z-axis, The alae points are filtered based on estimating the normal vector of the data point on the z-axis. Extract points on the upper surface of the nose, and estimate the orientation boundary box of the extracted points on the upper surface of the nose. Rotate the normal direction of the maximum boundary length of the estimated orientation boundary box around the y-axis. The processor includes, Computing device.

12. The computing device according to claim 1, further comprising assigning a score to one or more patient attributes for each of the one or more facial device recommendations.

13. The computing device according to claim 12, wherein the patient's attributes include at least one of deviated septum, nasal congestion, seasonal allergies, sleeping posture, facial hair, skin hypersensitivity, and / or claustrophobia.

14. A method for adaptively generating facial device selection based on visually determined anatomical dimension data, Receive image data representing the user's face, Based on the received image data, anatomical dimension data related to the user's facial region is determined. Based on the recommended model and the anatomical dimension data, one or more facial device recommendations are generated, and generating such one or more facial device recommendations includes determining that the anatomical dimension data falls within the upper and lower limits of the expected facial device specifications. To provide recommendations for one or more facial devices for display in the user interface, A method having the following characteristics.

15. A non-temporary computer-readable medium storing machine-interpretable instructions, wherein, when executed by a processor, the machine-interpretable instructions cause the processor to execute a computer-implemented method for adaptively generating facial device selections based on visually determined anatomical dimensional data, the method being Receive image data representing the user's face, Based on the received image data, anatomical dimension data related to the user's facial region is determined. Based on the recommended model and the anatomical dimension data, one or more facial device recommendations are generated, and generating such one or more facial device recommendations includes determining that the anatomical dimension data falls within the upper and lower limits of the expected facial device specifications. To provide recommendations for one or more facial devices for display in the user interface, A non-temporary computer-readable medium having the ability to do so.

16. The computing device according to claim 1, wherein the received image data includes a photograph of at least a portion of the user's face captured by an image capture device.

17. The computing device according to claim 1, wherein the received image data excludes photographic data of at least a portion of the user's face.

18. The computing device according to claim 1, further comprising determining the anatomical dimension data relating to the region of the user's face, and then deleting the received image data from the computing device.

19. The computing device according to claim 18, wherein the received image data is deleted before generating the one or more facial device recommendations.

20. The computing device according to claim 1, wherein the processor executable instruction is configured, when executed, to further transmit a transmission to a server, the server is configured to perform the generation of one or more facial device recommendations, the transmission excluding personally identifiable user information.

21. The computing device according to claim 1, wherein the facial device has a device configured to be positioned on the nose of the user's face.

22. The computing device according to claim 1, wherein the facial device comprises a breathing mask, a gas mask, and / or a headgear device.

Images