Systems and methods for adaptively generating facial device selection based on visually determined anatomical dimension data

Abstract

An apparatus and method for adaptively generating facial device selections based on visually determined anatomical dimension data is disclosed. The apparatus may include a processor and a memory. The memory may include processor-executable instructions for receiving image data representing a user's face and determining anatomical dimension data associated with a nose-mouth region of the user's face based on the received image data. The processor-executable instructions may be for generating one or more facial device recommendations based on the recommendation model and the anatomical dimension data, and providing the one or more facial device recommendations for display in a user interface.

Description

[Technical Field]

[0001] This claims the benefit of U.S. Provisional Patent Application No. 63 / 166,723, filed March 26, 2021, the entire contents of which are incorporated herein by reference.

[0002] FIELD Embodiments of the present disclosure relate to the field of facial devices, and more particularly to adaptively generating facial device selections. [Background technology]

[0003] A facial device or garment may be positioned adjacent to a user's nasal and / or oral cavity for a particular function. For example, a facial mask may be positioned adjacent to a user's nasal and / or oral cavity to provide a physical barrier between the user's environment and the user's respiratory system. In another example, a facial mask of a continuous positive airway pressure (CPAP) machine may be positioned adjacent to a user's nasal (e.g., over the user's nose) and / or oral (e.g., over the user's mouth) cavity as an interface for providing air pressure to the user's nose and / or mouth while the user is asleep. Other facial device examples may also be contemplated. In some scenarios, the effectiveness of a facial device may be correlated to how well the facial device fits to the user's face. Summary of the Invention

[0004] In one aspect, the present disclosure provides a computing device for adaptively generating facial device selections based on visually determined anatomical dimension data. The device may include a processor and a memory coupled to the processor. The memory may store processor-executable instructions, when executed, that configure the processor to: receive image data representing a user's face; determine anatomical dimension data associated with a nose-mouth region of the user's face based on the received image data; generate one or more facial device recommendations based on a recommendation model defined based on a Bayesian-based scoring metric and the anatomical dimension data; and provide the one or more facial device recommendations for display in a user interface.

[0005] In another aspect, the present disclosure provides a method for adaptively generating facial device selections based on visually determined anatomical dimension data, which may include receiving image data representing a user's face, determining anatomical dimension data associated with a nose-mouth region of the user's face based on the received image data, generating one or more facial device recommendations based on a recommendation model defined based on a Bayesian-based scoring metric and the anatomical dimension data, and providing the one or more facial device recommendations for display in a user interface.

[0006] In another aspect, one or more non-transitory computer-readable media store machine-interpretable instructions that, when executed by a processor, can cause the processor to perform one or more methods described herein.

[0007] In various further aspects, the present disclosure provides corresponding systems and devices, as well as logical structures, such as machine-executable coded instruction sets, for implementing such systems, devices, and methods.

[0008] In this regard, before describing at least one embodiment in detail, it is to be understood that such embodiments are not limited in their application to the details of construction and the arrangements of components set forth in the following description or illustrated in the drawings. Also, it is to be understood that the phraseology and terminology used herein is for the purpose of description and should not be regarded as limiting.

[0009] Numerous additional features and combinations of the embodiments described herein will become apparent to those skilled in the art after reading this disclosure. [Brief explanation of the drawings]

[0010] Embodiments are illustrated by way of example in the drawings, in which: It is to be clearly understood that the description and drawings are for purposes of illustration only and as an aid to understanding.

[0011] Embodiments will now be described, by way of example only, with reference to the accompanying drawings, in which: [Figure 1] 1 illustrates a system platform according to an embodiment of the present disclosure. [Figure 2] 2 shows a block diagram illustrating the operation of the system platform of FIG. 1; [Figure 3] 3-5 illustrate sample programming code according to embodiments of the present disclosure. [Figure 4] 3-5 illustrate sample programming code according to embodiments of the present disclosure. [Figure 5] 3-5 illustrate sample programming code according to embodiments of the present disclosure. [Figure 6] 1 illustrates a sample object inheritance table according to an embodiment of the present disclosure. [Figure 7] 7-9 illustrate 3D point cloud renderings according to embodiments of the present disclosure. [Figure 8]7-9 illustrate 3D point cloud renderings according to embodiments of the present disclosure. [Figure 9] 7-9 illustrate 3D point cloud renderings according to embodiments of the present disclosure. [Figure 10A] 10A and 10B illustrate contouring of a user's face according to an embodiment of the present disclosure. [Figure 10B] 10A and 10B illustrate contouring of a user's face according to an embodiment of the present disclosure. [Figure 11] 1 illustrates sample programming pseudocode according to an embodiment of the present disclosure. [Figure 12A] 12A and 12B show example visual data representing a topology of a face according to an embodiment of the present disclosure. [Figure 12B] 12A and 12B show example visual data representing a topology of a face according to an embodiment of the present disclosure. [Figure 13A] 13A and 13B show close-up views of a 3D mesh model according to an embodiment of the present disclosure. [Figure 13B] 13A and 13B show close-up views of a 3D mesh model according to an embodiment of the present disclosure. [Figure 14] 1 illustrates a method for estimating head pose based on image data, according to an embodiment of the present disclosure. [Figure 15A] 15A, 15B, 16A, 16B, 17A, 17B, 18A, 18B, 19-21, 22A, and 22B show images based on 3D point cloud data related to facial features according to embodiments of the present disclosure. [Figure 15B] 15A, 15B, 16A, 16B, 17A, 17B, 18A, 18B, 19-21, 22A, and 22B show images based on 3D point cloud data related to facial features according to embodiments of the present disclosure. [Figure 16A]15A, 15B, 16A, 16B, 17A, 17B, 18A, 18B, 19-21, 22A, and 22B show images based on 3D point cloud data related to facial features according to embodiments of the present disclosure. [Figure 16B] 15A, 15B, 16A, 16B, 17A, 17B, 18A, 18B, 19-21, 22A, and 22B show images based on 3D point cloud data related to facial features according to embodiments of the present disclosure. [Figure 17A] 15A, 15B, 16A, 16B, 17A, 17B, 18A, 18B, 19-21, 22A, and 22B show images based on 3D point cloud data related to facial features according to embodiments of the present disclosure. [Figure 17B] 15A, 15B, 16A, 16B, 17A, 17B, 18A, 18B, 19-21, 22A, and 22B show images based on 3D point cloud data related to facial features according to embodiments of the present disclosure. [Figure 18A] 15A, 15B, 16A, 16B, 17A, 17B, 18A, 18B, 19-21, 22A, and 22B show images based on 3D point cloud data related to facial features according to embodiments of the present disclosure. [Figure 18B] 15A, 15B, 16A, 16B, 17A, 17B, 18A, 18B, 19-21, 22A, and 22B show images based on 3D point cloud data related to facial features according to embodiments of the present disclosure. [Figure 19] 15A, 15B, 16A, 16B, 17A, 17B, 18A, 18B, 19-21, 22A, and 22B show images based on 3D point cloud data related to facial features according to embodiments of the present disclosure. [Figure 20]15A, 15B, 16A, 16B, 17A, 17B, 18A, 18B, 19-21, 22A, and 22B show images based on 3D point cloud data related to facial features according to embodiments of the present disclosure. [Figure 21] 15A, 15B, 16A, 16B, 17A, 17B, 18A, 18B, 19-21, 22A, and 22B show images based on 3D point cloud data related to facial features according to embodiments of the present disclosure. [Figure 22A] 15A, 15B, 16A, 16B, 17A, 17B, 18A, 18B, 19-21, 22A, and 22B show images based on 3D point cloud data related to facial features according to embodiments of the present disclosure. [Figure 22B] 15A, 15B, 16A, 16B, 17A, 17B, 18A, 18B, 19-21, 22A, and 22B show images based on 3D point cloud data related to facial features according to embodiments of the present disclosure. [Figure 23] 1 illustrates a method for determining anatomical dimension data of facial features according to an embodiment of the present disclosure. [Figure 24A] 24A, 24B, 25A, 25B, 26, 27A, and 27B show RGB images for determining data representing facial features according to embodiments of the present disclosure. [Figure 24B] 24A, 24B, 25A, 25B, 26, 27A, and 27B show RGB images for determining data representing facial features according to embodiments of the present disclosure. [Figure 25A] 24A, 24B, 25A, 25B, 26, 27A, and 27B show RGB images for determining data representing facial features according to embodiments of the present disclosure. [Figure 25B]24A, 24B, 25A, 25B, 26, 27A, and 27B show RGB images for determining data representing facial features according to embodiments of the present disclosure. [Figure 26] 24A, 24B, 25A, 25B, 26, 27A, and 27B show RGB images for determining data representing facial features according to embodiments of the present disclosure. [Figure 27A] 24A, 24B, 25A, 25B, 26, 27A, and 27B show RGB images for determining data representing facial features according to embodiments of the present disclosure. [Figure 27B] 24A, 24B, 25A, 25B, 26, 27A, and 27B show RGB images for determining data representing facial features according to embodiments of the present disclosure. [Figure 28] 1 illustrates a method for determining one or more facial device recommendations according to an embodiment of the present disclosure. [Figure 29] 1 illustrates sample boundary ranges related to facial device sizes, according to embodiments of the present disclosure. [Figure 30] 1 illustrates a flowchart of a method for updating boundary ranges associated with generating facial device recommendations, according to an embodiment of the present disclosure. [Figure 31] 31-33 illustrate user interfaces for providing facial device recommendations according to embodiments of the present disclosure. [Figure 32] 31-33 illustrate user interfaces for providing facial device recommendations according to embodiments of the present disclosure. [Figure 33] 31-33 illustrate user interfaces for providing facial device recommendations according to embodiments of the present disclosure. [Figure 34] 1 is an example table showing scores assigned to each facial device type for patient demographics and clinical patient data. [Figure 35]1 is an example of a user interface for providing ranked facial device recommendations according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0012] Systems and methods for adaptively generating facial device selection based on visually determined anatomical dimensional data are described in this disclosure. For ease of explanation, embodiments of the present disclosure may be described based on examples of facial devices associated with positive airway pressure (PAP) devices. PAP devices may include continuous positive airway pressure (CPAP), bilevel or BiPAP machines, non-invasive ventilators (NIV), and adaptive servo ventilators (ASV). PAP machines may be configurable to provide a flow of air pressure to a user's nose or mouth while the user is asleep. Such a flow of air pressure to the user's nose or mouth may help keep the user's airway open, thereby supporting normal breathing. An example of such a facial device may be an interface between the PAP device and the user's respiratory system, where a combination of fit and comfort contribute to optimal operation of such a facial device.

[0013] In some scenarios, there may be several different types or manufacturers of PAP devices. Each manufacturer's PAP device may be associated with a facial device with unique characteristics related to delivering air pressure to the user's nose or mouth. Since the fit and comfort of a facial device for a particular user may be an important consideration related to the use of a PAP device, there may not be a one-size-fits-all mask device. It may be beneficial to provide a system and method for dynamically generating facial device recommendations for each user.

[0014] Although the present disclosure describes examples related to facial devices associated with PAP machines, systems and methods for dynamically providing recommendations for other types of facial devices are also contemplated. For example, embodiments of the systems and methods described herein may be for dynamically providing recommendations for respiratory mask device fitting (e.g., N95 mask fitting), headgear device fitting, or the like, among other examples.

[0015] 1, a system platform 100 according to an embodiment of the present disclosure is shown. The system platform 100 may include multiple computing devices that send and receive data messages to other computing devices via a network 150.

[0016] In some embodiments, system platform 100 may be configured to dynamically generate facial device recommendations based on one or more of quantitative data or qualitative data. In some embodiments, the quantitative data may include acquired image data associated with a potential user of the facial device. In some scenarios, the image data may represent anatomical dimension data associated with the user. For example, the anatomical dimension data may provide the distance between the user's nostrils, the width or length of the user's nose, and the width of the user's face, among other examples. As described in this disclosure, system platform 100 may include an application for dynamically determining the quantitative data based on the acquired image data.

[0017] In some embodiments, the quantitative data may include data related to facial device specifications retrieved from a client device or a service provider device. For example, the facial device specifications may include PAP device tuning parameters obtained via one or more service provider devices.

[0018] In some scenarios, a substantially exact fit may not result in optimal comfort and performance for all users. In some embodiments, the qualitative data may include data related to user feedback data associated with a facial device category or an anatomical data category. For example, a category of facial device users with an inter-nostril distance of approximately 0.5 centimeters may not all be well-fitted with a given facial device. Thus, in some embodiments, system platform 100 may dynamically generate facial device recommendations based on one or a combination of quantitative or qualitative data.

[0019] The system platform 100 may include multiple computing devices, such as, for example, a server 110 , a web interface platform 112 , one or more client devices 120 , or one or more service provider devices 130 .

[0020] One or more client devices 120 may be associated with a respective facial device user. For example, a mask device user may be a PAP device user. It may be beneficial to generate recommendations for the facial device user to provide the facial device user with an optimally sized facial device with respect to functionality and comfort, among other attributes. These recommendations may be a subset of facial devices recommended based on qualitative or quantitative data related to the facial device user, previous facial device recommendations associated with users with substantially similar anatomical characteristics to the facial device user, or facial device configuration specifications provided by a service provider user (e.g., a clinician, a technician, among other examples). For example, the qualitative or quantitative data may include measurement data related to the user's facial features, user demographic data, clinical data, PAP machine device setting data, data representative of previous user satisfaction, user preference data (e.g., data representative of a given user's previous satisfaction), or data representative of dimensions associated with the facial device.

[0021] In some embodiments, one or more client devices 120 may be configured to receive, via a user interface, user data that provides mask device recommendations. For example, the user data may include user demographic data, user preference data, and PAP therapy prescription data, among other examples.

[0022] In some embodiments, one or more client devices 120 may be configured to capture image data related to a mask device user via an image capture device. The one or more client devices 120 may perform an operation to extract anatomical feature data based on the image data. The one or more client devices 120 may transmit the extracted anatomical feature data and user data to server 110 so that an operation for mask device recommendation may be performed.

[0023] One or more service provider devices 130 may be associated with a clinician or technician user. In this example, a clinician or technician user may include a medical professional engaged in providing medical care or therapy to a facial device user. One or more service provider devices 130 may be configured to receive, via a user interface, clinician data regarding a facial device user or a treatment plan associated with a facial device user. In some embodiments, one or more service provider devices 130 may receive patient management data, including desired facial device specification data.

[0024] In some embodiments, server 110 may receive user data or clinician data as input for generating mask device recommendations based on facial device recommendation models via network 150. Server 110 may include a machine learning architecture for performing operations for generating one or more facial device recommendation models, as well as for generating facial device recommendations based on a combination of datasets representing qualitative and quantitative data. In some embodiments, server 110 may be based on Amazon® Relational Database Services (RDS), accessible via an application programmable interface.

[0025] In some embodiments, server 110 may include memory including one or a combination of computer memory such as, for example, random access memory, read-only memory, electro-optical memory, magneto-optical memory, erasable programmable read-only memory, electrically erasable programmable read-only memory, ferroelectric random access memory, or the like. In some embodiments, memory may be a storage medium such as, for example, a hard disk drive, a solid-state drive, an optical drive, or other type of memory.

[0026] The memory may include or store a database structure. The database structure may store data records received from one or more client devices 120 or service provider devices 130. In some embodiments, the database structure may include a large data set representing previous mask device recommendation results.

[0027] In some embodiments, datasets may be managed based on the characteristics of the PostgreSQL 12 database. PostgreSQL may be an open-source object-relational database system that uses or extends the SQL language combined with features for storing and scaling large dataset operations. PostgreSQL may manage internal security by role. In some examples, roles may be associated with users or groups (e.g., roles of which other roles may be members). In some scenarios, permissions may be granted or revoked at database column-level granularity, and may allow / prevent the creation of new objects within database structure, schema, or table-level granularity.

[0028] In some embodiments, server 110 may send or receive data to or from other computing devices via interface platform 112. For example, interface platform 112 may be a Django-based platform. The Django platform may include security mechanisms configured to counter potential data security threats associated with transmitting or receiving data between multiple computing devices. In some other examples, interface platform 112 may be configured based on PHP scripts to implement mechanisms to counter potential data security threats.

[0029] In some embodiments, the interface platform 112 may be based on features such as cross-site scripting protection, cross-site request forgery protection, SQL injection protection, clickjacking protection, or secure password hashing, among other examples.

[0030] Network 150 may include a wired or wireless wide area network (WAN), a local area network (LAN), a combination thereof, or other network for carrying communication signals. In some embodiments, network communications may be based on HTTP post requests or TCP connections. Other network communication operations or protocols may also be contemplated. In some embodiments, network 150 may include other networks, including the Internet, Ethernet, plain old telephone service lines, public switched telephone networks, integrated services digital networks, digital subscriber lines, coaxial cable, optical fiber, satellite, mobile, wireless, SS7 signaling networks, fixed lines, local area networks, wide area networks, or combinations of one or more of these networks.

[0031] In some embodiments, one or more client devices 120 may store and execute a facial device application. Embodiments of the facial device application may be configured for a mobile device operating system, such as iOS® or Android®.

[0032] In some examples, native iOS® applications may be based on the SwiftUI framework. In some examples, native Android® applications may be based on Java® and OpenCV / MLKit. In some examples, one or more applications may be developed based on React JS, a JavaScript® library package for generating user interfaces or user interface components.

[0033] In some embodiments, one or more client devices 120 may include an image capture device and may be configured to capture image data of a facial device user. As described in this disclosure, embodiments of the facial device recommendation application may generate facial device recommendations based on anatomical dimensions (e.g., dimensions of the user's face) determined from the captured image data.

[0034] In some embodiments, one or more servers 110 may store datasets for generating and training a facial device recommendation model. In some embodiments, the facial device recommendation model may generate facial device recommendations based on input datasets related to a facial device user. For example, the input datasets may include anatomical dimension data. In some embodiments, the anatomical dimension data may include data representing facial height, nose height, nose width, nose depth, nostril height, or nostril width, among other examples. In some examples, the input datasets may include user demographic data, user preference data, or other user input received from client device 120. In some embodiments, the input datasets may include CPAP therapy prescription data or clinician-provided data received from service provider device 130.

[0035] In some embodiments, the server 110 may generate a recommendation model that may be based on a deep learning neural network trained on a big data set.

[0036] In some embodiments, it may be beneficial to reduce the amount of personally identifiable information (PII) transmitted over network 150. PII may include user data or user image associated with the facial device user. In some embodiments, the operations for capturing image data and generating anatomical dimension data based on the captured image data may occur locally at one or more client devices 120. One or more client devices 120 may then transmit aggregated data or a subset of the mask device user's data to server 110 for generating a mask device recommendation. One or more client devices 120 may not store in persistent memory any of the image data associated with the facial device user, the three-dimensional model based on the image data, or the anatomical dimensions.

[0037] In some embodiments, client device 120 or service provider device 130 may not have direct access to data sets or data records stored on server 110. Client device 120 or service provider device 130 may send data retrieval and data modification requests to server 110, which may be configured to validate such requests and generate responses thereto. Such implementations may be based on the assumption that “front-end” devices, such as client device 120, may be untrusted or unprotected devices. Establishing rules governing the flow of data requests can improve or reduce behavior that may compromise data security or data integrity of system platform 100.

[0038] In some embodiments, client device 120 or service provider device 130 may not directly transmit or access data stored on server 110. Rather, server 110 may be configured to receive data requests via an application programmable interface, and may be configured to create or modify database structures or data sets stored in memory.

[0039] In some embodiments, data messages transmitted between computing devices may be via network 150 and interface platform 112. In some embodiments, transmitted data messages may be based on the HTTPS (TLS version 1.2) protocol.

[0040] In some embodiments, the interface platform 112 may be a web interface platform based on an Nginx 1.19 reverse proxy server. In some embodiments, the interface platform 112 may include a cloud computing platform including a Django 2.2 web application on Amazon EC2 (HTML, CSS, Javascript®). In some embodiments, the interface platform 112 may include an authentication process based on Django session authentication. In some embodiments, the web interface platform 112 may include a Representational State Transfer (REST) ​​Application Programmable Interface (API) based on the Django REST Framework 3.10.

[0041] In some embodiments, data messages transmitted over network 150 may be based on the HTTPS protocol, which has the capability to encrypt data messages. The protocol may be Transport Layer Security (TLS version 1.2). The protocol may secure communications based on an asymmetric public key infrastructure. Even if intercepted, data packets transmitted based on the HTTPS protocol may appear as meaningless data to a malicious or unintended user.

[0042] In some embodiments, server 110 may be based on Django 2.2 Server. In some embodiments, server 110 may include a database structure based on a PostgreSQL 12 database on Amazon RDS. In some scenarios, Amazon RDS may encrypt the database structure based on encryption keys managed using the AWS Key Management Service (KMS). On database instances running with Amazon RDS encryption, data stored at rest in the underlying storage may be encrypted, as may automated backups, read replicas, or database snapshots. RDS encryption may be based on the industry-standard AES-256 encryption algorithm for encrypting data.

[0043] It will be appreciated that the above examples are illustrative and alternative mechanisms may be contemplated.

[0044] Facial device recommendations based primarily on the anatomical dimensions of a facial device user may be suitable to address the user's fit, preferences, or a combination of qualitative characteristics that may contribute to optimal facial device operation and fit. In some scenarios, optimal facial device functionality or facial device user comfort may be based on a combination of both quantitative data (e.g., physical dimensions) and qualitative data (e.g., user preferences related to material stiffness, how the mask sits on the user's face, among other examples). In some examples, qualitative data for generating facial device recommendations may include user demographic data (e.g., age, gender, ethnicity, geographic location, or the like) or clinical data (e.g., the user's tendency to be sensitive to or experience nasal congestion, seasonal allergies, feelings of claustrophobia, skin sensitivities, or the like). In some embodiments, the qualitative data may represent the user's preferred sleeping position. In some embodiments, quantitative data for generating facial device recommendations may include, among other examples, physical dimensions of the user's facial features, target PAP machine or facial device settings (e.g., minimum or maximum pressure settings), etc. As described in this disclosure, server 110, one or more client devices 120, or one or more service provider devices 130 may include a machine learning architecture for generating and improving facial device recommendations based on a dataset representing a combination of previous mask device recommendations and user feedback.

[0045] Referring to FIG. 2, a block diagram 200 illustrating the operation of system platform 100 of FIG. 1 is shown, in accordance with an embodiment of the present disclosure. Operations may be performed on one or more of the computing devices of system platform 100. As an example, computing device 220 may include a mobile device running a mobile operating system such as iOS® or Android®. In some embodiments, one or more computing devices 220 may be client device 120 or service provider device 130 shown in FIG. 1. In some embodiments, server 110 may include secure storage that provides database 280 (e.g., a MaskFit AR® database).

[0046] In some embodiments, one or more computing devices 220 may be configured to receive input data from a facial device user, a service provider user (e.g., a clinician or technician providing medical care or treatment), or other user who provides data relevant to generating facial device recommendations for the facial device user.

[0047] In a scenario in which the computing device 220 is associated with a facial device user, the computing device 220 may capture image data (e.g., a photograph) of the facial device user via an image capture device. The image data may be a photograph that may show anatomical features such as the user's nose, mouth, chin, cheeks, forehead, and eyes, among other features.

[0048] In some embodiments, the computing device 220 may perform image processing operations 240 to determine anatomical dimension data associated with the facial device user. For example, the computing device 220 may determine, among other dimensions, facial height, nose height, nose width, nose depth, nostril height, or nostril width based on the captured image data. In some embodiments, the image processing operations 240 may be based on one or a combination of two-dimensional image data and three-dimensional point cloud data to identify the anatomical dimension data.

[0049] In some embodiments, computing device 220 may transmit determined anatomical dimension data associated with the facial device user to server 110 (FIG. 1). Server 110 may include a facial device application that includes operations for generating facial device recommendations 250 (e.g., one or more mask selections) based on the trained facial device recommendation model and user input received from one or more computing devices 220. In some embodiments, facial device recommendations 250 may be based on a dataset stored in database 280 and anatomical dimension data determined by image processing operations 240.

[0050] The database 280 may include a dataset storing features and attributes associated with a plurality of facial devices. The features and attributes of the facial devices may include facial device type, material type, dimensions, or the like. For example, the facial devices may be categorized based on type, such as a full facial device, a nose-fitting device, a nose pillow device, or the like. Each facial device type may be subcategorized, such as standard, "under the nose," etc. In some examples, each facial device type may be subcategorized as being an over-the-head fitting device or a non-over-the-head fitting device.

[0051] In some embodiments, database 280 may include data representing previous user feedback data associated with a plurality of facial devices. User feedback may include quantitative (e.g., user star ratings) or qualitative (e.g., text-based comments) feedback related to previous facial device recommendations for users with particular facial dimensions, preferences, or the like.

[0052] In some scenarios, providing a facial device to a user based solely on matching the physical dimensions of a given facial device to the corresponding anatomical dimensions of the user's face may not necessarily result in optimal facial device fit, comfort, and operation. For example, some facial device users of PAP machines may prefer a particular type of facial device material, such as silicone, plastic, or a combination thereof. In some other instances, some facial device users may prefer a facial device that can cover a larger or smaller portion of their face. In some other instances, some facial device users may prefer a facial device with a tighter or looser fit.

[0053] In some scenarios, despite having substantially similar anatomical dimensions, mask device users may have different facial contours or facial muscle stiffness, which may lead to different experiences with fit, comfort, or facial device performance. Accordingly, in some embodiments, server 110 may be configured to generate facial device recommendations based on machine learning operations or recommendation models. In some embodiments, the recommendation model may be based on a dataset representing previous facial device user data, previously generated facial device recommendations, ratings of previous facial device users, or the like.

[0054] In some embodiments, server 110 may perform the operations of a facial device application to generate facial device recommendations 250 and then transmit facial device recommendations 250 to computing device 220 for display in a user interface. Facial device recommendations 250 may include a subset of multiple facial devices that may be optimal for the facial device user. In some embodiments, facial device recommendations 250 may be a sorted list that includes a combination of multiple facial devices or subtypes.

[0055] In some embodiments, facial device recommendation 250 may be based on user input received at computing device 220. Such user input may include data representing a patient profile, clinician-provided treatment requirements, and user preferences, among other examples. In some embodiments, facial device recommendation 250 may include Bayesian-based operations such that recommendations may be based on data representing previously received user feedback by the target user or in combination with other facial device users with similar user profiles or similar facial anatomy.

[0056] In some embodiments, the server 110 may perform operations for centralized patient management 260 and may receive data from one or more computing devices 220 or from the interactive user interface 230. Such data may be stored in a database 280 (e.g., a MaskFit AR® database).

[0057] For example, user input may include facial device user profile data, treatment prescription data, user messages, or other data related to patient management operations 260. In some embodiments, computing device 220 may receive facial device user input to create a user profile or to communicate with a clinician-user via messaging operations. In some embodiments, computing device 220 may receive image data via an image capture device to determine anatomical dimensional data related to the facial device user. In some embodiments, computing device 220 may receive data related to responses to a facial device user questionnaire.

[0058] In some embodiments, a computing device 220 associated with a clinician device user may be configured to display the facial device user's data and / or receive data representing edits to the facial device user's data via a display interface.

[0059] In some embodiments, a computing device 220 associated with a clinician device user may be configured to display the facial device user's data and / or receive data representing edits to the facial device user's data via a display interface. In some embodiments, centralized patient management 260 may include operations for adding existing patient data for storage on server 110, adding new patient data for storage on server 110, obtaining data representing patient information based on a questionnaire, and saving and modifying patient / user data, among other examples. In some embodiments, operations associated with centralized patient management 260 may include operations for providing remote consultations (e.g., video or audio) between a patient user and a clinician user (e.g., similar to a telemedicine platform). In some embodiments, operations associated with centralized patient management 260 may include operations for accessing patient data and transmitting preliminary patient assessments to a computing device associated with a patient user. Other operations for centralized patient management 260 may also be contemplated.

[0060] In some embodiments, system platform 100 may be configured to provide an interactive user interface 230, such as a web portal, on a computing device. In some embodiments, interactive user interface 230 may be provided on any internet-accessible computing device. In some embodiments, interactive user interface 230 may be configured to receive input from a care manager, clinician, or the like. In some embodiments, interactive user interface 230 may be configured to receive patient profile data, provide messaging functionality between users of system platform 100, and receive a facial device user list, among other examples.

[0061] In some embodiments, server 110 can perform operations for centralized facial device application management 270. For example, server 110 can receive a facial device user list, data representing facial device parameters, or the like, and can perform user management and application configuration 270 operations.

[0062] The operation for providing a facial device recommendation may be based at least in part on the anatomy of the facial device user. In some scenarios, a clinician may provide treatment to a patient remotely. For example, a clinician providing treatment for sleep apnea may provide a consultation to a patient regarding a PAP machine. Providing multiple sample facial devices to a facial device user (e.g., a patient) for testing may be time-consuming or resource-intensive (e.g., requiring numerous physical samples or proper hygiene / cleaning). It may be beneficial to provide a facial device recommendation based on image data representing the facial device user.

[0063] The operations for generating anatomical dimension data associated with the face of the facial device user may include operations for facial recognition. Identifying facial features may be based on identifying a plurality of characteristic landmarks or a plurality of reference points. Because determining sufficient characteristic landmarks based on image data representing two-dimensional data may be difficult in some scenarios, it may be beneficial to provide image processing operations based on three-dimensional point cloud data based on the received image data.

[0064] Client device 120 (FIG. 1) may be configured to receive one or more images of a facial device user via an image capture device. In some embodiments, client device 120 may be configured to determine anatomical dimension data of the facial device user based on point cloud data associated with the received image data.

[0065] In some embodiments, a point cloud can be a set of data points in three-dimensional space that can serve as reference / anchor points for facial features. The point cloud can include data points generated based on features that represent depth or perception through positioning in space and time. In some embodiments, client device 120 can include an accelerometer device, and the client device can perform operations based on accelerometer data to track the facial device user's movements in space.

[0066] For ease of explanation, an example operation in the Apple iOS® operating system may be provided. Some examples of augmented reality (AR) operations (e.g., the RealityKit framework, the ARKit framework, the SceneKit framework) may not be able to provide point cloud processing. For operations that may require increased data accuracy or three-dimensional geometry reconstruction, the above-mentioned examples of AR operations may not be able to utilize a direct point cloud rendering model based on image data captured from an imaging device that provides a combination of visual data and real-time depth data (e.g., a TrueDepth® device). As an example, the face tracking model of the ARKit framework may include .line and .fill rendering modes, but may not be able to perform operations based on depth data. To address some of the challenges alluded to above, it may be beneficial to provide a system and method including a three-dimensional image data rendering engine.

[0067] As described herein, one or more client devices 120 may include a facial device application that includes operations for determining anatomical dimension data based on image data of a facial device user. In some embodiments, the facial device application may include operations of the Metal® framework. Metal® may be a hardware-accelerated 3D graphics and compute shader application programming interface based on the Apple iOS® platform. Metal may include functionality similar to OpenGL or OpenCL. The Metal API may include operations for rendering 3D graphics and for performing data-parallel computation based on a graphics processor.

[0068] In some embodiments, one or more client devices 120 may acquire image data representing anatomical features of a facial device user via an image capture device, and one or more client devices 120 may perform the operations of a three-dimensional rendering engine to determine anatomical dimensional data associated with the facial device user.

[0069] In some embodiments, a three-dimensional (3D) rendering engine, disclosed as JMetalRenderer, may be an abstraction around a graphical processing unit. The 3D rendering engine may be a programming class for providing a device and command buffer. The 3D rendering engine may generate a single scene object that may be populated with content. In some embodiments, the 3D rendering engine may instantiate multiple other objects to generate anatomical dimension data.

[0070] Referring to FIG. 3, sample programming code for a 3D rendering engine according to an embodiment of the present disclosure is shown.

[0071] In some scenarios, a scene may include multiple different root nodes to which related content can be added. In some embodiments, the 3D rendering engine may be based on at least four types of nodes: system nodes, root nodes, depth nodes, and face nodes. In addition to storing node instance instances, operations of the scene programming class may store node instances and clear colors. In some scenarios, a color may be used for each new frame as a background. In some scenarios, the 3D rendering engine may store a camera within the scene, which can be used to view the scene from multiple locations, similar to moving a camera in three-dimensional space.

[0072] Referring to FIG. 4, sample programming code for a JScene class is shown in accordance with an embodiment of the present disclosure.

[0073] In some embodiments of a 3D rendering engine, a node may specify where in the world a mesh object should be rendered based on a rotation, scaling, or translation transformation. For example, a mesh may define a model in a local space, and a node may contain data representing operations for taking local coordinates and mapping them to a position in 3D space. In some embodiments, a 3D rendering engine may include a hierarchical structure containing child nodes. In some embodiments, a node need not be associated with a mesh, so the mesh value may be optional.

[0074] In some embodiments, a programming class disclosed as a JNode class can represent operations for transforming 3D local data into points in real-world 3D space. Nodes can include position, orientation, or scale properties that can provide movement of content in real-world 3D space. In some embodiments, the JNode class can include optional mesh properties. Mesh properties can describe the 3D data associated with the node. Nodes can include other child nodes that provide a scene hierarchy of transformations.

[0075] In some embodiments, the JDepthNode programming class can be a child of a JNode object, which can be used to directly render depth data as points in a Metal shader. For illustration, see Figure 5, which shows sample programming code for the JDepthNode programming class.

[0076] In some embodiments, the 3D rendering engine includes additional programming classes for rendering objects. For example, a JMesh programming class may include 3D vertex data used to render an object. A JMesh programming class may include references to materials used to render a model. A JMaterial programming class may define how a model is presented on a display screen (e.g., whether an object should be rendered solid or textured). In some embodiments, a JMaterial programming class may be an abstraction of a vertex or fragment shader setup.

[0077] In some embodiments, the JTexture abstraction class can include MTLTexture and MTLSamplerState objects and can provide helper methods for loading new textures.

[0078] In some scenarios, the 3D rendering engine may include a camera for traversing the captured scene and for setting properties such as a field of view to control how the captured scene can be zoomed in or out. In some embodiments of the 3D rendering engine, perspective projection features may be provided. In some embodiments, orthogonal projection features may be provided based on creating a camera protocol that may include perspective camera and orthographic camera features. In some embodiments, the camera may provide view matrix and projection matrix properties, which may be recalculated based on any changes to the camera properties. Such matrices may be provided to a vertex shader with a uniform buffer.

[0079] In some embodiments, the 3D rendering engine may include JMesh vertex data in combination with a mechanism for importing other models. In some scenarios, 3D model formats in real-world 3D space may include .obj, .ply, .usd, or the like. In some embodiments, the Model I / O programming interface developed by Apple® may be incorporated to import and export data based on multiple formats.

[0080] In some embodiments, the Model I / O programming classes may include an MDLAsset programming class that includes a mechanism for loading external data into the Model I / O data structure. In scenarios where the MDLAsset programming class is instantiated, the MetalKit programming classes can create a MetalKit mesh, MTKMesh. The MTKMesh may include operations to access an MTLBuffer instance used to render the model.

[0081] Referring to FIG. 6, a sample object inheritance table for the custom 3D rendering engine disclosed herein is shown, in accordance with an embodiment of the present disclosure.

[0082] 7-9, 3D point cloud rendering based on the operation of the 3D rendering engine described above is illustrated, according to an embodiment of the present disclosure. Figures 7-9 show example diagrams representing test results of custom operation of the 3D rendering engine on 3D point cloud data. In some embodiments, image and depth data acquired from an image capture device (e.g., a TrueDepth® camera) can be processed to provide image data for determining anatomical dimension data related to facial features.

[0083] In some scenarios, one or more client devices 120 may capture image data representing the field of view of an image capture device. iOS®-based augmented reality programming interfaces may not provide operations for identifying and displaying the identity of a user's face (e.g., the ARKit® framework in iOS® may not include such functionality). It may be beneficial to perform operations that generate a coarse-grained capture of the user's face in substantially real time.

[0084] 10A and 10B, there is shown contour identification of a user's face in a captured image and a 3D point cloud representation of the identified contours, according to an embodiment of the present disclosure.

[0085] In some embodiments, the facial device application may include operations for detecting the user's facial contours in substantially real time and displaying the detected user's facial contours on a display interface. As an example, in FIG. 10A , the facial device application may display a front view of the user's face with a polygonal boundary around the user's face.

[0086] In some embodiments, the facial device application may include operations for displaying 3D point cloud data associated with the user's facial features. As an example, in FIG. 10B, the facial device application may include visual markers that identify contour or depth data associated with the user's facial features.

[0087] In some embodiments, the facial device application may include operations for detecting a user's face based on captured image data. In some embodiments, a programming class disclosed as ARFaceGeometry may provide 3D geometric or topological data related to the user's face. In some embodiments, data markers 1050 that may combine to represent or describe the contours of the user's face may be selectable and displayed in a user interface. In some embodiments, data points 1050 shown in 3D space based on the ARFaceGeometry programming class may be converted to 2D pixel points for display on a user interface.

[0088] In some embodiments, the facial device application may include operations for determining facial boundary markers 1055 based on 2D pixel points provided on a user interface. As an example, Figure 11 shows sample programming pseudocode for determining facial boundary markers 1055 according to one embodiment of the present disclosure.

[0089] In some embodiments, a facial device application may include operations based on the ARKit programming class (developed by Apple®). In a scenario where the ARKit operations determine that a unique face has been identified based on captured image data, the ARSession programming class operations may provide an ARFAceAnchor object. The ARFaceAnchor may include data representing the pose, topology, or expression of the face. Additionally, a geometry property may represent an ARFaceGeometry object that represents detailed topology data representing the user's face. FIGS. 12A and 12B show example visual data representing the topology of a detected user's face.

[0090] In some embodiments, the ARFaceGeometry programming class may provide topology data representing a user's face in the form of a 3D mesh diagram, which may be suitable for rendering by multiple third-party technologies or export as 3D digital data.

[0091] In some embodiments, a facial device application may include operations for determining face geometry from an ARFaceAnchor object during a face-tracking AR session. During the face-tracking AR session, a face model may determine the dimensions, shape, or current expression of a detected face. In some embodiments, the ARFaceAnchor object may be used to generate face mesh data based on stored shape coefficients, thereby providing a detailed description of the face's current expression.

[0092] In one example of an AR session, a facial device application can utilize the model as a basis for overlaying content based on the contours of the user's face, such as for applying virtual makeup or tattoos. In some embodiments, the face model can generate occluding geometric objects to mask virtual content behind the 3D shape of the user's face.

[0093] In some embodiments, a facial device application may determine facial features and generate data by point indexing of a general face tracking model. In some scenarios, vertex indexing of the ARFaceGeometry programming class may be useful for generating facial geometry data. Figures 13A and 13B show close-up views of a 3D mesh model representing a user's face, according to an embodiment of the present disclosure. The close-up views show virtual vertex indexing.

[0094] In some embodiments, a facial device application may include operations for detecting facial features or related geometric data based on the generated vertex indices. As an example, the ARFaceGeometry programming class described herein includes 1220 vertices based on the ARKit framework. Referring again to FIG. 10B , a contour identification of a user's face shows the vertex indices based on the operations of the ARFaceGeometry programming class. For example, a facial device application may include operations for generating automated 3D sensing measurement data based on an ARFaceGeometry generic face tracking model of the entire face or nose structure.

[0095] In some embodiments, the facial device application may include operations for determining a head pose based on 3D point cloud data representing a user's head dynamically or substantially in real time. The operations for head pose estimation may be performed as a preprocessing operation for facial feature recognition. In some embodiments, the operations for head pose estimation may include detecting a nose tip point based on a convex hull of the 3D point data.

[0096] 14, a method 1400 for estimating head pose based on image data is shown, according to an embodiment of the present disclosure. Method 1400 may be executed by a processor of one or more computing devices described in this disclosure. In some embodiments, processor-readable instructions may be stored in memory and associated with a facial device application of one or more client devices 120 (FIG. 1). Method 1400 may include operations such as data retrieval, data manipulation, data storage, or the like, and may also include other computer-executable operations.

[0097] At operation 1402, the processor may generate a convex hull based on a 3D point cloud associated with the image data of the user's face. For illustration purposes, Figure 15A shows a 3D point cloud associated with the image data of the user's face. Figure 15B shows a convex hull generated based on the 3D point cloud shown in Figure 15A.

[0098] At operation 1404, the processor may filter the nose tip point 1510, shown schematically in FIG. 15B, based on the minimum z-axis value of the current 3D Euclidean space. To illustrate, in some scenarios, the tip of the convex hull may correspond to the nose tip point of a typical human face. After constructing the convex hull, the processor may identify the data point with the lowest z-axis coordinate (e.g., the z-axis may represent depth) and determine that the identified data point represents the tip of the nose.

[0099] After identifying the nose tip point, the processor may generate a polygonal volume or sphere with its center located at the nose tip point at operation 1406. In some embodiments, the center may be the centroid of the nose tip point data points. The processor may generate a visual indicator identifying the polygon or sphere shape 1610. In some scenarios, the polygon or sphere may be other shapes depending on the anatomical shape of the user's face.

[0100] At operation 1408, the processor may estimate surface normals 1620 of the above-described filtered face-based region points 1610, as shown in Figure 16B. For example, Figure 16B shows an estimated face XY plane.

[0101] In step 1410, the processor may align the estimated surface normal to the Z axis, as shown in Figure 17A. In particular, Figure 17A shows a front view aligned with the Z axis.

[0102] At operation 1412, the processor may filter the alar points 1730 (FIG. 17B) by estimating the normals in the Z axis of the data points. In particular, FIG. 17B shows the estimated alar point set.

[0103] At operation 1414, the processor may extract the nose top points and estimate an orientation bounding box for the extracted nose top points. Figure 18A shows the extracted nose top points 1810, and Figure 18B shows the orientation bounding box for the extracted nose top points.

[0104] At act 1416, the processor may rotate the normal direction of the maximum bounding length of the estimated orientation bounding box around the Y-axis. FIG. 19 shows the aligned 3D face point cloud data. At act 1416, the processor may rotate the data representing the user's face to obtain a better pose. For example, if the user's face is tilted to the left or right when acquiring image data representing the user's face, act 1416 may include rotating the user's head to compensate for the tilted position.

[0105] In some scenarios, it may be beneficial to dynamically or substantially real-time determine a user's facial feature parameters based on image data representing the user's face, hi some embodiments, an aligned 3D point cloud representing the face may be associated with an action to determine the user's facial feature parameters.

[0106] As an example, referring to Figure 20, a 3D point cloud of a user's face illustrating the detection of nose depth parameters according to an embodiment of the present disclosure is shown. Figure 20 shows a front view plane in the XY plane of 3D Euclidean space. In some embodiments, determining the feature parameters may be based on a nose tip point 2010 extracted from the convex hull of raw scanned face points.

[0107] As an example, to determine a user's nose depth parameter, the facial device application may include an operation for locating a spherical region center at the centroid of a nose tip point. The facial device application may include an operation for estimating a face base surface from the spherical region center. Further, the processor may detect a distance between the nose tip point and the face base surface to determine the nose depth.

[0108] In another example, the facial device application may determine the width of the nose point based on a 3D point cloud of the user's face by estimating the normal in the z-axis of the candidate nose tip point 2110. Referring to FIG. 21 , a 3D point cloud of the user's face showing the candidate nose tip point 2110 is shown. The processor may filter the 3D points 2120 representing the user's nose. Based on the filtered 3D points 2120 representing the user's nose, the processor may determine an estimate of the width of the nose point.

[0109] In another example, the facial device application may determine the nose length based on the face base surface. Referring to Figures 22A and 22B, a detected nose top surface and a determined nose length are shown, respectively, according to an embodiment of the present disclosure.

[0110] FIG. 22A shows a detected bounding box 2210 of the nose top point. The nose top point may be filtered based on the filtered 3D points 2120 (FIG. 21) representing the user's nose by estimating a normal in the z-axis. The recognized bounding box of the nose top point may be extended to the y-axis to generate candidate points. Furthermore, the processor may estimate a predicted nasal root point 2220 parallel to the xy plane by comparing the angle between the normal and the z-axis. For example, FIG. 22B shows an example of the nasal root point 2220. Based on at least the above, the nose length may be calculated within the face-based plane.

[0111] 23 , a method 2300 for determining anatomical dimension data for a user's facial features is shown, according to an embodiment of the present disclosure. Method 2300 may be executed by a processor of one or more computing devices described in this disclosure. In some embodiments, the processor-readable instructions may be stored in memory and associated with a facial device application of one or more client devices 120 (FIG. 1). Method 2300 may include operations such as, for example, data retrieval, data manipulation, data storage, or the like, and may also include other computer-executable operations.

[0112] As one non-limiting example, method 2300 may include operations for determining the width or height of a user's nostrils. Method 2300 may be based on depth data associated with one or more image frames captured by a TrueDepth® camera. Method 2300 may be based on detecting transitions in pixel color values ​​in RGB images associated with each ARFrame. Thus, embodiments of the operations may automatically determine the width or height of the nostrils in substantially real time.

[0113] In some embodiments, determining anatomical dimension data related to the user's nostrils may be based on operations related to 2D single-frame image data. Such operations may include detecting transitions in pixel color values ​​in an RGB image. In some embodiments, the operations for determining anatomical dimension data may be based on point cloud data, which may be generated based on depth data obtained from an image capture device (e.g., a TrueDepth® camera or the like).

[0114] At operation 2302, the processor may identify a nose or nasal ridge based on the acquired image data. In some embodiments, the processor may include vision framework operations to identify a user's nose or nasal ridge. For example, FIG. 24A shows landmark 2410 representing a nasal ridge feature. FIG. 24B shows landmark 2420 representing a nasal feature.

[0115] At operation 2304, the processor may determine a valid evaluation area for the nostrils. In some embodiments, determining the valid evaluation area may be based on one or a combination of landmarks 2410 representing nasal ridge features or landmarks 2420 representing nose features.

[0116] Figure 25A shows a valid evaluation area 2550 determined based on a combination of landmarks representing nasal ridge features 2410 and landmarks representing nose features 2420. Figure 25B shows the determined valid evaluation area 2550 without showing the landmarks mentioned above.

[0117] At operation 2306, the processor may determine a contour of at least one nostril based on pixel RGB value transitions based on the diagonals of the respective quadrilateral boundaries. For example, FIG. 26 shows diagonal marking segments 2660 that may correspond to a user's nostrils. In FIG. 26, pixel RGB values ​​may transition as they transition "inward" from the sides of the respective quadrilateral boundaries. In some embodiments, the processor may determine one or more contour points 2662 for the left or right nostril.

[0118] In some embodiments, the processor may determine a "maximum" RGB value based on a boundary point of the determined effective evaluation area 2550. The processor may determine a "minimum" RGB value based on an identified central sub-area of ​​the effective evaluation area 2550. In some embodiments, a threshold for identifying a contour or boundary may be based on a determined RGB value range across a "maximum" RGB value and a "minimum" RGB value. In some embodiments, the processor may include operations that may be based on the average pixel RGB value of a neighboring region of a given pixel RGB value.

[0119] At operation 2308, the processor may adjust the determined nostril height and width based on the offset and rotation angles to provide nostril measurement parameters. For example, as shown in FIGS. 27A and 27B, initial green nostril height and width lines may be optimized with offset and rotation angles to obtain optimal measurement results. In some embodiments, the intersection of the nostril height line and width line may be identified as the center point of the rotation or offset operation. In some embodiments, an Amoeba numerical algorithm may be applied to determine the maximum height or width value. In some embodiments, a nonlinear optimization numerical algorithm may be applied to determine optimal targets for the rotation angle and offset distance values.

[0120] 28, a method 2800 for determining one or more facial device recommendations according to an embodiment of the present disclosure is shown. Method 2800 may be executed by a processor of one or more computing devices described in this disclosure. In some embodiments, the processor-readable instructions may be stored in memory and associated with a facial device application of one or more client devices 120 (FIG. 1). Method 2800 may include operations such as data retrieval, data manipulation, data storage, or the like, and may also include other computer-executable operations.

[0121] In some embodiments, operations of method 2800 may include operations for determining whether a target facial device can fit a user's face for optimal operation or comfort. In some embodiments, operations for determining whether a target facial device can fit a user's face may be based on facial device dimensions, such as upper bounding dimensions or lower bounding dimensions, and anatomical dimension data describing the user's face. In some embodiments, method 2800 may be for providing one or more recommendations for one or a combination of facial device types, such as, for example, a full-face device, a nasal device, or a nasal pillows device.

[0122] At operation 2802, a processor may obtain facial measurement data related to a user's face. In some embodiments, the facial measurement data may be determined based on operational embodiments described in this disclosure. For example, the processor may perform an operation to determine anatomical dimension data related to the size of the nostril openings. In another example, the processor may perform an operation to determine the width or length of the user's nose.

[0123] In some embodiments, determining the facial measurement data may be based on one or a combination of 2D image data, such as image data captured based on a TrueDepth® camera, or 3D point cloud image data.

[0124] At operation 2804, the processor may determine that the acquired facial measurement data may be within the upper boundary (AUB) and lower boundary (ALB) of a particular facial device. In some embodiments, each facial device may be associated with dimensional data suitable for users with a particular range of anatomical dimensional data (e.g., users with a particular nose length, nostril opening size, or the like).

[0125] The processor may perform operations at operation 2804 to determine whether the acquired facial measurement data (associated with a particular user) is within the AUB and ALB of multiple facial devices.

[0126] In response to determining that the acquired facial measurement data is within the AUB and ALB of a particular facial device, the processor may add the particular facial device to a list of facial device recommendations at operation 2806.

[0127] In some embodiments, upon determining a list including one or more facial device recommendations, the processor may perform an operation to sort the list of facial device recommendations based on quantitative or qualitative data (e.g., user reviews, star ratings, among other examples). Upon sorting the one or more facial device recommendations, the processor may perform a scoring operation to score the facial devices to identify the facial devices with the top scores for presentation to the user interface.

[0128] At operation 2808, the processor may perform operations to sort or rank the list of facial devices based on patient information, the device used, and / or patient-specific clinical data. In some embodiments, the patient information may include one or more of age, gender, body mass index (BMI), ethnicity, or the like. In some embodiments, the device used may include one or more of CPAP, BiPAP, ASV, and various pressure settings including minimum and maximum pressure settings. In some embodiments, the patient-specific clinical data may include one or more of a deviated septum, nasal congestion, seasonal allergies, sleep position (e.g., prone, supine, side, restless), facial hair (e.g., moustache, beard), skin sensitivity, claustrophobia, and the like.

[0129] In some embodiments, a score may be assigned to each facial device type that may perform better for different clinical parameters. An example table of scores assigned to various facial device types for particular characteristics is provided in Figure 34. As shown, scores range from 0 to 3, but it will be understood that any suitable scoring scale or scoring system may be used, and the table in Figure 34 is merely an example.

[0130] As shown in Figure 34, if the patient is claustrophobic, a nasal pillows mask has the highest recommendation score (score 3), followed by a nasal mask (score 2), and then a full face mask (score 1). Similarly, if the device pressure setting is high (e.g., greater than 10 cmH2O), a nasal mask is likely to provide the best seal (score 3), followed by a full face mask (score 2), and then a nasal pillows mask (score 1). As another example, if the patient has a moustache, a nasal pillows mask would receive the highest score (score 3), followed by a full face mask (score 2), and then a nasal mask (score 1).

[0131] In some embodiments, an overall score for each facial device type may then be obtained. In some embodiments, the overall score may be the sum of the scores for each attribute. In some embodiments, the overall score may be a weighted sum of the scores for each attribute, where the weights may be selected to emphasize or de-emphasize the importance of a patient quality or attribute. In some embodiments, a scoring system may then be used to rank the list of facial devices generated in 2806. FIG. 35 is an example of a user interface that provides a user with a list of facial device recommendations.

[0132] In some embodiments, a patient's history and / or mask preferences may be taken into consideration when ranking masks. For example, a patient may have previously used a particular facial device type and subsequently been successfully treated with it. For example, if the scoring system provided herein ranks a nasal pillow mask higher than a nasal mask on the recommendation list, but the patient has already successfully used a nasal mask in the past, the patient may feel more comfortable using the same facial device type with which the patient is already familiar (rather than introducing the patient to a new type of mask). In some embodiments, the scoring system may include an indication of whether the patient has previously used a particular facial device type, and if so, the patient's rating of that facial device type based on satisfaction and comfort. If a previously used facial device type is highly rated, the ranking system may promote that facial device type higher in the ranked list of facial devices.

[0133] In some embodiments, the AUB / ALB ranges can overlap between facial device sizes. To illustrate, with reference to Figure 29, boundary ranges associated with sizes (e.g., small, medium, large) for an example facial device according to an embodiment of the present disclosure are shown.

[0134] In some embodiments, the facial device application can define overlapping size boundary values. For example, the size boundary values ​​(e.g., S1, S2, M1, etc.) can progress upward or downward based on feedback data received from previous users of the respective facial devices. In some embodiments, one or more computing devices described in this disclosure can include a machine learning architecture for generating a facial device recommendation model. Such a facial device recommendation model can dynamically update the model to adjust size boundary ranges or values ​​based on feedback data received from previous users of the facial devices.

[0135] In some scenarios, the facial device application may include operations for updating boundaries of a particular size when the scoring or star rating of a mask changes over time. Referring to FIG. 30, a flowchart of a method 3000 for updating boundary ranges associated with generating facial device recommendations is shown, according to an embodiment of the present disclosure. For example, ranking / data manipulation when star rating changes: when a mask is ranked, the ranking may change. The ALB and AUB may be relative to the size of the selected and ranked mask.

[0136] Method 3000 may be executed by a processor of one or more computing devices described in this disclosure. In some embodiments, the processor-readable instructions may be stored in memory and associated with a facial device application on one or more client devices 120 (FIG. 1). Method 3000 may include operations such as data retrieval, data manipulation, data storage, or the like, and may also include other computer-executable operations.

[0137] In operation 3002, the processor may determine whether AUB - 3% < measured data input (DI) <= AUB + 3%, or ALB - 3% <= measured data input < ALB + 3%. In the present disclosure, 3% is provided for ease of explanation, but the + / -% value may be larger or smaller based on machine learning operations to improve the recommended model.

[0138] If the processor determines yes in operation 3002, in operation 3004, the processor may determine whether (AUB + DI) / 2 >= ALB ラージサイズ or (ALB + DI) / 2 <= AUB スモールサイズ

[0139] If the processor determines yes in operation 3004, the processor may replace ALB or AUB with a new average value in operation 3006.

[0140] In operation 3006, the processor may sort the display list by weighted ratings (WR). In some embodiments, the weighted ratings may be the sum of Sn values (see the following relationship) calculated based on Bayesian analysis. The processor may include an operation of showing a mask with a specific star rating (SR = 0) at the end of the list in alphabetical order.

[0141] In embodiments where the processor determines "no" in operation 3002 or operation 3004, the processor may perform operation 300 to sort the display list by WR. Further, the processor may show a facial device with SR = 0 at the end of the list in alphabetical order.

[0142] ​In some embodiments, the facial device application described herein may obtain user input including a facial device rating. In some embodiments, the facial device rating may be a rating associated with a new mask or a rating of an existing mask. In some embodiments, the processor may determine or update a facial device rank based on the following relationship:

[0143] Each one is s k Suppose we have K possible ratings indexed by k, which is worth points. For a "star" rating system, s k = k (e.g., 1 point, 2 points, etc.). A given item has n points for k. k Assume that a facial device application receives a total of N ratings based on the criteria:

number

[0144] Based on the reference embodiment described above, the facial device application may include operations for displaying a facial device mask list that may be ranked based on "S number."

[0145] In some embodiments, the scoring of each facial device may be based on a weight or weighted rating that represents the likelihood of an optimal fit for a given user. In some scenarios, the weight may be associated with a score or other metric for generating facial device recommendations for the user.

[0146] In some embodiments, a computing device may generate a user interface for display on one or more computing devices (of FIG. 1) in accordance with embodiments of the present disclosure. The one or more computing devices may perform automated 3D sensing measurement operations based on a generic tracking model for analyzing a user's face for a full-face / nose facial device.

[0147] 31-33 illustrate user interfaces for display on one or more computing devices (of FIG. 1) according to embodiments of the present disclosure. The user interfaces may include facial device recommendation lists or details associated with each facial device. In some embodiments, the user interfaces may include a user interface for displaying anatomical dimension data based on image processing operations described in this disclosure. In some embodiments, the user interfaces may be associated with operations for dynamically determining anatomical dimension data relevant to generating recommendations for nasal pillow facial devices.

[0148] In some embodiments described herein, a computing device may determine anatomical dimensional data associated with a user's facial features. In some other embodiments, a computing device may determine dimensional data for a facial device or other nose-mouth appliance based on image data representing the same. For example, a computing device may be configured to receive image data (e.g., a photograph) of a nose-mouth appliance and to determine physical dimensions of the nose-mouth appliance. Thus, the computing device may generate a dimensional data set associated with multiple nose-mouth appliances based on a combination of at least (i) device manufacturer specifications and (ii) dimensional data determined based on the image data.

[0149] The terms "connected" or "coupled" can include both direct coupling (where the two elements coupled together are in contact with each other) and indirect coupling (where at least one additional element is located between the two elements).

[0150] Although the embodiments have been described in detail, it is to be understood that various changes, substitutions, and alterations may be made therein without departing from the scope thereof. Moreover, the scope of the present disclosure is not intended to be limited to the particular embodiments of the process, machine, manufacture, composition of matter, means, methods and steps described in the specification.

[0151] Those skilled in the art will readily appreciate from this disclosure that any now existing or later developed processes, machines, manufacture, compositions of matter, means, methods, or steps may be used which perform substantially the same function or achieve substantially the same results as the corresponding embodiments described herein, and accordingly, the appended claims are intended to include within their scope such processes, machines, manufacture, compositions of matter, means, methods, or steps.

[0152] This description provides numerous example embodiments of the inventive subject matter. While each embodiment represents a single combination of inventive elements, the inventive subject matter is considered to include all possible combinations of the disclosed elements. Thus, if one embodiment has elements A, B, and C and a second embodiment has elements B and D, the inventive subject matter is considered to include any other remaining combinations of A, B, C, or D, even if not explicitly disclosed.

[0153] The device, system, and method embodiments described herein may be implemented in a combination of both hardware and software. These embodiments may be implemented on programmable computers, each computer including at least one processor, a data storage system (including volatile or non-volatile memory or other data storage elements, or a combination thereof), and at least one communication interface.

[0154] Program code is applied to input data to perform the functions described herein and to generate output information, which is provided to one or more output devices. In some embodiments, the communication interface may be a network communication interface. In embodiments where multiple elements may be combined, the communication interface may be a software communication interface, such as for inter-process communication. In yet other embodiments, there may be a combination of communication interfaces implemented as hardware, software, and combinations thereof.

[0155] Throughout the above description, numerous references are made to servers, services, interfaces, portals, platforms, or other systems formed from computing devices. It should be understood that use of such terms is considered to describe one or more computing devices having at least one processor configured to execute software instructions stored on a computer-readable, tangible, non-transitory medium. For example, a server may include one or more computers operating as a web server, database server, or other type of computer server to perform the described roles, responsibilities, or functions.

[0156] The technical solutions of the embodiments may be in the form of a software product. The software product may be stored in a non-volatile or non-transitory storage medium, which may be a compact disc read-only memory (CD-ROM), a USB flash disk, or a removable hard disk. The software product includes a number of instructions that enable a computer device (personal computer, server, or network device) to execute the method provided by the embodiments.

[0157] The embodiments described herein are implemented by physical computer hardware, including computing devices, servers, receivers, transmitters, processors, memory, displays, and networks. The embodiments described herein provide useful physical machines and specially configured computer hardware configurations.

[0158] As can be understood, the examples described and illustrated above are intended to be illustrative only.

Claims

1. 1. A computing device for adaptively generating facial device selection based on visually determined anatomical dimensional data, comprising: a processor; a memory coupled to the processor and storing processor-executable instructions that, when executed, receiving image data representing a user's face; determining anatomical dimension data associated with a nose-mouth region of the user's face based on the received image data, wherein determining the anatomical dimension data includes determining nostril opening dimensions based on transitions in RGB image data values ​​that exceed one or more thresholds; generating one or more facial device recommendations based on a recommendation model defined based on a Bayesian-based scoring metric and the anatomical dimension data; providing the one or more facial device recommendations for display in a user interface; a memory, the memory configuring the processor to 1. A computing device having:

2. The memory, when executed, receiving, via the user interface, target therapy data including at least one of user preference or target facial device specification data; processor-executable instructions to configure the processor to: the one or more facial device recommendations generated are based on the target therapy data combined with the anatomical dimension data. The computing device of claim 1 .

3. The computing device of claim 1 , wherein the one or more facial device recommendations include at least one of a full facial / nasal mask associated with a PAP device or a nasal pillows mask.

4. The computing device of claim 1 , wherein the received image data includes a combination of RGB image data and infrared depth data.

5. A computing device for adaptively generating facial device selections based on visually determined anatomical dimensional data, comprising: a processor; a memory coupled to the processor and storing processor-executable instructions that, when executed, receiving image data representing a user's face; determining anatomical dimension data associated with a nose-mouth region of the user's face based on the received image data; generating one or more facial device recommendations based on a recommendation model defined based on a Bayesian-based scoring metric and the anatomical dimension data, wherein generating the one or more facial device recommendations includes determining that the anatomical dimension data is within upper and lower bound specifications associated with a prospective facial device, wherein at least one bound specification for a given facial device size overlaps with another bound specification for an adjacent facial device size; providing the one or more facial device recommendations for display in a user interface; a memory, the memory configuring the processor to 1. A computing device having:

6. 10. The computing device of claim 1, wherein determining anatomical dimension data associated with the nose-mouth region of the user's face is based on at least one of three-dimensional (3D) point cloud data or two-dimensional image data.

7. The computing device of claim 1 , wherein the determined anatomical dimension data includes at least one of nostril opening size, nose width, nose depth, nose length, or face width.

8. The computing device of claim 1 , wherein the memory includes processor-executable instructions that, when executed, configure the processor to estimate head pose based on 3D point cloud data.

9. A computing device for adaptively generating facial device selections based on visually determined anatomical dimensional data, comprising: a processor; a memory coupled to the processor and storing processor-executable instructions that, when executed, receiving image data representing a user's face; determining anatomical dimension data associated with a nose-mouth region of the user's face based on the received image data; generating one or more facial device recommendations based on a recommendation model defined based on a Bayesian-based scoring metric and the anatomical dimension data; providing the one or more facial device recommendations for display in a user interface; a memory, the memory configuring the processor to and The memory, when executed, Identifying facial features based on the received image data; determining an effective evaluation area of ​​the facial feature, the facial feature including the nostrils; determining a contour of at least one nostril based on pixel RGB value transitions based on a diagonal of each quadrilateral boundary marker; adjusting the determined nostril height and width based on at least one of the offset or rotation angle to provide nostril measurement parameters; processor-executable instructions to configure the processor to Computing equipment.

10. A computing device for adaptively generating facial device selections based on visually determined anatomical dimensional data, comprising: a processor; a memory coupled to the processor and storing processor-executable instructions that, when executed, receiving image data representing a user's face; determining anatomical dimension data associated with a nose-mouth region of the user's face based on the received image data; generating one or more facial device recommendations based on a recommendation model defined based on a Bayesian-based scoring metric and the anatomical dimension data; providing the one or more facial device recommendations for display in a user interface; a memory, the memory configuring the processor to and The memory, when executed, generating a convex hull based on a 3D point cloud associated with the image data of the user's face; filtering the nose tip points based on the minimum z-axis value in the 3D Euclidean space; generating a sphere centered on the nose tip; generating surface normals for the filtered face-based region; aligning the estimated surface normal to the z-axis; filtering the alar points based on estimating the normals of the data points in the z-axis; Extracting nose top points and estimating an orientation bounding box for the extracted nose top points; Rotate the normal direction of the maximum bounding length of the estimated oriented bounding box to the y-axis. processor-executable instructions to configure the processor to Computing equipment.

11. The computing device of claim 1 , further comprising assigning a score for one or more of a plurality of patient attributes for each of the one or more facial device recommendations.

12. The computing device of claim 11 , wherein the patient attributes include at least one of a deviated septum, nasal congestion, seasonal allergies, sleep position, facial hair, skin sensitivity, and / or claustrophobia.

13. 1. A method for adaptively generating facial device selection based on visually determined anatomical dimensional data, comprising: receiving image data representing a user's face; determining anatomical dimension data associated with a nose-mouth region of the user's face based on the received image data, wherein determining the anatomical dimension data includes determining nostril opening dimensions based on transitions in RGB image data values ​​that exceed one or more thresholds; generating one or more facial device recommendations based on a recommendation model defined based on a Bayesian-based scoring metric and the anatomical dimension data; providing the one or more facial device recommendations for display in a user interface; How to have that.

14. A non-transitory computer-readable medium having stored thereon machine-interpretable instructions that, when executed by a processor, cause the processor to perform a computer-implemented method for adaptively generating facial device selection based on visually determined anatomical dimensional data, the method comprising: receiving image data representing a user's face; determining anatomical dimension data associated with a nose-mouth region of the user's face based on the received image data, wherein determining the anatomical dimension data includes determining nostril opening dimensions based on transitions in RGB image data values ​​that exceed one or more thresholds; generating one or more facial device recommendations based on a recommendation model defined based on a Bayesian-based scoring metric and the anatomical dimension data; providing the one or more facial device recommendations for display in a user interface; 1. A non-transitory computer-readable medium having:

15. The computing device of claim 1, wherein the received image data includes a photograph of the user captured by an image capture device.

16. The computing device of claim 1, wherein the received image data excludes photographic data of at least a portion of the user's face.

17. A computing device as described in claim 1, further comprising deleting the received image data from the computing device after determining the anatomical dimension data relating to the nose-mouth region of the user's face.

18. The computing device of claim 17, wherein deleting the received image data is performed before generating the one or more facial device recommendations.

19. The computing device of claim 1, wherein the processor-executable instructions, when executed, further configure the processor to send a transmission to a server, the server configured to perform generating the one or more facial device recommendations, the transmission excluding personally identifiable information of the user.

Images