Clinical Data Analysis

Abstract

A method for clinical data analysis includes receiving a first 3D representation representing clinical data. The first three-dimensional (3D) representation includes one or more mesh elements. The method includes calculating one or more mesh element features for the one or more mesh elements and providing the one or more mesh element features as input to a first machine learning (ML) module. The method includes executing the first ML module to encode the first 3D representation into one or more latent representations and providing the one or more latent representations to a second ML module different from the first ML module. The method includes executing the second ML module to classify the clinical data represented in the first 3D representation into at least one predicted classification label.

Description

[Technical Field]

[0001] Priority and related documents This application claims priority to U.S. Provisional Application No. 63 / 432,627, filed December 14, 2022, No. 63 / 460,563, filed April 19, 2023, and No. 63 / 462,855, filed April 28, 2023. The entire disclosures of PCT Application Nos. PCT / IB2023 / 056144, PCT / IB2023 / 056142, PCT / IB2023 / 056151, PCT Publication No. WO2022123402(A1), and U.S. Provisional Application No. 63 / 432,627 are incorporated herein by reference. [Background technology]

[0002] Appropriate treatment of a patient, e.g., treatment of a wound, skin, appendage, etc., may require careful clinical analysis. For example, the condition of the wound, skin, or appendage must be assessed before an appropriate treatment can be selected and applied. In some instances, the condition of the wound, skin, or appendage must be monitored during treatment. As a result, there is a need for machine learning models and training methods to improve systems that can automate clinical analysis. Summary of the Invention

[0003] The present disclosure provides a method for clinical data analysis. The method includes receiving a first three-dimensional (3D) representation representing clinical data. The first 3D representation includes one or more mesh elements. The method further includes calculating one or more mesh element features for the one or more mesh elements and providing the one or more mesh element features as input to a first machine learning (ML) module. The method further includes executing the first ML module to encode the first 3D representation into one or more latent representations and providing the one or more latent representations to a second ML module different from the first ML module. The method further includes executing the second ML module to classify the clinical data represented in the first 3D representation into at least one predicted classification label.

[0004] The present disclosure also provides a computing device. The computing device includes an interface configured to receive a first three-dimensional (3D) representation representing clinical data. The first 3D representation includes one or more mesh elements. The computing device further includes a memory communicatively coupled to the interface and configured to store the first 3D representation. The computing device further includes a processor communicatively coupled to the interface and the memory. The processor is configured to calculate one or more mesh element features for the one or more mesh elements, provide the one or more mesh element features as input to a first machine learning (ML) module, execute the first ML module to encode the first 3D representation into one or more latent representations, provide the one or more latent representations to a second ML module different from the first ML module, and execute the second ML module to classify the clinical data represented in the first 3D representation into at least one predicted classification label.

[0005] The present disclosure also provides a method for detecting an abnormality. The method includes receiving a first three-dimensional (3D) representation representing clinical data and providing the first 3D representation as an input to a first machine learning (ML) module. The method further includes executing the first ML module to encode the first 3D representation into one or more latent representations and reconstructing the one or more latent representations into a second 3D representation that is a replica of the first 3D representation. The method further includes calculating a reconstruction error that quantifies a difference between the first 3D representation and the second 3D representation and determining at least one region of the first 3D representation having a reconstruction error greater than a predetermined threshold. The method further includes determining that the at least one region corresponds to an abnormality.

[0006] The present disclosure also provides a method for detecting swelling. The method includes receiving a first three-dimensional (3D) representation representing clinical data. The clinical data represents a patient's skin or an appendage of the patient. The method further includes providing the first 3D representation as input to a first machine learning (ML) module and executing the first ML module to encode the first 3D representation into one or more latent representations. The method further includes providing the one or more latent representations to a second ML module different from the first ML module and executing the second ML module to classify the clinical data into a current state of swelling or a future state of swelling.

[0007] This disclosure describes systems and techniques for training and using one or more machine learning (ML) models, such as neural networks, to analyze a patient's skin, analyze a wound on the patient's skin, or otherwise analyze the condition of a patient's appendages for the purpose of guiding a clinician in treating the patient. The techniques described herein may use representation learning to train a neural network to perform such analyses. Clinical data may be provided to an ML model trained to perform one or more of the techniques described herein, enabling the ML model to generate an indicator of the patient's health or status that can be used by a clinician in treating the patient. The clinical data may include one or more digital images of a side of the patient's body (e.g., a two-dimensional (2D) raster image comprising a grid of pixels, such as a 2D color digital photograph, a heat map, a depth map, or a map of some other sensor-generated modality), or one or more 3D representations of the side of the patient's body (e.g., a 3D point cloud, a 3D mesh, a 3D surface, a voxelized representation, etc.). The 2D aspects of the patient's body may include a wound, a dressing on the skin, an appendage, or a 2D image of the patient's skin. The 3D aspects of the patient's body may include a 3D representation of a wound (e.g., a voxelized representation of an arm that may be affected by lymphedema), a 3D representation of a dressing on the skin (e.g., a 3D mesh of a dressing over a wound, where the dressing may have wrinkles that may lead to leakage), a 3D representation of an appendage (e.g., a 3D point cloud of a swollen appendage or torso), etc. Clinicians may include doctors, nurses, physician assistants, technicians, emergency medical technicians (EMTs), firefighters, dentists, orthodontists, dermatologists, chiropractors, physical therapists, or any other practitioner treating patients.

[0008] In some implementations, one or more instances of clinical data (e.g., 2D color digital photographs or 3D meshes) may be provided to a first ML module. The first ML module (e.g., one or more linear layers or some portion of an encoder-decoder structure such as an autoencoder or a transformer) may be trained to generate one or more latent representations (or latent embeddings) of the clinical data. The encoder-decoder structure may include at least one encoder and / or at least one decoder. Non-limiting examples of encoder-decoder structures include a transformer, an autoencoder such as a variational autoencoder, a regularized autoencoder, a masked autoencoder, or a capsule autoencoder. The one or more latent representations may be provided to a second ML module (e.g., a convolutional neural network, a set of fully connected layers, or some portion of an encoder-decoder structure). The second ML module may be trained to generate an indicator of the patient's health or status based on the one or more latent representations.

[0009] In some implementations, when a 3D representation of clinical data is provided to the first ML module, the first ML module may include at least one reconstruction autoencoder having an encoder-decoder structure. Reconstruction autoencoders are particularly well-suited for generating representations (i.e., one or more latent representations) of clinical data because they can reduce the data size of the clinical data (e.g., 3D data that may include thousands or tens of thousands of mesh elements) while preserving much of the information about the shape and / or structure of the clinical data. This reduced-dimensional form of the clinical data (e.g., a vector of 512 or 1024 real numbers, among other possible sizes) may occupy a latent space due to the reduced complexity of the representation of the clinical data and / or may be more easily processed by the second ML module. Reconstruction autoencoders (e.g., variational autoencoders, optionally utilizing continuous normalization flows) have an additional advantage in that a received 3D representation of clinical data, such as a wound or appendage, may first be converted into a latent form or representation (i.e., one or more latent representations) and then reconstructed. For example, the latent representations may be reconstructed using a decoder of the reconstruction autoencoder. The reconstructed version of the 3D representation can then be compared to the original 3D representation by calculating the reconstruction loss. A low reconstruction loss can indicate that the reconstruction autoencoder has been successfully trained to encode shape and / or structural aspects of the 3D representation of the latent form (e.g., latent vectors can be generated by a variational autoencoder, or latent capsules can be generated by a capsule autoencoder), and that the latent representation of the 3D representation of the clinical data is suitable for processing by a second ML module.

[0010] Continuous normalizing flows (CNFs) can include a series of invertible mappings that can transform probability distributions. In some implementations, CNFs may be implemented by a series of blocks in an autoencoder's decoder. Such blocks can constrain complex probability distributions, thereby enabling the autoencoder's decoder to learn to map simple distributions to more complex distributions and vice versa, leading to a data accuracy-related technical improvement in that the distribution of shapes of 3D representations of clinical data after reconstruction (e.g., during unfolding) is more representative of the distribution of shapes of 3D representations of clinical data in the training dataset provided during training. The invertibility of CNFs provides the technical advantage of improved mathematical efficiency during training, thereby providing a technical improvement related to resource usage.

[0011] Aspects of the present disclosure may provide a technical solution to the technical problem of predicting the current or future state of a patient's skin lesion, an article attached to the skin, a wound on the skin, a patient's limb or appendage, or an abnormality on the skin using a 2D or 3D representation of 2D or 3D clinical data. In particular, implementing the techniques disclosed herein improves computing systems specifically adapted for classifying 2D or 3D clinical data. For example, aspects of the present disclosure improve the performance of computing systems for 3D representations of clinical data by reducing the consumption of computing resources. In particular, aspects of the present disclosure reduce computing resource consumption by decimating the 3D representation of clinical data (e.g., reducing the number of mesh elements used to describe aspects such as the patient's wound, skin, limb, or article attached to the skin), thereby preventing computing resources from being unnecessarily wasted by processing an excessive number of mesh elements. In addition, decimating the mesh does not reduce the overall predictive accuracy of the computing system (and may actually improve predictions, as the input provided to the ML model after decimation may be a more accurate (or better) representation of the 3D clinical data). For example, noise or other artifacts that are not important (and that may reduce the accuracy of the predictive model) are removed. That is, aspects of the present disclosure provide for more efficient allocation of computing resources in a manner that improves the accuracy of the underlying system.

[0012] Furthermore, aspects of the present disclosure may need to be performed in a time-constrained manner, such as when a wound or abnormality on a patient's skin requires immediate evaluation while the patient is waiting in a clinical situation or environment. Thus, aspects of the present disclosure are necessarily rooted in underlying computer techniques that potentially encode 2D or 3D clinical data using encoder-decoder structures (or other neural networks), and cannot be performed by humans even with the aid of pen and paper. For example, implementations of the present disclosure should be able to: 1) store thousands or millions of mesh elements within 3D clinical data (e.g., a 3D representation of a patient's skin, limbs, wounds, appendages, or articles attached to the patient's body) in a manner that can be processed by a computer processor; 2) perform calculations on the thousands or millions of mesh elements within the 3D clinical data, for example, to quantify aspects of the shape and / or structure of the 3D representation of the clinical data; and 3) predict, based on a machine learning model, one or more class labels to assign to a patient's body part (e.g., the patient's skin, limbs, wounds, appendages, or articles attached to the patient's body), and do so over the course of a short clinic visit. [Brief explanation of the drawings]

[0013] Example implementations disclosed herein may be more fully understood by considering the following detailed description in conjunction with the following drawings: The drawings are not necessarily drawn to scale. Like numbers used in the figures refer to like components. However, it will be understood that the use of a number to refer to a component in a given figure is not intended to limit the component in another figure labeled with the same number. [Figure 1] FIG. 1 illustrates a schematic block diagram of a computing device according to one implementation of the present disclosure. [Figure 2] FIG. 2 shows a flowchart of a method for clinical data analysis according to one implementation of the present disclosure. [Figure 3] FIG. 3 shows a schematic block diagram of the process of the method shown in FIG. 2 according to one implementation of the present disclosure. [Figure 4] FIG. 4 shows a schematic block diagram of a process for training a first ML module according to one implementation of the present disclosure. [Figure 5] FIG. 5 shows a schematic block diagram of a process for training a second ML module according to one implementation of the present disclosure. [Figure 6A] FIG. 6A shows code implementing a 3D encoder and a 3D decoder for the first ML module according to one implementation of the present disclosure. [Figure 6B] FIG. 6B shows code implementing a 3D encoder and a 3D decoder for the first ML module according to one implementation of the present disclosure. [Figure 7] FIG. 7 shows code for implementing a 2D encoder and a 2D decoder for the first ML module according to another implementation of the present disclosure. [Figure 8] FIG. 8 shows a schematic block diagram of a segmentation process for segmenting a first 3D representation of clinical data according to one implementation of the present disclosure. [Figure 9] FIG. 9 shows a flowchart of a method for detecting anomalies according to one implementation of the present disclosure. [Figure 10] FIG. 10 shows a schematic block diagram of the process of the method shown in FIG. 9 according to one implementation of the present disclosure. [Figure 11A] FIG. 11A illustrates different types of skin abnormalities that can be classified by the second ML module according to one implementation of the present disclosure. [Figure 11B] FIG. 11B illustrates identifying different locations of skin abnormalities according to one implementation of the present disclosure. [Figure 11C] FIG. 11C illustrates different stages of skin abnormalities that can be classified by the second ML module according to one implementation of the present disclosure. [Figure 12A] FIG. 12A shows different types of skin abnormalities that can be classified by the second ML module according to another implementation of the present disclosure. [Figure 12B]FIG. 12B illustrates identifying different locations of skin abnormalities according to another implementation of the present disclosure. [Figure 12C] FIG. 12C illustrates different stages of skin abnormalities that can be classified by the second ML module according to another implementation of the present disclosure. [Figure 13] FIG. 13 illustrates an implant site including an implant on a patient's skin, according to one implementation of the present disclosure. [Figure 14] FIG. 14 shows a flowchart of a method for detecting swelling according to one implementation of the present disclosure. [Figure 15] FIG. 15 illustrates different states / stages of swelling in an appendage, according to one implementation of the present disclosure. [Figure 16] FIG. 16 illustrates an article disposed on a patient's skin according to one implementation of the present disclosure. [Figure 17] FIG. 17 shows a schematic block diagram of data augmentation according to one implementation of the present disclosure. [Figure 18] FIG. 18 shows an input 3D mesh and a corresponding reconstructed mesh, respectively, according to one implementation of the present disclosure. [Figure 19] FIG. 19 shows an input 3D mesh and a corresponding reconstructed mesh, respectively, according to one implementation of the present disclosure. [Figure 20] FIG. 20 shows a plot of reconstruction error from reconstructed teeth according to one implementation of the present disclosure. [Figure 21] FIG. 21 shows a bar graph depicting the average absolute distance of all vertices involved in the tooth reconstructions in the data. DETAILED DESCRIPTION OF THE INVENTION

[0014] In the following description, reference is made to the accompanying drawings, which form a part hereof, and in which various implementations are shown by way of illustration. It is to be understood that other implementations are contemplated and may be made without departing from the scope and spirit of the present disclosure. Accordingly, the following detailed description is not to be taken in a limiting sense.

[0015] In the disclosure that follows, the following definitions are adopted:

[0016] As used herein, all numbers should be considered to be modified by the term "about." As used herein, "a," "an," "the," "at least one," and "one or more" are used interchangeably.

[0017] The term "generally," when used herein as a modifier to a characteristic or attribute, unless specifically defined otherwise, means that the characteristic or attribute is readily recognizable by one of ordinary skill in the art, but does not require absolute precision or perfect agreement (e.g., within + / - 20% for quantifiable characteristics).

[0018] The term "substantially," unless otherwise specifically defined, means a high degree of approximation (e.g., within + / - 10% for quantifiable properties), but again does not require absolute precision or perfect agreement.

[0019] The term "about," unless otherwise specifically defined, means a high degree of approximation (e.g., within + / - 5% for quantifiable properties), but again does not require absolute precision or exact agreement.

[0020] As used herein, the terms "first" and "second" are used as identifiers. Therefore, such terms should not be construed as limiting the present disclosure. The terms "first" and "second" when used in conjunction with a feature or element are interchangeable throughout multiple implementations of the present disclosure.

[0021] As used herein, "at least one of A and B" should be understood to mean "A only, B only, or both A and B."

[0022] 1 is a schematic block diagram of a computing device 100 according to one implementation of the present disclosure. The computing device 100 may be used for clinical data analysis. In some implementations, the computing device 100 is deployed within a clinical environment 109.

[0023] The computing device 100 includes an interface 102. The computing device 100 further includes a memory 104 communicatively coupled to the interface 102. The computing device 100 further includes a processor 106 communicatively coupled to the interface 102 and the memory 104. In some implementations, the processor 106 may be interchangeably referred to as a "first processor 106." In some implementations, the interface 102 and the memory 104 may further be communicatively coupled to a second processor 107 that is different from the first processor 106. In some implementations, another device (not shown) may include the second processor 107. In some implementations, the first processor 106 and the second processor 107 may have different data processing capabilities.

[0024] The interface 102 is configured to receive a first three-dimensional (3D) representation 108 representing the clinical data 101. In some implementations, the interface 102 is configured to receive a first two-dimensional (2D) representation representing the clinical data 101.

[0025] The first 3D representation 108 has a different type of data structure than that of the first 2D representation, such as a 2D image (which may include a rectilinear grid of pixels of various colors or intensities).

[0026] In some implementations, the first 3D representation 108 includes at least one of a 3D point cloud, a 3D surface, a 3D mesh, and a voxelized representation (i.e., voxels used in sparse computation, etc.). In some implementations, the first 2D representation includes a 2D raster image including a grid of pixels, such as a 2D color digital photograph, an X-ray image, a heat map, a depth map, or a map of some other sensor-generated modality, or the like.

[0027] The memory 104 is configured to store a first 3D representation 108. The 3D mesh may include edges, faces, or vertices. In some implementations, the memory 104 is configured to store a first 2D representation.

[0028] In some implementations, the clinical data 101 represents the patient's skin 10 (shown in FIG. 11A).

[0029] In some implementations, the clinical data 101 represents an appendage 12 (shown in FIG. 12A ). In some implementations, the appendage 12 may include a limb (such as an arm or leg), a hand, a foot, a digit (such as a finger or toe), or a head of a patient. In some implementations, the clinical data 101 may represent a torso (not shown). In some implementations, the torso may include a groin, abdomen, chest, or shoulder of a patient.

[0030] In some implementations, the clinical data 101 represents an article 16 (shown in FIG. 16) disposed on the patient's skin 10.

[0031] In some implementations, the article 16 is a wrapping 18 (shown in FIG. 16 ) on the patient's skin 10. The wrapping 18 may be, for example, a 3M Corban wrapping. In some implementations, the wrapping 18 may be a dressing. The dressing may include a bandage, a hydrocolloid dressing, a hydrogel dressing, an alginate dressing, a collagen dressing, a foam dressing, a transparent dressing (such as a 3M TEGADERM product), a cloth dressing, or the like.

[0032]

[0023] Figure 2 is a flowchart of a method 200 for clinical data analysis according to one implementation of the present disclosure. Figure 3 is a schematic block diagram of a process 300 of the method 200 shown in Figure 2 according to one implementation of the present disclosure.

[0033] 1-3, at step 202, the method 200 includes receiving a first 3D representation 108 representing the clinical data 101. The first 3D representation 108 includes one or more mesh elements 803 (shown schematically in FIG. 8). Examples of the one or more mesh elements 803 may include vertex coordinates or vertex colors, as described herein.

[0034] In some implementations, the mesh pre-processor module 802 (shown in FIG. 8) may rearrange the mesh elements 803 of the first 3D representation 108 into one or more lists of mesh elements 803 .

[0035] In some implementations, at least one of the one or more mesh elements 803 has at least one associated metadata value. In some implementations, the first 2D representation may also have at least one associated metadata value.

[0036] In some implementations, the at least one associated metadata value includes data regarding at least one of the color of the object, the temperature of the object, the surface impedance of the object, or other aspect of the object that can be measured using one or more sensors or one or more imaging devices. The object may be a portion of any component of the first 3D representation 108 of the clinical data 101, such as skin 10, article 16, wrapping 18, etc. In some implementations, the at least one associated metadata value includes data regarding blood oxygenation or wound oxygenation.

[0037] In step 204 , the method 200 includes calculating one or more mesh element features 303 for one or more mesh elements 803 .

[0038] In some implementations, the one or more mesh element features 303 may be computed by the mesh element feature module 302. In some implementations, the processor 106 is configured to compute the one or more mesh element features 303 for one or more mesh elements 803. In some implementations, the processor 106 is configured to execute the mesh element feature module 302 to compute the one or more mesh element features 303 for one or more mesh elements 803.

[0039] At step 206, the method 200 includes providing the one or more mesh element features 303 as input to a first machine learning (ML) module 304. In some implementations, the first ML module 304 is an autoencoder neural network (e.g., a 3D autoencoder neural network). The first ML module 304 may further include one or more transformers, one or more fully connected layers, one or more combinations of 3D convolutional layers and 3D pooling layers, etc.

[0040] In some implementations, the autoencoder neural network includes a variational autoencoder (VAE) neural network. In some implementations, the autoencoder neural network includes a capsule autoencoder neural network. Additional types of autoencoder neural networks that can be trained for use in the first ML module 304 include convolutional autoencoders (which may include U-Net convolutional models), incomplete autoencoders, shrinkage autoencoders, deep-belief autoencoders (such as those composed of restricted Boltzmann machines for the encoder and decoder), sparse autoencoders, and denoising autoencoders.

[0041] An autoencoder neural network may be trained for use in a 2D domain by training the autoencoder neural network on 2D data. An autoencoder neural network may be trained for use in a 3D domain by training the autoencoder neural network on 3D data. The autoencoder neural network may improve the signal-to-noise ratio in the input data (e.g., the first 2D representation or the first 3D representation 108).

[0042] In some implementations, the processor 106 is configured to provide one or more mesh element features 303 as input to the first ML module 304 .

[0043] In some implementations, image texture features (e.g., SIFT, SURF, ORB, BRIEF, etc.) may be calculated for the first 2D representation and then provided to the first ML module 304 along with the first 2D representation.

[0044] At step 208, the method 200 includes executing a first ML module 304 to encode the first 3D representation 108 into one or more latent representations 305 (or one or more latent embeddings). Thus, the first ML module 304 may be a representation generation module. The one or more latent representations 305 may be information-rich, reduced-dimensionality representations of the first 3D representation 108 of the clinical data 101. For example, the one or more latent representations 305 may include latent vectors or latent capsules.

[0045] Additionally, in some implementations, either the transformer encoder or the transformer decoder may generate one or more latent representations 305 that may be output by the first ML module 304. In some implementations, the encoder portion of a variational autoencoder may generate one or more latent representations 305. In some implementations, the capsule encoder portion of a capsule autoencoder may generate one or more latent representations 305.

[0046] As described above, in some implementations, the processor 106 is configured to provide one or more mesh element features 303 as input to the first ML module 304. The mesh element features 303 can improve the ability of the first ML module 304 to encode the shape and / or structure of the clinical data 101 into a latent form.

[0047] In some implementations, the processor 106 is configured to execute the first ML module 304 to encode the first 3D representation 108 into one or more latent representations 305. Thus, in some implementations, executing the first ML module 304 to encode the first 3D representation 108 into one or more latent representations 305 includes executing the first ML module 304 by the first processor 106 to encode the first 3D representation 108 into the one or more latent representations 305.

[0048] At step 210, the method 200 includes providing one or more latent representations 305 to a second ML module 306 that is different from the first ML module 304. In some implementations, the second ML module 306 may include a neural network (e.g., a convolutional neural network, a set of fully connected layers, or some portion of an encoder-decoder structure, etc.) or a non-neural network ML model (e.g., a support vector machine (SVM) model, a logistic regression model, or other ML model described herein).

[0049] In some implementations, the second ML module 306 may include a multi-layer perceptron (MLP) (e.g., two, three, four, or more fully connected layers with optional skip connections), a transformer, an autoencoder, a decision tree, a K-nearest neighbor model, a naive Bayes model, a random forest model, a gradient boosting model, or others described herein.

[0050] In some implementations, the processor 106 is configured to provide the one or more latent representations 305 to a second ML module 306.

[0051] At step 212, the method 200 includes executing the second ML module 306 to classify the clinical data 101 represented in the first 3D representation 108 into at least one predicted classification label 308. In some implementations, the method 200 includes executing the second ML module 306 to classify the one or more latent representations 305 into at least one predicted classification label 308.

[0052] In some implementations, the processor 106 is configured to execute the second ML module 306 to classify the clinical data 101 represented in the first 3D representation 108 into at least one predicted classification label 308. Thus, in some implementations, executing the second ML module 306 to classify the clinical data 101 represented in the first 3D representation 108 into at least one predicted classification label 308 includes performing, by the first processor 106, a classification of the clinical data 101 represented in the first 3D representation 108 into the at least one predicted classification label 308.

[0053] However, in some other implementations, executing the second ML module 306 to classify the clinical data 101 represented in the first 3D representation 108 into at least one predicted classification label 308 includes executing the classification of the clinical data 101 represented in the first 3D representation 108 into at least one predicted classification label 308 by a second processor 107 different from the first processor 106.

[0054] In some implementations, the second ML module 306 is configured to classify the clinical data 101 into at least one of the following: a type of skin abnormality 20 (shown in FIG. 11A ), a current state of the skin abnormality 20, a future state of the skin abnormality 20, a current state of the implant site 13 (shown in FIG. 13 ) including the implant 14 on the skin 10 (shown in FIG. 13 ), and a future state of the implant site 13 on the skin 10.

[0055] In some implementations, the type of skin abnormality 20 is at least one of a tumor, a wound, a burn, a rash, a puncture, a cyst, an infection, a skin growth, a contusion, a cut, a laceration, an abrasion, a scratch, an ulcer, and a laceration. The skin abnormality 20 may further include a gash or an abrasion.

[0056] In some implementations, the implant 14 is at least one of a skin graft (eg, a tissue graft) and a device implant (eg, a prosthetic device).

[0057] The implant 14 may include a cosmetic implant, a medical implant (e.g., an indwelling port or a subcutaneous port), a ventricular assist device, a near field communication (NFC) chip such as used to make electronic payments, a microchip, a wireless key (e.g., to unlock a car), a radio-frequency identification (RFID) tag, blood flow tracking, a device that monitors body vitals such as temperature or heart rate, an automatic blood glucose monitor or regulator, a device that vibrates in response to environmental conditions (e.g., facing due north), and the like.

[0058] Skin grafting may include regenerative cell therapy, limb reattachment, skin reattachment, skin flap replacement (e.g., from alopecia injuries), bone reconstruction, organ transplantation, and the like.

[0059] In some implementations, the second ML module 306 is configured to classify the clinical data 101 into a current state of swelling 22 (shown in FIG. 15 ) in the appendage 12 or a future state of swelling 22 in the appendage 12. In some implementations, the second ML module 306 is configured to classify the clinical data 101 into a current state of torso swelling or a future state of torso swelling.

[0060] In some implementations, the second ML module 306 is configured to classify the clinical data 101 into a current condition of the article 16 (shown in FIG. 16 ). In some implementations, the current condition of the article 16 includes the fit of the wrapping 18 (shown in FIG. 16 ) on the patient's skin 10.

[0061] In some implementations, the first ML module 304 may be a 2D autoencoder neural network. In such implementations, the first 2D representation may be provided as an input to the first ML module 304. In some implementations, the first ML module 304 may encode the first 2D representation into one or more latent representations 305. In such implementations, the second ML module 306 may be executed to classify the clinical data 101 represented by the first 2D representation into at least one predicted classification label 308.

[0062] The at least one predicted classification label 308 may be associated with a body part (such as an appendage, torso, or skin), an item attached to the body part (e.g., item 16, implant 14, etc.), and / or a health condition of the patient. The at least one predicted classification label 308 may then be used by a clinician in treating the patient. For example, the at least one predicted classification label 308 may be used to make a treatment decision for the patient or to recommend a treatment for the patient. In some examples, the process 300 may be enabled to be deployed and operate on a handheld device (e.g., computing device 100) for use in a clinical environment 109.

[0063] The first and second ML modules 304, 306 may be deployed in a cloud computing environment, on a mobile device, on a laptop, on a desktop computer, in an augmented reality headset, or on another computing device.

[0064] In some implementations, the output generated by either or both of the first and second ML modules 304, 306 may be sent to the clinician via notification (e.g., via SMS, email, or other electronic means). In some implementations, a visualization of a portion of the body part (e.g., related to the clinical data 101) may be generated, and graphics or other indicia may be inserted on top of the visualization to highlight the output of the second ML module 306. In some cases, the visualization may be shown to the clinician via a mobile device, smartphone, laptop, or desktop computer. Treatment may then be provided to the patient by the clinician as a result of examining the output of the second ML module 306.

[0065] FIG. 4 is a schematic block diagram of a process 400 for training the first ML module 304 according to one implementation of the present disclosure.

[0066] For example, when the first ML module 304 includes one or more autoencoder neural networks, the one or more autoencoder neural networks may be trained on 2D or 3D representations of clinical data 101 such as skin 10 (including healthy skin, wounds, incisions, burns, rashes, skin growths, tissue grafts, bruises, etc.) and / or skin-related healthcare materials such as articles 16 (including dressings, casts, splints, compression garments, etc.).

[0067] 1, 2, and 4, in some implementations, the method 200 further includes executing the first ML module 304 to reconstruct the one or more latent representations 305 into a second 3D representation 406 that is a replica of the first 3D representation 108. In some implementations, the processor 106 is further configured to execute the first ML module 304 to reconstruct the one or more latent representations 305 into the second 3D representation 406 that is a replica of the first 3D representation 108.

[0068] In some implementations, when the first ML module 304 may encode a first 2D representation into one or more latent representations 305, the first ML module 304 may be executed to reconstruct the one or more latent representations 305 into a second 2D representation that is a replica of the first 2D representation.

[0069] In some implementations, the first ML module 304 has an encoder-decoder structure including one or more encoders or one or more decoders. Examples of encoder-decoder structures include U-Net, autoencoder, pyramid encoder-decoder, transformer, etc. In some implementations, the first ML module 304 has one or more sets of 3D convolutional layers and 3D pooling layers.

[0070] As described above, in some implementations, the first ML module 304 is an autoencoder neural network (e.g., a 3D autoencoder neural network having a 3D encoder and a 3D decoder, or a 2D autoencoder neural network having a 2D encoder and a 2D decoder) including an encoder 402 and a decoder 404. Specifically, in some implementations, the 3D encoder of the 3D autoencoder neural network is configured to encode the first 3D representation 108 into one or more latent representations 305, and the 3D decoder of the 3D autoencoder neural network is configured to reconstruct the one or more latent representations 305 into a second 3D representation 406. Similarly, in some implementations, the 2D encoder of the 2D autoencoder neural network is configured to encode the first 2D representation into one or more latent representations 305, and the 2D decoder of the 2D autoencoder neural network is configured to reconstruct the one or more latent representations 305 into a second 2D representation.

[0071] In such an implementation, the first ML module 304 may further be used to improve the security or transmission speed of the clinical data 101 .

[0072] For example, a first 2D or 3D representation of clinical data 101 may be collected by a clinician and then undergo subsequent encoding using a 2D or 3D encoder in the first ML module 304 to generate one or more latent representations 305 for each instance of clinical data 101 provided to the first ML module 304. For example, a 2D image of a wound may be generated by a patient at a remote site (e.g., while hiking) and then provided to the first ML module 304 (which may, for example, be executed locally) to generate one or more latent representations 305. When the second ML module 306 is located in a cloud server or on another remote server, the one or more latent representations 305 may be uploaded to the remote server (e.g., via the second processor 107 shown in FIG. 1 ) for subsequent classification by the second ML module 306. The one or more latent representations 305 may be compressed or dimensionality-reduced versions of the original clinical data 101, which may reduce the bandwidth required for transmission.

[0073] In some implementations, the decoder 404 of the autoencoder neural network is located remotely from the encoder 402. The decoder 404 is located on a remote server and may be used to reconstruct one or more latent representations 305 of the clinical data 101 to generate, for example, a second 2D representation (e.g., a reconstructed photograph of the wound). In some implementations, the reconstructed photograph of the wound may be inspected by a clinician to make a treatment decision for the patient. This conversion of the clinical data 101 into one or more latent representations 305 and subsequent transmission of the one or more latent representations 305 may preserve the anonymity and confidentiality of the patient.

[0074] In some implementations, the method 200 further includes calculating a reconstruction loss 408 that quantifies a difference between the first 3D representation 108 and the second 3D representation 406. In some implementations, the method 200 further includes training the first ML module 304 using the reconstruction loss 408. In some implementations, the method 200 further includes training the first ML module 304 using backpropagation using the reconstruction loss 408. In some implementations, training the first ML module 304 using the reconstruction loss 408 includes providing the reconstruction loss 408 to at least one of the encoder 402 and the decoder 404 to train the first ML module 304.

[0075] In some implementations, the reconstruction loss may quantify the difference between the first 2D representation and the second 2D representation.

[0076] In one example, a mesh element labeling operation can be applied to the clinical data 101 (e.g., mesh segmentation to isolate wounds or mesh cleanup to remove extraneous material from a 3D mesh, such as 3D scan artifacts). In some implementations, a registration step can be performed to align a template mesh (e.g., a template of an appendage 12 or other anatomical object) to the clinical data 101 (e.g., using an iterative nearest neighbor technique, etc.). This can provide technical enhancements that improve the accuracy of the mesh correspondence calculation and data precision. Furthermore, correspondences between example 3D meshes of the clinical data 101 and the corresponding template meshes can be calculated using technical improvements that prepare the clinical data 101 so that it is ready to be provided to the reconstruction autoencoder, i.e., the first ML module 304. The prepared dataset of example clinical data can be split into training, validation, and holdout test sets and then used to train the reconstruction autoencoder. The reconstruction autoencoder can be trained using a combination of the reconstruction loss and the KL divergence loss, and optionally other examples of loss functions described herein.

[0077] FIG. 5 is a schematic block diagram of a process 500 for training the second ML module 306 according to one implementation of the present disclosure.

[0078] 2 and 5, in some implementations, the method 200 further includes receiving at least one ground truth classification label 502 for the first 3D representation 108. In some implementations, at least one ground truth classification label for the first 2D representation may be received. The ground truth classification label 502 may be provided by an authority known to be correct.

[0079] In some implementations, the method 200 further includes calculating a loss 504 that quantifies a difference between the at least one ground truth classification label 502 and the at least one predicted classification label 308. In some implementations, the loss 504 may quantify a difference between the at least one ground truth classification label for the first 2D representation and the at least one predicted classification label 308 for the first 2D representation.

[0080] In some implementations, the method 200 further includes training the second ML module 306 using the loss 504. In some implementations, the method 200 further includes training the second ML module 306 using backpropagation using the loss 504.

[0081] 6A and 6B show code 600 for implementing a 3D encoder (e.g., the encoder 402 shown in FIG. 4) and a 3D decoder (e.g., the decoder 404 shown in FIG. 4) for the first ML module 304 according to one implementation of the present disclosure. Specifically, in FIGS. 6A and 6B, the code 600 is for a 3D autoencoder neural network.

[0082] These implementations may include convolutional layers, batch normalization layers, linear neural network layers, Gaussian operations, and continuous normalized flows (CNFs), among others.

[0083] One of the steps that can be performed in VAE training data preprocessing is the calculation of mesh correspondences. Mesh correspondences can be calculated between mesh elements of an input mesh and mesh elements of a reference mesh or template mesh having a known structure (e.g., a template representation). The template representation can include one or more mesh elements arranged in a standardized order (e.g., to match the arrangement used in training the autoencoder neural network). In development, a trial 3D representation (e.g., a mesh of a patient's unrestored teeth, an appliance component undergoing modification, or a fixture model undergoing modification) can undergo correspondence calculations to calculate one or more correspondences between the trial 3D representation and the corresponding template representation. These correspondences allow the mesh elements of the trial 3D representation to be rearranged in an order that matches the arrangement of the mesh elements of the training examples used in training the autoencoder neural network. This results in improved autoencoder reconstruction accuracy due to an improved signal-to-noise ratio.

[0084] In other words, the purpose of mesh correspondence calculation is to calculate correspondence between mesh elements of the surface of a trial input mesh and a template (reference) mesh (e.g., a template representation). Mesh correspondence may generate point-to-point correspondence between the trial input mesh and the template mesh by mapping each vertex from the trial input mesh to at least one vertex in the template mesh. Correspondence may be calculated between mesh elements of the trial input mesh and mesh elements of a reference mesh or template mesh having a known or pre-determined structure.

[0085] The use of mesh correspondence may provide improved data accuracy in mesh reconstruction, as mesh correspondence may reduce sampling errors by the encoder 402, improve alignment, and improve mesh generation quality. In some implementations, an iterative closest point (ICP) algorithm may be performed between the clinical data 101 and the corresponding 3D template during calculation of mesh correspondence. Mesh correspondence may be calculated to establish inter-vertex relationships for use in calculating reconstruction error (as described herein with respect to FIGS. 9 and 10 ).

[0086] 7 shows code 700 for implementing a 2D encoder (e.g., encoder 402 shown in FIG. 4) and a 2D decoder (e.g., decoder 404 shown in FIG. 4) for the first ML module 304 according to one implementation of the present disclosure. Specifically, in FIG. 7, the code 700 is for a 2D autoencoder neural network.

[0087] FIG. 8 shows a schematic block diagram of a segmentation process 800 for segmenting the first 3D representation 108 of the clinical data 101 according to one implementation of the present disclosure.

[0088] In some implementations, geometric deep learning (GDL) techniques for segmenting 3D representations, such as 3D mesh segmentation, may be applied to segments of the first 3D representation 108 using a generative adversarial network (GAN). The segmentation process 800 further illustrates the use of a GAN to train a neural network to segment the first 3D representation 108. Techniques for segmenting 3D representations may be applied to the first 3D representation 108, such as skin 10, appendages, torso, skin anomalies 20, etc.

[0089] The technique for segmenting the 3D representation may further be applied to the first 3D representation 108 that includes objects such as the article 16 or the implant 14. This may be to locate those objects or to facilitate their removal from the first 3D representation 108.

[0090] A first 3D representation 108 of clinical data 101 and corresponding ground truth input 804 (i.e., ground truth mesh element labels) are provided to a segmentation process 800.

[0091] As described above, in some implementations, the mesh pre-processor module 802 may convert the first 3D representation 108 into one or more mesh elements 803. Additionally, the mesh element feature module 302 may calculate one or more mesh element features 303 for the one or more mesh elements 803.

[0092] In some implementations, the first 3D representation 108 may be provided to a mesh element feature module 302 to calculate one or more mesh element features 303 for one or more mesh elements 803. The output of the mesh element feature module 302 (i.e., the one or more mesh element features 303) may be provided to a generator 810.

[0093] The generator 810 can benefit from training that includes a discriminator 822, although the generator 810 may alternatively be trained without the discriminator 822. The generator 810 receives input (e.g., one or more mesh elements 803 and one or more mesh element features 303). According to a particular implementation, the generator 810 uses the received input to determine a predicted output 812 for the first 3D representation 108. For example, for segmentation, the generator 810 may be configured to predict mesh element labels for use in segmentation or mesh cleanup.

[0094] The segmented output, or predicted output 812, of the segmentation process 800 may include mesh element labels for one or more mesh elements 803 (eg, one or more lists of mesh elements).

[0095] The ground truth input 804 may describe labels that are verified or otherwise known to be accurate labels for one or more mesh elements 803 associated with a segmentation output performed on the first 3D representation 108 (e.g., ground truth mesh element labels “true” and “false”). According to particular implementations, the mesh element labels described with respect to the segmentation operation (or mesh cleanup operation) may be used to specify a particular set of one or more mesh elements 803 (e.g., “point” elements, “edge” elements, “face” elements, “vertex” elements, “voxel” elements, etc.) for a particular aspect of the first 3D representation 108 of the clinical data 101. For example, a single triangular polygon of a 3D mesh includes three edge elements, three vertex elements, and one face element. It should be understood, therefore, that a segmented 3D representation consisting of many polygons may have multiple labels associated with the first 3D representation 108.

[0096] The differences between the predicted outputs 812 and the ground truth inputs 804 can be used to calculate one or more loss values ​​814. For example, the loss values ​​814 can represent a regression loss between the predicted outputs 812 and the ground truth inputs 804. That is, according to one implementation, the loss values ​​814 reflect the rate at which the predicted outputs 812 deviate from the ground truth inputs 804. The loss values ​​814 can include an L2 loss, a smoothed L1 loss, or some other type of loss. According to a particular implementation, the L1 loss is defined as follows:

[0097] TIFF2026500182000002.tif8170

[0098] Furthermore, according to a particular implementation, the L2 loss may be defined as follows:

[0099] TIFF2026500182000003.tif9170 where P represents the predicted output 812 and G represents the ground truth input 804.

[0100] Additionally, as described in more detail below, the loss values ​​814 may be provided to the generator 810 to further train the generator 810, for example, by modifying one or more weights in the neural network of the generator 810 to train the underlying model and improve its ability to generate predicted outputs 812 that reflect or substantially reflect the ground truth inputs 804.

[0101] Any of these losses (i.e., the L1 loss and the L2 loss) can be used to provide a loss value (i.e., the loss value 814) for use in training the neural network of the generator 810 via an appropriate training algorithm, such as backpropagation. In some cases, an accuracy score may be used in training the neural network. The accuracy score may quantify the difference between the data structure of the predicted output 812 and the data structure of the ground truth input 804. The accuracy score (e.g., in normalized form) may be fed back to the neural network during the course of training it, for example, through backpropagation.

[0102] For segmentation, the accuracy score may count matching mesh labels between the predicted mesh and the ground truth mesh (i.e., each mesh element has an associated label). The higher the percentage of matching mesh labels, the better the prediction (i.e., when comparing the predicted label, i.e., predicted output 812, to the ground truth label, i.e., ground truth input 804).

[0103] A similar accuracy score may be calculated for mesh cleanup, which also predicts labels for mesh elements, which in some implementations may perform operations on labeled mesh elements, such as transforming or removing mesh elements.

[0104] In general, the intersection on union metric specifies the percentage of correctly predicted edges, faces, and vertices in the predicted mesh after an operation such as completing a segmentation.

[0105] The average boundary distance specifies the distance between the predicted output 812 (or predicted representation 818) for the first 3D representation 108 (such as a 3D mesh, a 3D point cloud, a voxelized representation, or a 3D surface) and the ground truth input 804 (or ground truth representation 820).

[0106] The boundary percentage specifies the percentage of the mesh boundary length of a 3D mesh, such as a segmented 3D mesh, that the distance between the ground truth input 804 (or ground truth representation 820) and the predicted outputs 812 (or predicted representation 818) is less than a threshold. For example, the threshold may determine whether one or more of the predicted outputs 812 are sufficiently close to the ground truth input 804, such as a small line segment between each pair of boundary points.

[0107] 8 is used to train the segmentation process, a line segment (e.g., or any other mesh element) may be labeled as a complete boundary segment if the distance is less than a threshold. The percentage represents the proportion of the segment that lies within the boundary of the predicted output 812 compared to the boundary of the ground truth input 804. The over-segmentation ratio specifies the percentage of the mesh boundary length at which the wound (or other facets of the skin 10 or articles 16 attached to the skin 10) is over-segmented, according to a particular implementation. One or more intersections on the union metric may be used to further train the generator 810 and / or the discriminator 822.

[0108] Fig. 9 is a flowchart of a method 900 for detecting an anomaly 1002 shown in Fig. 10 according to one implementation of the present disclosure. Fig. 10 is a schematic block diagram of a process 1000 of the method 900 shown in Fig. 9 according to one implementation of the present disclosure.

[0109] 9 and 10, in step 902, the method 900 includes receiving a first 3D representation 108 representing the clinical data 101.

[0110] In step 904 , the method 900 includes providing the first 3D representation 108 as an input to the first ML module 304 .

[0111] As described above, in some implementations, the first 3D representation 108 includes one or more mesh elements 803 shown in Figure 8, and one or more mesh element features 303 are calculated for at least one of the one or more mesh elements 803. In some implementations, the one or more mesh element features 303 are further provided to a first ML module 304.

[0112] In step 906, the method 900 includes executing the first ML module 304 to encode the first 3D representation 108 into one or more latent representations 305 and reconstructing the one or more latent representations 305 into a second 3D representation 406 that is a replica of the first 3D representation 108.

[0113] In step 908 , the method 900 includes calculating a reconstruction error that quantifies the difference between the first 3D representation 108 and the second 3D representation 406 .

[0114] In step 910, the method 900 includes determining at least one region 1004 of the first 3D representation 108 that has a reconstruction error greater than a predetermined threshold.

[0115] At step 912, the method 900 includes determining that at least one region 1004 corresponds to an anomaly 1002. In some implementations, the anomaly 1002 includes at least one of a skin anomaly 20 (shown in FIG. 11A ) and an article 16 (shown in FIG. 16 ) placed on the patient's skin 10. In some implementations, the anomaly 1002 includes abnormal material. The abnormal material may include excess material, dead material, or crusted material.

[0116] In some implementations, the method 900 may be performed by the computing device 100 shown in FIG. 1 and may be integrated with a software application (e.g., a mobile application) and used in treating a patient.

[0117] The first ML module 304 (e.g., a 3D mesh reconstruction variational autoencoder (VAE) with optional continuous normalization flow) may be trained to identify anomalies 1002 (e.g., abnormal material) in the first 3D representation 108 (e.g., to identify damaged, infected, or wounded tissue in the first 3D representation 108 of the clinical data 101). For example, the 3D mesh reconstruction VAE may be trained to reconstruct healthy clinical data, such as one or more of examples of expected first 3D representations 108, e.g., healthy skin, healthy tissue, healthy appendages, etc.

[0118] After training, when an unexpected first 3D representation 108 is presented to the 3D mesh reconstruction VAE, the reconstructed mesh (i.e., the reconstructed mesh of the second 3D representation 406) may have a high reconstruction error. In some cases, under conditions in which the 3D mesh reconstruction VAE is trained entirely on healthy (or non-abnormal) clinical data examples, the high reconstruction error may be localized to mesh elements 803 associated with an abnormal portion (i.e., at least one region 1004) of the unexpected first 3D representation 108 (e.g., the high reconstruction error may flag mesh elements 803 associated with damage to the skin 10 or having abnormal skin growths).

[0119] Once the abnormality 1002 is identified, treatment may be administered by a clinician. In some instances, in the case of abnormal material that may be present in the vicinity of a wound or any other breach in the skin 10, the abnormal material may be targeted for debridement.

[0120] Using the 3D mesh reconstruction VAE to identify the anomaly 1002 may facilitate the determination of the resection margin and / or depth (e.g., if the anomaly 1002 is a skin growth to be removed). In some cases, an offset boundary may be calculated around a subset of mesh elements 803 that are identified as anomalies (i.e., anomalies 1002). Such a boundary may be used as the resection margin.

[0121] The first ML module 304 is trained to encode the first 3D representation 108 (or a list of mesh elements corresponding to the first 3D representation 108) using the encoder 402, resulting in one or more latent representations 305 (e.g., one or more latent vectors). The one or more latent representations 305 may then be reconstructed into a second 3D representation 406 (i.e., a replica of the received first 3D representation 108). This reconstructed second 3D representation 406 may then be compared to the input 3D representation (i.e., the first 3D representation 108) using a reconstruction loss calculation, a KL divergence loss calculation, or other losses described herein to calculate a reconstruction error. The one or more loss values ​​(i.e., the reconstruction error) may be used to at least partially train the encoder 402 or the decoder 404.

[0122] Through the training process, this 3D mesh reconstruction VAE very well reconstructs (e.g., with low reconstruction error) a 3D mesh of the first 3D representation 108 that reflects the distribution of the training dataset (e.g., healthy clinical data).

[0123] When a first 3D representation 108 containing anomalous geometry and / or anomalous structure is introduced, the 3D mesh reconstruction VAE may attempt to decompose and reconstruct the first 3D representation 108. As a result, the reconstruction error may be high or greater than a predetermined threshold, which may flag the presence of an anomaly 1002.

[0124] Similarly, a reconstruction error (eg, a 2D reconstruction error) may be determined by comparing an input 2D image (ie, a first 2D representation) with its corresponding reconstructed image.

[0125] In some implementations, one or more mesh element features 303 (e.g., mesh dimension information) may be provided to the encoder 402 to improve the accuracy of the one or more latent representations 305. Similarly, image texture features may be provided to the encoder 402 to improve the accuracy of the one or more latent representations 305. In some implementations, the first 2D representation of the thermochromic dye on the skin 10 may reveal a temperature.

[0126] Figure 11A shows different types of skin abnormalities 20 that can be classified by the second ML module 306 (shown in Figure 3). In the example shown in Figure 11A, the skin abnormalities 20 are different types of wounds.

[0127] 3, 5, and 11A, as described above, a first 2D representation or a first 3D representation 108 of clinical data 101 including a skin abnormality 20 (e.g., a burn, incision, sore, wound, puncture, contusion, other skin injury, rash, infection, or cyst) may be provided to a first ML module 304, which may generate one or more corresponding latent representations 305, which may be provided to a second ML module 306, which may classify the one or more latent representations 305 of the skin abnormality 20 according to the type of the skin abnormality 20. Additionally, the second ML module 306 may classify the skin abnormality 20 according to its clinical concern or severity, such as "superficial" (e.g., as shown in the left-most image of FIG. 11A), "moderate" (e.g., as shown in the center image of FIG. 11A), or "severe" (e.g., as shown in the right-most image of FIG. 11A). In some implementations, the degree of clinical concern or severity may be classified by the second ML module 306 on a scale having, for example, 5 or 10 levels.

[0128] In some implementations, the skin abnormality 20 may be an infection. Accordingly, the second ML module 306 may classify the infection according to a clinical category, such as infected, non-infected, or gangrenous.

[0129] Additionally, in some cases, the skin abnormality 20 may be an opening in the skin 10. The first 2D representation or first 3D representation 108 of the skin 10, including the opening in the skin 10, may be provided to a first ML module 304. Further, a second ML module 306 may classify the corresponding one or more latent representations 305 according to clinical categories, such as an opening from a surgical incision, a burn, a gunshot wound, a puncture wound from various types of objects, etc. Such a classifier may be used in emergency rooms, battlefields, fire scenes, etc.

[0130] In some implementations, a first 3D representation 108 including a 3D mesh of the patient's skin 10 may be generated and then provided to a first ML module 304 including, for example, an encoder 402 (e.g., one that may be trained as part of a 3D VAE with an optional continuous normalization flow) that may generate one or more latent representations 305 of the 3D mesh. A second ML module 306 may classify the one or more latent representations 305 of the skin anomaly 20 to identify the type of the skin anomaly 20. The 3D analysis of the skin anomaly 20 may involve information of the skin anomaly 20, such as high-precision shape, depth, or texture of the skin anomaly 20, which may assist the second ML module 306 in determining the classification.

[0131] Further, when the first 3D representation 108 of the clinical data 101 including the skin abnormality 20 is analyzed, in some implementations, one or more mesh element features 303 may be calculated for one or more mesh elements 803 of the first 3D representation 108. The one or more mesh element features 303 may, in some implementations, be provided to the first ML module 304 to improve the accuracy of the one or more latent representations 305 generated by the first ML module 304.

[0132] Further, as described above, in some implementations, at least one of the one or more mesh elements 803 has at least one associated metadata value including data regarding at least one of the object's color, object's temperature, and object's surface impedance, or some other measurement (e.g., a value associated with the object measured by a sensor). For example, color (e.g., expressed as HSV or RGB) may be associated with one or more mesh elements 803 of the first 3D representation 108. In some implementations, the associated metadata value including data regarding color may be provided to the first ML module 304 to improve the accuracy of the one or more latent representations 305. In some implementations, the object may be illuminated with either UV light or IR light. Color information derived from the resulting image may be associated with one or more mesh elements 803 as an associated metadata value in some implementations. In some implementations, data from an X-ray, a CT scan, an MRI scan, an fMRI scan, or other type of medical scan may be associated with one or more mesh elements 803 as an associated metadata value.

[0133] In some implementations, at least one associated metadata value including data regarding at least one of the object's color, the object's temperature, and the object's surface impedance may also be provided to the second ML module 306 to assist in classification and / or improve classification accuracy.

[0134] For example, data regarding color (e.g., color pixels in the first 2D representation or color as mesh element features 303 in the first 3D representation 108) may assist in classifying burns, bruises, skin rashes, or skin infections, among other skin abnormalities 20. Furthermore, data regarding temperature may assist in classifying insufficient blood flow in a limb, or classifying a vascular access site as "normal / healthy," "infected," "phlebitis," etc.

[0135] In some implementations, the second ML module 306 can classify the status or current state of the skin anomaly 20, such as the broken or damaged skin (i.e., wound, puncture, incision, burn, contusion, etc.), at a particular time point based on the first 3D representation 108 of the broken or damaged skin. In some implementations, the second ML module 306 can be trained (e.g., according to techniques described herein) to label one or more mesh elements 803 to identify healthy tissue, e.g., patches of healthy skin adjacent to the broken or damaged skin.

[0136] 5 and 11A, the second ML module 306 may be trained on a dataset in which at least one example of clinical data 101 reflects a condition in which the skin 10 has an abnormality (i.e., a skin abnormality 20, a deposit, or a foreign object, etc.) and at least one example of clinical data 101 reflects a condition in which the skin 10 is free of the abnormality. The second ML module 306 may be trained, at least in part, by computing a loss (e.g., loss 504 shown in FIG. 5) that compares predicted class labels (e.g., at least one predicted classification label 308) with a reference class label (or at least one ground truth classification label 502). A loss, such as cross-entropy or others described herein, may be computed as part of the training.

[0137] The first ML module 304 may be used to segment the first 3D representation 108 of the skin 10 with openings in the skin 10. This segmentation may classify the mesh elements 803 as "openings within the skin," or "other types of skin abnormalities," "healthy skin surrounding the opening," or any other category. Once this labeling is complete, mesh processing techniques may be used to remove excess material. The resulting cleaned-up mesh may be classified according to the process 300 shown in FIG. 3. In some implementations, the opening categories may include healing incisions, incision dehiscences, abscess drains, and catheters.

[0138] 11B illustrates, according to one implementation of the present disclosure, the identification of different locations of the skin abnormality 20. Specifically, FIG. 11B illustrates the identification of different locations of the skin abnormality 20 that are different types of wounds.

[0139] 8 , 9 , 10 , and 11B , the segmentation techniques described for segmentation process 800 or process 1000 may be applied to the skin abnormality 20 described herein, for example, to quantify the size of the skin abnormality 20 or to show the progression of healing of the skin abnormality 20 over time. A first 3D representation 108 of a patient's skin 10 (e.g., a portion of the skin 10 including a skin abnormality 20, such as a rash, infection, cut or incision, or bruise) may be segmented to isolate areas of the abnormal portion of the skin 10 that include the skin abnormality 20. The areas of the skin abnormality 20 may be quantified at one or more time points using the segmentation techniques described herein. In some implementations, these areas may be plotted over time to show the progression of healing of the skin abnormality 20 (as shown in FIG. 11C ). The areas may be surrounded by a bounding box 1110.

[0140] 11C illustrates different stages of a skin abnormality 20 that can be classified by the second ML module 306 (shown in FIG. 3 ) according to one implementation of the present disclosure. In particular, in FIG. 11C , the skin abnormality 20 is a wound. As shown in FIG. 11C (from left to right), the wound is healing over time due to, for example, treatment (e.g., a dressing, a topical antibiotic, an oral antibiotic, stem cells, endothelial cells, fibroblast growth factor, steroids, or hepatocyte growth factor, etc.).

[0141] 3, 4, and 11C, the second ML module 306, in some implementations, may be trained to assess the state of and / or detect skin abnormalities 20 on the skin 10 (or appendage 12). The second ML module 306, in some implementations, may analyze the skin 10 (or appendage 12) across multiple time points. In some implementations, the state of recovery may be assessed based on one or more of the first 2D representation or the first 3D representation 108 of the clinical data 101, including broken, incised, burned, rashed, bruised, infected, or otherwise damaged skin, for example, after application of a treatment.

[0142] In some examples, tissue healing after a medical procedure (such as surgery) can be assessed over one or more time points using those described herein. For example, a first 3D representation 108 of a healing incision (e.g., with optional data regarding color) can be provided to a first ML module 304, which can then use an encoder 402 to encode the first 3D representation 108 into one or more latent representations 305 (i.e., in latent vector format). The one or more latent representations 305 can be provided to a second ML module 306 trained to classify the incision into classes such as "fresh incision," "early healing in progress," or "fully healed."

[0143] In one example, the healing of a skin abnormality 20, such as a wound, may be monitored. In some implementations, the progress of healing may be expressed in terms of the surface area or percentage area of ​​the body part containing the skin abnormality 20.

[0144] 3, 5, and 11C, the second ML module 306 may be trained to predict future states of the skin abnormality 20 based on tuples comprising either the first 2D representation or the first 3D representation 108 from one or more time points. For example, the training dataset may include historical clinical data (such as clinical data 101) that includes many tuples.

[0145] Each tuple may include one or more 2D images or one or more first 3D representations 108 of the patient's skin 10 (optionally having one or more mesh elements 803 with at least one associated metadata value), and the skin abnormality 20 is included in at least one of the first 3D representations 108.

[0146] Each tuple may include ground truth label information (e.g., ground truth classification label 502) that provides an indication of the future state (or healing trajectory) of the skin abnormality 20 indicated by one or more of the first 3D representations 108 (e.g., “healed” or “improved,” “unchanged,” “worsening,” etc.).

[0147] The first ML module 304 may be trained to generate one or more latent representations 305 from each of the first 3D representations 108 in the tuple (e.g., to generate a latent representation 305 of a 3D mesh of an incision or wound having one or more mesh element features 303 associated with one or more mesh elements 803, optionally having at least one associated metadata value).

[0148] In some implementations, one or more latent representations 305 associated with the tuple may be provided to a second ML module 306, which may be trained to generate at least one predicted classification label 308, such as "cured" or "improved," "unchanged," "worsened," etc. The latent representations 305 for multiple time points may be provided to the second ML module 306, which may include a transformer.

[0149] The Transformer may be provisioned to consume multiple inputs (i.e., latent representations 305) and may be trained to generate a determination as to whether a patient's condition over two or more time points indicates that the patient's skin 10 is "healing" or "improving," "unchanged," or "worsening."

[0150] 12A illustrates different types of skin anomalies 20 that can be classified by the second ML module 306 (shown in FIG. 3) according to another implementation of the present disclosure. In the example shown in FIG. 12A, the skin anomalies 20 are different types of skin growths, such as tumors.

[0151] A partial list of categories of skin growths includes skin tags (acrochordons), warts, dermatofibromas, dermoid cysts, birthmarks (hemangiomas, port wine stains, etc.), freckles, keloids, keratoacanthomas, lipomas, moles (nevus), atypical moles (dysplastic nevi), seborrheic keratosis, melanoma, basal cell carcinoma, squamous cell carcinoma, and cutaneous horns.

[0152] Referring to Figures 3, 5, and 12A, in some implementations, the second ML module 306 may be trained to classify the status of one or more tumors at individual time points based on the first 3D representation 108 of clinical data 101 including one or more tumors.

[0153] In some implementations, the second ML module 306 may be used to label a portion of the skin 10, such as skin 10 containing one or more tumors, according to the health state of that portion of the skin 10.

[0154] In some cases, the second ML module 306 may be trained on the 3D representation of the tumor (i.e., the first 3D representation 108) to apply a label of "cancerous" or "benign" to the 3D representation. In some cases, the second ML module 306 may be trained on the 3D representation of the tumor to apply a label of "dangerous" or "non-dangerous" to the 3D representation.

[0155] The second ML module 306 may be trained to classify the first 3D representation 108 of the skin 10 (or appendage 12), including the tumor and / or healthy tissue next to the tumor, and to predict at least one predicted classification label 308 for the first 3D representation 108. The at least one predicted classification label 308 may include, but is not limited to, “cancerous” or “benign.” In some implementations, the ML model for mesh element labeling may be trained to label mesh elements as either “cancerous” or “benign,” for example, by training the ML model as shown and described with respect to FIG.

[0156] 12B illustrates, according to another implementation of the present disclosure, the identification of different locations of skin abnormalities 20. Specifically, FIG. 12B illustrates the identification of different locations of skin abnormalities 20 that are different types of skin growths.

[0157] 12B , in some implementations, skin growths with cancerous growths may be enclosed within bounding box 1210, while other growths not identified as cancerous growths may not be enclosed within bounding box 1210. Alternatively, in some implementations, skin growths with cancerous growths are enclosed within bounding box 1210, and benign / non-cancerous growths are enclosed by circle 1220. In some other implementations, different types of tumors or skin growths may be enclosed by any enclosed shape, such as a rectangle, square, oval, polygon, etc., according to desired application attributes.

[0158] 12C illustrates different stages of a skin abnormality 20 that may be classified by the second ML module 306 (shown in FIG. 3) according to another implementation of the present disclosure. In particular, in FIG. 12C, the skin abnormality 20 is a skin growth. As depicted in FIG. 12C, the skin growth shrinks over time, for example, due to treatment (e.g., after radiation, chemotherapy, or surgical removal).

[0159] Similarly, as described with reference to FIG. 11C , in a further example, the status of a tumor can be predicted using a tuple including data from multiple time points (obtained over days, weeks, months, etc.). In some implementations, the state of one or more tumor changes may be evaluated, for example, after the application of a treatment. The second ML module 306 can be trained using the process 500 shown in FIG. 5 to classify one or more of the first 3D representations 108 including the tumor and / or healthy tissue adjacent to the tumor and predict one or more class labels (at least one predicted classification label 308) for the one or more of the first 3D representations 108. The at least one predicted classification label 308 may include, but is not limited to, “tumor is growing,” “tumor is shrinking,” or “tumor size is unchanged.”

[0160] 13 illustrates an implant site 13 including an implant 14 on skin 10 according to one implementation of the present disclosure. In the implementation illustrated in FIG. 13, implant 14 is a device implant.

[0161] 3 and 13 , in some implementations, the second ML module 306 may classify the condition of the implant site 13 on the skin 10 based on one or more of the first 3D representations 108 of the skin 10 and / or the implant 14. Such clinical data analysis may be performed at a single time point or over a series of time points, for example, to monitor the healing process after implantation of the implant 14.

[0162] The first 3D representation 108 of the implant site 13 on the skin 10 and / or implant 14 may be provided to a first ML module 304 that may generate one or more latent representations 305 of the first 3D representation 108. The one or more latent representations 305 may be provided to a second ML module 306 that may be trained to classify the one or more latent representations 305. The second ML module 306 may generate at least one predicted classification label 308 for the implant site 13, including, but not limited to, infected, healthy, accepted, rejected, healed, not healed, dehiscence, etc.

[0163] Figure 14 is a flowchart of a method 1400 for detecting swelling 22 shown in Figure 15 according to an implementation of the present disclosure. Figure 15 illustrates different states / stages of swelling 22 in an appendage 12 according to an implementation of the present disclosure. In some implementations, the method 1400 is used to detect lymphedema.

[0164] 3, 5, 14 and 15, in step 1402, the method 1400 includes receiving a first 3D representation 108 representing clinical data 101. The clinical data 101 represents the patient's skin 10 or the patient's appendage 12.

[0165] In step 1404 , the method 1400 includes providing the first 3D representation 108 as an input to the first ML module 304 .

[0166] In step 1406 , the method 1400 includes executing the first ML module 304 to encode the first 3D representation 108 into one or more latent representations 305 .

[0167] In step 1408 , the method 1400 includes providing the one or more latent representations 305 to a second ML module 306 that is different from the first ML module 304 .

[0168] In step 1410, the method 1400 includes executing the second ML module 306 to classify the clinical data 101 into a current state of swelling 22 or a future state of swelling 22. In some implementations, the current state of swelling 22 or the future state of swelling 22 is used to detect lymphedema.

[0169] In some implementations, receiving the first 3D representation 108 includes receiving at least two first 3D representations representing the clinical data 101 acquired at different time intervals.

[0170] In some implementations, providing the first 3D representation 108 as input to the first ML module 304 includes providing at least two first 3D representations as input to the first ML module 304.

[0171] In some implementations, executing the first ML module 304 to encode the first 3D representation 108 into one or more latent representations 305 includes executing the first ML module 304 to encode at least two first 3D representations into one or more corresponding latent representations.

[0172] In some implementations, providing the one or more latent representations 305 to the second ML module 306 includes providing the one or more corresponding latent representations to the second ML module 306.

[0173] In some implementations, executing the second ML module 306 to classify the clinical data 101 into a current state of swelling 22 or a future state of swelling 22 includes comparing one or more corresponding latent representations by the second ML module 306 to classify the clinical data 101 into a current state of swelling or a future state of swelling.

[0174] A first 2D or first 3D representation 108 of a patient's appendage 12 (e.g., finger, arm, leg, foot, hand, etc.) or torso (e.g., groin, abdomen, chest, shoulder, etc.) may be provided to a first ML module 304, which may generate one or more corresponding latent representations 305 and provide them to a second ML module 306, which may classify the corresponding one or more latent representations 305 according to clinical categories such as "swelling present" or "swelling absent." In some implementations, the degree of clinical concern or severity of the swelling 22 may be classified by the second ML module 306, for example, on a scale having 5 or 10 levels.

[0175] In some implementations, the method 1400 may be used to inform a clinician about the severity of the swelling 22. In some cases, clinical data 101 about a patient's appendages or torso may be collected at multiple time points.

[0176] The second ML module 306 may be trained on a dataset in which at least one example of the clinical data 101 reflects the absence of swelling 22 and at least one example of the clinical data 101 reflects a condition in which swelling 22 is present.

[0177] The second ML module 306 may be trained, at least in part, by computing a loss (e.g., loss 504) that compares a predicted class label (e.g., at least one predicted classification label 308) to a reference class label (e.g., at least one ground truth classification label 502). A loss, such as cross-entropy or others described herein, may be computed as part of the training.

[0178] In some cases, the appendage 12 or torso may be scanned (e.g., to generate the first 3D representation 108) when the appendage 12 or torso does not include swelling 22. At a later point in time, the patient's appendage 12 or torso may be re-scanned. The first 3D representation 108 from a given point in time may be provided to a first ML module 304, which may then generate one or more latent representations 305.

[0179] The one or more latent representations 305 may then be provided to a second ML module 306, which may classify the first 3D representation 108 according to the current state of swelling 22 at that time. This analysis of clinical data of the appendage 12 or torso over multiple time points may be applied to the treatment of lymphedema, which affects a patient's lymphatic system and causes fluid to accumulate in the soft tissues of the appendage 12 or torso. A patient can use method 1400 to detect swelling 22 over the course of a day. Once swelling 22 is detected, the patient can apply a wrap or compression garment.

[0180] 16 illustrates an article 16 placed on a patient's skin 10, according to one implementation of the present disclosure. Specifically, in the implementation illustrated in FIG. 16, the article 16 is a wrapping 18 on the patient's skin 10.

[0181] FIG. 16 illustrates different fits of the wrapping 18 to the patient's skin 10. Specifically, FIG. 16 illustrates the wrapping 18 having different tensions on the patient's skin 10. The fit of the wrapping 18 on the patient's skin 10 can be a loose fit / tension (shown in the leftmost image of FIG. 16), a correct fit / tension (shown in the center image of FIG. 16), or a tight fit / tension (shown in the rightmost image of FIG. 16). Note that the texture of the wrapping 18 can change with the change in fit / tension of the wrapping 18 on the patient's skin 10. Thus, the texture of the wrapping 18 can be useful in assessing the proper fit / tension of the wrapping 18 on the skin 10.

[0182] A first 2D representation (e.g., one or more images) or first 3D representation 108 of the appendage 12 or any other body part of the patient with the wrapping 18 attached on the skin 10 may be provided to a first ML module 304, which may generate one or more corresponding latent representations 305, which may be provided to a second ML module 306, which may classify the corresponding one or more latent representations 305 of the appendage 12 or other body part with the wrapping 18 into clinical categories such as “too tight,” “tight,” “loose,” “too loose,” or “just right” according to the current state of the wrapping 18.

[0183] In some implementations, the segmentation techniques described for segmentation process 800 or process 1000 may be performed on a first 3D representation 108 (e.g., a 3D mesh) of wrapping 18 attached on skin 10 to label mesh elements 803 as "healthy skin," "wrapping," or some other category. The portion of the 3D mesh that belongs to wrapping 18 may be provided to a first ML module 304 so that mesh classification may be performed.

[0184] As mentioned above, in some implementations, the wrapping 18 may be a dressing. In such implementations, the second ML module 306 may classify one or more latent representations 305 of a dressing on the skin 10 of the patient's appendage 12 or other body part according to the current state of the dressing or bandage into clinical categories, such as wrinkles present, wrinkles that may lead to leakage or introduction of infection present, or no wrinkles present. Wrinkles in the dressing may predict leakage. Specifically, wrinkles may lead to leakage. Thus, the techniques of the present disclosure may be trained to analyze wrinkles in a dressing or bandage.

[0185] In some implementations, the segmentation techniques described for segmentation process 800 or process 1000 may be performed on a first 3D representation 108 (e.g., a 3D mesh) of the dressing and the skin 10 surrounding the dressing to label mesh elements 803 according to their membership in respective portions of the 3D mesh (e.g., healthy skin, wound, dressing, etc.).

[0186] This type of segmentation process can further aid in clinical data analysis (e.g., analyses that seek to detect and classify wrinkles that may lead to leakage) by isolating portions of the 3D mesh that represent the dressing (or by isolating specific portions of the 3D mesh that correspond to wrinkles). Once the portions of the 3D mesh that represent the dressing are isolated, the process of identifying wrinkles and predicting whether those wrinkles may lead to leakage can be greatly improved and strengthened (because neural networks for wrinkle classification can isolate and examine the dressing).

[0187] In some cases, the segmentation process may further isolate one or more particular wrinkles on the dressing (e.g., applying a separate label to each of the mesh elements 803 found on a particular wrinkle, and applying different labels to each of the mesh elements 803 found on different wrinkles). This refined segmentation may further aid a neural network that is trained to classify wrinkles as "likely to cause leakage" or "unlikely to cause leakage" (among other categories).

[0188] FIG. 17 is a schematic block diagram of a data augmentation process 1700 that can be applied to one or more mesh elements 803 of the first 3D representation 108 of the clinical data 101 shown in FIG. 2, according to one implementation of the present disclosure.

[0189] Various neural network models of the present disclosure can benefit from data augmentation. Data augmentation, such as by the data augmentation process 1700 shown in FIG. 17, can increase the size of a training dataset of clinical data (e.g., clinical data 101). Data augmentation can provide additional training examples by adding random rotations, translations, and / or rescaling to copies of existing clinical data. In some implementations of the techniques of the present disclosure, data augmentation can be performed by perturbing or jittering the vertices of a 3D mesh in a manner similar to that described in “Equidistant and Uniform Data Augmentation for 3D Objects,” IEEE Access, Digital Object Identifier 10.1109 / ACCESS.2021.3138162). The positions of the vertices may be perturbed, for example, through the addition of Gaussian noise with 0 mean and 0.1 standard deviation. Other mean and standard deviation values ​​are possible according to the techniques of the present disclosure.

[0190] The data augmentation process 1700 may be used by a system (e.g., the computing device 100 shown in FIG. 1) to apply to clinical data 101 shown in FIG. 2. As mentioned above, a non-limiting example of clinical data is a 3D mesh describing the contours of a wound or skin 10 adjacent to a wound. At block 1702, the clinical data 101 (e.g., the 3D mesh) is received as input. At block 1704, the system of the present disclosure may generate a copy of the clinical data 101. At block 1706, the system of the present disclosure may apply one or more stochastic rotations to the clinical data 101. At block 1708, the system of the present disclosure may apply a stochastic transformation to the clinical data 101. At block 1710, the system of the present disclosure may apply a stochastic scaling operation to the clinical data 101. At block 1712, the system of the present disclosure may apply a stochastic perturbation to one or more mesh elements 803 of the clinical data 101. At block 1714, the system of the present disclosure may output augmented 3D clinical data, which may be formed by the data augmentation process 1700 of FIG.

[0191] 18 and 19 each show an input 3D mesh on the left and a corresponding reconstructed mesh on the right, according to one implementation of the present disclosure. Specifically, FIG. 18 and FIG. 19 each show an input 3D mesh of a tooth on the left and a corresponding reconstructed mesh of the tooth on the right, according to one implementation of the present disclosure. The first ML module 304 (i.e., the reconstruction autoencoder shown in FIG. 4) can be trained to reconstruct a first 3D representation 108 of the clinical data 101, such as skin 10, an appendage 12, a wound, a dressing, or a type of anatomical structure (e.g., a tooth).

[0192] FIG. 20 shows a plot of the reconstruction error from the reconstructed teeth, called a reconstruction error plot, according to an implementation of the present disclosure.

[0193] Specifically, Figure 20 shows the reconstruction error for the results described above with respect to Figures 18 and 19 in what is called a reconstruction error plot in millimeters (mm). Note that the reconstruction error is less than 50 microns at the cusp tips and much less than 50 microns over most of the tooth surface. Compared to a typical tooth having a size of 1.0 cm, an error rate of 50 microns (or less) can mean that the tooth surface was reconstructed with an error rate of less than 0.5%.

[0194] FIG. 21 is a bar graph, where each bar represents an individual tooth and represents the average absolute distance of all vertices involved in the reconstruction of that tooth in the data used to evaluate the performance of the reconstruction model (e.g., the first ML module 304).

[0195] Various loss calculation techniques are generally applicable to the techniques of this disclosure, for example, to calculate reconstruction loss 408 (shown in FIG. 4) or loss 504 (shown in FIG. 5).

[0196] These losses include L1 and L2 losses (as described above), mean squared error (MSE) losses, and cross-entropy losses, among others. Losses may be computed and used in training neural networks such as multilayer perceptrons (MLPs), U-Net structures, generators and discriminators (e.g., for GANs), autoencoders, variational autoencoders, regularized autoencoders, masked autoencoders, and transformer structures. Some implementations may use either triplet or contrastive losses, for example, in sequence learning.

[0197] The loss may also be used to train the encoder and decoder structures. The KL divergence loss may be used at least in part to train one or more of the neural networks of this disclosure, taking advantage of the Gaussian behavior it imparts to the optimization space. This Gaussian behavior may enable a reconstruction autoencoder to generate better reconstructions (e.g., when a latent vector representation is modified and the modified latent vector is reconstructed using a decoder, the resulting reconstruction is more likely to be a valid instance of the input representation). There are other techniques for computing losses that may be described elsewhere in this disclosure. Such losses may be based on quantifying the difference between two or more 3D representations.

[0198] The MSE loss calculation may include calculating the mean squared distance between two sets, vectors, or datasets. The MSE may be minimized generally. The MSE may be applicable to regression problems where predictions generated by a neural network or any other ML model may be real numbers. In some implementations, the neural network may include one or more linear activation units on the output to generate the MSE predictions. Mean absolute error (MAE) loss and mean absolute percentage error (MAPE) loss may also be used in accordance with the techniques of this disclosure.

[0199] Cross-entropy, in some implementations, may be used to quantify the difference between two or more distributions. Cross-entropy loss, in some implementations, may be used to train neural networks of the present disclosure. Cross-entropy loss, in some implementations, may involve comparing predicted probabilities to ground truth probabilities. Other names for cross-entropy loss include "logarithmic loss," "logistic loss," and "logarithmic loss." A smaller cross-entropy loss may indicate a better (e.g., more accurate) model. Cross-entropy loss may be logarithmic. Cross-entropy loss, in some implementations, may be applied to binary classification problems. In some implementations, the neural network may include a sigmoid activation unit at the output to generate probability predictions. In the case of multi-class classification, cross-entropy can also be used. In such cases, a neural network trained to make multi-class predictions may include one or more softmax activation functions at the output (e.g., if there is one output node for the class to be predicted). Other loss calculation techniques that may be applied to training the neural networks of the present disclosure include one or more of Huber loss, Hinge loss, Categorical hinge loss, Cosine similarity, Poisson loss, Logcosh loss, or Mean Squared Logarithmic Error loss (MSLE). Other loss calculation methods are described herein and may be applied to training any of the neural networks described in this disclosure.

[0200] In some implementations, one or more of the neural networks of the present disclosure may be trained at least in part by a loss based on at least one of Point-wise Mesh Euclidean Distance (PMD) and Earth Mover's Distance (EMD). Some implementations may incorporate a Hausdorff distance (HD) calculation into the loss calculation. Calculating the Hausdorff distance between two or more 3D representations (such as 3D meshes) may provide one or more technical improvements in that HD not only considers the distance between the two meshes, but also considers how those meshes are oriented and the relationship between the mesh shapes in those orientations (or positions or poses). The Hausdorff distance can improve the comparison of two or more 3D clinical data, such as two or more instances of 3D clinical data in different poses.

[0201] The techniques of this disclosure may include operations such as 3D convolution, 3D pooling, 3D deconvolution, and 3D unpooling. 3D convolution can assist the segmentation process, for example, when downsampling a 3D mesh. 3D deconvolution undoes 3D convolution, for example, in U-Net. 3D pooling can assist the segmentation process, for example, in summarized neural network feature maps. 3D unpooling undoes 3D pooling, for example, in U-Net. These operations may be implemented by one or more layers within the predictive or generative neural networks described herein. These operations can be applied directly to mesh elements, such as mesh edges or mesh faces. These operations are invariant to changes in mesh rotation, scale, and translation, providing a technical improvement over other approaches. Generally, these operations depend on edge (or face) connectivity, and therefore remain invariant to mesh changes in 3D space as long as edge (or face) connectivity is preserved. That is, operations can be applied to 3D clinical data and produce the same output regardless of the orientation, position, or scale of the 3D clinical data, which can lead to improved data accuracy. MeshCNN is a general-purpose deep neural network library for 3D triangular meshes and can be used for tasks such as 3D shape classification or mesh element labeling (e.g., segmentation or mesh cleanup). MeshCNN performs these operations on mesh edges. Other toolkits and implementations may operate on edges or faces.

[0202] In some implementations of the techniques of the present disclosure, a neural network may be trained to operate on a 2D representation (e.g., a first 2D representation including one or more images). In some implementations of the techniques of the present disclosure, a neural network may be trained to operate on a 3D representation (e.g., a first 3D representation 108 including a 3D mesh or a 3D point cloud). An imaging device may capture 2D images of the side of the patient's body from various views. A 3D scanner may also (or alternatively) capture a 3D mesh or a 3D point cloud that describes aspects of the patient's body. According to various techniques, an autoencoder (or other neural network described herein) may be trained to operate on either or both of the 2D and 3D representations.

[0203] A 2D autoencoder (comprising a 2D encoder and a 2D decoder) can be trained on 2D image data to convert the input 2D image into a latent form (such as a latent vector or latent capsule) using the 2D encoder, and then reconstruct a replica of the input 2D image using the 2D decoder. For handheld mobile applications developed for such analysis (e.g., analysis of clinical data), the 2D images can be easily captured using one or more on-board cameras. In other examples, the 2D images can be captured using a 2D scanner configured for such functionality. Among the operations that can be used in implementations of a 2D autoencoder (or any other 2D neural network) for 2D image analysis are 2D convolution, 2D pooling, and 2D reconstruction error calculation.

[0204] 2D convolution may involve "sliding" a kernel across a 2D image, computing element-wise multiplications, and adding those element-wise multiplications to an output pixel. The output pixels resulting from each new position of the kernel are stored in an output 2D feature matrix. In some implementations, neighboring elements (e.g., pixels) may be at well-defined locations (e.g., above, below, left, and right) in a rectilinear grid.

[0205] A 2D pooling layer can be used to downsample a feature map and summarize the presence of some features within that feature map.

[0206] The 2D reconstruction error can be calculated between pixels in the input image and pixels in the reconstructed image. The mapping between pixels can be well understood (e.g., assuming both images have the same dimensions, the top pixel [23, 134] in the input image is directly compared to pixel [23, 134] in the reconstructed image).

[0207] Among the advantages provided by the 2D autoencoder-based techniques of the present disclosure is the ease of capturing 2D image data using a handheld device. In some cases, when an external data source provides data for analysis, there may be instances where only 2D image data is available. When only 2D image data is available, analysis using a 2D autoencoder is warranted.

[0208] Modern mobile devices (such as commercially available smartphones) may also have the capability to generate 3D data (e.g., using multiple cameras and stereo photogrammetry, or a single camera moved around an object to capture multiple images from different views, or both), which in some implementations may be arranged into a 3D representation, such as a 3D mesh, a 3D point cloud, and / or a 3D voxelized representation. Analysis of a 3D representation of an object may, in some cases, provide a technical improvement over a 2D analysis of the same object. For example, a 3D representation may describe the object's geometry and / or structure with less ambiguity than a 2D representation (which may include shadows and other artifacts that complicate the depiction of depth from the object and the object's texture). In some implementations, 3D processing may enable technical improvements due to inverse optics problems that can sometimes adversely affect 2D representations. Inverse optics problems refer to the phenomenon whereby the size of an object, the orientation of the object, and the distance between the object and the imaging device may sometimes be confused in a 2D image of that object. Any given projection of an object onto the imaging sensor of an imaging device can be mapped to an infinite number of {size, orientation, distance} pairings. 3D representations may enable technical improvements in that they remove ambiguities introduced by inverse optics problems.

[0209] Devices configured for the dedicated purpose of 3D scanning, such as 3D scanners (or CT or MRI scanners), can generate 3D representations of objects (e.g., aspects of a patient's body) with significantly higher fidelity and accuracy than is possible with handheld devices. When such high-fidelity 3D data is available (e.g., in applying the classification techniques described herein), the use of a 3D autoencoder provides technical improvements (such as increased data accuracy) to extract the best possible signal from those 3D data.

[0210] A 3D autoencoder (comprising a 3D encoder and a 3D decoder) can be trained on a 3D data representation, converting the input 3D representation into a latent form (such as a latent vector or latent capsule) using the 3D encoder, and then reconstructing a replica of the input 3D representation using the 3D decoder. Among the operations that can be used to implement a 3D autoencoder for analysis of a 3D representation (e.g., a 3D mesh or a 3D point cloud) are 3D convolution, 3D pooling, and 3D reconstruction error calculation.

[0211] For each mesh element, a 3D convolution can be performed to aggregate local features from nearby mesh elements. Processing can be performed above and beyond techniques for 2D convolution to account for different counts and locations of neighboring mesh elements (relative to a particular mesh element). A particular 3D mesh element may have a variable number of neighbors, and those neighbors may not be found in expected locations (as opposed to pixels in 2D convolution, which may have a fixed number of neighboring pixels that may be found in known or expected locations). In some cases, the order of neighboring mesh elements may be relevant to the 3D convolution.

[0212] A 3D pooling operation may enable the combination of features from a 3D mesh (or other 3D representation) at multiple scales. 3D pooling may iteratively reduce a 3D mesh to the mesh elements that are most highly relevant to a given application (e.g., a neural network trained on). Similar to 3D convolution, 3D pooling may benefit from special processing beyond that involved in 2D convolution to account for different counts and locations of neighboring mesh elements (relative to a particular mesh element). In some cases, the order of neighboring mesh elements may be less relevant for 3D pooling than for 3D convolution.

[0213] The 3D reconstruction error may be calculated using one or more of the techniques described herein, such as calculating the Euclidean distance between corresponding mesh elements or between two meshes. Other techniques are possible according to aspects of the present disclosure. The 3D reconstruction error may generally be calculated on 3D mesh elements rather than on 2D pixels as in the 2D reconstruction error. Because the 3D representation may, in some cases, have less ambiguity than the 2D representation (i.e., less ambiguity in form, shape, and / or structure), the 3D reconstruction error may allow for technical improvements over the 2D reconstruction error. In some implementations, additional processing may be required for 3D reconstruction over that of the 2D reconstruction due to the complexity of the mapping between the input mesh elements and the reconstructed mesh elements (i.e., the input mesh and the reconstructed mesh may have different mesh element counts, and the mapping between mesh elements may be less clear than the mapping between pixels in the 2D reconstruction). Technical improvements in 3D reconstruction error calculation include improved data precision.

[0214] The 3D representation may be generated using a 3D scanner, a computed tomography (CT) scanner, an ultrasound scanner, a magnetic resonance imaging (MRI) machine, or a mobile device capable of performing stereophotogrammetry. The 3D representation may describe the shape and / or structure of an object. The 3D representation may include one or more of a 3D mesh, a 3D point cloud, and / or a 3D voxelized representation, among others. A 3D mesh includes edges, vertices, or faces. Although interrelated in some cases, these three types of data are distinct. Vertices are points in 3D space that define the boundary of the 3D mesh. These points are alternatively described as a point cloud, apart from further information about how the points are connected to each other, as described by edges. An edge is described by two points and may also be called a line segment. A face is described by several edges and vertices. For example, in the case of a triangular mesh, a face includes three vertices, which are interconnected to form three adjacent edges. Some 3D meshes may include degenerate elements, such as non-manifold mesh elements, that can be removed for the benefit of later processing. Other mesh preprocessing operations are possible according to aspects of the present disclosure. 3D meshes are generally formed using triangles, but in other implementations, they may be formed using quadrilaterals, pentagons, or some other n-gons. In some implementations, the 3D mesh may be converted into one or more voxelized geometries (i.e., including voxels), such as when sparse processing is performed. Techniques of the present disclosure that operate on 3D meshes may receive as input one or more meshes describing 3D clinical data (e.g., a patient's skin with deposits). Each of these meshes may generally undergo preprocessing before being input into a predictive architecture (e.g., including at least one of an encoder, decoder, pyramid encoder-decoder, and U-Net). This preprocessing may include converting the mesh into a list of mesh elements, such as vertices, edges, faces, or, in the case of sparse processing, voxels. A feature vector may be generated for a selected mesh element type or types (e.g., vertices). In some examples, one feature vector is generated for each vertex of the mesh.Each feature vector may contain a combination of spatial and / or structural features as specified in the table below.

[0215] TIFF2026500182000004.tif97155

[0216] Table 1 discloses non-limiting examples of mesh element features. In some implementations, color (or other visual cue / identifier) ​​may be considered a mesh element feature in addition to the spatial or structural mesh element features described in Table 1. Color may be represented in RGB, HSV, or the like. As used herein (e.g., in Table 1), a point differs from a vertex, which is part of a 3D point cloud, but a vertex is part of a 3D mesh and may have an incident face or edge. A dihedral angle (which may be expressed in either radians or degrees) may be calculated as the angle (e.g., a signed angle) between two connected faces (e.g., two faces connected along an edge). The sign of the dihedral angle can reveal information about the convexity or concavity of the mesh surface. For example, a positively signed angle may indicate a convex surface in some implementations. Furthermore, a negatively signed angle may indicate a concave surface in some implementations. To calculate the principal curvatures of a mesh vertex, first, directional curvatures may be calculated for each neighboring vertex around the vertex. These directional curvatures may be sorted into circular order (e.g., 0, 49, 127, 210, 305 degrees) in proximity to the vertex normal vector, or may comprise a subsampled version of the full curvature tensor. Circular order means sorted by angle around an axis. The sorted directional curvatures can contribute to a linear system of equations amenable to a closed-form solution from which the two principal curvatures and directions that can characterize the full curvature tensor can be estimated.

[0217] Consistent with Table 1, a voxel may also have features calculated as a collection of other mesh elements (e.g., vertices, edges, and faces) that either intersect with the voxel or, in some implementations, are primarily or completely contained within the voxel. Rotating a mesh may not change structural features, but may change spatial features. Also, as explained elsewhere in this disclosure, the term mesh should be considered in a non-limiting sense to include 3D meshes, 3D point clouds, and 3D voxelized representations. In some implementations, alternative methods exist for describing the geometry of a mesh apart from mesh element features, such as 3D keypoints and 3D descriptors. Examples of such 3D keypoints and 3D descriptors can be found in TONIONI A et al., "Learning to detect good 3D keypoints," Int J Comput Vis. 2018 Vol. 126, pp. 1-20. In some implementations, 3D keypoints and 3D descriptors may describe extrema (either minima or maxima) of the surface of the 3D representation. In some implementations, one or more mesh element features may be computed, at least in part, via deep feature synthesis (DFS), for example, as described in J.M. Khanter and K. Veeramachaneni, "Deep feature synthesis: Towards automating data science endeavors," 2015 IEEE International Conference on Data Science and Advanced Analytics (DSAA), 2015, pp. 1-10, doi:10.1109 / DSAA.2015.7344858.

[0218] Representation-generating neural networks based on autoencoders, U-Nets, transformers, other types of encoder-decoder structures, convolutional and pooling layers, or other models can benefit from the use of mesh element features. Mesh element features can convey aspects of the surface shape and / or structure of the 3D representation to the neural network model of the present disclosure. Each mesh element feature describes distinct information about the 3D representation that may not be redundantly present in other input data provided to the neural network. For example, vertex curvature can quantify concave or convex aspects of the surface of the 3D representation that would otherwise not be understood by the network. In other words, mesh element features can provide a processed version of the structure and / or shape of the 3D representation, data that would not otherwise be available to the neural network. This processed information is often more accessible or amenable for the neural network to encode into its weights. A system embodying the techniques disclosed herein has been utilized to perform several experiments on 3D representations of teeth. For example, mesh element features have been provided to a representation generation neural network based on a U-Net model, and also to a representation generation model based on a variational autoencoder with continuous normalized flow. Based on experiments, it was found that a system that uses the full complement of mesh element features (e.g., "XYZ" coordinate tuples, "normal vectors," "vertex curvatures," point gyrations, and normal gyrations) is at least 3% more accurate than a system that does not.

[0219] The point-pivot describes an "XYZ" coordinate tuple with a local coordinate system (e.g., the tooth centroid). The normal-pivot describes a "normal vector" with a local coordinate system (e.g., the tooth centroid). Furthermore, training converges more quickly when the full complement of mesh element features is used. In other words, machine learning models trained using the full complement of mesh element features tended to be more accurate quickly (earlier) than systems that were not trained. For a system that was previously 91% accurate, a 3% improvement in accuracy reduces the actual error rate by more than 30%.

[0220] Such feature vectors may be presented as inputs to predictive models, and in some implementations, such feature vectors may be presented to one or more inner layers of neural networks that are part of one or more of those predictive models.

[0221] According to a particular implementation, convolutional layers in the various 3D neural networks described herein may use edge data to perform mesh convolution. The use of edge data ensures that the 3D neural network is not sensitive to different input orders of 3D elements. In addition to, or instead of, using edge data, convolutional layers may use vertex data to perform mesh convolution. The use of vertex data is advantageous in that there are typically fewer vertices than edges or faces, and thus vertex-oriented processing may lead to lower processing overhead and lower computational costs. In addition to, or instead of, using edge or vertex data, convolutional layers may use face data to perform mesh convolution. Furthermore, in addition to, or instead of, using edge, vertex, or face data, convolutional layers may use voxel data to perform mesh convolution. The use of voxel data is advantageous in that, depending on the selected granularity, there may be significantly fewer voxels to process compared to the vertices, edges, or faces in the mesh. Low density processing (using voxels) can lead to lower processing overhead and lower computational costs (especially in terms of computer memory or RAM usage).

[0222] Because the generator network of the present disclosure may be implemented as one or more neural networks, the generator may include an activation function. When executed, the activation function outputs a decision on whether a neuron in the neural network should fire (e.g., send output to the next layer). Some activation functions may include binary step functions or linear activation functions. Other activation functions impart nonlinear behavior to neural networks and include sigmoid / logistic activation functions, Tanh (hyperbolic tangent) functions, rectified linear units (ReLU), leaky ReLU functions, parametric ReLU functions, exponential linear units (ELU), softmax functions, swish functions, Gaussian error linear units (GELU), or scaled exponential linear units (SELU). Linear activation functions may be well suited to some regression applications (among others) in the output layer. A sigmoid / logistic activation function may be well suited to some binary classification applications (among other applications) at the output layer. A softmax activation function may be well suited to some multi-class classification applications (among other applications) at the output layer. A sigmoid activation function may be well suited to some multi-label classification applications (among other applications) at the output layer. A ReLU activation function may be well suited to some convolutional neural network (CNN) applications (among other applications) at the hidden layer. Tanh and / or sigmoid activation functions may be well suited to some recurrent neural network (RNN) applications, e.g., at the hidden layer (among other applications).There are several optimization algorithms that can be used in training the neural networks of the present disclosure (such as when updating neural network weights), including gradient descent (which uses first-order derivatives to determine training gradients and is commonly used in training neural networks), Newton's method (which may utilize second-order derivatives in loss calculations to find better training directions than gradient descent, but may require calculations involving Hessian matrices), and conjugate gradient methods (which may have faster convergence than gradient descent, but do not require Hessian matrix calculations that may be required by Newton's method). In some implementations, additional methods can be used to update weights in addition to or instead of the techniques described above. These further methods include the Levenberg-Marquardt method and / or simulated annealing. A backpropagation algorithm is used to transfer the results of the loss calculations back to the neural network so that the neural network weights can be adjusted and learning can proceed.

[0223] Neural networks contribute to many of the functions of the disclosed applications and may embody some or all of a variety of different neural network models. Examples include U-Net architecture, multi-layer perceptron (MLP), transformer, pyramid architecture, recurrent neural network (RNN), autoencoder, variational autoencoder, regularized autoencoder, conditional autoencoder, capsule neural network, capsule autoencoder, stacked capsule autoencoder, denoising autoencoder, sparse autoencoder, conditional autoencoder, long / short term memory (LSTM), gated recurrent unit (GRU), deep belief network (DBN), deep convolutional network (DCN), deep convolutional inverse graphics network (DCIGN), liquid state machine (LSM), extreme learning machine (ELM), echo state network (ESN), deep residual network (DRN), Kohonen network (KN), and others. Examples of such models include neural network (KN), neural Turing machine (NTM), or generative adversarial network (GAN). In some implementations, an encoder or decoder structure may be used. Each of these models offers one or more of its own particular advantages. For example, certain neural network architectures may be particularly well suited to particular ML techniques. For example, autoencoders are particularly well suited to classifying 3D clinical data due to their ability to convert 3D clinical data into a more easily classifiable form.

[0224] In some implementations, the neural networks of the present disclosure can be adapted to operate on 3D point cloud data (alternatively, on 3D mesh or 3D voxelized representations). Numerous neural network implementations may be applied to processing 3D representations and training predictive and / or generative models for clinical applications, including PointNet, PointNet++, SO-Net, spherical convolution, Monte Carlo convolution and dynamic graph networks, PointCNN, ResNet, MeshNet, DGCNN, VoxNet, 3D-ShapeNets, Kd-Net, Point GCN, Grid-GCN, KCNet, PD-Flow, PU-Flow, MeshCNN, and DSG-Net.

[0225] Some implementations of the techniques of this disclosure incorporate the use of autoencoders. Autoencoders that can be used in accordance with aspects of this disclosure include, but are not limited to, AtlasNet, FoldingNet, and 3D-PointCapsNet. Some autoencoders can be implemented based on PointNet.

[0226] Representation learning can be applied to the techniques of this disclosure by training a neural network to learn representations of clinical data and then using another neural network to classify the representations. Some implementations can use a VAE or capsule autoencoder to generate a representation of the reconstructed features of one or more 3D representations of clinical data (e.g., of a wound or appendage). That representation (either a latent vector or a latent capsule) can then be used as input to a classification module.

[0227] The systems of the present disclosure may implement end-to-end training. Some of the end-to-end training-based techniques of the present disclosure may involve two or more neural networks that are trained together (i.e., weights are updated simultaneously during the processing of each batch of input clinical data). End-to-end training, in some implementations, may be applied to the classification techniques described herein.

[0228] According to some of the transfer learning-based implementations of the present disclosure, a neural network (e.g., U-Net) may be trained for a first task. The neural network trained on the first task may be used to provide one or more starting neural network weights for training another neural network trained to perform a second task. The first neural network may learn low-level neural network features of clinical data and be shown to perform well on the first task. The second neural network may demonstrate faster training and / or improved performance by using the first neural network as a starting point in training. Some layers may be trained to encode neural network features for clinical data that were in the training dataset. These layers may then be fixed (or undergo slight changes over the course of training) and combined with other neural network components, such as additional layers, to be trained for other tasks. In this way, a portion of a neural network for one or more of the techniques of the present disclosure may receive initial training on another task, which may result in significant learning in the trained network layers. This encoded learning can then be built upon with further task-specific training of other networks.

[0229] In some implementations, a neural network trained to output predictions based on clinical data may first be partially trained on one of the following publicly available datasets before being further trained on the clinical data: the Google PartNet dataset, the ShapeNet dataset, the ShapeNetCore dataset, the Princeton Shape Benchmark dataset, the ModelNet dataset, the ObjectNet3D dataset, the Thingi10K dataset (particularly relevant to 3D printed part verification), the ABC: Big CAD Model Dataset for Geometric Deep Learning, ScanObjectNN, VOCASET, 3D-FUTURE, MCB: Mechanical Parts Benchmark, the PoseNet dataset, the PointCNN dataset, the MeshNet dataset, the MeshCNN dataset, the PointNet++ dataset, the PointNet dataset, or the PointCNN dataset.

[0230] Transfer learning can be used to further train any of the following networks: GCN (Graph Convolutional Network), PointNet, ResNet, or any of the other neural networks from the published literature listed above.

[0231] The disclosed system may train ML models using representation learning. Advantages of representation learning include the fact that the generative network is guaranteed to receive input data with a known size and / or standard format, as opposed to receiving inputs with variable size or structure. Representation learning may produce improved performance over other methods because noise in the input data may be reduced (e.g., because the representation-generating model extracts important aspects of the input representation (e.g., a mesh or point cloud) through a loss calculation or network architecture selected for that purpose). Such loss calculation methods may include the KL divergence loss, the reconstruction loss, or other losses disclosed herein. Representation learning may reduce the size of the dataset required to train the model because the representation model learns the representation, allowing the generative network to focus on learning the generation task. The result may be improved model generalization because meaningful features (e.g., local and / or global features) of the input data are made available to the generative network. In some cases, transfer learning may first train a representation-generating model. The representation-generating model (in whole or in part) can then be used to pre-train a subsequent model, such as a classification model. Representation generation models may benefit from employing mesh element features as input to improve understanding of the structure and / or shape of the 3D clinical data in the training dataset.

[0232] One or more of the neural network models of the present disclosure may have an attention gate integrated therein. Attention gate integration provides an enhancement that allows the associated neural network architecture to focus resources on one or more input values. In some implementations, an attention gate may be integrated with a U-Net architecture, which has the advantage of allowing the U-Net to focus on specific inputs. Attention gates may also be integrated with an encoder or autoencoder to improve resource efficiency, according to aspects of the present disclosure.

[0233] The mesh comparison module can compare two or more meshes, for example, to calculate a loss function or a reconstruction error. Some implementations may involve comparing the volume and / or area of ​​the two meshes. Some implementations may involve calculating the minimum distance between corresponding vertices / faces / edges / voxels of the two meshes. For a point in one mesh (e.g., a vertex, a midpoint on an edge, or a triangle center), the minimum distance between that point and a corresponding point in the other mesh is calculated. If the other mesh has a different number of elements or if there is no clear mapping between corresponding points in the two meshes, different approaches may be considered. For example, the open-source software packages CloudCompare and MeshLab each have mesh comparison tools that can function in the mesh comparison module for the present disclosure. In some implementations, the Hausdorff distance can be calculated to quantify the difference in shape between two meshes. The open-source software tool Metro, developed by Visual Computing Lab, can also serve to quantify the difference between two meshes. The following paper, "Metro: Measuring Error on Simplified Surfaces," P. Cignoni, C. Rocchini and R. Scopigno, Computer Graphics Forum, Blackwell Publishers, vol. 17(2), June 1998, pp. 167-174, describes the approach taken by Metro that can be adapted by the neural network applications of the present disclosure for use in mesh comparison and difference quantification.

[0234] Some techniques of the present disclosure may incorporate the act of casting a ray normal to the mesh surface for one or more points on a first mesh and calculating the distance before the ray enters a second mesh. The length of the resulting line segment may be used to quantify the distance between the first mesh and the second mesh. According to some techniques of the present disclosure, the distance may be assigned a color based on the magnitude of the distance, and the color may be applied to the first mesh by visualization.

[0235] An autoencoder, such as a variational autoencoder (VAE), can be trained to encode 3D clinical data in a latent space vector A, which may reside in an information-rich, low-dimensional latent space. Because the latent space vector A allows complex 3D clinical data (e.g., a 3D mesh containing thousands of mesh elements) to be efficiently manipulated, this latent space vector A may be particularly suitable for subsequent processing by the techniques disclosed herein. Such a VAE can be trained to reconstruct the latent space vector A into a replica of the input mesh. In some implementations, the latent space vector A can be strategically modified to introduce changes to the reconstructed mesh. The term mesh should be considered to include, in a non-limiting sense, 3D meshes, 3D point clouds, and 3D voxelized representations.

[0236] The 3D representation reconstruction VAE can advantageously utilize loss functions, nonlinearities (also known as neural network activation functions), and / or solvers not addressed by existing techniques. Examples of loss functions may include mean absolute error (MAE), mean squared error (MSE), L1 loss, L2 loss, KL divergence, entropy, and reconstruction loss. Such loss functions allow each generated prediction to be compared against corresponding ground truth values ​​in a quantified manner, resulting in one or more loss values ​​that can be used to at least partially train one or more of the neural networks. Examples of solvers may include dopri5, bdf, rk4, midpoint, adams, explicit_adams, and fixed_adams. The solver may enable the neural network to solve simultaneous equations and corresponding unknown variables. Examples of nonlinearities may include tanh, relu, softplus, elu, swish, square, and identity. Activation functions can be used to introduce nonlinear behavior into neural networks, allowing them to better represent the training data. Losses can be calculated through the process of training a neural network via backpropagation. Neural network layers such as: ignore, concat, concat_v2, squash, concatsquash, scale, and concatscale can be used.

[0237] The reconstruction loss can compare the predicted output to the ground truth (or reference) output. The system of the present disclosure implements the following line of pseudocode: reconstruction_loss=0.5 * L1(all_points_target,all_points_predicted)+0.5 *The reconstruction loss can be calculated as a combination of the L1 loss and the MSE loss, as shown in MSE(all_points_target, all_points_predicted). In the above example, all_points_target is a 3D representation (e.g., a 3D mesh or point cloud) corresponding to ground truth data (e.g., ground truth examples of 3D clinical data). In the above example, all_points_predicted is a 3D representation (e.g., a 3D mesh or point cloud) corresponding to generated or predicted data (e.g., generated examples of 3D clinical data). Other implementations of the reconstruction loss may additionally (or alternatively) involve an L2 loss, a mean absolute error (MAE) loss, or a Huber loss term.

[0238] The reconstruction error may compare the reconstructed output data (e.g., as generated by a reconstruction autoencoder) with the original input data (e.g., the data provided to the input of the reconstruction autoencoder). The system of the present disclosure may implement the following line of pseudocode: reconstruction_error=0.5 * L1(all_points_input,all_points_reconstructed)+0.5 * The reconstruction error may be calculated as a combination of the L1 loss and the MSE loss, as shown in MSE(all_points_input, all_points_reconstructed). In the above example, all_points_input is a 3D representation (e.g., a 3D mesh or point cloud) corresponding to the input data (e.g., 3D clinical data provided to the input of the ML model). In the above example, all_points_reconstructed is a 3D representation (e.g., a 3D mesh or point cloud) corresponding to the reconstructed (or generated) data (e.g., generated 3D clinical data).

[0239] In other words, reconstruction loss involves calculating the difference between the predicted output and the reference output, while reconstruction error involves calculating the difference between the reconstructed output and the original input from which the reconstructed data is derived.

[0240] In some implementations, a 3D representation reconstruction autoencoder (e.g., a reconstruction VAE model for reconstructing 3D clinical data) may be trained on examples of 3D clinical data. FIG. 4 illustrates a method for training such a reconstruction autoencoder, which may be used by the first ML module 304 to generate representations (i.e., latent representations 305). According to the reconstruction autoencoder training illustrated in FIG. 4, a reconstruction loss 408 may be calculated between the reconstructed output (i.e., second 3D representation 406) and the ground truth (i.e., first 3D representation 108) using a loss calculation method described herein (e.g., reconstruction loss or KL divergence loss, among others). Backpropagation may be used to train the encoder 402 and decoder 404, at least in part, using the reconstruction loss 408.

[0241] The reconstruction autoencoder of FIG. 4 , of which the reconstruction VAE model is an example, can be trained to encode clinical data 101 into a reduced-dimensional form called a latent space vector (i.e., latent representation 305). The clinical data 101 is provided to an encoder 402, encoded into the latent space vector, and then reconstructed using a decoder 404 into a replica of the input mesh (i.e., second 3D representation 406). One advantage of this process is that the encoder 402 can be trained to convert the clinical data 101 (or a mesh of an aspect of a patient's body) into a reduced-dimensional form that can be used in training and deploying the classification techniques of the present disclosure. This reduced-dimensional form of the clinical data 101 can enable the second ML module 306 (e.g., classification module) shown in FIG. 3 to more efficiently encode the reconstruction characteristics of the clinical data 101 and better learn to classify the clinical data 101, thereby providing technical improvements in terms of both data accuracy and resource footprint.

[0242] The reconstructed mesh (i.e., second 3D representation 406) may be compared to the input mesh (i.e., first 3D representation 108), for example, using a reconstruction error (as described herein) that quantifies the difference between the reconstructed mesh and the input mesh. This reconstruction error, in some implementations, may be calculated using the Euclidean distance between corresponding mesh elements between the two meshes (i.e., the reconstructed mesh and the input mesh). Other methods of calculating this error exist and may be described elsewhere in this disclosure.

[0243] In some implementations, the 3D representation provided to the reconstruction VAE may first be rearranged into a list of mesh elements (e.g., a 3D mesh may be rearranged into a list of vertices, a 3D point cloud may be rearranged into a list of points, etc.) before being provided to the encoder 402.

[0244] In some implementations, the performance of the reconstruction VAE can be measured using a reconstruction error calculation. In some examples, the reconstruction error can be calculated as the element-to-element distance between two meshes, for example, using Euclidean distance. According to various implementations of the techniques of this disclosure, other distance measures are possible, such as cosine distance, Manhattan distance, Minkowski distance, Chebyshev distance, Jaccard distance (e.g., intersection over a union of meshes), Hevershyn distance (e.g., distance over a surface), and Sørensen-Dice distance.

[0245] The performance of the reconstruction VAE may, in some implementations, be verified via reconstruction error plots and / or other key performance indicators.

[0246] An autoencoder (such as a VAE or capsule autoencoder) of the present disclosure may be trained on other types of data (e.g., 2D images, text data, categorical data, spatiotemporal data, real-time data, and / or vectors of real numbers). Such an autoencoder may be provisioned to reconstruct examples of those other types of data. The data may be qualitative or quantitative. The data may be nominal or ordinal. The data may be discrete or continuous. The data may be structured, unstructured, or semi-structured. An autoencoder of the present disclosure may convert such data into a latent representation (e.g., a latent vector or latent capsule) for classification by the second ML module 306 shown in FIG. 3.

[0247] Techniques of the present disclosure (e.g., the first ML module 304 shown in FIG. 3 ) may, in some implementations, use PointNet, PointNet++, or a derived neural network (e.g., a network trained via transfer learning using either PointNet or PointNet++ as a basis for training) to extract local or global neural network features from a 3D point cloud or other 3D representation (e.g., a 3D point cloud describing an aspect of a patient's body, such as an appendage or skin). Techniques of the present disclosure (e.g., the first ML module 304) may, in some implementations, use U-Net to extract hierarchical neural network features (e.g., local, intermediate, or global neural network features) from a 3D point cloud or other 3D representation.

[0248] It should be understood that 3D clinical data, as used herein, is intended to be used non-limitingly to encompass any representation in three or higher dimensions (e.g., 4D, 5D, etc.), and that ML models may be trained using the techniques disclosed herein to operate on higher dimensional representations.

[0249] A U-Net may comprise an encoder followed by a decoder. The architecture of a U-Net may resemble a U. The encoder may be trained to extract one or more global neural network features, zero or more intermediate-level neural network features, or one or more local neural network features (at the most local level as opposed to the most global level) from an input 3D representation. The output from each level of the encoder may be passed to a corresponding input of the decoder (e.g., by a skip connection). Similar to the encoder, the decoder may operate at multiple levels of neural network features from global to local. For example, the decoder may output a representation of the input data that may include global, intermediate, or local information about the input data. In some implementations, a U-Net can generate an information-rich (optionally dimensionally reduced) representation of the input data, which may be more easily consumed by other generative or discriminative machine learning models.

[0250] The Transformers can be trained to at least partially generate the representation of the 3D clinical data using self-attention. The Transformers can encode long-range dependencies (e.g., encode relationships between multiple inputs). The Transformers may comprise an encoder or a decoder. The encoder, in some implementations, may operate in a bidirectional manner or may operate a self-attention mechanism. The decoder, in some implementations, may operate a masked self-attention mechanism, a mutual attention mechanism, or an autoregressive manner. The self-attention operation of the Transformers described herein, in some implementations, can associate different positions or aspects of individual instances of the 3D clinical data to compute a reduced-dimensional representation of that 3D clinical data. The mutual attention operation of the Transformers described herein, in some implementations, may mix or combine aspects of two (or more) different 3D clinical data. The autoregressive operation of the Transformers described herein, in some implementations, can consume previously generated aspects of the 3D clinical data as additional inputs when generating new or modified 3D clinical data. Either the Transformer Encoder or the Transformer Decoder, in some implementations, may generate a latent form of the 3D clinical data, which may be used as an information-rich, reduced-dimensionality representation of the 3D clinical data and may be more easily consumed by other generative or discriminative machine learning models (e.g., the second ML module 306 shown in FIG. 3).

[0251] The techniques of the present disclosure may, in some instances, be trained using federated learning. Federated learning may enable multiple remote clinicians to iteratively improve a machine learning model (e.g., one or both of the first ML module 304 and the second ML module 306) while preserving data privacy (e.g., the clinical data 101 may not need to be transmitted “over the wire” to a third party). Data privacy is particularly important for clinical data 101 protected by applicable law. A clinician may receive a copy of the machine learning model, use a local machine learning program to further train the ML model using locally available data from their local clinic, and then transmit the updated ML model back to a central hub or third party. The central hub or third party may combine the updated ML models from multiple clinicians into a single updated ML model that benefits from learning recently collected patient data at various clinical sites. In this way, a new ML model can be trained that benefits from additional and updated patient data (possibly from multiple clinical sites), without the patient data actually being transmitted to a third party. Training on local in-clinic devices may, in some cases, occur when the devices are idle or otherwise during off-time (e.g., when patients are not being treated in the clinic). Devices in clinical environments for collecting data and / or training ML models for the techniques described herein may include smartphones equipped with stereo cameras, smartphones equipped with a single camera running software that enables stereo photography through the capture of images from multiple views, other handheld devices, 3D scanners, intraoral scanners, CT scanners, X-ray machines, laptop computers, servers, or desktop computers. In addition to federated learning techniques, in some implementations, contrastive learning may be used to at least partially train the ML models described herein.Contrastive learning may, in some cases, augment samples in a training dataset to highlight differences between samples from different classes and / or to increase similarity between samples of the same class, which may improve the accuracy of the classification techniques described herein.

[0252] Unless otherwise indicated, all numbers expressing feature sizes, quantities, and physical properties used in the specification and claims should be understood as modified by the term "about." Accordingly, unless otherwise indicated, the numerical parameters set forth in the foregoing specification and appended claims are approximations that may vary depending upon the desired properties sought to be obtained by those of ordinary skill in the art utilizing the teachings of the present disclosure.

[0253] While specific implementations have been illustrated and described herein, those skilled in the art will recognize that various alternative and / or equivalent implementations may be substituted for the specific implementations shown and described without departing from the scope of the present disclosure. This application is intended to cover any adaptations or variations of the specific implementations discussed herein. Accordingly, it is intended that the present disclosure be limited only by the claims and their equivalents.

Claims

1. 1. A method for clinical data analysis, comprising: receiving a first three-dimensional (3D) representation representing clinical data, the first 3D representation including one or more mesh elements; calculating one or more mesh element characteristics for the one or more mesh elements; providing the one or more mesh element features as input to a first machine learning (ML) module; executing the first ML module to encode the first 3D representation into one or more latent representations; providing the one or more latent representations to a second ML module different from the first ML module; executing the second ML module to classify the clinical data represented in the first 3D representation into at least one predicted classification label.

2. receiving at least one ground truth classification label for the first 3D representation; calculating a loss that quantifies a difference between the at least one ground truth classification label and the at least one predicted classification label; The method of claim 1 , further comprising: using the loss to train the second ML module.

3. 2. The method of claim 1, wherein the clinical data represents the patient's skin, and the second ML module is configured to classify the clinical data into at least one of a type of skin abnormality, a current state of the skin abnormality, a future state of the skin abnormality, a current state of an implant site on the skin including an implant, and a future state of the implant site on the skin.

4. 4. The method of claim 3, wherein the type of skin abnormality is at least one of a tumor, a wound, a burn, a rash, a puncture, a cyst, an infection, a skin growth, a contusion, a cut, a laceration, an abrasion, a scratch, an ulcer, and a laceration.

5. The method of claim 3 , wherein the implant is at least one of a skin graft and a device implant.

6. 2. The method of claim 1, wherein the clinical data represents an appendage, and the second ML module is configured to classify the clinical data into a current state of swelling in the appendage or a future state of swelling in the appendage.

7. 2. The method of claim 1, wherein the clinical data represents a torso, and the second ML module is configured to classify the clinical data into a current state of swelling in the torso or a future state of swelling in the torso.

8. 10. The method of claim 1, wherein the clinical data describes an article placed on the skin of a patient, and the second ML module is configured to classify the clinical data into a current state of the article.

9. The method of claim 8 , wherein the article is a wrapping on the skin of the patient, and the current state of the article includes the fit of the wrapping on the skin of the patient.

10. The method of claim 1 , further comprising executing the first ML module to reconstruct the one or more latent representations into a second 3D representation that is a replica of the first 3D representation.

11. The method of claim 10 , further comprising calculating a reconstruction error that quantifies a difference between the first 3D representation and the second 3D representation.

12. determining at least one region of the first 3D representation having the reconstruction error greater than a predetermined threshold; 12. The method of claim 11, further comprising: determining that the at least one region corresponds to at least one of a skin abnormality and an article placed on the patient's skin.

13. The method of claim 1 , wherein at least one of the one or more mesh elements has at least one associated metadata value.

14. The method of claim 13 , wherein the at least one associated metadata value includes data regarding at least one of a color of an object, a temperature of the object, and a surface impedance of the object.

15. The method of claim 1 , wherein the first 3D representation further comprises at least one of a 3D point cloud, a 3D surface, a 3D mesh, and a voxelized representation.

16. The method of claim 10 , wherein the first ML module is an autoencoder neural network.

17. 17. The method of claim 16, wherein an encoder of the autoencoder neural network is configured to encode the first 3D representation into the one or more latent representations.

18. 18. The method of claim 17, wherein a decoder of the autoencoder neural network is configured to reconstruct the one or more latent representations into the second 3D representation.

19. calculating a reconstruction loss quantifying a difference between the first 3D representation and the second 3D representation; 20. The method of claim 18, further comprising: using the reconstruction loss to train the first ML module.

20. 20. The method of claim 18, wherein the decoder of the autoencoder neural network is located remotely from the encoder.

21. The method of claim 16 , wherein the autoencoder neural network comprises a variational autoencoder (VAE) neural network.

22. 2. The method of claim 1 , wherein executing the first ML module to encode the first 3D representation into the one or more latent representations comprises executing the first ML module by a first processor to encode the first 3D representation into the one or more latent representations, and executing the second ML module to classify the clinical data represented in the first 3D representation into the at least one predicted classification label comprises executing the second ML module by a second processor different from the first processor to classify the clinical data represented in the first 3D representation into the at least one predicted classification label.

23. an interface configured to receive a first three-dimensional (3D) representation representing clinical data, the first 3D representation including one or more mesh elements; a memory communicatively coupled to the interface and configured to store the first 3D representation; a processor communicatively coupled to the interface and the memory, calculating one or more mesh element characteristics for the one or more mesh elements; providing the one or more mesh element features as input to a first machine learning (ML) module; executing the first ML module to encode the first 3D representation into one or more latent representations; providing the one or more latent representations to a second ML module different from the first ML module; a processor configured to execute the second ML module to classify the clinical data represented in the first 3D representation into at least one predicted classification label.

24. 24. The computing device of claim 23, wherein the processor is further configured to execute the first ML module to reconstruct the one or more latent representations into a second 3D representation that is a replica of the first 3D representation.

25. The computing device of claim 24 , wherein the computing device is deployed in a clinical environment.

26. 1. A method for detecting an anomaly, comprising: receiving a first three-dimensional (3D) representation representative of clinical data; providing the first 3D representation as an input to a first machine learning (ML) module; Executing the first ML module, encoding the first 3D representation into one or more latent representations; reconstructing the one or more latent representations into a second 3D representation that is a replica of the first 3D representation; calculating a reconstruction error quantifying a difference between the first 3D representation and the second 3D representation; determining at least one region of the first 3D representation having the reconstruction error greater than a predetermined threshold; determining that the at least one region corresponds to the anomaly.

27. 27. The method of claim 26, wherein the abnormality comprises at least one of a skin abnormality and an article placed on the patient's skin.

28. 27. The method of claim 26, wherein the first 3D representation includes one or more mesh elements, and wherein one or more mesh element features are calculated for at least one of the one or more mesh elements.

29. 30. The method of claim 28, wherein the one or more mesh element features are provided to the first ML module.

30. 30. The method of claim 28, wherein at least one of the one or more mesh elements has at least one associated metadata value.

31. 31. The method of claim 30, wherein the at least one associated metadata value comprises data regarding at least one of a color of an object, a temperature of the object, and a surface impedance of the object.

32. 27. The method of claim 26, wherein the first 3D representation comprises at least one of a 3D point cloud, a 3D surface, a 3D mesh, and a voxelized representation.

33. 27. The method of claim 26, wherein the first ML module is an autoencoder neural network.

34. 34. The method of claim 33, wherein an encoder of the autoencoder neural network is configured to encode the first 3D representation into the one or more latent representations.

35. 35. The method of claim 34, wherein a decoder of the autoencoder neural network is configured to reconstruct the one or more latent representations into the second 3D representation.

36. 1. A method for detecting swelling, comprising: receiving a first three-dimensional (3D) representation representing clinical data, the clinical data representing the patient's skin or an appendage of the patient; providing the first 3D representation as an input to a first machine learning (ML) module; executing the first ML module to encode the first 3D representation into one or more latent representations; providing the one or more latent representations to a second ML module different from the first ML module; and executing the second ML module to classify the clinical data into a current state of the swelling or a future state of the swelling.

37. 37. The method of claim 36, wherein the current state of swelling or the future state of swelling is used to detect lymphedema.

38. receiving the first 3D representations includes receiving at least two first 3D representations representing the clinical data acquired at different time intervals; providing the first 3D representations as inputs to the first ML module includes providing the at least two first 3D representations as inputs to the first ML module; executing the first ML module to encode the first 3D representation into the one or more latent representations includes executing the first ML module to encode the at least two first 3D representations into corresponding one or more latent representations; providing the one or more latent representations to the second ML module includes providing the corresponding one or more latent representations to the second ML module; 37. The method of claim 36, wherein executing the second ML module to classify the clinical data into the current state of the swelling or the future state of the swelling comprises comparing the corresponding one or more latent representations by the second ML module to classify the clinical data into the current state of the swelling or the future state of the swelling.

Images