Artificial intelligence techniques for generating predicted future images of wounds

Abstract

An example system includes a processor configured to: obtain image capture data representing one or more image sequences of an appearance of a wound at a corresponding image capture time, each of the images being separated by a sampling time interval between the image and the next image; passing the image capture data of the image sequences through a machine learning model trained to generate image data representing one or more predicted images of a future appearance of the wound at a corresponding future time, wherein a predicted time interval between the future time and a capture time of a last image of the sequence of images is greater than each of the sampling time intervals; and outputting image data representing the one or more predicted images of the future appearance of the wound.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of priority to U.S. Provisional Application No. 63 / 351,954, filed on June 14, 2022, which is incorporated herein by reference in its entirety. Background Art

[0002] Many real-world processes, especially those of a chemical and biological nature, evolve slowly over time. For example, a wound may take several weeks to fully heal, depending on the nature of the wound, its size, and the treatments used during the healing period. Summary of the invention

[0003] In general, this disclosure describes techniques for generating a predicted image that indicates the future appearance of a wound. More specifically, this disclosure describes example techniques for generating a predicted image of the future appearance of a wound based on applying a machine learning model to a time series of actual images of the wound. The predicted image may represent the likely appearance of the wound in the coming days or weeks. This may enable a medical practitioner to determine early on a treatment or treatment parameter to be used to treat a wound based on a reasonably accurate prediction of the likely future appearance of the wound under a particular treatment or treatment parameter.

[0004] As described herein, a prediction system may receive a time series of image captures, images of a wound prior to treatment and / or during a preliminary phase of treatment, and generate a predicted image of how the wound may appear at a future time point given a particular treatment or treatment parameters. A processing unit of a prediction device receives the image data and provides the image data to a machine learning model that has been trained to generate a predicted image of the future appearance of the wound based on the image data, the treatment method, and / or the treatment parameters. In various examples set forth herein, a prediction system may receive a time series of images from an initial period, e.g., an initial image of the wound prior to treatment and an image of the wound captured after treatment has begun, and process the time series of images to generate a predicted image of how the wound may appear after a few days or weeks of treatment.

[0005] Existing methods of wound treatment typically rely on the current state of the wound to guide treatment decisions. The technology disclosed herein may provide at least one technical advantage over existing methods. For example, a practical application of the technology disclosed herein is a prediction system that can generate a predicted image of the future appearance of a wound, which can be used to guide decisions about treatments and / or treatment parameters that will result in improved results regarding wound healing and wound appearance. As treatment progresses, a prediction system using the technology disclosed herein may receive further image captures of the wound and may generate new predicted images of the future appearance of the wound. These new predicted images can be used to determine whether the current treatment plan is optimal or whether the treatment of the wound needs to be modified or replaced with a new treatment.

[0006] In one example, the present disclosure describes a system comprising a memory; and a processing unit having one or more processors coupled to the memory, the one or more processors being configured to execute instructions, the instructions causing the processing unit to: obtain image capture data of one or more image sequences representing the appearance of a wound at a corresponding image capture time, each of the images being separated by a sampling time interval between the image and a next image before a final image of the image sequence; pass the image capture data of the image sequence through a machine learning model, the machine learning model being trained to generate image data of one or more predicted images representing a future appearance of the wound, each of the one or more predicted images representing a future appearance of the wound at a corresponding future time, the machine learning model being trained using historical image data, the historical image data comprising one or more historical image data sets, each of the one or more historical image data sets comprising image data of a historical sequence of images of the appearance of a corresponding historical wound, wherein the predicted time interval between the future time and the capture time of the last image of the image sequence is greater than each of the sampling time intervals; and output image data of the one or more predicted images representing the future appearance of the wound.

[0007] In another example, the present disclosure describes a method comprising: obtaining, by a processing unit comprising one or more processors, image capture data of one or more image sequences representing the appearance of a wound at a corresponding image capture time, each of the images being separated by a sampling time interval between the image and a next image before a final image of the image sequence; passing the image capture data of the image sequence through a machine learning model, the machine learning model being trained to generate image data of one or more predicted images representing a future appearance of the wound, each of the one or more predicted images representing a future appearance of the wound at a corresponding future time, the machine learning model being trained using historical image data, the historical image data comprising one or more historical image data sets, each of the one or more historical image data sets comprising image data of a historical sequence of images of the appearance of a corresponding historical wound, wherein the predicted time interval between the future time and the capture time of the last image of the image sequence is greater than each of the sampling time intervals; and outputting image data of the one or more predicted images representing the future appearance of the wound.

[0008] In another example, the present disclosure describes a method comprising: receiving historical image data, the historical image data comprising a plurality of historical image data sets, each of the historical image data sets comprising image data of a historical sequence of images corresponding to a wound, each image of the image historical sequence being separated by a sampling time interval between the image and a next image before a final image of the image historical sequence; for each of the plurality of historical image data sets, training the machine learning model to generate one or more predicted images of a future appearance of the wound, each image corresponding to a future time from the image historical sequence, wherein the predicted time interval between the future time and a capture time of the last image of the image historical sequence is greater than each of the sampling time intervals; and adjusting weights in a layer of the machine learning model based on differences between the one or more predicted images and one or more target images associated with the wound.

[0009] In a further example, the present disclosure describes a system comprising: a device for obtaining image capture data of one or more image sequences representing the appearance of a wound at a corresponding image capture time, each of the images being separated by a sampling time interval between the image and the next image before a final image of the image sequence; a device for passing the image capture data of the image sequence through a machine learning model, the machine learning model being trained to generate image data of one or more predicted images representing a future appearance of the wound, each of the one or more predicted images representing a future appearance of the wound at a corresponding future time, the machine learning model being trained using historical image data, the historical image data comprising one or more historical image data sets, each of the one or more historical image data sets comprising image data of a historical sequence of images of the appearance of a corresponding historical wound, wherein the predicted time interval between the future time and the capture time of the last image of the image sequence is greater than each of the sampling time intervals; and a device for outputting image data of the one or more predicted images representing the future appearance of the wound.

[0010] In yet a further example, the present disclosure describes a system comprising: a device for receiving historical image data, the historical image data comprising a plurality of historical image data sets, each of the historical image data sets comprising image data of a historical sequence of images corresponding to a wound, each image of the image historical sequence being separated by a sampling time interval between the image and a next image before a final image of the image historical sequence; a device for training the machine learning model to generate, for each of the plurality of historical image data sets, one or more predicted images of a future appearance of the wound, each image corresponding to a future time from the image historical sequence, wherein the predicted time interval between the future time and a capture time of a last image of the image historical sequence is greater than each of the sampling time intervals; and a device for adjusting weights in a layer of the machine learning model based on differences between the one or more predicted images and one or more target images associated with the wound.

[0011] The details of at least one example of the present disclosure are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the present disclosure will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figures 1A to 1C is a block diagram illustrating a system for generating a predicted image of the future appearance of a wound according to at least one example technique described in this disclosure.

[0013] Figure 2A and Figure 2B is a block diagram illustrating a training system according to at least one example technology described in this disclosure.

[0014] Figure 3 is a block diagram illustrating another training system according to at least one example technology described in this disclosure.

[0015] FIG. 4A to FIG. 4C 2 and 3 are diagrams illustrating at least one example technique described in the present disclosure. Figure 3 Block diagram of an example training framework for a training system shown in .

[0016] Figure 5 is a block diagram of an example processing unit of a system for generating a predicted image of the future appearance of a wound according to at least one example technique described in this disclosure.

[0017] Figure 6 is a flow chart illustrating example operations of a method for generating a predicted image of the future appearance of a wound in accordance with at least one example technique described in the present disclosure.

[0018] Figure 7 is a flow diagram illustrating example operation of a training system in accordance with one or more techniques of this disclosure. DETAILED DESCRIPTION

[0019] Systems and techniques are described for generating a predicted image of a future image of a wound based on a current image capture of the wound and metadata associated with the image capture. A prediction system may receive an image capture of a wound taken by a patient using the patient's own image capture device (e.g., a smart phone camera, a digital camera, etc.). The image capture may be a time series of samples over an initial sampling period. The prediction system may generate a predicted image of the future appearance that the wound will appear at a future time point (which may be days or weeks in the future). The difference between the future time point and the last sample of the time series may be referred to as a prediction interval. The sampling period may be relatively much shorter than the entire treatment period of the wound, i.e., the sampling period may be much shorter than the prediction interval.

[0020] Figure 1A 1 is a block diagram illustrating a system for generating a predicted image of the future appearance of a wound according to at least one example technology described in the present disclosure. In some aspects, system 100 includes a prediction system 102 and a client device 132. Prediction system 102 and client device 132 can be communicatively coupled to each other via network 130. Network 130 can be any type of network, including a local area network, a wide area network, or a network that is part of the Internet.

[0021] The client device 132 may be any type of computing device having the image capture device 110. In some aspects, the client device 132 may be a smartphone, such as a smartphone belonging to the patient or another person associated with the patient. In some aspects, the client device 132 may be a camera at a doctor's office, hospital, or other medical facility.

[0022] The image capture device 110 obtains one or more images 103 of the wound. Figure 1A In the example shown in , the image capture device 110 captures three images at different time points after the patient is injured. In this example, the image capture device 110 captures an image of the wound 109A of the patient on the first day of the patient's injury, an image of the wound 109B of the patient three days after the patient's injury, and an image of the wound 109C of the patient seven days after the patient's injury.

[0023] The image capture device 110 may be a camera or other component configured to capture image data representing a wound. The image capture device 110 may include a component capable of capturing image data, such as a video recorder, an infrared camera, a CCD (charge coupled device) array, or a laser scanner. Figure 1A1 , one image capture device 110 is shown, but there may be multiple image capture devices 110. Images 109 may all be captured from the same device, or they may be captured from different devices. For example, image of wound 109A may be captured by client device 132 at a medical facility treating or diagnosing a patient's wound, while images of wounds 109B and 109C may be captured by client devices 132 owned by the patient.

[0024] In some aspects, image 103 may be represented as a two-dimensional image. In some aspects, image 103 may be a three-dimensional (3D) volume of an image. For example, an image may be represented as a 3D volume of image data recorded over a relatively small period of time. As an example, a 3D volume may be a video recording. The three dimensions of the volume may be an x-dimension, a y-dimension, and a time dimension. Thus, capturing image data may refer to capturing a 2D image over a period of time or recording multiple frames of image data as a 3D volume.

[0025] The image capture device 110 may store the captured image 103 on a storage unit 107 of the client device 132. The client device 132 may transmit the captured image 103 to the prediction system 102 via the network 130. In some aspects, the client device 132 may send the captured image 103 to the prediction system 102 alone (i.e., before capturing another image of the wound 109). In some aspects, the client device 132 may store multiple images 103 of the wound on the storage unit 107 and send the multiple images together to the prediction system 102.

[0026] The prediction system 102 may receive the image 103 sent by the client device 132 and store the received image in the storage unit 105 as part of the wound image sequence 112. In some aspects, the wound image sequence may be a single image of the wound 109. In some aspects, the wound image sequence 112 may be a time sequence of images of the wound 109 (e.g., images of the wound 109A to 109C captured over a period of time). The prediction system 102 may store one or more timestamps as part of the metadata 114 associated with the wound sequence 112. The timestamp may be a timestamp obtained from the image data 103 indicating when the image was captured. The timestamp indicating when the image was received may be generated by the prediction system. The timestamp may be stored with the image, or it may be stored as metadata 114. The metadata 114 may also include data such as the type of treatment used to treat the wound (e.g., wound closure and healing treatment), products used to clean the skin and / or the wound, and products used to dress the wound. As an example, the metadata 114 may include one or more parameters for controlling a negative pressure wound therapy (NPWT) system. The NPWT system can be configured to control fluid at the wound site based on one or more input parameters that adjust wound irrigation and / or instillation. Details regarding the NPWT system can be found in U.S. Provisional Patent Application No. 63 / 201,319, entitled “Wound Therapy System,” filed on April 23, 2021, the entire contents of which are incorporated herein by reference.

[0027] The metadata 114 may also include characteristics of the wound itself, such as the size of the wound, the location of the wound, the type of tissue affected, signs and / or symptoms of infection, and exudate (discharge) associated with the wound or other data associated with the wound. The metadata 114 may also include demographic information about the patient (age, gender, race, etc.) or other data associated with the patient. For example, the metadata 114 may include data from patient records (such as weight, body mass index, etc.), personal and / or family medical history and comorbidities, past and current diagnoses (e.g., diabetes, obesity, cardiovascular disease, cholesterol, blood pressure, etc.), prescribed medications (including dosage and frequency of use), blood laboratory results and values, genetic test results, allergy and allergy test results, and any other suitable patient health record data.

[0028] The processing unit 104 of the prediction system 102 may read and process the wound image sequence 112. For example, the prediction system 102 may read and process the wound sequence 112 in response to receiving a command from a user via the user interface 111. The processing unit 104 may utilize an artificial intelligence (AI) engine 106 and a machine learning model 108 to process the image data of the wound image sequence 112 and optionally process the metadata 114 to generate predicted wound image data 116 and optionally generate predicted metadata 117. In some aspects, the AI ​​engine 106 and the machine learning model 108 may implement a neural network. For example, the machine learning model 108 may define a layer of a neural network that has been trained using the techniques described herein to receive the wound image sequence 112 as input and generate predicted wound image data 116 as output. In some aspects, the predicted wound image data 116 has the same form as the image data of the images in the wound image sequence 112. For example, if the images in the wound image sequence 112 are 2D images, the predicted wound image data 116 may represent the 2D images. Similarly, if the images in the wound image sequence 112 are 3D volumes, the predicted wound image data 116 represents the 3D volumes. In some aspects, the predicted wound image data 116 may have a different form than the image data of the images in the wound image sequence 112. For example, the images in the wound image sequence 112 may be 3D volumes. The prediction system 102 may generate the predicted wound image data 116 as 2D images.

[0029] In some aspects, the prediction system may extract image processing features (e.g., differences from an initial image over time, gradient-based images, etc.) and use these features as additional inputs to the prediction system 102 and the machine learning model 108.

[0030] In some aspects, the predicted wound image data 116 may be data for a single predicted wound image. In some aspects, the predicted wound image data 116 may be a sequence of images (e.g., a 2D image or a 3D volume). In some aspects, the sequence of images may be a temporal sequence of predicted images having a temporal order. For example, the first image of the predicted wound image data 116 may be the earliest predicted image, and the last image may be the predicted image at the furthest time point in the sequence.

[0031] exist Figure 1A In the example shown in , the wound image sequence 112 is assumed to be a plurality of image sequences. In some aspects, the wound image sequence 112 may be a single image (e.g., a single 2D image or a single 3D volume), and the machine learning model 108 may be trained to generate predicted wound image data 116 from a single input image.

[0032] In some examples, user interface 111 allows a user to control system 100. User interface 111 may include any combination of a display screen, a touch screen, buttons, audio input, or audio output. In some examples, user interface 111 is configured to power on or off any combination of elements of system 100, provide configuration information and other input to prediction system 102 and / or processing unit 104, and display output from prediction system 102.

[0033] Figure 1B is a block diagram illustrating another system for generating a predicted image of the future appearance of a wound according to at least one example technique described in this disclosure. Figure 1B In the example shown in , the prediction system 102 includes a preprocessor 136 that can process the wound image 103 before the prediction system 102 generates the predicted wound image data 116. As described above, the wound image 103 can be generated using the image capture device 110 of the client device 132. In some aspects, the client device 132 can be a smart phone or other handheld device. The wound images captured over time can be captured under different conditions. For example, images of wounds can be captured at different distances, different angles, and different lighting conditions. The image can also have a varying amount of background elements included in the image. These images are referred to as unregistered wound image sequences 142 in their pre-processed form. The preprocessor 136 can segment the wound from the image data using image segmentation techniques to exclude non-wound elements, such as background elements and / or unaffected body parts. The preprocessor 136 can then align the segmented wound images with respect to scale and angle. The segmented and aligned wound images can be referred to as a registered wound image sequence 144. In some aspects, the processing unit 132 may use the registered wound image sequence 142 as input to generate the predicted wound image data 116. Figure 1B In the example shown in , the prediction system 132 generates three predicted images 140A-140C of the future appearance of the wound, which represent the appearance of the wound one week in the future (140A), two weeks in the future (140B), and three weeks in the future (140C).

[0034] In some aspects, in addition to or in lieu of the predicted image 116, the prediction system 102 may generate predicted metadata 117. The predicted metadata 117 may include predicted future wound properties, such as predicted wound geometry (e.g., wound area, wound depth, wound location), wound healing stage, etc. Figure 1B In the example shown in , the prediction system 132 generates three prediction metadata sets associated with the wound, metadata 118A to 118C. These metadata sets represent the properties of the wound one week in the future (metadata 118A), two weeks in the future (metadata 118B), and three weeks in the future (metadata 118C).

[0035] Figure 1C is a block diagram illustrating input image data for a system for generating a predicted image of the future appearance of a wound according to at least one example technique described in the present disclosure. Figure 1C In the example shown in , the image capture device 110 has captured image data of images 112A to 112C of the example wound sequence 112 at various time points within the input time interval 120. In this example, the time interval between the image capture of image 112A and image 112B may be m days. The time interval between the image capture of image 112B and image 112C is m ± k days. The predicted time interval 122 between the capture time image 112C and the generation of predicted wound image data 116 is m + h days. The predicted time interval 122 does not represent the actual amount of time that has passed. Instead, the predicted time interval m + h represents a simulated time interval between image 112C and the predicted wound image data 116. The actual time interval between image 112C and the generation of predicted wound image data 116 may simply be the amount of time that the prediction system 102 takes to generate the predicted wound image data 116. The predicted time interval 122 may be much longer than the input time interval 120. For example, the predicted time interval 122 may be twice as long as the input time interval 120, and in some examples, may even be much longer, such as several weeks longer. However, the prediction time interval 122 is not limited in this regard, and in some cases, may be equal to or nearly equal to the input time interval.

[0036] As an example, the image capture device 110 may capture the wound 109 ( Figure 1A ) to create a wound image sequence 112. The prediction system 102 can process the wound image sequence 112 to generate predicted wound image data 116, which represents a predicted image of the future appearance of the wound at a future time point (e.g., the next two weeks). The predicted wound image data 116 can be used to determine whether the current treatment plan for the wound is acceptable to the physician and / or patient, or whether a different or modified treatment plan should be considered. Using the techniques described herein, a user (or user system) can use the predicted images of the future appearance of the wound to draw conclusions about the treatment of the wound earlier than is possible using currently existing methods. In the above example, the user can draw conclusions about the treatment of the wound days or weeks earlier than current methods.

[0037] exist Figure 1CThe example shown in shows several aspects of the wound image sequence 112 and predicted wound image data 116. The first aspect is that there may be long sampling intervals. Each input image may be several days apart from each other. Within these intervals, there may be many changes controlled by the underlying nonlinear processes involved in wound healing. Therefore, it is challenging to generate accurate future images or 3D volumes of the wound.

[0038] The second aspect is that the time intervals between image captures can be inconsistent and uneven. Figure 1C As shown in FIG, the first two samples may be m days apart, while the time between the next two samples may be greater or less than m days (i.e., m ± k days). The expected intervals may also be relatively large due to missing or corrupted data. Therefore, k may even be higher when wound image capture is lost or corrupted.

[0039] A third aspect is that the prediction time interval (e.g., m+h) associated with the predicted wound image data 116 may be very long compared to the interval between image captures of the wound. For example, the prediction time interval 122 between the last captured image (e.g., image 112C) of the wound image sequence 112 and the predicted wound image data 116 may be several days to several weeks apart.

[0040] A user, such as a clinician or medical practitioner, may operate the prediction system 102 to determine the effects that different treatments or different treatment parameters (e.g., NPWT parameters) will likely have on wound healing and the predicted future appearance of the wound. For example, one or more current image sets of a wound may be provided to the prediction system 102 along with metadata describing the wound. The user may provide further metadata, such as data describing a treatment method or parameters of a wound treatment system, as input to the prediction system 102. The prediction system 102 may generate a predicted future image of the wound based on the current image of the wound and the input metadata. The user may change different input parameters, such as proposed treatment and / or treatment system parameters, and may select a treatment and / or treatment parameters for application to the wound that produces a desired result with respect to the predicted image of the future appearance of the wound.

[0041] Figure 2A is shown for training a system (such as the following reference Figure 2B2. Block diagram of training data for a training system discussed in the context of FIG. 21. In some aspects, the training data includes multiple historical wound image sequences. The historical wound image sequence 212 includes images 232A to 232N, which include image data of a corresponding wound image sequence captured at different time points. Images 232A to 232M of the historical wound image sequence may be images captured during a sampling period 210. The sampling period 210 may include images captured before the completion of wound treatment, which may include images captured a few days or weeks before the expected completion of treatment. As an example, when a patient first visits a practitioner to seek treatment for his or her wound, image 232A may be captured before treatment begins. Generally speaking, images 232B to 232M may be captured at any point before treatment is completed, such as during the early stages of wound treatment. Images 232M+1 to 232N may be images captured during the later stages of the treatment cycle 208 of the wound. The final image of the sequence (image 232N) may be the target image of the sequence. That is, final image 232N may be used as ground truth regarding the appearance of a corresponding wound undergoing a given treatment at a desired point in the treatment of the wound (eg, at a point in time at the end of treatment or after treatment has been completed).

[0042] Figure 2B 2 is a block diagram illustrating a training system according to at least one example technology described in the present disclosure. The training system 202 may include a machine learning framework 204, which includes a machine learning engine 206. The machine learning framework 204 may receive training data 203 and process the training data to generate a machine learning model 224. In some aspects, the machine learning framework 204 includes a machine learning engine 206, which may train the machine learning model 224 using supervised or unsupervised machine learning techniques. In some aspects, the machine learning engine 206 may be a deep learning engine that implements a convolutional neural network (CNN). In some aspects, the machine learning engine 206 may be, for example, a generative adversarial network (GAN). As an example, the machine learning engine may be a T-adversarial GAN. In some aspects, the machine learning engine 206 may be a U-Net-based machine learning engine, including U-Net 2D and U-Net 3D architectures. The U-Net architecture may be used to preserve content such as spatial information in the training data. The U-Net architecture typically has contraction and expansion paths, and in combination with skip connections in the layers, may be used to link corresponding feature maps on the encoder and decoder. The linking of feature maps may facilitate the reuse of features in the encoder, thereby reducing information loss. Additionally, the U-Net architecture can be computationally efficient and can be trained with relatively small datasets.

[0043] In some aspects, the machine learning framework 204 can implement multiple machine learning techniques that can be applied together when training the machine learning model 224. For example, the machine learning engine 206 can be a U-Net engine, and the machine learning framework can use the machine learning engine 206 to apply a recurrent learning technique. FIG. 4A to FIG. 4C Provides further details on machine learning frameworks and recurrent learning.

[0044] The training data 203 may include historical wound image sequences 212A to 212N (collectively referred to as historical wound image sequences 212). Each historical wound image sequence 212 in the training data is a time series of images of a particular wound captured or recorded during a period of time prior to training the machine learning model 224. For example, the historical wound image sequence 212A may be image data of an image sequence showing the appearance of a first wound over time, the historical wound image sequence 212B may be image data of an image sequence showing the appearance of a second wound over time, the historical wound image sequence 212C may be image data of an image sequence showing the appearance of a third wound over time, and so on.

[0045] Each historical wound image sequence 212A to 212N in the training data 203 may have a corresponding target image 220A to 220N. The target image of an image sequence is a "ground truth" final image, eg, an actual image of the wound associated with the image sequence captured at the end of a treatment cycle.

[0046] The training data 203 may also include metadata 214 that may be used to train the machine learning model 224. The metadata 214 may include a timestamp indicating when an image in the historical wound image sequence 212 was captured. The metadata 214 may include patient demographic information, such as data from patient records (such as weight, body mass index, etc.), personal and / or family medical history and comorbidities, past and current diagnoses (e.g., diabetes, obesity, cardiovascular disease, cholesterol, blood pressure, etc.), prescribed medications (including dosages and frequency of use), blood laboratory results and values, genetic test results, allergy and allergy screening results, and any other suitable patient health record data. The metadata 214 may include wound information, such as wound geometry (e.g., wound location, wound depth, wound size, etc.), affected tissue type, signs or symptoms of infection, and / or healing stage. The metadata 214 may also include data such as the type of treatment used to treat the wound (e.g., wound closure and healing treatments), products used to clean the skin and / or wound, and products used to dress the wound. As an example, the metadata 214 may include one or more parameter values ​​of an NPWT system used to treat the wound. The metadata 214 may also include characteristics of the wound itself, such as the size of the wound, the location of the wound, the type of tissue affected, signs and / or symptoms of infection, and exudate (secretions) associated with the wound when the historical image was captured. The metadata 214 may also include demographic information about the patient (age, gender, etc.) or other data associated with the patient or wound when the historical image was captured. In some aspects, the metadata 214 may be added to the training system 202 (or the prediction system 102 of FIG. 1 ) by padding the metadata information on the border of the image. In some aspects, the metadata 214 may be added as a vector to the potential feature vector generated in the intermediate layer of the machine learning model 224.

[0047] The training system 202 provides the training data 203 to the machine learning framework 204 for processing by the machine learning engine 206. The machine learning engine 206 processes the historical wound image sequence 212 to generate predicted image data 218. The predicted image data 218 may include a predicted image sequence of the future appearance of the wound, each predicted image having an associated future time. The machine learning framework 204 may compare the predicted image with a target image 220 associated with the historical wound image sequence 212 to determine the difference between the predicted image and the target image 220. The difference between the predicted image and the target image 220 is used to update the training weights in the machine learning model 224 in an attempt to improve the model's ability to generate accurate predicted images of the future appearance of the wound. In some aspects, a loss function (such as a reconstruction loss or a GAN loss) may be used to adjust the weights in the machine learning model 224.

[0048] In some aspects, the machine learning framework 204 may also train the machine learning model 224 to generate the predicted metadata 217 using historical metadata information (e.g., metadata 214) associated with the historical wound image sequence 212. The metadata 214 may be the historical sequence and target metadata corresponding to metadata of the historical wound image sequence 212 and the target image 220. The machine learning framework may compare the predicted metadata 217 to the target metadata and adjust the machine learning model based on the difference between the predicted metadata 217 and the target metadata.

[0049] After the training system 202 trains the machine learning model 224, the model can be deployed to the prediction system 216. The prediction system 216 can be a specific implementation of the prediction system 102 of Figure 1. The prediction system 216 can receive the wound image sequence 112 and use the AI ​​engine 222 and the deployed machine learning model 224 to process the image sequence to generate predicted wound image data 116 and / or predicted metadata 117.

[0050] As in Figure 2B As shown in FIG. 1 , the machine learning framework 204 may generate predicted image data 218 that may include a sequence of images (e.g., 2D images or 3D volumes). In some aspects, the machine learning framework 204 may generate predicted image data 218 that may be a single 2D image or a 3D volume. Additionally, the historical wound image sequence 212 may be as shown in FIG. Figure 2B In some aspects, the wound image sequence 212 may be a single image (eg, a single 2D image or a single 3D volume), and the machine learning framework 204 may train the machine learning model 224 to generate the predicted image data 218 from the single input image.

[0051] In some aspects, the machine learning engine 206 may implement a weighted loss function that assigns different weights to images in the predicted image data 218. For example, the weighted loss function may assign a greater weight to images later in a sequence of images than to images earlier in the sequence. In other words, a first predicted future image having an associated predicted future time earlier than a predicted future time associated with a second predicted future image will have a weight less than that of the second predicted future image. This can be beneficial because a predicted future image that is accurate and later in time in a predicted image sequence is more valuable to an end user than another predicted image predicted at an earlier future time in the sequence. In some examples, these weights may also be learned from data. For example, a machine learning model may automatically learn the relevance and importance of each image data in the input.

[0052] Figure 3 is a block diagram illustrating further aspects of a training system according to at least one example technology described in this disclosure. Figure 3In the example shown in , the training system 300 includes a loading and formatting unit 302, a data segmentation unit 304, a spatial enhancement unit 306, a temporal enhancement unit 308, a sampling unit 310, a batch processing unit 312, a preprocessing unit 313, a machine learning framework 314, a testing unit 320, and a result visualization unit 322. The loading and formatting unit 302, the data segmentation unit 304, the spatial enhancement unit 306, the temporal enhancement unit 308, the sampling unit 310, the batch processing unit 312, the preprocessing unit 315, the machine learning framework 314, the testing unit 320, and the result visualization unit 322 can be implemented as a configurable pipeline to process the candidate image dataset 301 into multiple batches of image datasets for use by the machine learning framework 314 in training the machine learning model 319.

[0053] The loading and formatting unit 302 can process the candidate image dataset 301 to format the image sequences in the candidate image dataset 301 into a form that can be processed by the training system. For example, the images can be scaled, resized, cropped, etc. so that they have a format compatible with the machine learning framework 314.

[0054] The data segmentation unit 304 may divide the candidate image dataset 301 into training data, test data, and / or validation data. For example, the input parameter may specify the percentage of the dataset used as training data, test data, and / or validation data.

[0055] The spatial enhancement unit 306 can increase the amount of training data by transforming an existing image into one or more additional training images. For example, an image can be transformed by taking a portion of an image and moving the portion left, right, rotating the image, mirroring the image, etc. along a diagonal axis to create a new image that can be included in the training data.

[0056] The time enhancement unit 308 may control the selection of images from the candidate image dataset 301 based on the temporal aspects of the candidate training data. The time enhancement unit 308 may select an image sequence based on where the images are located on the time axis. As an example, the time enhancement unit 308 may select images based on a start time and an end time.

[0057] The sampling unit 310 may select images from the training data based on a skip factor 311. For example, the sampling unit 310 may select a subset of images in the candidate data set instead of including each image in the candidate image data set 301. The skip factor 311 may be used to control the manner in which images are selected. For example, a skip factor of four may cause the sampling unit 310 to skip four images of the candidate data set before selecting the next image for inclusion in the training data.

[0058] Configuration data 324 may include data that determines data sources, hyperparameters, machine learning parameters, machine learning types, etc. for use by machine learning framework 314.

[0059] The batch processing unit 312 creates and controls batches of training data to be processed as a unit. For example, a first batch of training data may be used to train a first machine learning model 319, while a second batch of training data may be used to train a second machine learning model 319. The batch processing unit 312 may use configuration data 324 to determine which data sources to use for a batch of training data. The batch processing unit 312 may also use configuration data 324 to specify configuration parameters to be used by the machine learning framework 314 when training the machine learning model 319 using a corresponding batch of training data.

[0060] The batch processing unit 312 may provide a batch of training data to the machine learning framework 314 for training the machine learning model 319. In some aspects, the batch processing unit 312 may provide the training data to the preprocessing unit 313. The preprocessing unit 313 may apply an image segmentation technique to each wound image in the training image set to segment the wound from the other image data to exclude non-wound elements, such as background elements and / or unaffected body parts. The preprocessor 136 may then align the segmented wound images with respect to scale and angle. The segmented and aligned wound images may be referred to as an aligned wound image sequence 315.

[0061] The machine learning framework 314 may include a machine learning engine 316. In some aspects, the machine learning framework 314 and / or the machine learning engine 316 may be Figure 2B The machine learning framework 204 and / or machine learning engine 206 of the present invention may be a specific implementation of the machine learning framework 204 and / or machine learning engine 206. As discussed above, the machine learning engine 316 may be a deep learning engine that implements CNN, GAN, U-Net based machine learning engine (including U-Net2D and U-Net 3D architectures). The machine learning framework 314 may train the machine learning model 319 using the techniques described herein to generate a predicted image of the future appearance of the wound.

[0062] The testing unit 320 can test the machine learning model 319 to determine the accuracy of the predicted future wound images generated using the machine learning model 319. As described above, the candidate image dataset 301 can be segmented into training data and test data. The machine learning model 319 is trained using the training data. The testing unit 320 can receive a sequence of historical wound images in the test data as input, and can use the machine learning model 319 to generate a predicted wound image as output.

[0063] As an example, the test data may include a historical sequence of images of a wound, wherein a first portion of the images in the sequence are captured during a sampling period 210, and other images in the sequence are captured after the sampling period 210. The last image in the sequence may be a target wound image. The test unit 320 may apply the machine learning model 319 to the first portion of the image history sequence to generate a predicted wound image. The test unit 320 may then compare the predicted wound image with the target wound image, and determine the accuracy of the predicted wound image based on the comparison. The test unit 320 may determine various measures of the performance of the machine learning model 319, and compare these measures with other machine learning models that may have been generated using different training parameters and / or training data. The comparison results may be used to determine a machine learning model 319 that produces a better (e.g., more accurate) predicted wound image.

[0064] The result visualization unit 322 can provide feedback to the user regarding the training of the machine learning model 319. For example, the result visualization unit 322 can output statistics regarding the accuracy of the predicted images generated by the machine learning model 319. In some aspects, the result visualization unit 322 can output examples of a sequence of input wound images and predicted wound images generated by the machine learning model 319. The user can use the output of the result visualization unit 322 to determine whether any adjustments need to be made regarding training the machine learning model 319. For example, the user can adjust hyperparameters, prediction time intervals, or other configuration data 324 and signal the batch processing unit 312 to begin providing another batch of training data to train the new machine learning model 319. The result visualization unit 322 can provide an output that can be used to compare the performance of the machine learning model 319 with other machine learning models.

[0065] Training system 300 need not include Figure 3 All components shown in FIG. 1 and in various implementations, the training system 300 may include Figure 3 Various combinations of one or more of the components shown in FIG.

[0066] FIG. 4A to FIG. 4C 2 and 3 are diagrams showing at least one example technique described in the present disclosure. Figure 3 A block diagram of an example bidirectional machine learning framework used in the training system shown in FIG. FIG. 4A to FIG. 4C In the example shown in , a machine learning model will be trained to generate a predicted image of the future appearance of a wound using a sequence 406 of input wound images. FIG. 4A to FIG. 4C In the example shown in FIG. 4 , the image sequence 406 is k images IMG1-IMG k A sequence of images, where IMG1 is the first image in the sequence and IMG kis the last image in the sequence. The input image sequence 406 may be images of the wound captured during the sampling period. out is a predicted future image generated by the machine learning framework using image data selected from image 406. IMG label An image captured at the end of treatment or after a wound has been treated. label The images may be captured at a time point a few days or even weeks after the input image 406 was captured. label represents the “ground truth” image, also known as the target image.

[0067] Figure 4A is a block diagram illustrating a machine learning framework 402 that performs two passes over a sequence of images to train a machine learning model to predict future images of the appearance of a wound. The machine learning framework 402 may be Figure 2B Machine learning framework 204 and / or Figure 3 A specific implementation of the machine learning framework 314. The machine learning framework 402 includes two deep learning architectures 404A and 404B. The deep learning architecture 404A is used to train the machine learning model 405 to generate predicted future images from a sequence of past images, and the deep learning architecture 404B is used to train the machine learning model 405 to reconstruct past images from later images and predicted future images. The deep learning architectures 404A and 404B can each be a CNN (including U-Net 2D and U-Net3D), a GAN, a T-adversarial GAN, a time-cycle GAN, or a GAN using privileged information. In some aspects, the deep learning architectures 404A and 404B share layers in the machine learning model 405. The shared layers provide constraints on learning, where past information is linked to future information so that predicted future images can be used to reconstruct past images. Figure 4A In the example shown in FIG. 4 , the goal of the first pass is to use the deep learning architecture 404A to generate an image IMG that is consistent with the ground truth image. label The same or similar predicted future image IMG out . IMG out With IMG label Compare and use IMG out With IMG label to update the trained weights in the deep learning architecture 404A in an attempt to improve the generated predicted future image IMG out In some aspects, a loss function (such as a reconstruction loss or a GAN loss) can be used to adjust the weights in the machine learning model 405. The above process is generally the same as the process for unidirectional training for some specific implementations.

[0068] In the second pass, the side goal is to use the deep learning architecture 404B to use the subsequent images IMG2 to IMG k and predict future images IMG out A reconstructed first image IMG1' in the sequence is generated as input to the deep learning architecture 404B, which is the same as or similar to the actual first image IMG1 in the sequence. IMG1' is compared to IMG1 (which is now considered the target image), and the difference is used to adjust the weights in the layers of the machine learning model 405. This second pass can make the layer weights more robust and can avoid overfitting the machine learning model to the training data.

[0069] Figure 4B 4 is a block diagram illustrating a machine learning framework 410 that performs two stages of a bidirectional traversal to train a machine learning model to predict future images of the appearance of a wound. The machine learning framework 410 includes deep learning architectures 411A, 411B, and 411C (collectively referred to as "deep learning architectures 411"). The deep learning architectures 411 can each be a CNN (including U-Net 2D and U-Net 3D), a GAN, a T-adversarial GAN, a time-recurring GAN, or a GAN using privileged information.

[0070] In some aspects, the deep learning architecture 411A is similar to the above reference Figure 4A That is, the deep learning architecture 411A may be bidirectional and may perform two passes over the image sequence, i.e., generating predicted future images IMG from the initial images in the image sequence 406. out The first traversal, and based on the predicted future image IMG out and an image after the first image in the image sequence to generate a second pass of the reconstructed first image IMG1. In some aspects, the deep learning architecture 411A is similar to Figure 4A The deep learning architectures 404A and 404B differ in that a longer time range of image data is included when training the machine learning model. For example, in addition to input images 406 collected before wound treatment and images of the initial stages of treatment, the deep learning architecture 411A may also include images 408 collected during the later stages of treatment of the wound.

[0071] exist Figure 4B In the example shown in FIG. 4 , the first stage of generating the machine learning model 415 includes using the machine learning architecture 411A to train the machine learning model 415 to generate images that are consistent with the ground truth image IMG label The same or similar predicted future image IMG out . Machine Learning Framework 410 IMG out With IMG labelCompare and use IMG out With IMG label to update the trained weights in the machine learning model 415 in an attempt to improve the generated predicted future images IMG out In some aspects, a loss function (such as a reconstruction loss or a GAN loss) can be used to adjust the weights in the machine learning model 415. In addition, the deep learning architecture 411A trains the machine learning model 415 to obtain the IMG out and IMG1 after the first image IMG1 is reconstructed. As described above, Figure 4B In the example shown in FIG. 4 , the input to the deep learning architecture 411A during the first stage of machine learning may include two images captured before treatment (e.g., IMG1 to IMG k ) and images captured after treatment has begun but before treatment is completed (e.g., image IMG k+n 、IMG k+n+1 、IMG k+n+2 Thus, in the first phase of training, the machine learning framework 410 utilizes data from a longer time frame to improve the accuracy of the machine learning model 415.

[0072] While it may be advantageous to use a longer time frame to improve the accuracy of predicted future images, an aspect of the technology disclosed herein is a machine learning model that can use images captured prior to treatment of a wound and images captured during the early stages of treatment of a wound to generate predicted future images without relying on images captured during the later stages of treatment of a wound. Figure 4B In the example shown in , during the second stage of machine learning, the machine learning framework 410 continues to train the machine learning model 415' using the images 406 captured before treatment of the wound and during the early stages of treatment. As with the first stage, the second stage of training can be bidirectional, with the deep learning architecture 411B sharing layers of the machine learning model 415' with the deep learning architecture 411C. For example, the deep learning architecture 411B trains the machine learning model 415' to use the input images 406 (e.g., IMG1 to IMG2). k ) to generate the predicted future image IMG out , and IMG out With IMG label The comparison is performed to determine the adjustment of the weights of the machine learning model 415'. In addition, the deep learning framework 411C trains the machine learning model 415' to use IMG out and IMG kIMG2 is used as input to generate a reconstructed image IMG1', and IMG1' is compared with IMG1 to determine adjustments to the weights of the machine learning model 415'.

[0073] The machine learning framework 410 can impose constraints 412 on the training of the machine learning model 415'. For example, the machine learning framework 410 can implement a constraint that certain layers of the machine learning model 415' match the weights of corresponding layers of the machine learning model 415. In some aspects, the constraint can be that the weights of the last layer of the machine learning model 415' match the weights of the last layer of the machine learning model 415. In some aspects, the constraint can be that the weights of the intermediate layers of the machine learning model 415' match the weights of the corresponding intermediate layers of the machine learning model 415.

[0074] In addition to the aspects discussed above, Figure 4B A further aspect of the present disclosure illustrated in is that additional data obtained in the future may be used to train the machine learning model 415'. For example, an initial training data set may be used to train the machine learning model 415. As further data becomes available at future points in time, the machine learning model 415' may be trained as discussed above to potentially improve the accuracy of the predicted image.

[0075] exist Figure 4B In the example shown in , the machine learning framework 410 implements bidirectional training (i.e., loop training) to train both the machine learning model 415 and the machine learning model 415'. However, bidirectional training is not required, and in some aspects, the machine learning framework 410 can train either or both of the machine learning models 415 and 415' using a single direction.

[0076] Figure 4C 4 is a block diagram illustrating another machine learning framework 420 that performs two stages of a bidirectional traversal to train a machine learning model to predict the future appearance of a wound. The machine learning framework 420 includes deep learning architectures 422A, 422B, and 422C (collectively referred to as "deep learning architectures 422"). The deep learning architectures 422 can each be a CNN (including U-Net 2D and U-Net 3D), a GAN, a T-adversarial GAN, a time-recurring GAN, or a GAN using privileged information.

[0077] In some aspects, the deep learning architecture 425A is similar to that described above in Figure 4A The deep learning architectures 404A and 404B described in Figure 4B That is, the deep learning architecture 422A can be bidirectional and can perform two passes on the image sequence, for example, generating a predicted future image IMG from an initial image in the image sequence 406. outThe first traversal, and based on the predicted future image IMG out and the image after the first image in the image sequence to generate a second pass of the reconstructed first image IMG1. Similar to the deep learning architecture 411A, the deep learning architecture 422A includes Figure 4A However, in Figure 4C In the example of , the additional training data obtained may include a greater number of images from images captured prior to treatment of the wound. Figure 4C In the example shown in , the deep learning architecture 422A initially uses images IMG1 to IMG6 to train the machine learning model 425. Therefore, in the first stage of training, the deep learning architecture 422A utilizes more image samples to improve the accuracy of the machine learning model 425.

[0078] During the second phase, the deep learning architecture 422B uses fewer images from the images 406 to train the machine learning model 425'. Figure 4C In the example shown in , deep learning architectures 422B and 422C use half the number of images (e.g., IMG1, IMG3, and IMG5). As with the first phase, the second phase of training can be bidirectional, where deep learning architecture 422B shares layers of machine learning model 425' with deep learning architecture 422C. For example, deep learning architecture 422B trains machine learning model 425' to use input images 406 (e.g., IMG1, IMG3, and IMG5) to generate predicted future images IMG out , and IMG out With IMG label The comparison is performed to determine the adjustment of the weights of the machine learning model 425'. In addition, the deep learning framework 422C trains the machine learning model 425' to use IMG out , IMG5 and IMG3 are used as input to generate a reconstructed image IMG1', and IMG1' is compared with IMG1 to determine adjustments to the weights of the machine learning model 425'.

[0079] The machine learning framework 420 can impose constraints 424 on the training of the machine learning model 425'. For example, the machine learning framework 420 can implement a constraint that certain layers of the machine learning model 425' match the weights of corresponding layers of the machine learning model 425. In some aspects, the constraint can be that the weights of the last layer of the machine learning model 425' match the weights of the last layer of the machine learning model 425. In some aspects, the constraint can be that the weights of the intermediate layers of the machine learning model 425' match the weights of the corresponding intermediate layers of the machine learning model 425.

[0080] In addition to the aspects discussed above, Figure 4C A further aspect of the present disclosure illustrated in is that despite utilizing fewer samples during an initial phase of training (eg, a testing phase), the machine learning framework 420 can still produce a machine learning model 425' that can have the same or similar accuracy as a machine learning model trained using more samples.

[0081] with Figure 4B Similar to the example shown in Figure 4C In the example shown in FIG. 4 , the machine learning framework 420 implements bidirectional training (i.e., loop training) to train both the machine learning model 425 and the machine learning model 425′. However, bidirectional training is not required, and in some aspects, the machine learning framework 410 can train either or both of the machine learning models 415 and 415′ using a single direction.

[0082] Figure 5 is a block diagram of an example processing unit of a system for generating a predicted future image of the appearance of a wound according to at least one example technique described in the present disclosure. Figure 5 is a block diagram illustrating an example processing unit 500 according to at least one example technique described in this disclosure. The processing unit 500 may be Figure 1A and Figure 1B An example or alternative implementation of processing unit 104. Figure 5 The architecture of processing unit 500 shown in FIG. 5 is shown for example purposes only. Processing unit 500 should not be limited to the example architecture shown. In other examples, processing unit 500 can be configured in various ways. Figure 5 In the example shown in , the processing unit 500 includes a prediction unit 510 configured to generate a predicted wound image based on an input wound image sequence. The prediction unit 510 may include an AI engine 512 configured to process the wound image sequence using a machine learning model 514 to generate a predicted wound image as an output.

[0083] In some aspects, the machine learning model 514 may include data defining a CNN. In some aspects, the machine learning model 514 may include data defining a generative adversarial network (GAN), a T-adversarial GAN, a U-Net (including a U-Net 2D and a U-Net 3D).

[0084] Processing unit 500 may be implemented as any suitable computing system (e.g., at least one server computer, workstation, mainframe, appliance, cloud computing system, and / or other computing system) capable of performing operations and / or functions described according to at least one aspect of the present disclosure. In some examples, processing unit 500 represents a cloud computing system, server farm, and / or server cluster (or portion thereof) configured to be connected to system 100 via a wired or wireless connection. In other examples, processing unit 500 may represent or be implemented by at least one virtualized computing instance (e.g., a virtual machine or container) of a data center, cloud computing system, server farm, and / or server cluster. In some examples, processing unit 500 includes at least one computing device, each computing device having a memory and at least one processor.

[0085] As in Figure 5 As shown in the example of , the processing unit 500 includes a processing circuit 502, at least one interface 504, and at least one storage unit 506. The prediction unit 510 (including the AI ​​engine 512) can be implemented as program instructions and / or data stored in the storage unit 506 and executable by the processing circuit 502. The storage unit 506 can store a machine learning model 514. The storage unit 506 of the processing unit 500 can also store an operating system (not shown) that can be executed by the processing circuit 502 to control the operation of the components of the processing unit 500. The components, units, or modules of the processing unit 500 can be coupled (physically, communicatively, and / or operationally) using a communication channel for communication between the components. In some examples, the communication channel includes a system bus, a network connection, an inter-process communication data structure, or any other method for transmitting data.

[0086] In one example, the processing circuit 502 may include at least one processor configured to implement functions and / or process instructions for execution within the processing unit 500. For example, the processing circuit 502 may be capable of processing instructions stored by the storage unit 506. The processing circuit 502 may include, for example, a microprocessor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or equivalent discrete or integrated logic circuitry, or a combination of any of the foregoing devices or circuitry.

[0087] There may be multiple instances of processing circuit 502 within processing unit 500 to facilitate parallel processing of inspection operations. The multiple instances may be of the same type, such as a multi-processor system or a multi-core processor. The multiple instances may be of different types, such as a multi-core processor with associated multiple graphics processor units (GPUs).

[0088] Processing unit 500 can communicate with external systems via at least one network using interface 504. In some examples, interface 504 includes an electrical interface configured to electrically couple processing unit 500 to prediction system 102. In other examples, interface 504 can be a network interface (e.g., an Ethernet interface, an optical transceiver, a radio frequency (RF) transceiver, Wi-Fi, or via wireless technology under the "Bluetooth" trademark), a telephone interface, or any other type of device that can send and receive information. In some examples, processing unit 500 communicates wirelessly with external systems using interface 504.

[0089] The storage unit 506 may be configured to store information within the processing unit 500 during operation. The storage unit 506 may include a computer-readable storage medium or a computer-readable storage device. In some examples, the storage unit 506 includes at least a short-term memory or a long-term memory. The storage unit 506 may include, for example, a random access memory (RAM), a dynamic random access memory (DRAM), a static random access memory (SRAM), a magnetic disk, an optical disk, a flash memory, a magnetic disk, an optical disk, a flash memory, or an electrically programmable memory (EPROM) or an electrically erasable programmable memory (EEPROM). In some examples, the storage unit 506 is used to store program instructions executed by the processing circuit 502. The storage unit 506 may be used by software or applications running on the processing unit 500 to temporarily store information during program execution.

[0090] Figure 6 is a flow chart illustrating an example operation of a prediction system according to one or more techniques of the present disclosure. The prediction system may receive image data of a sequence of images representing the appearance of a wound by a processing unit including one or more processors, the images being captured at a plurality of times, each of the images being separated by a sampling time interval between the image and the next image before a final image of the sequence of images (605). Next, the prediction system may pass the image data of the sequence of images through a machine learning model, the machine learning model being trained to generate image data representing a predicted image of a future appearance of the wound at a future time, the machine learning model being trained using historical image data, the historical image data comprising a plurality of historical image data sets, each of the historical image data sets comprising image data of a historical sequence of images of a corresponding historical wound, wherein a predicted time interval between the future time and the capture time of the final image of the sequence of images is greater than each of the sampling time intervals in the sampling time intervals (610). Next, the prediction system may output image data representing the predicted image of the future appearance of the wound (615).

[0091] Figure 7is a flow chart illustrating an example operation of a training system according to one or more techniques of the present disclosure. The training system may receive historical image data, the historical image data comprising a plurality of historical image data sets, each of the historical image data sets comprising image data of a historical sequence of images of a corresponding wound, each image in the historical sequence of images being separated by a sampling time interval between an image and a next image before a final image of the historical sequence of images (705). In some aspects, the sampling time interval may be variable. That is, the sampling time interval between images in the historical sequence need not be uniform across all images, and for some images, the sampling interval may be different from the sampling interval of other images. Next, the training system may train a machine learning model to generate a predicted image of a future appearance of the wound at a future time from the historical sequence of images, wherein the predicted time interval between the future time and the capture time of the last image of the historical sequence of images is greater than each of the sampling time intervals in the sampling time intervals (710). Next, the training system may adjust weights in a layer of the machine learning model based on a difference between the predicted image and a target image associated with the wound (715).

[0092] The above discussion has been presented in the context of predicting future images of a wound based on images taken prior to treatment and / or during the early stages of treatment of the wound. However, the techniques discussed herein may be applied to predict other properties of a wound and wound treatment. For example, a machine learning model may be trained to predict wound area, wound depth, and / or healing stage based on input image data, input metadata, or a combination of both.

[0093] In addition, the technology discussed in this article can also be easily applied to other fields. For example, the technology can be applied to images of microbial growth to generate predicted future images of microbial colonies based on image sequences of microbial colonies.

[0094] The technology disclosed herein can also be applied to agriculture. Plant growth behaviors, such as bacterial colony growth and wound healing, can have slow and long progressions. Using the technology described herein, image sequences of fields can be used to predict new varieties and fields that are most resistant to disease.

[0095] The techniques described in the present disclosure may be implemented, at least in part, in hardware, software, firmware, or any combination thereof. For example, various aspects of the techniques may be implemented in at least one processor, including at least one microprocessor, DSP, ASIC, FPGA, and / or any other equivalent integrated or discrete logic circuits, and any combination of these components. The term "processor" or "processing circuit" may generally refer to any of the aforementioned logic circuits, alone or in combination with other logic circuits, or any other equivalent circuits. A control unit including hardware may also perform at least one of the techniques of the present disclosure.

[0096] These hardware, software and firmware may be implemented in the same device or in separate devices to support the various operations and functions described in this disclosure. In addition, any of the units, modules or components may be implemented together or separately as discrete but cooperable logic devices. Depicting different features as modules or units is intended to emphasize different functional aspects and does not necessarily imply that such modules or units must be implemented by separate hardware or software components. On the contrary, the functions associated with at least one module and / or unit may be performed by separate hardware or software components, or integrated in common or separate hardware or software components.

[0097] The techniques described in this disclosure may also be embodied or encoded in a computer-readable medium (such as a non-transitory computer-readable medium or a computer-readable storage medium) containing instructions. The instructions embedded or encoded in the computer-readable medium may cause a programmable processor or other processor to perform the method (e.g., when the instructions are executed). The computer-readable storage medium may include RAM, read-only memory (ROM), programmable read-only memory (PROM), EPROM, EEPROM, flash memory, hard disk, CD-ROM, floppy disk, cassette, magnetic media, optical media, or other computer-readable storage media. The term "computer-readable storage medium" refers to a physical storage medium, rather than a signal or carrier wave, although the term "computer-readable medium" may also include transient media such as signals in addition to physical storage media.

Claims

1. A system, comprising: Memory; as well as a processing unit having one or more processors coupled to the memory, the one or more processors being configured to execute instructions that cause the processing unit to: obtaining image capture data of one or more sequences of images representing the appearance of the wound at corresponding image capture times, each of the images being separated by a sampling time interval between the image and the next image prior to a final image of the sequence of images, passing the image capture data of the image sequence through a machine learning model, the machine learning model being trained to generate image data of one or more predicted images representing a future appearance of the wound, each of the one or more predicted images representing a future appearance of the wound at a corresponding future time, the machine learning model being trained using historical image data, the historical image data comprising one or more historical image datasets, each of the one or more historical image datasets comprising image data of a historical sequence of images of the appearance of a corresponding historical wound, wherein a predicted time interval between the future time and a capture time of a last image of the image sequence is greater than each of the sampling time intervals, and Image data of the one or more predicted images representing a future appearance of the wound is output. 2 . The system of claim 1 , wherein the image capture data includes metadata identifying a treatment method or one or more treatment method parameters.

3. The system of claim 2, wherein the therapy method parameters include negative pressure wound therapy (NPWT) parameters.

4. The system of claim 1 , wherein the machine learning model is trained bidirectionally, wherein a first training direction trains the machine learning model to generate one or more predicted future images from the image history sequence, and wherein a second training direction trains the machine learning model to generate a reconstructed first image from the one or more predicted images and an image subsequent to the first image in the image history sequence.

5. The system of claim 4, wherein a layer in the machine learning model is shared by the first training direction and the second training direction.

6. The system of claim 4, wherein: The machine learning model includes a second machine learning model; training the first machine learning model prior to the second machine learning model using a first training image dataset, the first training image dataset comprising a first subset of images of the image history sequence captured during a sampling period associated with historical wound images and a second subset of images captured after the sampling period; and The second machine learning model is constrained to include one or more layers of the first machine learning model.

7. The system of claim 6, wherein the first machine learning model is trained bidirectionally.

8. The system of claim 6, wherein the one or more layers include a last layer, a penultimate layer, or one or more intermediate layers.

9. The system of claim 1, wherein: The machine learning model includes a second machine learning model; training the first machine learning model prior to the second machine learning model using a first training image dataset, the first training image dataset comprising a first subset of images of the historical sequence of images captured during a sampling period associated with the historical wound and a second subset of images captured during the sampling period, wherein the number of images in the first subset of images is greater than the number of images in the second subset of images; and The second machine learning model is constrained to use one or more layers of the first machine learning model.

10. The system of claim 9, wherein the first machine learning model is trained bidirectionally.

11. The system of claim 1, wherein the prediction time interval is greater than an input time interval associated with the image sequence.

12. The system of claim 1, wherein the machine learning model is trained using historical metadata corresponding to historical image sequences, and wherein the processing unit is further configured to: obtaining metadata including wound properties of the wound corresponding to the sequence of images, the wound properties including one or more of wound area, wound depth, or wound healing stage; passing the metadata through the machine learning model to generate predicted metadata for the wound at the corresponding future time; and The predicted metadata is output.

13. A method comprising: obtaining, by a processing unit comprising one or more processors, image capture data of one or more sequences of images representing the appearance of the wound at corresponding image capture times, each of the images being separated by a sampling time interval between the image and the next image prior to a final image of the sequence of images; passing the image capture data of the image sequence through a machine learning model, the machine learning model being trained to generate image data of one or more predicted images representing a future appearance of the wound, each of the one or more predicted images representing a future appearance of the wound at a corresponding future time, the machine learning model being trained using historical image data, the historical image data comprising one or more historical image data sets, each of the one or more historical image data sets comprising image data of a historical sequence of images of the appearance of a corresponding historical wound, wherein a predicted time interval between the future time and a capture time of a last image of the image sequence is greater than each of the sampling time intervals; and Image data of the one or more predicted images representing a future appearance of the wound is output.

14. The method of claim 13, wherein the machine learning model is trained using a weighted loss that assigns a first weight to a first image that is less than a second weight assigned to a second image that has a corresponding predicted future time that is later than the predicted future time corresponding to the first image.

15. The method of claim 13, wherein the machine learning model is trained bidirectionally, wherein a first training direction trains the machine learning model to generate the one or more predicted images from the image history sequence, and wherein a second training direction trains the machine learning model to generate a reconstructed first image from the one or more predicted images and an image subsequent to the first image in the image history sequence.

16. The method of claim 15, wherein a layer in the machine learning model is shared by the first training direction and the second training direction.

17. The method of claim 15, wherein: The machine learning model includes a second machine learning model; training the first machine learning model prior to the second machine learning model using a first training image dataset, the first training image dataset comprising a first subset of images of the image history sequence selected from a sampling period associated with the wound and a second subset of images selected from images of the wound captured after the sampling period; and During a training phase of the second machine learning model, the second machine learning model is constrained to include layers of the first machine learning model.

18. The method of claim 17, wherein the first machine learning model is trained bidirectionally.

19. The method of claim 17, wherein the layer comprises a last layer.

20. The method of claim 15, wherein: The machine learning model includes a second machine learning model; training the first machine learning model prior to the second machine learning model using a first training image dataset, the first training image dataset comprising a first subset of images of the historical sequence of images captured during a sampling period associated with the historical wound and a second subset of images captured during a treatment period of the wound, wherein the number of images in the first subset of images is greater than the number of images in the second subset of images; and The second machine learning model is constrained to use layers of the first machine learning model.

21. The method of claim 20, wherein the first machine learning model is trained bidirectionally.

22. The method of claim 13, wherein the image capture data includes metadata identifying a treatment method or one or more treatment method parameters.

23. The method of claim 22, wherein the therapy method parameters include negative pressure wound therapy (NPWT) parameters.

24. The method of claim 13, wherein the machine learning model is trained using historical metadata corresponding to the historical image sequence, and wherein the method further comprises: obtaining metadata including wound properties of the wound corresponding to the sequence of images, the wound properties including one or more of wound area, wound depth, or wound healing stage; passing the metadata through the machine learning model to generate predicted metadata for the wound at the corresponding future time; and The predicted metadata is output.

25. A method comprising: receiving historical image data, the historical image data comprising a plurality of historical image data sets, each of the historical image data sets comprising image data of a historical sequence of images corresponding to a wound, each image of the historical sequence of images being separated by a sampling time interval between the image and a next image prior to a final image of the historical sequence of images; training the machine learning model to generate, for each historical image dataset of the plurality of historical image datasets, one or more predicted images of a future appearance of the wound, each image corresponding to a future time from the image history sequence, wherein a predicted time interval between the future time and a capture time of a last image of the image history sequence is greater than each of the sampling time intervals; and Weights in a layer of the machine learning model are adjusted based on differences between the one or more predicted images and one or more target images associated with the wound.

26. The method according to claim 25, further comprising: training the machine learning model to generate a reconstructed first image from the predicted future image and an image subsequent to the first image in the image history sequence; as well as Weights in the layer of the machine learning model are adjusted based on a difference between the reconstructed first image and a first image in the image history sequence.

27. The method of claim 25, wherein training the machine learning model comprises training the machine learning model using a weighted loss that assigns a first weight to a first output image that is less than a second weight assigned to a second output image that has a corresponding predicted future time that is later than a predicted future time corresponding to the first image.

28. The method of claim 25, wherein the machine learning model comprises a second machine learning model, and wherein the method further comprises: Prior to training the second machine learning model, training the first machine learning model using a first training image dataset, the first training image dataset comprising a first subset of images of the image history sequence captured during a sampling period associated with the wound and a second subset of images captured after the sampling period; as well as Constrain the second machine learning model to include layers of the first machine learning model.

29. The method of claim 28, wherein the layer comprises a last layer.

30. The method of claim 25, wherein the machine learning model comprises a second machine learning model, and wherein the method further comprises: Prior to training the second machine learning model, training the first machine learning model using a first training image dataset, the first training image dataset comprising a first subset of images of the image history sequence captured during a sampling period associated with the historical wound and a second subset of images captured during the sampling period, wherein the number of images in the first subset of images is greater than the number of images in the second subset of images; as well as Constrain the second machine learning model to use layers of the first machine learning model.

31. The method of claim 25, wherein the image capture data includes metadata identifying a treatment method or one or more treatment method parameters.

32. The method of claim 31 , wherein the therapy method parameters include negative pressure wound therapy (NPWT) parameters.

Images