AI-based peripheral vascular disease patient monitoring system

Abstract

The invention relates in particular to a device for monitoring a health risk of a patient. The device (400) comprises means (410) (e.g., a smartphone camera) for obtaining visual information (430) (e.g., a video sequence) relating to the skin (420) of the patient, and means (430) for detecting an anomaly on the skin of the patient based at least in part on the obtained visual information, and means (440) for assigning a score value (445) to the detected anomaly, wherein the score value is associated with a health risk of the patient. The score value may help diagnose or predict peripheral vascular disease.

Description

Background Art

[0001] In recent years, changes in lifestyle have led to an increase in various different dysfunctions of the circulatory system, which have corresponding hazards to the health status of patients. The burden on the health of patients may lead to an increase in treatment costs, an increase in the need for medical staff, etc.

[0002] These dysfunctions of the circulatory system are usually caused by damage, occlusion, and / or inflammation of the arteries and / or veins of the patient. These dysfunctions particularly involve peripheral vascular diseases (PVD), and represent a major cause of morbidity and mortality. Peripheral artery disease (PAD), chronic venous disease (CVD) (which includes chronic venous insufficiency (CVI) and deep vein thrombosis (DVT)) are common types of PVD, and are most prevalent in the lower extremities.

[0003] Early diagnosis and treatment of the above diseases are crucial for addressing the risks of these dysfunctions and avoiding death and morbidity in affected patients. This goal is complicated by the fact that approximately 50% of people suffering from one of the above diseases are asymptomatic and thus do not necessarily seek medical help and / or are not usually screened by clinicians without diagnosing the disease, as there may be no obvious reason for detailed screening.

[0004] The fact that the symptoms of PAD are not always safely recognized by patients as severe symptoms of PAD or as a worsening of known existing PAD diseases has been a major problem.

[0005] Therefore, establishing effective and efficient clinical non-invasive diagnostic tools for determining vascular diseases is a key task, especially for asymptomatic PVD and PAD patients who have the same morbidity and mortality risks as patients with more obvious symptoms. Although many non-invasive methods have been developed to at least support the diagnosis of PVD and PAD, the currently existing methods for diagnosing and treating PVD and PAD are not satisfactory (e.g., because they may require many manual and thus time-consuming diagnostic steps), especially given the increasing number of patients suffering from PVD and / or PAD and the drawbacks of current methods. Therefore, there is still a need for improved new non-invasive diagnostic tools. Summary of the Invention

[0006] According to a first aspect of the present invention, these drawbacks can be at least partially overcome. The first aspect of the present invention relates to a device for monitoring the health risk of a patient. The device may include means for obtaining visual information associated with the skin of the patient and means for detecting abnormalities on the skin of the patient based at least in part on the obtained visual information. In addition, the device may include means for assigning a score value to the detected abnormality, wherein the score value is associated with the health risk of the patient.

[0007] The device can be implemented as a remote entity (such as, for example, a server), but can also be implemented as a local entity (such as, for example, a smart phone).

[0008] The visual information can include information on a locally restricted area of the patient's skin (e.g., a locally restricted area of the skin at the patient's neck and / or a locally restricted area of the skin at the patient's thigh and / or any other (suitable) locally restricted area of the patient's skin). The locally restricted area can be, for example, between 1 and 30 cm 2 between, preferably between 5 and 25 cm 2 between, more preferably between 10 and 20 cm 2 between or any other suitable area.

[0009] An anomaly can be understood as a feature derivable from visual information indicating a health risk of the patient. Additionally or alternatively, an anomaly can be understood as a feature in the obtained visual information associated with an ill patient that is not present in the corresponding visual information associated with a patient considered to be healthy.

[0010] A score value can be provided as a number (e.g., an integer) within a predefined numerical interval, which can indicate the likelihood that the patient is suffering from a specific health risk. As an example, the score value can be any number between 0 and 100, where the number can indicate the likelihood (in percentage) that the patient is suffering from a specific health risk. Alternatively, the score value can also span an interval from 0 to 10, where 0 can be an indication of the likelihood that the patient (most likely) is not suffering from a specific health risk, and 10 can be understood as an indication of the likelihood that the patient (most likely) is suffering from a specific health risk.

[0011] In a preferred embodiment, the device for obtaining visual information is a device for continuously obtaining visual information, and / or the device for detecting anomalies is a device for continuously detecting anomalies, and / or the device for assigning score values is a device for continuously assigning score values. Thus, a more stringent monitoring of the patient's health risk can be achieved.

[0012] An apparatus according to one aspect of the present invention may allow for a non-invasive diagnostic and / or monitoring tool for potential health risks of a patient. The apparatus may be used by a patient (e.g., due to its simplicity and / or its preferably compact form factor) to track and identify the patient's existing symptoms, or as a tracking tool for a doctor to gain access to the progressive symptoms of a potential pandemic disease. This may allow for early diagnosis of the disease and thus may support better and safer treatment of the patient while reducing the costs of the health system (e.g., because potential treatment may be initialized at an early stage of the progressive disease). According to one aspect of the present invention, since the collection of visual information may be done solely by the patient, the burden on the doctor (e.g., in terms of workload) and other members of the medical staff (who are typically part of the diagnostic process) may be reduced, since the main part of the diagnosis may be replaced by the interaction between the apparatus for monitoring the patient's health risks and the patient without the need for a doctor. In addition, the risk that the patient may not be aware of the potential dangers caused by the potentially present health risks may be minimized, and the potential health risks may be determined before they may turn into an emergency.

[0013] Another aspect of the present invention relates to an additional apparatus for monitoring the health risks of a patient. The apparatus may include means for obtaining visual information associated with the skin of the patient and optionally means for detecting an abnormality on the skin of the patient at least in part based on the obtained visual information. In addition, the apparatus may include means for receiving a scoring value to be assigned to the detected abnormality, wherein the scoring value is associated with the health risk of the patient.

[0014] The means for obtaining visual information may include a sensor configured to acquire visual information associated with the skin of the patient. In some exemplary cases, the sensor may be provided as a camera, e.g., including a CCD chip.

[0015] The means for receiving the scoring value may include means for communicating with a remote entity such that the scoring value may be received from the remote entity. The means for communicating may also include means for sending the visual information to another device (e.g., a server). Then, the other device may detect the abnormality and / or assign the scoring value to the abnormality. In some exemplary cases, the apparatus may include means for at least partially determining the scoring value locally (e.g., by the apparatus itself). The other device, and accordingly, the means for determining the scoring value may be configured to execute an algorithm for determining or assisting in determining the scoring value.

[0016] By providing the device as described above, a portable device for monitoring a patient's health risks can be provided. Notably, by providing the device with means for receiving a score value, the device may not necessarily be provided with means for performing the calculation of the score value and / or detecting anomalies that may be accompanied by a corresponding high energy requirement, which may adversely affect the battery life of the device. Thus, by outsourcing the calculation of the score value, the battery life of the device can be extended and a more compact form factor can be provided for the device. Additionally, the manufacturing cost of the device can be reduced.

[0017] A device for monitoring a patient's health risks can be adapted to monitor the prevalence, presence, and / or progression of peripheral artery disease (PAD) and / or peripheral vascular disease (PVD) and / or diabetes.

[0018] A device for monitoring a patient's health risks can be adapted to monitor the likelihood, degree of progression, probability, and / or likelihood of complications of PAD, PVD, and / or diabetes in patients with, without, developing, and / or not developing PAD, PVD, and / or diabetes.

[0019] Monitoring of prevalence can include monitoring whether a patient has one or more of the above diseases and / or the likelihood that an existing disease will deteriorate over time.

[0020] By configuring a device for monitoring at least one or more of the aforementioned health risks of a patient, early diagnosis and / or treatment of the potential prevalence of at least one of the diseases can be supported. This can be particularly advantageous since the prevalence of the diseases may often progress without any symptoms or only with a negligible worsening of potentially present symptoms, such that the prevalence and / or worsening of any of the diseases may not be recognized in the early stages. Thus, the device can advantageously contribute to reducing the morbidity of diseased patients.

[0021] The visual information can be a video sequence.

[0022] The video sequence can relate to a video sequence having a duration of 1 - 10 s, preferably 2 - 8 s, more preferably 5 - 6 s. In some cases, the video sequence can be a color video. In an alternative case, the video can be a black and white video. Additionally or alternatively, the visual information can include a series of photographs.

[0023] In some cases, the video sequence can additionally include one or more freeze images.

[0024] By providing visual information as a video sequence, a larger skin area of the skin can be provided (compared to a single frozen image). This can allow for monitoring of a correspondingly larger area of the patient's skin and can thus more likely support early diagnosis (and preferably corresponding early treatment of a primary disease), as potential small abnormalities are less likely to be overlooked.

[0025] The device for detecting an abnormality can be adapted to detect a lesion at at least one location on a patient's skin and preferably detect the appearance of the lesion.

[0026] A lesion can be understood as any injury or abnormal change in a patient's skin. This can include wounds, hematomas, ulcers, discolorations, pigmentations, combinations thereof, or any other lesion that may be associated with a health risk to the patient.

[0027] Appearance can be understood as the size of the lesion (e.g., the maximum (e.g., top to bottom) extent in length units and / or the surface area of the lesion), the color and / or color change of the lesion (compared to a previously detected abnormality), and / or any other visually accessible parameter that may be associated with a health risk to the patient.

[0028] This can advantageously contribute to non-invasive monitoring of a patient's health risk, as the monitoring may only require visual information that can be derived from the patient's skin. This can simplify the monitoring itself, can reduce the total monitoring cost (e.g., because complex medical equipment may not be required and / or because a physically present doctor may not be required), can allow for gapless monitoring of the patient's health status (e.g., because monitoring of the patient's health status may not be limited to different screenings performed by, for example, a doctor), and can thus advantageously contribute to improved health monitoring of the patient.

[0029] The device for obtaining can include a device for receiving visual information from a local device.

[0030] The device for monitoring a patient's health risk can be physically separated from the patient and / or the local device, i.e., the device for monitoring can not be adjacent to the patient and / or the local device, but can be located in another room and / or another building as the patient and / or can be accessed via a network and / or an internet connection.

[0031] The device for receiving can include a device for communicating with the local device by means of a wired connection and / or preferably by means of a wireless connection (e.g., Bluetooth, WiFi, LTE, 5G, etc.).

[0032] The device for obtaining can further include a device for decompressing the received visual information in the case where the visual information is sent in a compressed manner.

[0033] By implementing a device for monitoring such that it includes means for receiving visual information, the device for monitoring can support implementing the device as a remote entity, such as a server. This can facilitate at least a partially centralized data warehouse system where the received visual information can be collected and / or analyzed. Thus, centralized data analysis can be facilitated. This can additionally allow the main computing power for detecting anomalies and / or for assigning scoring values to be provided by the device for monitoring, such that local devices may not need to be provided with dedicated data storage capabilities and / or may not be adapted with dedicated computing capabilities for detecting anomalies and / or for assigning scoring values. This can allow local devices to be adapted to a compact form factor, reduced manufacturing costs, and extended battery life.

[0034] In a preferred embodiment, the means for obtaining visual information is configured to obtain first visual information associated with a first part of the skin and second visual information associated with a second part of the skin, the means for detecting anomalies is configured to detect a first anomaly on the first part of the skin at least partially based on the obtained first visual information, and to detect a second anomaly on the second part of the skin at least partially based on the obtained second visual information, and the means for assigning scoring values is configured to assign a first scoring value to the first anomaly and a second scoring value to the second anomaly. The device further includes means for making a first comparison between the first scoring value and the second scoring value, and means for assigning a first comparison scoring value to the first anomaly based on the first comparison.

[0035] The first part of the skin is different from the second part of the skin. For example, the first part of the skin can be a part of the patient's left leg, while the second part of the skin is a part of the right leg.

[0036] If the first part of the skin includes an anomaly, but the second part of the skin is healthy and does not include an anomaly (the second anomaly is within the range of anomalies related to healthy skin), then the device provides a more accurate and robust determination of the patient's health risk by comparing the healthy skin with the skin that may include an anomaly.

[0037] By comparison between the first scoring value, which is a scoring value and thus associated with the patient's health risk, and the second scoring value, which is a scoring value and thus associated with the patient's health risk, the first comparison scoring value is associated with the patient's health risk and allows for a more accurate and robust determination of the patient's health risk.

[0038] In another preferred embodiment, the device for obtaining visual information is further configured to obtain third visual information associated with a first part of the skin and fourth visual information associated with a second part of the skin, wherein the third visual information and the fourth visual information are obtained at times that are respectively later in time than the first visual information and the second visual information. The device for detecting anomalies is further configured to detect a third anomaly on the first part of the skin based at least in part on the obtained third visual information, and to detect a fourth anomaly on the second part of the skin based at least in part on the obtained fourth visual information. The device for assigning a scoring value is configured to assign a third scoring value to the third anomaly and a fourth scoring value to the fourth anomaly. The device for making a first comparison is further configured to make a second comparison between the first scoring value and the third scoring value, and a third comparison between the second scoring value and the fourth scoring value, and / or a fourth comparison between the third scoring value and the fourth scoring value, and the device for assigning a first comparison scoring value is further configured to assign a second comparison scoring value to the third anomaly based on the second comparison and the third comparison and / or the first comparison scoring value. The second comparison and / or the fourth comparison.

[0039] The third visual information and the fourth visual information are obtained at a time later than the first visual information and the second visual information in order to capture the potential progression of a potential anomaly. A reasonable time between obtaining the first and second visual information respectively and obtaining the third and fourth visual information respectively can be hours, days, weeks or even months. In an even further preferred embodiment, the device for obtaining visual information continuously obtains visual information of the first and / or second part of the skin at more than two time points. According to this embodiment, the progression of the anomaly over time can be determined and compared with the progression of healthy skin over time. Thus, the device provides an even more accurate and robust determination of the health risk of the patient.

[0040] In an alternative embodiment, the device for obtaining visual information is configured to obtain first visual information and second visual information, wherein the first visual information and the second visual information are associated with a part of the patient's skin, and the second visual information is obtained at a time later than the first visual information. The device for detecting anomalies is configured to detect a first anomaly on the part of the skin based at least in part on the obtained first visual information, and to detect a second anomaly on the part of the skin based at least in part on the obtained second visual information, and the device for assigning a scoring value is configured to assign a first scoring value to the first anomaly and a second scoring value to the second anomaly. The device further includes means for comparing between the first scoring value and the second scoring value, and means for assigning a comparison scoring value to the second anomaly based on the comparison.

[0041] The second visual information is obtained at a time later than the first visual information in order to capture the potential progression of a potential anomaly. The preferred duration of the time period between obtaining the first visual information and the second visual information can be hours, days, weeks or even months. In an even further preferred embodiment, the device for obtaining visual information is configured to continuously obtain visual information of a part of the skin. According to this embodiment, the temporal evolution or progression of the anomaly can be determined.

[0042] By comparing a first score value, which is a score value and thus associated with the health risk of the patient, with a second score value, which is a score value and thus associated with the health risk of the patient, the comparison score value is associated with the health risk of the patient and allows for a more accurate and robust determination of the health risk of the patient, as it takes into account the temporal evolution of the anomaly.

[0043] The first comparison score value and the second comparison score value can also be provided as numbers (e.g., integers) within a predefined numerical range, which can indicate the likelihood that the patient is suffering from a specific health risk. As an example, the first comparison score value and the second comparison score value can also be any number between 0 and 100, where the number can indicate the likelihood (in percentage) that the patient is suffering from a specific health risk. Alternatively, the first comparison score value and the second comparison score value can also span a range from 0 to 10, where 0 can be an indication of the likelihood that the patient (most likely) does not suffer from a specific health risk, and 10 can be understood as an indication of the likelihood that the patient (most likely) suffers from a specific health risk.

[0044] In another preferred embodiment, the device for detecting anomalies on the patient's skin includes an artificial intelligence AI engine or model. The AI engine can include an autoencoder (AE), a variational autoencoder (VAE), a generative adversarial network (GAN) or a convolutional neural network (CNN) for accurately and robustly determining the health risk to the patient.

[0045] The device for monitoring the health risk of the patient can further include means for providing to the patient at least one question associated with the health state of the patient, where at least one question is preferably associated with the degree of pain experienced by the patient and / or the location of the pain experienced by the patient. The device can additionally or alternatively include means for receiving from the patient at least one answer in response to the at least one question (where the question can optionally be provided to the patient by a different device).

[0046] The degree of pain can be understood as the ranking of the currently experienced pain by numbers. As an example, the currently experienced pain can be indicated by an integer between 0 and 10, where 0 can indicate that no pain is currently being experienced, and where 10 can indicate that very strong pain is being experienced. It should be emphasized that any other suitable interval (e.g., from 0 to 5) can be equivalently implemented to rank the pain experienced by the patient.

[0047] The location of the pain can refer to the area of the patient's body where the pain is being experienced, e.g., an area including the patient's thigh, the patient's neck, etc.

[0048] In the case where the device is implemented as a local device, the means for providing at least one question to the patient can include means for displaying the question (such as, for example, a graphical display, and preferably a graphical user interface (GUI)) and / or can include means for acoustically providing at least one question to the patient (such as, for example, at least one speaker that can provide the question to the patient as an oral sequence).

[0049] In the case where the device is implemented as a local device, a response can be received by displaying possible responses to the patient (e.g., on a touch-sensitive display). As an example, after providing at least one question to the patient (e.g., "Please rank the current degree of pain you are experiencing in your upper right thigh"), a scale from 0 to 10 can be provided to the patient, where the patient can select one of the numbers as a measure of the degree of pain the patient is currently experiencing.

[0050] Additionally or alternatively, the means for receiving can further include a microphone. The microphone can be configured to receive an oral response to at least one question from the patient. As an example, in view of the above question, the patient can answer "5" as an indication of the degree of pain the patient is currently experiencing in his / her upper right thigh.

[0051] The received response can be further processed / evaluated at the local device. However, additionally or alternatively, the received response can also be sent to a remote entity.

[0052] In other cases, preferably, if the device is implemented as a remote entity, the means for providing can include a database, where the database can be provided with at least one question to be provided to the patient.

[0053] In this case, the means for providing can further include means for communicating with the local device, where the device can preferably include means for sending the question to the local device. Then, the local device can provide at least one question to the patient, as described above.

[0054] In this case, the device for receiving may include a device for receiving a response to at least one question from a local device. The local device may be implemented as described above such that it is capable of receiving a corresponding response to at least one question.

[0055] By providing at least one question to the patient, the monitoring of the health risks of the patient can be further improved, because the monitoring (and potential diagnosis) depends not only on a single parameter (e.g., visual information), but can be further refined by at least one question. Therefore, an early and more reliable diagnosis of the potential general health risks of the patient can be advantageously supported.

[0056] The device for monitoring the health risks of a patient may further include a device for weighting the at least one received answer and / or score value, the first score value and / or the second score value and / or the third score value and / or the fourth score value and / or the first comparison score value and / or the second comparison score value respectively to provide an estimate of the prevalence of PAD, PVD, and / or diabetes.

[0057] The device for weighting may include a device for analyzing the received response and extracting information from the response. As an example, at least one question may include the question "Is the estimated total distance you walked today more than 200m or less?" As a response, the patient may reply "more than 200m" ("more than 200m" may be considered as information). Based on the extracted response content, the device for weighting may assign a weighting value (e.g., an integer between 1 and 4) to the response, indicating to what extent the received answer and / or score value, the first score value and / or the second score value and / or the third score value and / or the fourth score value and / or the first comparison score value and / or the second score value can be associated with the prevalence of PAD, PVD, and / or diabetes.

[0058] More specifically, if the patient answers that the pain occurs at rest, a weight value of 4 may be assigned to the answer, if the patient answers that the pain occurs after walking less than 200m, a weight value of 3 may be assigned, if the patient answers that the pain occurs after walking more than 200mm, a weight value of 2 may be assigned, and if the patient answers that the pain occurs after walking 1km, a weight value of 1 may be assigned.

[0059] The device for weighting may be implemented at the local device (i.e., the local device may at least partially locally assign a weighting factor to the provided response) and / or may be implemented at a remote device (i.e., the remote device may at least partially assign a weighting factor to the provided response).

[0060] By providing a device having means for weighting, when monitoring a patient's health status, at least one question (in addition to visual information) regarding the patient's health status can be considered. The weighting of the at least one question can contribute to providing a probability value (encoded in the sum of the assigned weighting values associated with the at least one question) indicating the likelihood that the patient may have PVD, PAD, and / or diabetes.

[0061] The device for monitoring a patient's health risk may further include means for requesting (preferably periodically) visual information of the patient's skin.

[0062] The means for requesting may include means for outputting a push notification to the patient, the push notification indicating that new visual information of the patient's skin should be provided. The means for outputting may include a display (if the device is implemented as a local device) such that the push notification can be provided visually as a message on the display and / or may include an LED, where the LED may be configured to blink if new visual information is to be provided. Additionally or alternatively, the means for outputting may include a loudspeaker such that the indication of the need for new visual information can be output as a beep and / or an audible cue (e.g., an oral sequence).

[0063] In some cases, the (local) device may be internally configured (e.g., by means of hard-coded configuration) to request visual information.

[0064] Additionally or alternatively, the (local) device may be provided with means for receiving a request message from a remote device and may include means for outputting the received message.

[0065] The periodic request may involve a request output once a day, once a week, once a month, once a year, and / or at any other suitable interval. The means for requesting may additionally or alternatively be configured to output a request for visual information in response to an externally provided request (such as a request remotely provided by the patient's doctor).

[0066] By providing the device with means for (e.g., periodically) requesting visual information, gapless tracking and monitoring of the patient's health status can be ensured. This can reduce the risk that a deterioration in the patient's health status may still go undetected.

[0067] Another aspect of the present invention relates to a method for monitoring a patient's health risk. The method may include obtaining visual information related to the patient's skin and detecting abnormalities on the patient's skin at least partially based on the obtained visual information. Additionally, the method may include assigning a scoring value to the detected abnormalities, where the scoring value is associated with the patient's health risk.

[0068] Visual information, assignment of a score value, health risk, and / or an anomaly can be implemented as described above.

[0069] Another aspect of the invention relates to a method for monitoring a patient's health risk. The method can include obtaining visual information related to the patient's skin and, optionally, detecting an anomaly on the patient's skin at least in part based on the obtained visual information, where detecting the anomaly can include comparing the obtained visual information with previously obtained visual information. Additionally, the method can include receiving a score value to be assigned to the detected anomaly, where the score value is associated with the patient's health risk.

[0070] Visual information, assignment of a score value, health risk, and / or an anomaly can be implemented as described above.

[0071] The method for monitoring a patient's health risk can also include receiving the visual information at a remote entity and preferably performing the detection and / or assignment at least in part at the remote entity.

[0072] In such an embodiment, the visual information can initially be obtained by a local device (e.g., a camera, a medical diagnostic tool, and / or a portable device such as a smart phone, etc.). After being obtained, the obtained information can be provided to a remote device.

[0073] By receiving the visual information at a remote entity and preferably performing the detection and / or at least in part the assignment at the remote entity, computationally and / or storage-intensive tasks can be outsourced from the local device to the remote entity. This can allow for an extended battery life of the local device and generally can allow for a more compact form factor of the portable device. Additionally, by outsourcing computationally intensive tasks to a remote entity (which can preferably be optimized for computationally intensive tasks), faster detection and / or assignment can be supported. Further, if the detection / assignment is performed at the remote entity, the detection / assignment can be at least in part based on a previously acquired (historical) data set, which can further improve the reliability of the detection and / or assignment.

[0074] In a preferred embodiment, the method for monitoring a patient's health risk can also include obtaining second visual information associated with a second part of the patient's skin, detecting a second anomaly on the patient's skin at least in part based on the obtained second visual information, assigning a second score value to the detected second anomaly, where the second score value is associated with the patient's health risk, performing a first comparison between the first score value and the second score value, and assigning a first comparison score value to the first anomaly based on the first comparison.

[0075] In a case where a first part of the skin includes an abnormality, but a second part of the skin is healthy and does not include an abnormality (the second abnormality is within the range of abnormalities associated with healthy skin), the method provides a more accurate and robust determination of the health risk of a patient by comparing the healthy skin with the skin that may include an abnormality. In a case where both the first part and the second part of the skin include abnormalities, the method provides an improved determination of the degree of health risk of the patient. Accordingly, a first comparison score value is associated with the health risk of the patient.

[0076] In addition, a method for monitoring the health risk of a patient may include obtaining third visual information associated with a first part of the patient's skin after obtaining first visual information, detecting a third abnormality on the patient's skin at least in part based on the obtained third visual information, assigning a third score value to the detected third abnormality, wherein the third score value is associated with the health risk of the patient, making a second comparison between the first score value and the third score value, and assigning a second comparison score value to the third abnormality based on the second comparison.

[0077] By comparing the first score value and the third score value and assigning the second comparison score value, the progression and temporal evolution of the disease are determined. Accordingly, the second comparison score value may be associated with the health risk of the patient.

[0078] In a preferred embodiment, a first time period of hours, days, weeks, or months may be located between obtaining the third visual information associated with the first skin part of the patient and obtaining the first visual information. Accordingly, the frequency of obtaining visual information may be adjusted for the disease (PAD, PVD, and / or diabetes), the degree and / or progression of the disease.

[0079] In another embodiment, the method may include obtaining visual information and / or detecting abnormalities and / or assigning score values in a continuous manner. Accordingly, a more stringent monitoring of the health risk of the patient can be achieved.

[0080] In another preferred embodiment, the method may include: after obtaining the second visual information, obtaining fourth visual information related to the second skin part of the patient; detecting a fourth abnormality on the patient's skin at least in part based on the obtained fourth visual information; assigning a fourth score value to the detected fourth abnormality, wherein the fourth score value is associated with the health risk of the patient; making a third comparison between the second score value and the fourth score value, and / or making a fourth comparison between the third score value and the fourth score value, and assigning a third comparison score value to the third abnormality based on the second comparison and the third comparison and / or the first comparison and / or the fourth comparison.

[0081] Thus, this embodiment combines the advantages of embodiments that include comparing score values associated with different parts of the skin and embodiments that include comparing score values associated with the same part of the skin but at different time points.

[0082] In a preferred embodiment, the second time period of hours, days, weeks, or months can be located between obtaining the second visual information associated with the second skin part of the patient and obtaining the fourth visual information. In a further preferred embodiment, the first time period and the second time period are the same.

[0083] The method of monitoring a patient's health risk can further include providing the patient with at least one question related to the patient's health status, wherein the at least one question is preferably related to the degree of pain experienced by the patient and the location of the pain experienced by the patient, and receiving at least one answer from the patient in response to the at least one question. The method can also include preferably weighting the at least one received answer to provide an estimate of the prevalence of PAD.

[0084] In some cases, when the method is performed for the first time, only at least one question can be provided to the patient. Additionally or alternatively, the at least one question can be provided to the patient periodically (e.g., once a day, once a week, once a month, once a year, etc.) and / or irregularly (e.g., if a deterioration in the abnormality in the visual information is determined).

[0085] The detection of anomalies can include providing the obtained first visual information, second visual information, third visual information, and / or fourth visual information to a trained artificial intelligence (AI) engine, where the trained AI engine (preferably including an autoencoder) has been trained on previously obtained visual data that is considered healthy and preferably based on the patient's skin type. The detection can also include: deriving a first anomaly in the obtained first visual information by means of the trained AI engine, preferably by reconstructing the obtained first visual information by the autoencoder and comparing the reconstructed information with the obtained first visual information, and / or deriving a second anomaly in the obtained second visual information by means of the trained AI engine, preferably by reconstructing the obtained second visual information by the autoencoder and comparing the reconstructed information with the obtained second visual information, and / or deriving a third anomaly in the obtained third visual information by means of the trained AI engine, preferably by reconstructing the obtained third visual information by the autoencoder and comparing the reconstructed information with the obtained third visual information, and / or deriving a fourth anomaly in the obtained fourth visual information by means of the trained AI engine, preferably by reconstructing the obtained fourth visual information by the trained AI engine and comparing the reconstructed information with the obtained fourth visual information. The obtained fourth visual information is constructed by the autoencoder and the reconstructed information is compared with the obtained fourth visual information.

[0086] The training on the previously obtained visual data can be understood as training the AI engine such that it can reproduce one or more data sets included in the obtained visual data used for training. The AI engine can include a neural network that includes an input layer, an output layer, and one or more hidden layers between the input layer and the output layer. Each node of a certain layer can be interconnected to each node of the previous layer and / or the next layer. A weighting factor can be provided for each interconnection. The weighting factors can be determined by training the neural network.

[0087] The detection of anomalies can be at least partially based on determining the difference between the reconstructed obtained visual information and the obtained visual information that may include anomalies.

[0088] Training the AI engine for the patient's skin type can allow the anomaly detection to adapt to the different appearances of anomalies on the patient's skin relative to the patient's skin. As an example, in some cases, anomalies may look different on pigmented skin compared to Caucasian skin. Thus, training the AI engine for the patient's different skin can support more precise and reliable anomaly determination. It can be a separate aspect of the present invention.

[0089] The reliability of monitoring can be further improved by means of an AI-based determination of anomalies in the obtained visual information. This may be the case because the AI engine can be trained on large datasets of images and can thus also be trained on rare and / or small anomalies that may be overlooked by a doctor.

[0090] Another aspect of the invention relates to a system for monitoring a patient's health risk. The system may include one or more of the devices described above.

[0091] In a preferred embodiment, the system may include a device configured to act as a remote entity. Additionally, the system may preferably include a device configured to act as a local entity.

[0092] The system may allow for a beneficial synergistic interaction between the local device (as described above) and the remote entity (as described above). This synergistic interaction can be facilitated by combining one or more beneficial attributes of the local device (such as portability, ease of handling, ease of operability, optimized user experience, low cost, etc.) with one or more beneficial attributes of the remote entity (such as computing power, large storage capacity, centralized data storage, centralized maintenance, etc.). This can advantageously support the reliable monitoring of a patient's health risk.

[0093] Another aspect of the invention relates to a computer program including code which, when executed on a computer, performs the method described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0094] The following drawings are provided to support the understanding of the invention:

[0095] Figure 1 is a diagram of an exemplary process for monitoring a patient's health risk;

[0096] Figure 2 is a diagram of an exemplary limping position;

[0097] Figure 3 is a diagram of an exemplary location for determining a patient's pulse intensity;

[0098] Figure 4 is an exemplary system overview according to aspects of the invention;

[0099] Figure 5 is an exemplary flowchart of a method for anomaly detection in visual information;

[0100] Figure 6 is a diagram of an exemplary neural network configured to act as an autoencoder;

[0101] Figure 7A and 7BAn exemplary method for anomaly detection in visual information according to aspects of the present invention is shown;

[0102] Figure 8 An exemplary flowchart for assigning a score value to visual information obtained from both sides of a patient's body. Detailed Description

[0103] Attempts to diagnose PAD in a non-invasive manner have been around for a long time. Typical methods for diagnosing PAD as used in the art are shown and compared in the table below.

[0104]

[0105] Table 1: Overview of different methods for diagnosing PAD and / or PVD.

[0106] Figure 1 An exemplary flowchart for monitoring the health risks of 100 patients (especially PAD and PVD venous diseases) is shown. The commonly used monitoring flow is called "VESSEL", which is represented by the exemplary flowchart depicted in Figure 1 Among them, each of steps 110 - 160 corresponds to a letter of the term "VESSEL", and each of the steps in method 100 is intended to separate the potential pad from PVD, that is, during each of the 100 steps, a separate characterization step can be performed to separate the potential pad from PVD. Generally, arteries carry oxygen-rich blood away from the heart, while veins return oxygen-poor blood to the heart. The individual steps to be performed during each of method steps 110 - 160 are outlined below.

[0107] The monitoring method can start from step 110 corresponding to the letter "V". Step 110 is intended to characterize various positions that can help relieve the discomfort / pain of the patient.

[0108] As an example, to study whether a patient has PAD, it can be characterized whether hanging the legs down (dependent position) helps reduce potential general pain. In this case, it can be further elicited whether raising the legs causes a worsening of the experienced pain.

[0109] In the case of studying whether a patient has venous disease, it can be characterized whether raising at least one leg can reduce the swelling of at least one leg and whether raising at least one leg helps improve blood flow through at least one leg. In addition, in this regard, it can be characterized whether hanging or standing / sitting for a long time (e.g., for several minutes, several hours, etc.) of at least one leg causes an increase in pain and potential edema.

[0110] Subsequently, in step 120, corresponding to the letter "E", an investigation can be made regarding the explanation for pain. This can include the characterization of in which situations pain can be experienced and how it can be experienced (e.g., sharp, etc.).

[0111] In the case of investigating whether a patient has PAD, it can be characterized whether the pain is experienced severely (e.g., most severe at night). For example, whether the patient experiences "rest pain" (e.g., pain). For example, the patient wakes up from sleep with pain (when the leg is in a horizontal position, it may impede blood flow) and whether the patient hangs the limbs over the bed to relieve the pain. Additionally or alternatively, it can be studied whether there is intermittent claudication. This can include characterizing whether activities (e.g., running, walking, etc.) cause severe pain in the calf muscles, thighs, buttocks, etc., and whether the pain subsides when the activity stops. This may be due to the fact that the muscles are deprived of blood flow due to primary PAD, which may then result in the experienced pain.

[0112] In the case of investigating whether a patient has venous disease (PVD), it can be characterized whether the pain is experienced as severe, dull, cramping, or aching. Additionally, it can be characterized whether the pain is most severe when standing or sitting for a long time. Additionally or alternatively, it can be characterized whether elevating the leg relieves the pain and any potential swelling.

[0113] Subsequently, in step 130, corresponding to the letter "S", the skin of the lower extremities (e.g., at least one leg of the patient) can be characterized, preferably regarding its color and / or temperature.

[0114] In the case of studying whether a patient has PAD, it can be characterized whether the skin is thin, dry / squamous, hairless (or at least hair loss at at least one extremity (e.g., at least one leg)) and whether thick toenails can be observed. In some cases, it can be further characterized whether the dangling of at least one leg causes friction on at least one leg and whether the elevation of at least one leg causes pallor (or light blue) of at least one leg. Additionally or alternatively, the patient may have non-healing sores or wounds on at least one toe, at least one foot, or at least one leg.

[0115] In the case of studying whether a patient has venous disease, it can be characterized whether at least one leg shows thick and tough skin and / or whether at least one leg shows brown color.

[0116] Subsequently, in step 140, corresponding to the letter "S", the pulse intensity in at least one lower extremity can be characterized.

[0117] In the case of studying whether a patient has PAD, it can be characterized whether the pulse is very low or even absent. This may occur due to the fact that if PAD is dominant, the blood flow to at least one limb may be reduced.

[0118] In the case of investigating whether a patient has venous disease, the presence (and preferably the normality) of a pulse can be elicited. This can be explained by the fact that in the case of major venous disease, there is usually no obstruction in the blood flow from the heart to at least one limb. On the contrary, an obstruction may be present in the return path of the blood flow, for example, from at least one extremity to the heart.

[0119] Subsequently, in step 150, corresponding to the letter "E", the presence of edema can be investigated.

[0120] More specifically, in the case of investigating whether a patient has PAD, the presence of edema may be uncommon.

[0121] However, given the potential presence of venous disease, there may be edema, and it generally tends to worsen at the end of the day.

[0122] Subsequently, in step 160, corresponding to the letter "L", the presence (and preferably the appearance) of a lesion can be characterized.

[0123] In the case of investigating whether a patient has PAD, the presence of a lesion (e.g., an ulcer) can be characterized. If this is the case, its location can be characterized. In the case of the pad of the present invention, the lesion is most likely to be located at the end of at least one toe (e.g., near the corresponding toenail), on the top (e.g., the dorsum) of at least one foot, and / or in the lateral ankle region (e.g., the ankle) of at least one foot.

[0124] The presence of any kind of drainage (e.g., very little drainage), the presence of some (e.g., very little) tissue granulation (e.g., light pink / very light pink), or whether the tissue granulation is necrotic / black can be further characterized. In addition, in this case, it can be elicited whether a potential ulcer is deeply "punched out", having a distinct edge / margin that gives it a round appearance.

[0125] If it is investigated whether a patient has venous disease, the presence of a lesion (e.g., an ulcer) can also be characterized, and if this is the case, whether the ulcer is located in at least the medial part of the patient's calf and / or in the medial ankle region of the leg can be characterized. In addition, the appearance of the ulcer can be characterized. In this regard, it can be characterized whether the ulcer appears as a swollen one with drainage, whether there is granulation (e.g., colored between deep pink and red), and whether the edge of the ulcer is irregular and shallow in depth.

[0126] Figure 2 An exemplary map shows the relevant locations of potential claudication (e.g., stenosis or occlusion) of at least one artery in a patient's leg and the pain experienced by the patient due to stenosis or occlusion of a specific artery (corresponding to the above reference Figure 1The combination 200 between the letter "E" of the term "blood vessel" in the overview.

[0127] Claudication can typically be experienced by the patient as pain or burning in at least one leg muscle. Additionally, the pain can be reliably reproduced after walking a certain distance and can be relieved within a few minutes of rest. Additionally, in the case of current claudication, the pain may never be present at rest and the pain may not be exacerbated due to certain positions / orientations of the leg.

[0128] If claudication itself is present, the location of the pain can provide an indication of the site of the disease, e.g., which artery may be experiencing stenosis and / or occlusion.

[0129] As an example, stenosis or occlusion of the aorta 210a in the patient's leg may result in claudication in the bilateral buttocks, thighs, and calves 210b.

[0130] Stenosis or occlusion of the common iliac artery 220a may result in pain experienced in the buttock region 220b of the patient's leg.

[0131] Stenosis or occlusion of the common femoral artery 230a may result in pain experienced in the thigh region 230b of the patient's leg.

[0132] Stenosis and / or occlusion of the superficial femoral artery 240a may result in pain in the calf region 240f of the patient's leg.

[0133] Figure 3 Exemplarily shown are different locations 300 where the intensity of the patient's pulse can be determined. Note that the intensity of the pulse can thus be associated with the location where the pulse is determined. This location-specific determination of the patient's pulse can provide additional information about the potential prevalence of PAD, venous disease, and / or diabetes.

[0134] Thus, the intensity of the patient's pulse in the thigh of the patient's leg (such as in the patient's femoral artery 310) can be determined. Additionally or alternatively, the intensity of the pulse in the patient's femoral vein 320 can be determined.

[0135] Additionally or alternatively, the intensity of the patient's pulse in the popliteal cavity of the patient's leg (such as in the patient's popliteal artery 330) can be determined.

[0136] Additionally or alternatively, the intensity of the patient's pulse at the patient's foot (e.g., in the patient's dorsalis pedis artery 340) can be determined.

[0137] Additionally or alternatively, the intensity of the patient's pulse in the posterior tibial artery 350 can also be determined.

[0138] Figure 4 Exemplarily shown is an overview of the system 400 of a communication device according to an aspect of the present invention.

[0139] System 400 may include a local device 410, and the local device 410 may be adapted with a device for obtaining visual information 420 of a patient's skin.

[0140] The visual information 420 may include visual information associated with the patient's feet and / or legs and / or the patient's hands and / or arms.

[0141] In some cases, the visual information 420 may be a video sequence 430 or may be converted into a video sequence 430. The video sequence 430 may be forwarded to a remote entity 440. The forwarding of the video sequence 430 may preferably occur in an encrypted manner.

[0142] The remote entity 440 may include a cloud-based system. The remote entity 440 may include a device for at least storing the video sequence 430 received from the local device 410. The remote entity 440 may include an artificial intelligence (AI) engine for at least partially processing the received video sequence 430. The processing of the video sequence 430 may include detecting anomalies in the video sequence 430 and / or assigning a scoring value to a potentially detected anomaly.

[0143] If an anomaly is detected and / or if the prevalence of the patient's health risk is determined, an indication 445 may be provided to the local device 410. The indication may include, for example, a scoring value assigned to the detected anomaly and / or an answer to a question associated with the patient's health status. Additionally or alternatively, the indication may warn the patient that a potential existing health risk has deteriorated, that a health risk has been detected for the first time, and / or may indicate medical support.

[0144] The remote entity 440 may also include a device for communicating with a clinical system 450.

[0145] The clinical system 450 may include a hospital information system, one or more clinicians (e.g., doctors, nurses, scientists, etc.). By means of the clinical system 450, the patient's doctor and / or nurse may be facilitated to monitor the patient's potential health risks at least partially based on one or more video sequences 430 received from the remote entity 440.

[0146] Additionally or alternatively, the data received at the clinical system 450 (from the remote entity 440) may be provided to at least one scientist, and the at least one scientist may thus obtain access to at least one video sequence 430 for studying the nature / characteristics of PAD, PVD, and / or diabetes.

[0147] The data provided to the clinical system 450 may be limited to at least one video sequence 430. In some cases, the data provided to the clinical system 450 may also include a score value associated with the prevalence of the patient's health risk.

[0148] In some cases, a doctor, nurse, and / or scientist may interpret the data received from the remote entity 440 and may determine, for example, that an error occurred when assigning the score value and / or an anomaly was erroneously detected. In such a case, feedback 455 may be provided to the remote entity 440 such that the AI engine of the remote entity 440 can be updated based on the feedback 455. This may facilitate a more accurate determination of anomalies and / or a more accurate assignment of score values for subsequent video sequences 430.

[0149] Additionally or alternatively, the feedback 455 may include direct feedback sent by a doctor, nurse, and / or scientist to the patient. The feedback 455 may be received by the local device 410 and may be directed to the patient. The feedback may include, for example, a statement from the doctor that there is no health risk, a potentially pre-known health risk has worsened, or may include a warning that the patient should seek medical help. In some cases, the feedback 455 may include a request for an additional video sequence 430.

[0150] Figure 5 An exemplary flowchart of an anomaly detection method 500 according to an aspect of the present invention is shown and uses the AI engine as described above.

[0151] Before an anomaly is detected, a normal data set 510 (e.g., of a healthy patient) is provided to the above-described AI engine to train the AI engine (e.g., AI engine 440). Based on this, the AI engine can learn to encode and / or reconstruct the normal data set 510, where the training algorithm may be, for example, at least partially based on a variational autoencoder (VAE), a generative adversarial network (GAN), and / or a convolutional neural network (CNN or ConvNet). Additionally or alternatively, a vector quantization variational autoencoder-2 (VQVAE-2) may be used.

[0152] After the AI engine has successfully learned 520 (has been separately trained) a large number of normal data sets 510, the underlying AI model can be considered to be a constructed anomaly detection model 530.

[0153] To test the performance of the trained AI engine (e.g., its ability to predict anomalies when there are anomalies in the data set provided to the trained AI engine and its ability not to predict anomalies when there are no anomalies in the data set provided to the trained AI engine), a test data set 540 may be provided to the AI engine.

[0154] The test data set 540 can include at least one representative of the training data set 510 (and / or additional data sets considered normal that are not part of the initial training data set) and one or more data sets 545 that can include anomalies.

[0155] The test data set 540 can then be provided 550 to the trained AI engine.

[0156] Based on this, the trained AI engine can determine 560 whether the training data set 540 is similar to the normal data set 510 used to train the AI engine. If the training data set is similar 520 to the normal data set 510, the training data set can also be considered normal, i.e., it does not include anomalies.

[0157] However, if the trained AI engine determines 560 that the training data set 540 includes dissimilarities from the normal data set 510, anomalies can be identified 580.

[0158] Figure 6 An exemplary neural network 600 that can be used as an autoencoder (AE) is shown.

[0159] The AE can include an encoder 610, exemplarily including an input layer 620, a first hidden layer 630, and a second hidden layer 640. Since the number of nodes in subsequent layers decreases from the input layer 620 to the second hidden layer 640, this transition is accompanied by information loss, as will be further described below.

[0160] Following the encoder 610 can be a decoder 650, exemplarily including a second hidden layer 640, a third hidden layer 660, and an output layer 670. Since the number of nodes in each layer increases from the second hidden layer 640 to the output layer 670, the amount of information that can be stored in each layer increases, as will be further described below.

[0161] The input layer 620 may exemplarily include four nodes. The input layer 620 may be configured to represent an input image to be processed by the exemplary neural network 600. Note that the four nodes of the input layer 620 are shown only for simplicity, and in practical applications, the number of nodes in the input layer 620 may typically be higher. As an example, an image with a resolution of 1024×768 pixels may be transformed into a single 1×x vector including 1024×768 = 786432 rows, where each row exemplarily represents a grayscale value between 0 and 255. It should be emphasized that the neural network 600 is not limited to the processing of grayscale images and may also be configured to process color images, for example, in the RGB space, the YMCK space, and / or any other suitable color space. If an image represented in the RGB space is provided to the input layer 620, the image may exemplarily be transformed into a vector with 1024×768×3 = 2359296 rows. Here, the factor 3 takes into account the representation of the corresponding image in the RGB color space including three primary colors. The exact value of each entry in the vector takes into account, for example, the luminance of the pixel in the red space. In other words, the first set of 786432 (continuous) rows may represent the red color space (and each row may represent the luminance of a specific pixel in the red color space), the second set of 786432 (continuous) rows may represent the green color space (and each row may represent the luminance of a specific pixel in the green color space), and the third set of 786432 (continuous) rows may represent the blue color space (and each row may represent the luminance of a specific pixel in the blue color space). It should also be noted that the architecture of the input layer 620 is not limited to these examples and may also be configured with any other suitable architecture.

[0162] Each of the nodes in the input layer 620 may be interconnected to each of the three exemplary nodes assigned to the first hidden layer 630. Each interconnection may be accompanied by a weighting factor that multiplies the data propagated between two (subsequent) nodes (i.e., from a node of the input layer 620 and a node of the first hidden layer 630). Since the first hidden layer 630 may include only three nodes compared to the four nodes assigned to the input layer 620, the amount of information that can be represented by the first hidden layer 630 may be lower compared to the amount of information that can be represented by the input layer 620. Therefore, the transition from the input layer 620 to the first hidden layer 630 may be understood as a dimensionality reduction of the image input into the neural network 600. The associated loss of information may, for example, result in one or more of color loss (e.g., the transition from a color image to a black-and-white image, loss of contrast, loss of certain boundaries / margins in the image, etc.).

[0163] Each of the three nodes of the first hidden layer 630 can be interconnected with each of the two nodes (shown only as an example) assigned to the second hidden layer 640. Each interconnection can be accompanied by a weighting factor that is multiplied by the data propagated between the two (subsequent) nodes. Since the amount of information that can be represented by the second hidden layer 640 is reduced compared to the first hidden layer 630 (due to the reduced number of nodes), the transition from the first hidden layer 630 to the second hidden layer 640 can again be understood as a dimensionality reduction. This can again be understood as a loss of information of the image processed by the neural network 600. The output of the second hidden layer 640 can be referred to as a reduced image.

[0164] Processing of the originally rendered image by the decoder 650 may be performed after features are removed from the image fed to the input layer 620 (which may be associated with the encoder 610).

[0165] As the number of nodes increases with each layer included by the decoder 650, the amount of information that can be stored / processed in each layer can increase. This concept can be used by the neural network 600, and specifically by the layers / nodes assigned to the decoder 650, to reconstruct the originally presented image (as fed into the input layer 610) from the downscaled image generated at the second hidden layer 640.

[0166] This can be done, among other things, by adding additional features to the downscaled image in each of the third hidden layer 660 and the output layer 670 (e.g., by adding color information, contrast information, borders / margins, etc. to the image). This can be encoded in the respective weighting factors associated with the interconnection of the two nodes in the respective subsequent layer.

[0167] The image obtained at the output layer 670 may then be compared with the image originally provided to the input layer 620. The comparison may include the calculation of an error function, wherein the absolute value of the error function may be large if the deviation between the image provided at the output layer 670 and the image originally fed into the input layer 620 is large (i.e., the image provided at the output layer 670 is significantly different from the image originally provided to the input layer 620, wherein certain features of the original image are not present in the reconstructed image and / or wherein features that were not present in the original image appear in the reconstructed image).

[0168] The applied training algorithm may then attempt to determine the associated weighting factors such that the error function may be minimized, i.e., the weighting factors are considered to be at their respective optimal values ​​if the image provided at the output layer 670 is nearly identical (including at most negligible differences, e.g., within an average noise level) to the image originally provided to the input layer 620. If the value of the error function is below a predetermined threshold, preferably close to zero, the training may be considered successful and / or completed.

[0169] Figure 7A and 7B Exemplarily shows the training process of a GAN for determining anomalies in acquired visual information according to an aspect of the present invention.

[0170] It is worth noting that Figure 7A Shows a high-level overview of an exemplary training of a GAN and an exemplary determination of potential anomalies in the acquired visual information.

[0171] The raw data (i.e., the acquired visual information, e.g., the visual information 420 as described above with reference to Figure 4 can be preprocessed (e.g., color adjustment, contrast adjustment, sharpening filter, cropping, etc.) to obtain a training dataset 710.

[0172] The training dataset 710 can be provided to GAN training 720. GAN training 720 can be based on a set of preprocessed images 720A that can be considered healthy (i.e., they do not include anomalies) and a model 720B to be trained (e.g., included by an AI engine as described above with reference to Figure 4 and / or Figure 6 described).

[0173] GAN training 720 can exemplarily be based on the training of a generator. The generator can be based on a neural network, e.g., the generator can be based on layers 640 - 670 of the exemplary neural network 600 as described above with reference to Figure 6 (but it can also be implemented differently, e.g., include another generative neural network). Based on this, the neural network of GAN training 720 can be trained to reconstruct (or more generally: generate) an artificial image that can be recognized (e.g., by a discriminator as further outlined below) as representative of the preprocessed images 720A.

[0174] If the training has been successfully completed, then anomaly detection 730 can be initiated at least partially based on the trained model 720B. Anomaly detection 730 can include providing a set of unseen data 730A (i.e., data that has not been used to train model 720B) to the trained model 730B, identifying potential candidates 730C for anomalies, and determining 730D whether the potential candidates 730C represent anomalies.

[0175] The trained model 730B can additionally (compared to the trained model 720B) include a trained encoder, which can be based on a neural network architecture similar to that of layers 620 - 640 of the neural network 600 as described above with reference to Figure 6 described.

[0176] If the unseen data is provided to the trained model 730B, the unseen data can first undergo encoding (as described above with reference toFigure 6 as described), and subsequent reconstruction by means of the trained generator included in the trained model 720B. However, potential anomalies present in the unseen data 730A may not be correctly reconstructed by the trained model 730B because the underlying neural network has not been trained to reconstruct anomalies present in the provided dataset. More specifically, the neural network has not been trained to reconstruct anomalies in the unseen data 730A.

[0177] Identifying potential candidates 730C for anomalies can include comparing the reconstructed image of the unseen data 730A with the images of the training data 710 and / or the unseen data 730A initially provided to the training model 730B. Any deviation identified from the training data can thus be understood as an indicator for the presence of potential anomalies in the unseen data 730A. Only deviations exceeding a predefined noise level can be considered. In this case, noise can be understood as interference in the provided data (e.g., in the training data 720A and / or the unseen data 730A) that is not related to the remaining content of the corresponding data. If a potential anomaly can be separated from the noise, the potential anomaly can only be regarded as an anomaly (e.g., if the potential anomaly can be represented by more than 10 adjacent pixels and / or by a certain region in the image that appears darker than the surrounding regions in the image (including the potential anomaly), the potential anomaly can only be regarded as a potential anomaly). The difference that an anomaly must effectively exhibit compared to the image data that does not represent an anomaly can depend on the progression of the underlying disease. In other words, a potential disease monitored in an early stage can be represented by potential anomalies only slightly above the noise level, while a progressive disease can be represented by potential anomalies that are more significantly different from the noise level as anomalies associated with the early stage of the disease.

[0178] The deviation can be represented, for example, by the value of an error function. If the value of the error function exceeds a predefined threshold, the potential anomaly can be called a determined anomaly 730D.

[0179] Figure 7B is shown in more detail in reference Figure 7A the GAN training process described.

[0180] GAN training 740 can include a generative adversarial training sub-step 740A and can include encoder training 740B.

[0181] Sub-step 740A can include a generator 740C and a discriminator 740D (considered adversarial). The generator 740C can be implemented as the generator described above with reference to Figure 7A the generator described.

[0182] In the following, the training of the generator 740C in the context of generative adversarial training may be described. According to one aspect of the present invention, the generator 740C may generate artificial images that can represent visual information (based on the input data (e.g., random values, noise, encoded visual information, and / or any other suitable input parameters) provided to the corresponding one or more input layers of the generator 740C). The generated images are passed to the discriminator 740D, and the discriminator 740D may determine, for example, whether there is a difference between the artificially generated images and the real images 740E (which are considered healthy, i.e., do not include abnormalities, and which may be included in the training dataset 720A) based on calculating the value of an error function (and / or a reward function) as described above. The value of the error function may be fed back into the generator 740C, and the generator 740C may generate new artificial images at least partially based on the value of the error function, and the new artificial images may then be passed to the discriminator 740D again. This process may be iteratively repeated until the value of the error function is below a predefined threshold. In this case, the generative adversarial training sub-step 740A is considered to have been successfully terminated.

[0183] After the generative adversarial training sub-step 740A, the encoder training 740B may be performed. The encoder training 740F may utilize the generative adversarial training sub-step 740A and the encoder 740F.

[0184] The encoder training 740B may include training the underlying neural network such that a dimensionality reduction of the visual information provided to the encoder may occur and such that the input data provided to the generator 740C may be obtained as the output of the encoder 740F. The encoder 740F may be implemented as the encoder described above with reference to Figure 6 The training of the encoder 740F may be at least partially based on the training of the neural network as described elsewhere herein (e.g., based on the minimization of an error function).

[0185] If the encoder 740F has been successfully trained, then the actual anomaly detection 750 may then proceed. The anomaly detection 750 may include providing unseen data 750B (i.e., data that has not been part of the training data 740A) to the training model 750A, which may include the anomalies to be detected. The training model 750A may be based on the trained generator 740C and the trained encoder 740F.

[0186] To determine whether there is an anomaly in the unseen data 750B, the unseen data 750B (or at least one representation of the unseen data 750B, such as for example an unseen image) can be provided to the trained model 750A. The provided unseen data 750B can undergo dimensionality reduction by means of the trained encoder 740F. After encoding, the encoded unseen data 750B can be provided to the trained generator 740C as input data based on which the trained generator 740C can generate artificial images. The generated artificial images can be compared with the unseen data 750B initially provided to the encoder 740F. Any differences that can be derived from the comparison (wherein differences can be defined as described elsewhere herein) can be understood as an indication of a potential anomaly in the unseen data 750B, since both the encoder 740F and the generator 740C are only trained to encode and reconstruct (generate) real images 740E that are considered healthy (i.e., do not include anomalies). Additionally or alternatively, the generated artificial images can also be compared with one or more images of the real images 740E to derive an indication of a potential anomaly in the unseen data 750B. If the aforementioned differences exceed a predefined value (which preferably exceeds the noise level in the generated artificial images), then the potential anomaly in the unseen data 750B can be regarded as an anomaly.

[0187] It is further emphasized that a key aspect of training the GAN can be seen when obtaining the trained generator 740C, which is optimized (due to training) to provide images that will be received by the discriminator 740D. In other words, the trained generator 740C of the GAN can be optimized to generate artificial images that can be close to the real images 740 such that they cannot be rejected by the discriminator 740D (e.g., such that they cannot be distinguished from the real images 740 of the discriminator 740D).

[0188] In contrast, a trained autoencoder can be optimized to remember how to reconstruct the originally presented input data set with maximum detail and efficiency.

[0189] The GAN-based method for anomaly detection and the autoencoder-based concept for anomaly detection can be implemented as alternative solutions. Alternatively, the GAN-based method can also be combined with the autoencoder-based method (e.g., as described in Figure 6 the layers 640 - 670 of the neural network 600 can be replaced by the generator 640C and can be trained according to the interaction of the respective generator and the respective discriminator).

[0190] Figure 8 An exemplary flowchart of a method 800 for detecting that PAD (and / or PVD and / or diabetes) is developing on at least one side of a patient's body is shown.

[0191] Method 800 may be based on a first step 810A, which obtains visual information associated with a patient's skin and determines whether the visual information includes an abnormality according to the aspects presented herein. Step 810A may be performed on a first side of the patient's body (e.g., the right side) (e.g., the patient's right leg).

[0192] Method 800 may further include obtaining visual information associated with the patient's skin and determining whether the visual information includes an abnormality according to the aspects presented herein. Step 810B may be performed on a second side of the patient's body (e.g., the left side) (e.g., the patient's left leg).

[0193] Based on determining that the obtained visual information includes an abnormality, a scoring value may be assigned to the corresponding right side of the patient's body (scoring value 820A) and the corresponding left side of the patient's body (scoring value 820B).

[0194] The separately assigned scoring values (820A, 820B) may then be forwarded to a scoring comparison algorithm 830. The scoring comparison algorithm 830 may compare each of the scoring values 820A and 820B with a corresponding previously determined scoring value (e.g., historical data) associated with previously obtained visual information. This may allow determination of the temporal evolution of the previously assigned scoring value to the currently assigned scoring value and, thus, may allow determination of whether the deterioration of the abnormality has occurred over time. An increase between the currently assigned scoring values 820A and 820B may respectively be an indication of a worsening of the health risk of the patient (e.g., PAD and / or PVD and / or diabetes).

[0195] In addition, the scoring comparison algorithm 830 may also be configured to compare the currently assigned scoring values 820A, 820B with each other to determine the numerical difference between the currently assigned scoring values 820A, 820B. The determined numerical difference may be compared with a previously determined numerical difference to determine how the difference between the assigned scoring values 820A and 820B may change over time.

[0196] The method may include only step 810A or 810B, and the scoring comparison algorithm 830 may compare only the scoring value 820A or 820B with a corresponding previously determined scoring value (e.g., historical data) associated with previously obtained visual information. This may allow determination of the temporal evolution of the previously assigned scoring value to the currently assigned scoring value and, thus, may allow determination of whether the deterioration of the abnormality has occurred over time.

[0197] The calculation result of the scoring comparison algorithm 830 can be provided as feedback 840. The feedback 840 can include the currently assigned scoring values 820A and / or 820B. The feedback can additionally or alternatively include the difference between the currently assigned scoring values 820A and / or 820B. Additionally or alternatively, the feedback 840 can include one or more differences between the corresponding currently assigned scoring values 820A and / or 820B and the previously assigned scoring values.

[0198] In addition to the scoring value 820A and / or the scoring value 820B, a comparison scoring value can be determined to respectively describe the change of the scoring value 820A and / or the scoring value 820B over time and / or the change of the scoring value 820A over time, and the scoring values 820B are compared with each other.

[0199] As described above, after determining whether the obtained visual information includes an anomaly, it can be to provide the patient with at least one question about the patient's health status. Then a weighting factor can be assigned to the possible responses to the at least one question to estimate the association between the received responses and a specific disease (e.g., PAD, PVD, or diabetes).

[0200] The most common symptom of PAD is intermittent claudication (IC), which is muscle pain or discomfort in the legs and / or buttocks brought on by walking and relieved within a few minutes of rest. The occurrence of IC is due to the inability to adequately increase blood flow (and oxygen delivery) during lower limb exercise to match the metabolic demands of the lower limb muscles.

[0201] The walking distance or the speed at which symptoms occur can depend on multiple factors, including the severity and location of the arterial disease. Therefore, a higher score can be assigned in the "Explanation of the Pain Section" of the questionnaire associated with the "vascular" method as described above.

[0202] Providing the patient with at least one question about the patient's health status can be done through a weighting algorithm, which can be implemented, for example, in an app operating at a local device and / or a remote device as described above. Table 3 below states several exemplary questions / topics solved by the at least one question and the corresponding weighting coefficients (assigned at least in part based on the content information included in the responses to the provided questions) to support the monitoring of the patient's health risk (preferably the prevalence of PAD).

[0203]

[0204]

[0205] Table 3: Exemplary questions / topics provided to the patient and the corresponding weighting coefficients to monitor the patient's health risk.

[0206] When a patient first starts using the app, he / she can answer one or more of the above questions. When the patient can use the app at specified intervals (e.g., periodically), the answers will be the basis for future comparisons.

[0207] The weighted algorithm can be configured to first characterize the "E" of "vessels", and thus can provide the patient with at least one question that can be associated with an "explanation of pain", and can characterize pain as the first clinical symptom of PAD.

[0208] When the weighted coefficient in the "explanation of pain" is 6, the limit for the system can be reached. In this case, the patient can be automatically flagged for further testing without further examination of other symptoms. If the score for this part can be lower than 6 (and optionally at least 2 or at least 3), the system can look at the weighted coefficients associated with the lesions, as outlined herein. Next, it can move to other parts and their scores and can compare them with the previously assigned weighting factors.

[0209] If at least one of the weighted coefficients for other parts becomes higher over time, the patient can be considered more likely to be developing PAD. For example, if after 6 months of monitoring a patient's health risk, at least one of the assigned weighted coefficients associated with symptoms increases in at least one part other than the "explanation of pain", the patient can be flagged for further testing by a doctor. The overall history of symptom progression as well as the type of pain and the location of the pain can be provided to the patient or their doctor.

[0210] The above weighted algorithm can be optimally suited for use by patients with a history of heart and / or diabetes and may not be implemented as a consumer app (e.g., the app can only be used for (professional) monitoring purposes and / or can be specifically used by a clinician (such as a doctor)). If a patient needs to be called in, the symptom weighting report can be shared with the doctor for viewing and decision-making.

[0211] Based on how the weighted algorithm assigns weighting factors to symptoms, the weighted coefficients can be used to detect venous disease or diabetes, as these two diseases may also have skin manifestations.

[0212] Tables 4 and 5 below show two additional exemplary sets of questions that will be provided to the patient and the associated weighted coefficients that will be assigned to the received responses for monitoring the patient's health risk according to the "vascular" scheme as described above.

[0213] More specifically, Table 4 states a number of exemplary questions to be provided to a patient for monitoring whether a patient has formed a pad and / or whether an existing pad has deteriorated. Similar to Table 4, Table 5 states a number of exemplary questions to be provided to a patient for monitoring whether a patient has developed PVD and / or whether an existing PVD has worsened. In addition, Tables 4 and 5 also indicate whether the questions are directly related to PAD / PVD and the potential weighting factors assigned to responses to the questions provided to the patient.

[0214]

[0215]

[0216] Table 4: Exemplary Questions Provided to a Patient to Monitor the Patient's Health Risks

[0217] The following table shows another exemplary set of questions provided to a patient for monitoring the patient's health risks for potential venous diseases (e.g., PVD).

[0218]

[0219]

[0220] Table 5: Exemplary Questions Provided to a Patient to Monitor the Patient's Health Risks

[0221] The questions as described above can be provided to (potential) PAD and (potential) PVD patients.

[0222] By evaluating the responses to the questions and the assigned scoring values (visual information), a weighting algorithm can be able to assign diseased patients to PAD and / or PVD. It should be noted that some symptoms are common while other symptoms are completely different from others.

[0223] The difference between the PAD and PVD weighting assessments can be that the weighting algorithm can start with questions associated with pain and then can move on to edema. After that, it may focus on other symptoms compared to PAD.

Claims

1. An apparatus (440) for monitoring health risks for a patient, comprising: means for obtaining visual information (440) associated with the patient's skin; means for detecting an abnormality (440) on the patient's skin based at least in part on the obtained visual information; means for assigning a score value (440) to the detected abnormality, wherein the score value is associated with the patient's health risk.

2. The apparatus (440) according to claim 1, wherein the means for obtaining comprises means for receiving the visual information from a local device (410).

3. An apparatus (410) for monitoring health risks for a patient, comprising: means for obtaining visual information (420) associated with the patient's skin; means for receiving a score value to be assigned to an abnormality on the patient's skin based at least in part on the obtained visual information, wherein the score value is associated with the patient's health risk.

4. The apparatus according to any one of the preceding claims, wherein the apparatus for monitoring the patient's health risk is adapted to monitor the prevalence, presence and / or progression of peripheral artery disease PAD and / or peripheral vascular disease PVD and / or diabetes.

5. The apparatus according to any one of the preceding claims, wherein the visual information is a video sequence (430).

6. The apparatus according to claim 1, 2, 4 or 5, if claim 4 or 5 refers to claim 1 or 2, wherein the means for detecting an abnormality is adapted to detect a lesion at at least one location on the patient's skin and preferably detect the appearance of the lesion.

7. The apparatus according to claim 1, 2, 6 or claim 4 or 5, if claim 4 or 5 refers to claim 1 or 2, wherein: the means for obtaining visual information is configured to obtain first visual information and second visual information, wherein the first visual information and the second visual information are associated with a part of the patient's skin, and the second visual information is obtained later in time than the first visual information; the means for detecting an abnormality is configured to detect a first abnormality on the part of the skin based at least in part on the obtained first visual information and a second abnormality on the part of the skin based at least in part on the obtained second visual information; and the means for assigning a score value is configured to assign a first score value to the first abnormality and a second score value to the second abnormality; and the apparatus further comprises: means for comparing between the first score value and the second score value; and means for assigning a comparison score value to the second abnormality based on the comparison.

8. The apparatus according to claim 1, 2, 6 or claim 4 or 5, if claim 4 or 5 refers to claim 1 or 2, wherein: the means for obtaining visual information is configured to obtain first visual information related to a first part of the skin and second visual information related to a second part of the skin; The device for detecting anomalies is configured to detect a first anomaly on a first part of the skin based at least in part on the acquired first visual information, and to detect a second anomaly on a second part of the skin based at least in part on the acquired second visual information; and The device for assigning a scoring value is configured to assign a first scoring value to the first anomaly and a second scoring value to the second anomaly; and The device further comprises: A device for making a first comparison between the first scoring value and the second scoring value; and A device for assigning a first comparison scoring value to the first anomaly based on the first comparison.

9. The device according to claim 8, wherein The device for acquiring visual information is further configured to acquire third visual information related to the first part of the skin and fourth visual information related to the second part of the skin, wherein the third visual information and the fourth visual information are acquired at times respectively later than the first visual information and the second visual information; The device for detecting anomalies is further configured to detect a third anomaly on the first part of the skin based at least in part on the acquired third visual information, and to detect a fourth anomaly on the second part of the skin based at least in part on the acquired fourth visual information; The device for assigning a scoring value is configured to assign a third scoring value to the third anomaly and a fourth scoring value to the fourth anomaly; The device for making the first comparison is further configured to make a second comparison between the first scoring value and the third scoring value, a third comparison between the second scoring value and the fourth scoring value, and / or a fourth comparison between the third scoring value and the fourth scoring value; and The device for assigning the first comparison scoring value is further configured to assign a second comparison scoring value to the third anomaly based on the second comparison and the third comparison and / or the first comparison and / or the fourth comparison.

10. The device according to claim 1, 2, 6 to 9 or claim 4 or 5, if claim 4 or 5 refers to claim 1 or 2, wherein the device for detecting anomalies (440) on the patient's skin comprises an artificial intelligence AI engine.

11. The device according to any one of the preceding claims, further comprises: A device for providing to the patient at least one question (440) related to the patient's health status, wherein the at least one question is preferably related to the degree of pain experienced by the patient and / or the location of the pain experienced by the patient; A device for receiving from the patient at least one answer in response to the at least one question.

12. The device according to claim 11, further comprises: A device for weighting the at least one received answer and / or scoring value, first scoring value and / or second scoring value and / or third scoring value and / or fourth scoring value and / or first comparison scoring value and / or second comparison scoring value respectively to provide an estimate of the prevalence of PAD, PVD and / or diabetes.

13. A method for monitoring a patient's health risk, comprising: Acquiring first visual information (420) related to a first part of the patient's skin; Detecting a first anomaly on the patient's skin based at least in part on the acquired first visual information; Assign a first score value to the detected first anomaly, wherein the first score value is associated with the health risk of the patient.

14. The method according to claim 13, wherein the detection of the anomaly comprises: providing the obtained first visual information (420) to a trained artificial intelligence AI engine (600), wherein the trained AI engine, preferably comprising an autoencoder, has been trained on previously obtained visual data considered to be healthy and preferably based on the skin type of the patient (510); and / or deriving a first anomaly in the obtained first visual information (420) by means of the trained AI engine (600), preferably by reconstructing the obtained first visual information by the autoencoder and comparing the reconstructed information with the obtained first visual information (420).

15. A system for monitoring the health risk of a patient, comprising: the device (410, 440) according to any one of claims 1 to 12 and the device according to claim 3.

16. A computer program comprising code which, when executed on a computer, performs the method according to claim 13 or 14.

Images