Method for determining an anaemia condition

Abstract

The invention provides a method for determining an output anaemia condition, the method comprising: - obtaining a plurality of digital recordings, comprising at least one of the following: - a digital recording comprising static visual information of a subject's finger nails; - a digital recording comprising static visual information of the subject's conjunctiva; - a digital recording comprising static visual information of the subject's oral mucosa; - a digital recording comprising static visual information of the subject's tongue; and the plurality of digital recordings comprising at least - a digital recording comprising temporal visual information of a backlit finger of the subject; - for each of the obtained digital recordings, obtaining a respective anaemia condition assessment using a respective machine learning sub-model that has been trained to detect anaemia conditions in the respective obtained digital recording, - using an ensemble machine learning model to assess an output anaemia condition based on the obtained anaemia condition assessments.

Description

Method for determining an anaemia conditionField of the invention

[0001] The invention relates to a method for determining an anaemia condition. It also relates to a computing device and computer program product for anaemia condition determination.Background

[0002] Iron Deficiency Anaemia (IDA) is a prevalent nutritional disorder affecting millions worldwide, with a significant impact on children's health and development. The traditional methods for diagnosing IDA involve invasive procedures that can be distressing for children and their caregivers. These methods include venous blood draws, which, despite being the gold standard for iron status assessment, are highly invasive, costly, and require the involvement of multiple healthcare professionals. Alternatively, point-of-care testing, such as the HemoCue Hb 201+ System, offers a less invasive approach by requiring only a finger prick. However, this method still presents challenges in terms of scalability, cost, and patient discomfort.

[0003] In recent years, the development of non-invasive tools for iron status assessment has gained momentum. Devices like the Masimo Rad-67™ and FerroSens FIDscreen represent significant advancements in non-invasive technology, utilizing optical methods to estimate haemoglobin levels or detect markers of iron status without the need for blood samples. Similarly, mobile applications such as Sanguina's AnemoCheck have introduced approaches to haemoglobin estimation through image analysis of fingernail beds. These non-invasive methods offer several advantages, including reduced patient discomfort, ease of use, and the potential for widespread application.

[0004] Despite these advancements, current non-invasive tools for assessing iron status have limitations. Many of these tools are device-specific, requiring access to specialized equipment that may not be readily available or affordable for all populations. Additionally, the accuracy and reliability of some non-invasive methods, particularly those based on image analysis, can be influenced by factors such aslighting conditions and skin tone, potentially affecting their performance across diverse populations.

[0005] Furthermore, existing tools often focus on single anatomical sites for assessment, which may not provide a comprehensive view of an individual's iron status. The reliance on a single site can limit the sensitivity and specificity of the assessment, potentially leading to missed diagnoses or false positives.

[0006] The problem that the invention seeks to solve is the need for a more accessible, reliable, and comprehensive method for non-invasive iron status assessment.Summary of the invention

[0007] The disclosure provides a method for determining an output anaemia condition, the method comprising:- obtaining a plurality of digital recordings, comprising at least one of the following:- a digital recording comprising static visual information of a subject’s finger nails;- a digital recording comprising static visual information of the subject’s conjunctiva;- a digital recording comprising static visual information of the subject's oral mucosa;- a digital recording comprising static visual information of the subject’s tongue; and the plurality of digital recordings comprising at least- a digital recording comprising temporal visual information of a backlit finger of the subject;- for each of the obtained digital recordings, obtaining a respective anaemia condition assessment using a respective machine learning sub-model that has been trained to detect anaemia conditions in a training set of the respective obtained digital recording,- using an ensemble machine learning sub-model to assess an output anaemia condition based on the obtained anaemia condition assessments.

[0008] As such, the method comprises obtaining a plurality of digital recordings, which include static visual information of a subject’s fingernails, conjunctiva, oral mucosa, tongue, and temporal visual information of a backlit finger. For each obtained digital recording, a respective anaemia condition assessment is obtained using a respective machine learning sub-model trained to detect anaemia conditions of the respective obtained digital recording. For the training of each machine learning sub-model, a respective training set of prepared and annotated digital recordings may be used. Anensemble machine learning sub-model then assesses an overall anaemia condition based on these assessments. This method allows for a comprehensive and non- invasive assessment of anaemia, leveraging multiple physiological indicators for a more accurate diagnosis.

[0009] The term sub-model is used here to indicate that while the sub-model will give a anaemic I non-anaemic result and is in fact a fully functional model, it may on its own not be sufficiently reliable (compared to e.g. more traditional methods such as blood analysis) and is preferably combined with the result of at least one other submodel in an ensemble model in order to boost reliability.

[0010] A digital recording can be captured with a device, such as the camera in a smartphone or other digital camera. A digital recording comprising static visual information can be a digital photo or a video frame. A digital recording comprising temporal visual information can be a sequence of video frames. These definitions are exemplary and not limiting. A skilled person can come up with other means of obtaining digital recordings and types of digital recordings that can be used in the context of this disclosure.

[0011] In this disclosure, the term “conjunctiva” will be used interchangeably with (lower) eyelid, and “oral mucosa” will be used interchangeably with lower or upper inner lip.

[0012] In an embodiment of the disclosure, at least one of the digital recordings comprising static visual information includes a visual colour reference. This inclusion enhances the accuracy of colour analysis in the assessments, compensating for variations in lighting conditions and skin tones, which can affect the performance of image-based assessments. In an embodiment, a first machine learning sub-model is trained on images that do not have a visual colour reference. Another, second, machine learning sub-model may be trained on images that do have a visual colour reference. The ensemble machine learning sub-model may be able to work with both the first and the second machine learning sub-model. A user might initially supply a digital recording without the visual colour reference, because that is easier. The ensemble model will then use the first machine learning sub-model. If the ensemble model determines that the reliability (or confidence value) is too low, it may request the use to supply a new digital recording with the visual colour reference included init. The new digital recording may then be processed using the second machine learning sub-model. In this manner, the more complicated (from the user’s perspective) way of working is only used when the easier way does not yield sufficient results.

[0013] As was mentioned, in an embodiment of the disclosure, the plurality of digital recordings are obtained using a portable device, such as a mobile phone. This significantly improves the accessibility and convenience of the method, enabling users to perform assessments in various settings without the need for specialized equipment, thus facilitating wider adoption and use.

[0014] The disclosure further provides that one or more of the plurality of machine learning sub-models comprise a locating stage, an analysing stage, and a determining stage. This structured approach to image analysis ensures a systematic and thorough evaluation of the target areas, improving the accuracy and reliability of the anaemia condition assessments.

[0015] In an embodiment, the locating stage comprises a segmentation model. This allows for precise identification and isolation of the target area in the images, which is critical for accurate colour analysis and anaemia assessment, thereby enhancing the method’s overall effectiveness.

[0016] In an embodiment, the method further comprises obtaining personal information of the subject and suggesting a relative health level based on this information along with the output anaemia condition. This personalized assessment offers the advantage of providing tailored health insights, which can be more meaningful and actionable for the user, improving the utility of the method for individual health management.

[0017] Additionally, in an embodiment the method includes providing personalized nutrition information based on the relative health level. This feature extends the utility of the method beyond mere anaemia assessment, offering users valuable guidance on dietary adjustments that can help manage or improve their condition, thereby contributing to better health outcomes. In an embodiment, the method utilizes a database of food ingredients and linked values indicating to what extent said ingredients can reduce anaemia. The database may be adapted to also include atleast one of user preferences for food ingredients, food ingredient prices, food ingredient availabilities, and ease of preparation. The method may be adapted to use the database to provide food suggestions to improve anaemia conditions that are optimized for at least one of price, user preference, availability, and ease of preparation.

[0018] In another embodiment, the use of a machine learning model comprises communicating with a remote server on which the machine learning model is hosted. This facilitates the processing of complex data and the use of advanced machine learning algorithms without requiring extensive computing resources on the user’s device, enhancing the method’s scalability and performance.

[0019] In an embodiment, the ensemble machine learning model is based on Bayesian Model Analysis or Bayesian regression.

[0020] The disclosure also provides a computing device configured to implement the method, and specifically, a computing device in the form of a mobile phone. This highlights the method’s integration into widely used consumer electronics, making it highly accessible and convenient for a broad user base.

[0021] The disclosure also provides a computer program product comprising instructions, which, when executed on a computing device, causes the device to perform the method, is provided. This encapsulates the method in a form that can be easily distributed and utilized, further enhancing its accessibility and potential for widespread adoption.Brief description of the figures

[0022] The attached figures illustrate various aspects of the disclosed invention, providing detailed examples of the system and method for determining an output anaemia condition using digital recordings and machine learning models, wherein:- Figure 1 schematically shows an ensemble model according to the invention, highlighting the integration of multiple sub-models for assessing anaemia conditions from different anatomical sites;- Figure 2a provides a detailed schematic of one of the sub-models, showcasing the stages involved in processing a digital recording for anaemia condition assessment;- Figure 2b illustrates a simplified version of the sub-model shown in Figure 2a, with a streamlined process for extracting and processing information from a digital recording;- Figure 2c depicts a further simplified version of the model in Figure 2b, demonstrating direct processing of a digital recording by a machine learning model;- Figures 3a, 3b, 3c, and 3d each show a digital recording for a different anatomical model, illustrating the process of identifying and segmenting target areas and reference objects for anaemia assessment;- Figure 4 presents a fifth sub-model designed to analyse a video recording of a backlit digit, detailing the components involved in determining an anaemia indication from changes in blood flow and colouration;- Figure 5 shows a video recording for a backlit digit; and- Figure 6 schematically shows an iterative use of sub-models to calculate an result with sufficient confidence.Detailed description of the figures

[0023] Figure 1 schematically shows an ensemble model according to the invention, highlighting the integration of multiple sub-models for assessing anaemia conditions from different anatomical sites. The ensemble model, designated as model 100, is a system designed to aggregate and analyse data from up to five distinct anatomical sites, including the subject’s fingernails 10a, eyelid 10b, tongue 10c, lower or upper lip (oral mucosa) 10d, and digit 10e.

[0024] The digital recording of each body part, denoted as 10a through 10e, is processed by its corresponding machine learning sub-model, 11a through 11e (submodel 11 a to segment and process fingernail images, 11 b for eyelid images, 11 c for tongue images, 11 d for oral mucosa images, and 11e for videos of digits), to yield a respective intermediate result, 12a through 12e. These results are then input into a combiner model, item 13, which may employ a Bayesian approach to synthesize a comprehensive output, 14, indicating the overall anaemia condition. This ensemble approach allows for an improved assessment by considering multiple physiological indicators of anaemia, thereby enhancing the accuracy and reliability of the diagnosis. The use of multiple anatomical sites addresses the limitations of single-site assessments and provides a more holistic view of the subject’s iron status. An alternative to the Bayesian model could involve other statistical or machine learning methods capable of integrating diverse data types to produce a unified output.

[0025] As mentioned above, the model 100 integrates the outputs from up to five distinct sub-models, each trained on digital recordings from specific body parts: fingernails, eyelid, tongue, lip, and digit. These sub-models, labelled from 11a to 11e, process their respective digital recordings, denoted as 10a to 10e, to produce intermediate anaemia indication results (12a to 12e). The ensemble model employs a combiner model, item 13, potentially utilizing a Bayesian approach, to synthesize these intermediate results into a comprehensive output, 14, indicating the overall anaemia condition.

[0026] It is generally not necessary to include all five sub-models. Any one of the 5 sub-models, any combination of 2 sub-models (10 permutations), or any combination of 3 sub-models (also 10 permutations possible), or any combination of 4 sub-models (5 permutations possible) can also deliver suitable results. Applicant has found that a good result is generally achieved by combining at least one of the sub-models Hal id (that is, any one of the sub-models that detect anaemia in static images of fingernail(s), eyelid, tongue, and oral mucosa) and sub-model 11e (that is, the submodel that detects anaemia in video images of a digit).

[0027] Each sub-model within the ensemble can be exemplified by Convolutional Neural Networks (CNNs), which are particularly adept at handling image data. CNNs automatically and adaptively learn spatial hierarchies of features from images, making them suitable for tasks such as segmenting the target area of visual information in an image and analysing the colour of the target area with reference data.

[0028] The disclosure also considers the application of Vision Transformers (ViTs) as an alternative or complementary approach to Convolutional Neural Networks (CNNs) in the assessment of anaemia conditions from digital recordings. ViTs, leveraging the transformer architecture originally developed for natural language processing, have demonstrated significant potential in various computer vision tasks, including those relevant to medical image analysis.

[0029] The method for determining an output anaemia condition can be enhanced by incorporating ViTs in the analysis of digital recordings obtained from various anatomical sites. Unlike CNNs, which process images through the application of convolutional filters, ViTs divide the input image into a sequence of patches andlinearly embed these patches into vectors. This process, followed by the addition of positional embeddings, allows ViTs to preserve spatial information and utilize selfattention mechanisms to model dependencies between patches. Such capabilities can be particularly advantageous in identifying and analyzing subtle features indicative of anaemia in images of fingernails, conjunctiva, oral mucosa, tongue, and (backlit) fingers.

[0030] For instance, in the analysis of digital recordings of the subject’s fingernails, a ViT could first segment the image into patches corresponding to the fingernail area and its surroundings. Through self-attention mechanisms, the ViT would then model the relationships between these patches, effectively capturing both local and global features relevant to the assessment of anaemia. This approach could lead to a more nuanced analysis, improving the accuracy of anaemia condition assessments compared to methods relying solely on CNNs.

[0031] Moreover, the flexibility of ViTs in handling different image sizes and the ability to capture long-range dependencies between image patches make them well-suited for the comprehensive analysis required in the disclosed method. The ensemble machine learning model, which assesses the overall anaemia condition based on individual assessments from various anatomical sites, may benefit from the integration of ViTs by leveraging their capability to extract and synthesize complex features from diverse image types.

[0032] In addition to the methodological advantages, the use of ViTs also facilitates the implementation of the proposed method on a wide range of computing devices. Given that ViTs can be efficiently executed on modern hardware, including those with limited computational resources such as mobile phones, their incorporation into the method could enhance its accessibility and usability in various settings, from clinical environments to remote or resource-limited areas.

[0033] Therefore, the method for determining an output anaemia condition may include using Vision Transformers for the analysis of digital recordings in addition to or instead of Convolutional Neural Networks (CNNs). While in the following, reference will be made to CNNs, it is to be understood that wherever CNNs are mentioned as examples, this example could also utilize a ViT instead of or in combination with the CNN.

[0034] Turning back to the CNN example for now, in the sub-model processing digital recordings of the subject’s fingernails (11a), a CNN can first segment the fingernail area from the rest of the image. This involves identifying the edges and textures unique to fingernails and differentiating them from the background. Following segmentation, the CNN can analyse the colour characteristics of the fingernail area, comparing it to reference data to assess signs of anaemia.

[0035] The output layer of these CNNs can utilize a softmax function, which is particularly useful for classification tasks like determining the anaemia condition. The softmax function converts the raw output scores from the network (often called logits) into probabilities by taking the exponential of each output and then normalizing these values by dividing by the sum of all the exponentials. This ensures that the output values are in the range (0,1) and sum up to 1 , making them interpretable as probabilities.

[0036] For example, if a CNN sub-model analysing a fingernail image produces logits indicating varying levels of confidence in the “anemic” and “non-anemic” classes, the softmax function can convert these into a probability distribution. If the softmax output for “anemic” is 0.8 and for “non-anemic” is 0.2, it indicates a higher probability (or confidence) of the subject being anemic based on the fingernail image analysis.

[0037] In an alternative embodiment, the level of haemoglobin can be estimated using a regression model. Combining such a level with a threshold value can be an indicator of degrees of an anemic or non-anemic state.

[0038] As an example, consider a scenario where the ensemble model is assessing anaemia based on digital recordings from the specified anatomical sites. Each CNN sub-model processes its respective image and outputs a softmax probability distribution over the “anemic” and “non-anemic” classes. For instance:- Sub-model for fingernails (11a) outputs a softmax probability of 0.75 for “anemic” and 0.25 for “non-anemic”.- Sub-model for digit (11 e) outputs a softmax probability of 0.65 for “anemic” and 0.35 for “non-anemic”.

[0039] The posterior probabilities of the models are used as weights when averaging the predictions of each model. Given that we have two sub-models, the posterior probabilities of sub-model 11a and sub-model 11e are denoted as P(M1) and P(M2), respectively. The overall probabilities for “anemic” and for “non-anemic” can then be calculated as follows: for “anemic”: 0.75-P(M1)+0.25-P(M2) and for “non-anemic”: 0.65-P(M1)+0.35-P(M2), P(M1) + P(M2) = 1 as they are probabilities and must sum up to 1. Also, x + y = 1 as “anemic” and “non-anemic” are mutually exclusive events. The exact values of P(M1) and P(M2) depend on the prior probabilities of the submodels and the data at hand. They can be calculated using Bayes’ theorem if the necessary information is available. If the sub-models 11a and 11e, in this example, are assumed to be equally likely, then P(M1) = P(M2) = 0.5.

[0040] These probabilities serve as the likelihood indications for the Bayesian ensemble model. Utilizing the Bayesian model averaging (BMA), the ensemble model combines these probabilities, weighted by their posterior probabilities derived from the performance and reliability of each sub-model on validation data. This Bayesian approach allows the ensemble model to account for the uncertainty in each submodel’s predictions, leading to a more confident and robust overall anaemia condition assessment.

[0041] In addition to Bayesian Model Averaging (BMA), Bayesian regression offers an alternative or complementary approach for assessing anaemia conditions based on digital recordings. While BMA focuses on averaging over models, Bayesian regression applies Bayesian methods to estimate the parameters of a regression model, offering a structured way to incorporate prior knowledge and uncertainty in the analysis.

[0042] Bayesian regression allows for the incorporation of prior knowledge about the expected relationships between visual indicators in the digital recordings and anaemia conditions. For example, prior clinical studies indicating the correlation between certain colour characteristics in fingernails, conjunctiva, oral mucosa, tongue, and backlit finger videos with iron deficiency levels can be encoded as prior distributions in the Bayesian regression model. This enhances the model’s ability to make informed predictions even when faced with limited or noisy data.

[0043] In addition, Bayesian regression provides a natural framework for dealing with uncertainty. In the assessment of anaemia conditions from digital recordings, varioussources of uncertainty can arise, such as variations in lighting conditions, skin tones, and the quality of the digital recordings. Bayesian regression models can explicitly account for these uncertainties by treating model parameters as random variables with distributions that reflect our uncertainty about their true values. This approach allows for more robust and reliable anaemia condition assessments, as the model’s predictions include a measure of uncertainty or confidence intervals.

[0044] Implementing Bayesian regression in the context of the disclosed method may involve defining a likelihood function that describes how the observed digital recordings are generated from the underlying anaemia conditions, given certain model parameters. The prior distributions for these parameters may encapsulate existing knowledge or assumptions about the expected relationships between visual indicators and anaemia conditions. The posterior distributions of the model parameters, obtained via Bayes’ theorem, can then inform the ensemble machine learning model’s assessment of the overall anaemia condition.

[0045] For example, a Bayesian regression model may be used to predict the level of haemoglobin or the severity of anaemia from the colour metrics extracted from the digital recordings of the subject’s fingernails, conjunctiva, oral mucosa, tongue, and backlit finger. The ensemble machine learning model may then integrate these predictions, taking into account the posterior uncertainties, to provide an assessment of the subject’s anaemia condition.

[0046] The integration of Bayesian regression into the ensemble machine learning model offers several advantages. It enhances the model’s ability to leverage multiple anatomical indicators for a comprehensive and accurate assessment of anaemia, overcoming the limitations of current methods that rely on single-site assessments. Moreover, the probabilistic nature of Bayesian regression allows for the explicit consideration of uncertainty in the predictions, leading to more informed and reliable anaemia condition assessments.

[0047] The ensemble model’s output, therefore, is not merely an aggregation of the sub-models’ outputs but a synthesis that considers the confidence and reliability of each sub-model, offering a comprehensive and accurate assessment of the anaemia condition. This approach addresses the limitations of single-site assessments andprovides a holistic view of the subject’s iron status, enhancing the accuracy and reliability of the diagnosis.

[0048] In an embodiment, not shown in figure 1 , the ensemble model is connected to a database of food ingredients and linked values indicating to what extent said ingredients can reduce anaemia. The database may be adapted to also include at least one of user preferences for food ingredients, food ingredient prices, food ingredient availabilities, and ease of preparation. The model may be configured to use the database to provide food suggestions to improve anaemia conditions that are optimized for at least one of price, user preference, availability, and ease of preparation.

[0049] Figure 2a provides a schematic of one of the sub-models 11a-11d, showcasing the stages involved in processing a digital recording 10 for anaemia condition assessment. This figure illustrates the process starting with the segmentation stages, 20a and 20b, dedicated to isolating the target body part and a reference object within the digital recording, respectively. Following segmentation, the preprocessor stages, 21a and 21b, prepare the segmented images for analysis. Stage 21a adjusts the colour and luminance of the body part image based on reference data, while 21 b establishes a reference colour vector or correction function from the segmented reference object. The processed images are then analysed by a machine learning model, 22, which determines an intermediate anaemia indication result. This machine learning model can be a CNN or ViT, which, as was noted, is suited for image processing. Other machine learning models can be used as well, however.

[0050] A special form of the sub-models of figure 2a omits the segmentation stage 20b, in other words: the sub-model does not rely on the presence of a reference object within the digital recording. In that case, the stages 21a and 21b may work with other data to, respectfully, adjust the colour and luminance of the body part image and obtaining the reference colour vector or correction function for said adjustment. That other data can come from an overall analysis of the image and it’s colours. It may, in addition or instead come from external data, such as EXIF data of the used image capture device.

[0051] It is advantageous to train for each sub-model a version with and without reference object in the image data. Sub-models with such a reference object willgenerally have a higher confidence, but it may well be that the sub-models without the reference object also provide sufficient confidence. For the user, it is easier to obtain images without the reference object than with it. This is further explained in the embodiment of figure 6.

[0052] Figure 2b illustrates a simplified version of the sub-model shown in Figure 2a, with a streamlined process for extracting and processing information from a digital recording. In this configuration, a single segmenter, 20, performs the dual function of extracting both the target body part and any necessary reference information from the digital recording. The simplified preprocessor stage, 22, then prepares the image for analysis by the machine learning model, which assesses the anaemia condition. This simplified model may be particularly advantageous in scenarios where computational resources are limited or when rapid assessments are required. As an alternative, the model could incorporate adaptive segmentation techniques to improve efficiency in identifying the target areas within the digital recordings.

[0053] Figure 2c depicts a further simplified version of the model in Figure 2b, demonstrating direct processing of a digital recording by a machine learning model. This model represents the most streamlined approach, where the machine learning model, 22, is trained to perform segmentation, preprocessing, and anaemia assessment tasks in a single step. By eliminating separate segmentation and preprocessing stages, this model can potentially offer faster processing times, making it well-suited for applications where immediate results are essential. A drawback can be that it can be considerably harder to train such an all-in-one model, which incorporates the previously separated steps of segmenting and preprocessing. This alternative could involve the use of deep learning techniques to enhance the model’s ability to directly analyse raw digital recordings without the need for explicit segmentation or preprocessing steps.

[0054] Figures 3a, 3b, 3c, and 3d each show a digital recording for a different anatomical model, illustrating the process of identifying and segmenting target areas and reference objects for anaemia assessment. Each figure demonstrates the application of the segmentation stages, 20a and 20b, to isolate the specific body part (fingernails 31a, eyelid 31 b, tongue 31c, and lower lip 31 d, respectively) and any included reference object 32a, 32b, 32c, 32d within the digital recording. Thesesegmented images are used for the accurate analysis of anaemia indicators, such as colour and luminance changes associated with iron deficiency.

[0055] As was mentioned already in the context of figures 2a-2c, it is important to note that the reference object is optional. When provided, it offers the model a chance to calibrate colours and luminance, improving the detection results. However, a model can also be trained to estimate the calibration from other parts of the digital recording is taken, or from metadata such as the time of day the digital recording and whether or not it is likely recorded outside in daylight conditions or inside under artificial light. The model can be trained to take Exchangeable Image File (EXIF) data from the digital recording as input, to provide information on the conditions in which the recording was captured.

[0056] By incorporating the optional visual references in the digital recordings, as shown in these figures, the method can more effectively compensate for variations in environmental conditions, ensuring consistent and reliable assessments across different settings. An alternative approach could involve the use of augmented reality (AR) technology to assist users in capturing optimal digital recordings by guiding the placement of the reference object and ensuring proper lighting conditions.

[0057] A training set for each of the four sub-models 11a - 11d can comprise a set of images with annotations indicating the segmented relevant subject’s parts (respectively the fingernail(s), eyelid, tongue and inner lip) and optionally the segmented colour reference, if present. The annotation also includes an anaemia classification, as obtained by a reliable method (e.g. using a blood sample and laboratory test). The training of each of the four sub-models using the training set for each sub-model is then as known to a skilled person in the art of machine learning and dependent on the implementation details of the sub-models.

[0058] Figure 4 presents a fifth sub-model 11e designed to analyse a video recording of a backlit digit, detailing the components involved in determining an anaemia indication from changes in blood flow and colouration. This approach leverages video analysis to observe the dynamic changes in the digit’s reddishness caused by the subject’s heartbeat, offering a perspective on blood flow characteristics that can be indicative of anaemia.

[0059] The model comprises an image extractor, 41 , temporal intensity and colour detectors, 42 and 43, and a colour vector composer, 44, which collectively process the video data to extract relevant features. These features are then analysed by a trained sub-model, 45, to produce an intermediate anaemia indicator, 12e. This method’s ability to capture temporal variations offers a potentially more sensitive indicator of anaemia than static images alone.

[0060] This embodiment assess the presence and severity of anaemia through dynamic analysis of blood flow and colouration changes in a backlit digit, typically a finger. This method capitalizes on the principle that the colour and intensity of light passing through the skin, modulated by the pulsatile nature of blood flow due to the heartbeat, can reveal critical information about the blood’s haemoglobin content.

[0061] The process begins with the capture of a video recording of the subject’s digit, illuminated from behind by a bright light source. This setup is for enhancing the visibility of the colour hue and intensity. The heartbeat will also affect the colour and brightness of the video images captured over time. The image extractor component, 41 , then processes the video to isolate frames that are most indicative of these changes, effectively filtering out irrelevant data and focusing on moments where blood flow dynamics are most apparent.

[0062] Subsequently, the temporal intensity and colour detectors, 42 and 43, analyse the extracted frames to quantify the changes in colour and intensity over time. These detectors are designed to identify the specific shades of redness and their variations that are characteristic of blood flow through the digit. The analysis is sensitive to fluctuations in colour and brightness that correspond to the pulsatile blood flow, and more specifically to fluctuations in intensity and hue which may be indicative of haemoglobin levels and, by extension, the presence of anaemia.

[0063] The colour vector composer, 44, then synthesizes the detected changes into a comprehensive colour vector. This vector encapsulates the temporal variations in colour and intensity, providing a profile of the blood flow characteristics through the digit. This vector translates the raw data into a format that can be effectively analysed for anaemia detection.

[0064] The trained sub-model, 45, equipped with knowledge from a dataset of individuals with known haemoglobin levels and anaemia status, analyses the colour vector. By comparing the subject’s colour vector against patterns derived from the dataset, the sub-model assesses the likelihood of anaemia. This analysis leverages machine learning techniques, based on e.g. CNNs or ViTs as disclosed earlier in this disclosure, to discern subtle patterns in the colour vector that correlate with anaemia, making it possible to detect the condition non-invasively.

[0065] Finally, the sub-model outputs an intermediate anaemia indicator, 12e, which reflects the model’s assessment of the subject’s anaemia status based on the video analysis. This indicator is then integrated with assessments from at least one other digital recording of the subject, using an ensemble machine learning model, to determine a comprehensive output anaemia condition.

[0066] By analysing the dynamic changes in blood flow and colouration in a backlit digit, the system can potentially uncover subtle signs of anaemia that static images or traditional non-invasive methods might miss. Applicant has found that it is particularly advantageous to combine the video based method with a static image method.

[0067] A training set for sub-models 11 e can comprise a set of videos with annotations indicating the segmented relevant subject’s digit. The annotation also includes an anaemia classification, as obtained by a reliable method (e.g. using a blood sample and laboratory test). The training of the sub-model using the training set then as known to a skilled person in the art of machine learning and dependent on the implementation details of the sub-model.

[0068] Figure 5 shows a video recording for a backlit digit 51. The centre of the recorded digit is shown as 52.

[0069] Figure 6 schematically shows an iterative use of sub-models to calculate an result with sufficient confidence. Initially, in step 61 , the user provides digital recordings for n sub-models, for example a static image of fingernails for sub-model 11a and a video of a backlit digit for sub-model 62 (n=2). Any of the other possible permutations of n=1 to 4 sub-models is also possible. In step 62, the ensemble result is calculated using e.g. BMA or Bayesian regression, as explained earlier. In step 63, the assessment is made whether or not the confidence in the ensemble result issufficient to proceed. For example, this can be done by comparing a confidence estimate of the ensemble result with a threshold value. If the confidence is sufficient, then the method proceeds to step 64 and an anaemia result is established.

[0070] If the confidence is deemed insufficient, the method proceeds to step 66. In this step, the method provides a recommendation (or multiple recommendations) to the user for obtaining a better result. Such a recommendation can be to provide an image for one of the sub-models not yet included in the initial n sub-models. It can also be a recommendation to repeat one of the n sub-models, but to now include a reference objection in order to have a better confidence value for the sub-model. For example, when BMA is used, a sub-model M with a reference object may have a higher a posteriori likelihood P(M) than the same model without the reference object. In step 65, the user provides the additional digital recording and the additional submodel is evaluated. Then the method returns to step 62 where the result is now calculated for the ensemble of n+1 sub-models, and again the confidence of the ensemble is evaluated in step 63. Alternatively, in step 62 the result may be calculated for n sub-models, where the most recent digital recording replaces an earlier digital recording with a low confidence value.

[0071] In the foregoing description of the figures, the invention has been described with reference to specific embodiments thereof. It will, however, be evident that various modifications and changes may be made thereto without departing from the scope of the invention as summarized in the attached claims.

[0072] In addition, many modifications may be made to adapt a particular situation or material to the teachings of the invention without departing from the essential scope thereof. Therefore, it is intended that the invention not be limited to the particular embodiments disclosed, but that the invention will include all embodiments falling within the scope of the appended claims.

[0073] In particular, combinations of specific features of various aspects of the invention may be made. An aspect of the invention may be further advantageously enhanced by adding a feature that was described in relation to another aspect of the invention.

[0074] It is to be understood that the invention is limited by the annexed claims and its technical equivalents only. In this document and in its claims, the verb “to comprise” and its conjugations are used in their non-limiting sense to mean that items following the word are included, without excluding items not specifically mentioned. In addition, reference to an element by the indefinite article “a” or “an” does not exclude the possibility that more than one of the element is present, unless the context clearly requires that there be one and only one of the elements. The indefinite article “a” or “an” thus usually means “at least one”.

Claims

Claims1. Method for determining an output anaemia condition, the method comprising:- obtaining a plurality of digital recordings, comprising at least one of the following:- a digital recording comprising static visual information of a subject’s finger nails;- a digital recording comprising static visual information of the subject’s conjunctiva;- a digital recording comprising static visual information of the subject's oral mucosa;- a digital recording comprising static visual information of the subject’s tongue; and the plurality of digital recordings comprising at least- a digital recording comprising temporal visual information of a backlit finger of the subject;- for each of the obtained digital recordings, obtaining a respective anaemia condition assessment using a respective machine learning sub-model that has been trained to detect anaemia conditions in the respective obtained digital recording,- using an ensemble machine learning model to assess an output anaemia condition based on the obtained anaemia condition assessments.

2. Method according to claim 1 , wherein the at least one digital recording comprising static visual information includes a visual colour reference.

3. Method according to claim 1 or 2, wherein the plurality of digital recordings are obtained using a portable device, such as a mobile phone.

4. Method according to any one of the previous claims, wherein one or more of the respective machine learning sub-models comprises:- a locating stage for locating a target area of visual information in an image;- an analysing stage for analysing and comparing the colour of the target area with reference data;- a determining stage for determining the level of anaemia.

5. Method according to claim 4, wherein the locating stage comprises a segmentation model.

6. Method according to any one of the preceding claims, further comprising:- obtaining personal information of the subject, said information including one or more of age, gender, weight, and length;- obtaining reference population information;- suggesting a relative health level based on the personal information of the subject, the reference population information, and the output anaemia condition.

7. The method according to claim 6, further comprising:- providing personalized nutrition information based on the relative health level.

8. The method according to claim 7, wherein providing personalized nutrition information based on the relative health level comprises accessing a local or remote database with nutrition information.

9. The method according to any one of the previous claims, wherein using a machine learning sub-model comprises communicating with a remote server on which the machine learning sub-model is hosted.

10. The method according to any one of the previous claims, wherein the ensemble machine learning model is based on Bayesian Model Analysis or Bayesian regression.

11. The method according to any one of the previous claims, wherein one or more of the sub-models comprise a Convolutional Neural Network, CNN, or a Visual Transformer, ViT.

12. The method according to any one of the previous claims, wherein the method makes a recommendation to provide an additional digital recording in case a confidence value of the output anaemia condition is below a predetermined value.

13. The method according to claim 12, comprising obtaining an additional anaemia condition assessment using a respective machine learning sub-model, and updating the output anaemia condition using the additional anaemia condition assessment.

14. Computing device configured to implement the method of any one of claims 1 - 13.

15. Computing device according to claim 14, in the form of a mobile phone.

16. Computer program product comprising instructions which, when executed on a computing device, causes said computing device to perform the method of any one of claims 1 - 13.

Images