System and method for obtaining a biosignal from video sequence

Abstract

The invention relates to a device for training a signal analysis machine learning model to be used for obtaining a global photoplethysmography PPG signal of at least a portion of a skin surface of a user comprised in a video sequence.

Description

SYSTEM AND METHOD FOR OBTAINING A BIOSIGNAL FROM VIDEO SEQUENCEFIELD OF INVENTION

[0001] The present invention relates to the field of medical informatics, particularly to remote photoplethysmography (rPPG), wherein physiological data is obtained from a video sequence of a user's skin surface. The present invention more particularly relates to a system for obtaining a global PPG signal of at least one portion of a skin surface of a user from a video sequence of the at least one portion of a skin surface of the user, and a corresponding computer-implemented method.BACKGROUND OF INVENTION

[0002] rPPG techniques are employed to measure cardiovascular parameters non- invasively by capturing optical variations of the skin due to cardiac activity. Conventional techniques run the entire rPPG pipeline either entirely on the device or on the cloud. Running the pipeline on the device may lead to challenges regarding the protection of the algorithms and limitations on the possible algorithms that can be used depending on the user's device capabilities. Conversely, running the pipeline on the cloud can lead to privacy issues and bandwidth constraints due to the transfer of raw image data to a remote server.

[0003] Further, methods transmitting average color values per image over the network for rPPG processing lack the capacity to retain useful information about the distribution of color on the face, which is crucial for isolating the physiological signal of interest from the noise.

[0004] Thus, there is a need for a system and computer-implemented method which solve the abovementioned drawbacks.SUMMARY

[0005] A first aspect of the present invention concerns device for training a signal analysis machine leaning model to be used for obtaining a global PPG signal of at least one portion of a skin surface of a user comprised in a video sequence, said device comprising: at least one input configured to receive a global training dataset comprising a plurality of global training samples associated to a plurality of subjects, each global training sample comprising, for one subject, at least one subject video sequence of at least one portion of skin surface of said subject and at least one reference photoplethysmography (PPG) signal, said at least one video sequence and said at least one reference PPG signal being acquired in one same time window; at least one processor configured to obtain said trained signal analysis machine learning model by training a machine learning architecture using said global training dataset, wherein training comprises iterating till convergence, for each global training sample, the following steps:• for each frame of said at least one subject video sequence, apply a mesh on the frame to define regions of interest on said at least one portion of skin surface, wherein each region of interest comprises at least one point and each point is associated to a value,• for each frame of said video sequence and for each region of interest, compute a raw trace value based on at least said value of at the least one point in said region of interest,• for each region of interest: compute over the frames of the at least one subject video sequence, a raw trace signal of the subject video sequence associated to said region of interest using said raw trace values computed for said region of interest, compute a raw photoplethysmography (PPG) signal based on said raw trace signal associated to said region of interest, obtain a PPG quality signal representative of the quality of the raw PPG signal associated to said region of interest by feeding said rawphotoplethysmography (PPG) signal as input to said machine learning architecture, compute a processed PPG signal based on said PPG quality signal and the raw PPG signal associated to said region of interest,• compute a global PPG signal of the at least one portion of the skin surface of the subject based on the processed PPG signals computed for said regions of interest,• modify the machine learning architecture parameters to minimize a loss function configured to compare said global PPG signal to the at least one reference PPG signal of the global training sample.

[0006] According to an alternative embodiment, the device for training comprises: at least one input configured to receive a labelled training dataset comprising a plurality of labelled training samples associated to a plurality of subjects, each labelled training sample comprising, for one subject, at least one labelled subject video sequence of at least one portion of skin surface of said subject, wherein each labelled frame of each labelled subject video sequence is labelled by defining regions of interest on said at least one portion of skin surface and associating to each region of interest a weight; at least one processor configured to obtain said trained signal analysis machine learning model by training a machine learning architecture using said labelled training dataset, wherein training comprises iterating till convergence, for each labelled training sample, the following steps:• for each frame of said at least one subject video sequence, apply a mesh on the frame to define regions of interest on said at least one portion of skin surface, wherein each region of interest comprises at least one point and each point is associated to a value,• for each frame of said video sequence and for each region of interest, compute a raw trace value based on at least said value of at the least one point in said region of interest,• for each region of interest:compute over the frames of the at least one subject video sequence, a raw trace signal of the subject video sequence associated to said region of interest using said raw trace values computed for said region of interest, compute a raw photoplethysmography (PPG) signal based on said raw trace signal associated to said region of interest, obtain a PPG quality signal representative of the quality of the raw PPG signal associated to said region of interest by feeding said raw photoplethysmography (PPG) signal as input to said machine learning architecture, compute a processed PPG signal based on said PPG quality signal and the raw PPG signal associated to said region of interest,• modify the machine learning architecture parameters to minimize a loss function configured to compare said processed PPG signal, for each region of interest, to reference processed PPG signals obtained from the labelled subject video sequence of the labelled training sample.

[0007] According to one embodiment, the machine learning architecture is ID convolutional neural network.

[0008] An object of the invention is a system for obtaining a biosignal of a user from a video sequence of at least one portion of a skin surface of the user. This system incorporates machine learning models to handle mesh application and raw PPG signal segmentation. The invention preserves user privacy and reduces computational load and bandwidth requirements for data transmission, ensuring a more reliable rPPG signal and consistent semantics for regions of interest (RO I). The system is designed to compute biosignals like heart rate, oxygen saturation, respiratory rate, heart rate variability, and arterial pressure.

[0009] Another aspect of the present invention concerns a system for obtaining a global PPG signal of at least one portion of a skin surface of a user from a video sequence of said at least one portion of a skin surface of said user using a signal analysis machine leaning model obtained from a device described above, said system comprising:at least one input configured to receive the video sequence of said at least one portion of skin surface of the user acquired from a camera, at least one processor configured to:• for each frame of said video sequence, apply a mesh on the frame to define regions of interest on said at least one portion of skin surface, wherein each region of interest comprises at least one point and each point is associated to a value,• for each frame of said video sequence and for each region of interest, compute a raw trace value based on at least said value of at the least one point in said region of interest,• for each region of interest: compute over the frames of the video sequence, a raw trace signal of the video sequence associated to said region of interest using said raw trace values computed for the region of interest, compute a raw photoplethysmography (PPG) signal based on said raw trace signal associated to said region of interest, obtain a PPG quality signal representative of the quality of the raw PPG signal associated to said region of interest using the signal analysis machine learning model feed in with said raw photoplethysmography (PPG) signal, compute a processed PPG signal based on the PPG quality signal and the raw PPG signal associated to said region of interest, so as to reduce contribution of low-quality raw PPG signals in said computed processed PPG signals,• compute a global PPG signal of the at least one portion of the skin surface of the user based on the processed PPG signals computed for said regions of interest, at least one output configured to provide said global PPG signal.

[0010]

[0011] According to one embodiment, the present invention relates to a system for obtaining a global PPG signal of at least one portion of a skin surface of a user from avideo sequence of said at least one portion of a skin surface of said user, said system comprising: at least one input configured to receive a video sequence of a skin surface of the user acquired from a camera, at least one processor configured to:• apply a mesh on the frames of said video sequence to define regions of interest on said at least one portion of a skin surface, each region of interest comprising at least one point,• compute for each frame and for each region of interest, a raw trace value based on the value of at the least one point in said region of interest,• for each region of interest, compute, over the video sequence, a raw trace signal of the video sequence in said region of interest using said raw trace values,• compute, for each region of interest, raw photoplethysmography (PPG) signals based on the raw trace signals,• segment each of the obtained raw PPG signals using a pre-trained machine learning model to obtain a PPG quality signal representative of the quality of the raw PPG signal,• compute, for each region of interest, processed PPG signals based on the respective PPG quality signal and the associated raw PPG signal, so as to reduce contribution of low-quality raw PPG signal in said computed processed PPG signals,• compute a global PPG signal of the portion of the skin surface of the user based on the processed PPG signals, at least one output configured to provide said global PPG signal.

[0012] According to one or several embodiments, the at least one processor may be further configured to compute at least one biosignal of the user based on the global PPG signal.

[0013] According to one or several embodiments, the at least one output may be further configured to output the at least one biosignal.

[0014] The claimed system allows for remote, non-contact global PPG signal (or biosignal) monitoring of a user, reducing the need for wearable sensors and enhancing user comfort. The use of video sequences enables dynamic and regular monitoring of a user's health status, providing a real-time health overview. The application of a mesh on video frames for defining regions of interest optimizes the analysis of photoplethysmographic data, improving the accuracy of the captured biosignals. The computation of raw PPG signals and subsequent segmentation using a machine learning model allows for quality assessment of the PPG signal, enhancing reliability of the resulting biosignal. The system's ability to process raw PPG signals into a global PPG signal and subsequently compute a biosignal provides comprehensive health metrics in a single integrated system. The system’s output functionality extends its utility by allowing the integration of the processed biosignal into various health monitoring applications, medical devices, or electronic health records.

[0015] According to an embodiment, the skin surface of the user may comprise at least part of a face of the user.

[0016] According to an embodiment, the mesh may be a dynamic mesh applied on the frames using a pre-trained machine learning model such that the face is tracked and regions of interest are consistent along the video sequence, the machine learning model being trained on a dataset of face images labelled with a specific mesh.

[0017] This ensures consistent regions of interest along the video sequence, improving the quality and consistency of the PPG signal. The system ensures tracking consistency across video frames, improving data reliability over time and motion scenarios. It maximizes region of interest relevancy, thereby reducing errors from movement or position change.

[0018] According to an embodiment, the pre-trained machine learning model used to segment each of the obtained raw PPG signals is a ID convolutional neural network, trained on a dataset of labelled PPG signals.

[0019] According to an embodiment, the processed PPG signals may be computed by multiplying the PPG quality signals and the corresponding PPG raw signals with each other.

[0020] According to an embodiment, the PPG quality signals may be thresholded, and the processed PPG signals are computed by multiplying the obtained thresholded PPG quality signals and the corresponding PPG raw signals with each other.

[0021] According to an embodiment, the biosignal may be one of the following: heart rate, oxygen saturation, respiratory rate, heart rate variability, diastolic blood pressure, systolic blood pressure, cardiac workload, hemoglobin level, glycemia.

[0022] According to an embodiment, the biosignal is heart rate, and the heart rate is computed using frequential analysis of the PPG signal.

[0023] According to an embodiment, the biosignal is heart rate, and the heart rate is computed using temporal analysis of the PPG signal.

[0024] The use of frequential analysis of the PPG signal for heart rate computation provides a robust and reliable method for determining the heart rate, contributing to the precision of cardiovascular monitoring.

[0025] According to an embodiment, the raw PPG signals may be obtained by filtering the raw trace signals, said filtering being adapted to the biosignal to obtain. The raw PPG signals may also be normalized.

[0026] The computation of raw trace signals by filtering, and specifically averaging, the values of the points in the region of interest, provides a robust method to mitigate signal noise and improve signal clarity.

[0027] According to an embodiment, the system may comprise a first processor and a second processor distinct and remote from the first processor, wherein the first processor is configured to: apply the mesh on the frames of the video sequence to define regions of interest on said at least one portion of skin surface, compute for each frame and for each region of interest, the raw trace value based on the value of the at least one point in saidregion of interest, for each region of interest, compute, over the video sequence, the raw trace signal of the video sequence in said region of interest using said raw trace values; wherein the first or the second processor is configured to compute, for each region of interest, the raw PPG signals based on the raw trace signals; and wherein the second processor is configured to: segment each of the received raw PPG signals using a pretrained machine learning model to obtain the PPG quality signal representative of the quality of the raw PPG signal, compute the processed PPG signals for each region of interest based on the respective PPG quality signals and the associated raw PPG signals, compute the global PPG signal of the skin surface of the user based on the processed PPG signals, and compute said biosignal based on said global PPG signal of the portion of the skin surface of the user.

[0028] In an example, the step of computing, for each region of interest, the raw PPG signals based on the raw trace signals, may be performed either by the first or the second processor. The selection of the processor for performing this step may be carried out dynamically, e.g. based on the bandwidth available for data communication between the first and the second processor, and / or based on the computational power available at the first processor. In other words, a computer-implemented method using a system as described above may also comprise a step of selecting the best processor to carry out the step of computing, for each region of interest, the raw PPG signals based on the raw trace signals.

[0029] This system allows for better privacy as the image of the skin of the user is never sent to the second remote processor, while reducing computational resources on the first local processor. The division of computational tasks between a first and second processor promotes efficient use of computational resources, improving the speed and reliability of biosignal computation. The remote nature of the second processor allows for scalability and distribution of processing load, enabling the system to handle larger user bases without compromising performance.

[0030] According to an embodiment, the system may further comprise a camera to acquire the video sequence. The integration of a camera within the system allows for complete in-situ data acquisition, enhancing the system’s convenience and ease of use.

[0031] According to an embodiment, the system comprises a first local device including the camera, the input, the first processor and the output, and a second remote device including the second processor. As mentioned, the partition of the system across a local device and a remote device fosters a balance between immediate data acquisition and heavy computational tasks, enhancing overall system efficiency.

[0032] According to an embodiment, the first local device may be a smartphone, a tablet or a computer, and the second remote device may be a cloud server. The potential implementation of the system in everyday devices like smartphones, tablets, or computers promotes user accessibility, contributing to better health monitoring and engagement. The utilization of a cloud server as the remote device leverages the advantages of cloud computing, such as scalability and cost-effectiveness, enhancing the system's feasibility for widespread adoption.

[0033] According to an embodiment, the video sequence comprises a plurality of channels, and wherein the at least one processor is configured to, for each channel: apply a mesh on the frames of the channel of the video sequence to define regions of interest on said at least one portion of skin surface, compute for each frame of the channel and for each region of interest, a raw trace value based on the value of at least one point in said region of interest, for each region of interest, compute, over the video sequence, a raw trace signal of the channel of the video sequence in said region of interest using said raw trace values, and wherein, the at least one processor is further configured to: compute, for each region of interest, combined raw trace signals by combining the raw trace signals of all channels of the video sequence, and compute, for each region of interest, raw PPG signals based on the combined raw trace signals.

[0034] The use of several channels for the video sequence permits to reduce the noise in the acquired and computed signals.

[0035] The invention also introduces a computer-implemented method that follows the same steps as the system for obtaining the biosignal.

[0036] In particular, the present invention relates to a computer-implemented method for obtaining a global PPG signal of at least one portion of a skin surface of a user from avideo sequence of said at least one portion of a skin surface of said user, said method comprising: receiving a video sequence of a skin surface of the user, for each frame of said video sequence, applying a mesh on the frame of the video sequence to define regions of interest on said at least one portion of skin surface, wherein each region of interest comprises at least one point and each point is associated to a value, for each frame of said video sequence and for each region of interest, computing a raw trace value based on at least said value of at least one point in said region of interest, for each region of interest: o computing, over the frames of the video sequence, a raw trace signal of the video sequence associated to said region of interest using said raw trace values computed for the region of interest, o computing a raw PPG signal based on said raw trace signal associated to said region of interest, o obtaining a PPG quality signal representative of the quality of the raw PPG signal associated to said region of interest using the signal analysis machine learning model feed in with said raw photoplethysmography (PPG) signal, o computing a processed PPG signal based on the PPG quality signal and the raw PPG signal associated to said region of interest, computing a global PPG signal of the at least one portion of the skin surface of the user based on the processed PPG signals computed for said regions of interest.

[0037] According to one embodiment, the computer-implemented method for obtaining a biosignal of a user from a video sequence of at least one portion of a skin surface of the said user, comprises the following steps: receive a video sequence of a skin surface of the user, apply a mesh on the frames of the video sequence to define regions of interest on said at least one portion of skin surface, each region of interest comprising at least one point, compute for each frame and for each region of interest, a raw trace value based on the value of at least one point in said region of interest, for each region of interest, compute, over the video sequence, a raw trace signal of the video sequence in said regionof interest using said raw trace values, compute, for each region of interest, raw PPG signals based on the raw trace signals, segment each of the obtained raw PPG signals using a pre-trained machine learning model to obtain a PPG quality signal representative of the quality of the raw PPG signal, compute, for each region of interest, processed PPG signal based on the respective PPG quality signal and the associated raw PPG signal, compute a global PPG signal of the portion of the skin surface of the user based on the processed PPG signals, and compute said biosignal based on said global PPG signal of the skin surface of the user

[0038] The method implements non-invasive, real-time, and remote biosignal monitoring, which can be integrated with various devices and applications, promoting user engagement and continuity of health monitoring. The application of machine learning techniques in the method allows for the automation and optimization of signal quality assessment and biosignal computation, enhancing the accuracy and reliability of the obtained biosignals.

[0039] In addition, the disclosure relates to a computer program comprising software code adapted to perform a computer implemented method compliant with any of the above execution modes when the program is executed by a processor.

[0040] The present disclosure further pertains to a non-transitory program storage device, readable by a computer, tangibly embodying a program of instructions executable by the computer to perform a computer implemented method compliant with the present disclosure.

[0041] Such a non-transitory program storage device can be, without limitation, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor device, or any suitable combination of the foregoing. It is to be appreciated that the following, while providing more specific examples, is merely an illustrative and not exhaustive listing as readily appreciated by one of ordinary skill in the art: a portable computer diskette, a hard disk, a ROM, an EPROM (Erasable Programmable ROM) or a Flash memory, a portable CD-ROM (Compact-Disc ROM).DEFINITIONS

[0042] As used in the context of the present invention, the term “machine learning (ML) model” refers to a computational algorithm or a set of algorithms that are capable of learning patterns or relationships in input data through a training process. The learned patterns or relationships can then be used to make predictions, classifications, or recommendations for new, previously unseen data. Machine learning models can be based on various techniques, including but not limited to supervised learning, unsupervised learning, semi- supervised learning, reinforcement learning, and deep learning. Examples of machine learning models include, but are not limited to, decision trees, support vector machines, artificial neural networks, k-means clustering, principal component analysis, and reinforcement learning agents. In the context of this invention, machine learning models are used to segment raw PPG signals and, advantageously, to apply a dynamic mesh to define regions of interest.

[0043] “Datasets” are collections of data used to build an ML mathematical model, so as to make data-driven predictions or decisions. In “supervised learning” (i.e. inferring functions from known input-output examples in the form of labelled training data), three types of ML datasets (also designated as ML sets) are typically dedicated to three respective kinds of tasks: “training”, i.e. fitting the parameters, “validation”, i.e. tuning ML hyperparameters (which are parameters used to control the learning process), and “testing”, i.e. checking independently of a training dataset exploited for building a mathematical model that the latter model provides satisfying results.

[0044] A “neural network (NN)” designates a category of ML comprising nodes (called “neurons”), and connections between neurons modeled by “weights”. Lor each neuron, an output is given in function of an input or a set of inputs by an “activation function”. Neurons are generally organized into multiple “layers”, so that neurons of one layer connect only to neurons of the immediately preceding and immediately following layers.

[0045] The above ML definitions are compliant with their usual meaning, and can be completed with numerous associated features and properties, and definitions of related numerical objects, well known to a person skilled in the ML field. Additional terms willbe defined, specified or commented wherever useful throughout the following description.

[0046] The terms “adapted”, “designed” and “configured” are used in the present disclosure as broadly encompassing initial configuration, later adaptation or complementation of the present device, or any combination thereof alike, whether effected through material or software means (including firmware).

[0047] The term “processor” should not be construed to be restricted to hardware capable of executing software, and refers in a general way to a processing device, which can for example include a computer, a microprocessor, an integrated circuit, or a programmable logic device (PLD). The processor may also encompass one or more Graphics Processing Units (GPU), whether exploited for computer graphics and image processing or other functions. Additionally, the instructions and / or data enabling to perform associated and / or resulting functionalities may be stored on any processor-readable medium such as, e.g., an integrated circuit, a hard disk, a CD (Compact Disc), an optical disc such as a DVD (Digital Versatile Disc), a RAM (Random- Access Memory) or a ROM (Read-Only Memory). Instructions may be notably stored in hardware, software, firmware or in any combination thereof.BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The present disclosure will be better understood, and other specific features and advantages will emerge upon reading the following description of particular and non- restrictive illustrative embodiments, the description making reference to the annexed drawings wherein:

[0049] Figure 1 illustrates a system according to an embodiment of the invention.

[0050] Figure 2 illustrates a system according to another embodiment of the invention.

[0051] Figure 3 details a flowchart representing the steps of a computer-implemented method according to an embodiment of the invention.

[0052] Figure 4 shows an example of mesh applied to a frame containing the face of a user.

[0053] Figure 5 exemplifies the segmentation of a raw PPG signal by a pre-trained machine learning model.

[0054] Figure 6 shows a comparative graph of the heart rate obtained by a method or system according to the invention and a heart rate obtained by a reference oximeter.

[0055] Figure ? shows a schematic representation of the signal analysis machine learning model according to one example.

[0056] Figure 8 shows a schematic representation of semantic divisions of zones on the face.

[0057] Figure 9A and 9B show graphs, each associated to one region of interest or the full face, comparing PPG quality signals, ground truth and raw PPG signals obtained according to the example 1.DETAILED DESCRIPTION

[0058] The present description illustrates the principles of the present disclosure. It will thus be appreciated that those skilled in the art will be able to devise various arrangements that, although not explicitly described or shown herein, embody the principles of the disclosure and are included within its scope.

[0059] All examples and conditional language recited herein are intended for educational purposes to aid the reader in understanding the principles of the disclosure and the concepts contributed by the inventor to furthering the art, and are to be construed as being without limitation to such specifically recited examples and conditions.

[0060] Moreover, all statements herein reciting principles, aspects, and embodiments of the disclosure, as well as specific examples thereof, are intended to encompass both structural and functional equivalents thereof. Additionally, it is intended that suchequivalents include both currently known equivalents as well as equivalents developed in the future, i.e., any elements developed that perform the same function, regardless of structure.

[0061] Thus, for example, it will be appreciated by those skilled in the art that the block diagrams presented herein may represent conceptual views of illustrative circuitry embodying the principles of the disclosure. Similarly, it will be appreciated that any flow charts, flow diagrams, and the like represent various processes which may be substantially represented in computer readable media and so executed by a computer or processor, whether or not such computer or processor is explicitly shown.

[0062] The functions of the various elements shown in the figures may be provided through the use of dedicated hardware as well as hardware capable of executing software in association with appropriate software. When provided by a processor, the functions may be provided by a single dedicated processor, a single shared processor, or a plurality of individual processors, some of which may be shared.

[0063] It should be understood that the elements shown in the figures may be implemented in various forms of hardware, software or combinations thereof. Preferably, these elements are implemented in a combination of hardware and software on one or more appropriately programmed general-purpose devices, which may include a processor, memory and input / output interfaces.

[0064] The invention thus relates to a system for obtaining a biosignal of a user from a video sequence of at least one portion of a skin surface of the said user, as shown in Figures 1 and 2, and a corresponding computer-implemented method as shown in Figure 3. The system may be a remote photoplethysmography system.

[0065] SYSTEM

[0066] In the first example shown in figure 1, the system 100 generally comprises an input 110 configured to receive a video sequence or stream of at least a portion of a skin surface of the user 1, a processor 120 configured to process the video sequence and obtainthe biosignal, and an output 130 configured to output the biosignal. It may also comprise a camera 111 to provide the video sequence to the input 110, and a memory 140.

[0067] The biosignal may be one of the following: heart rate, oxygen saturation, respiratory rate, heart rate variability, diastolic pressure, systolic pressure, cardiac workload, hemoglobin level, glycemia.

[0068] In this example, the system 100 may be integrated in a single computer (e.g. in a telemedicine cabin booth), smartphone 101 or tablet. Other devices may also be contemplated.

[0069] The input 110 is configured to receive the video sequence of at least a portion of a skin surface of the user 2 acquired from the camera 111. The video sequence comprises at least two frames or a plurality of frames, i.e., images. An example of image or frame 4 from a video sequence captured from a camera is shown at figure 4. Preferably, the portion of the skin surface comprises at least part of the face 3 of the user.

[0070] The camera 111 may be a single-color camera, or multi-channel camera, like a RGB camera. If the camera is a multi-channel camera, the video sequence then comprises as many channels.

[0071] The (at least one) processor may be configured to perform the following steps, which will be explained in greater details thereafter with respect to figure 3 and the computer-implemented method: a. Step S2: apply a mesh 5 (figure 4) on the frames 4 of the video sequence to define regions of interest 6 on said at least one portion of skin surface, b. Step S3: compute, for each frame and for each region of interest, a raw trace value based on the values of the points comprised in said region of interest, c. Step S4: for each region of interest, compute, over the frames 4 of the video sequence, a raw trace signal of the video sequence in said region of interest using said raw trace values computed for said region of interest, d. Step S5: for each region of interest, compute one raw photoplethysmography (PPG) signal based on the raw trace signals,e. Step S6: obtain a PPG quality signal representative of the quality of the raw PPG signal associated to said region of interest 6 using the signal analysis machine learning model feed in with said raw photoplethysmography (PPG) signal (or alternatively, segment each of the obtained raw PPG signals using a pre-trained machine learning model to obtain a PPG quality signal representative of the quality of the raw PPG signal), f. Step S7: for each region of interest, compute one processed PPG signal based on the respective PPG quality signal and the associated raw PPG signal, so as to reduce contribution of low-quality raw PPG signal in said computed processed PPG signals, g. Step S8: compute a global PPG signal of the skin surface of the user based on the processed PPG signals obtained from all the regions of interest, and h. Step S9: optionally, compute said biosignal based on said global PPG signal of the skin surface of the user.

[0072] The output 130 may be configured to provide the global PPG signal and / or the biosignal thus computed.

[0073] The detail concerning the different calculation steps implemented by the at least one processor will be discussed further in the following description in relation to the corresponding computer-implemented method.

[0074] The memory 140 may be used to store the video sequence, signals and data necessary to compute the biosignal (e.g. machine learning models parameters, etc.). The latter can take the form of storage resources available from any kind of appropriate storage means, which can be notably a RAM or an EEPROM (Electrically-Erasable Programmable Read-Only Memory) such as a Flash memory, possibly within an SSD (Solid-State Disk).

[0075] The system 100 may further comprise a display to display the biosignal provided by the output 130.

[0076] The system 100 may further comprise communication means to wirelessly communicate with remote devices, for example to transmit the biosignal to another device for remote health monitoring.

[0077] In the advantageous embodiment shown in figure 2, the system 200 comprises two subsystems (or devices) 210 and 220. In this example, the first subsystem 210 comprises a first input 211, a processor 212, an output 213, a camera 21 la and a memory 214. In this example, the second subsystem 200 comprises a second input 221, a second processor 222, a second output 223 and a second memory 224.

[0078] In this example, the first processor 212 is configured to: a. apply the mesh on the frames of the video sequence received at the first input 211 (e.g. from the camera 21 la) to define regions of interest on said at least one portion of skin surface, b. compute for each frame and for each region of interest, the raw trace value based on the value of at least one point in said region of interest, c. for each region of interest, compute, over the video sequence, the raw trace signal of the video sequence in said region of interest using said raw trace values, d. compute, for each region of interest, the raw PPG signals based on the raw trace signals, and send the raw PPG signals thus obtained to the second processor 222 (via the second input 221); and wherein the second processor 222 is configured to: e. segment each of the received raw PPG signals using a pre-trained machine learning model to obtain a PPG quality signal representative of the quality of the raw PPG signal, f. compute processed PPG signals for each region of interest based on the respective PPG quality signal and the associated raw PPG signal, so as to reduce contribution of low-quality raw PPG signal in said computed processed PPG signals,g. compute the global PPG signal of the skin surface of the user based on the processed PPG signals, and h. optionally compute said biosignal based on said global PPG signal of the skin surface of the user.

[0079] It should be noted that the step of computing, for each region of interest, the raw PPG signals based on the raw trace signals, may be performed either by the first 212 or the second 222 processor. The selection of the processor for performing this step may be carried out dynamically, e.g. based on the bandwidth available to communicate data between the first and the second processor, and / or based on the computational power available at the first processor.

[0080] The functions of memories 214 and 224 are similar as in system 100.

[0081] The subsystem 210 may comprise a first local device 21 (e.g. a smartphone, a tablet or a computer), including the first input 211, the camera 211a, the first processor 212, the first output 213 and the memory 214. The subsystem 220 may comprise a second remote device 22 (e.g. a cloud server), including the second input 221, the second processor 222, the second output 223 and the second memory 224.

[0082] In this example, the second output 223 may not be used, and the second processor 222 may send back the computed biosignal to the first processor 212 such that the first output 213 provides the biosignal at the first subsystem 210.

[0083] This example is advantageous as it increases privacy (no image of the subject is directly sent to a remote device), while distributing computation load between the first device 21 (which may have low computation capabilities, like a smartphone or tablet), and the second device 22 (which may have greater computation capabilities, like a cloud server).

[0084] COMPUTER-IMPLEMENTED METHOD

[0085] A computer-implemented method for obtaining a biosignal of a user from a video sequence of at least one portion of a skin surface of the said user according to an embodiment of the invention will be described in greater details with respect to figures 3,4 and 5. The biosignal may be one of the following: heart rate, oxygen saturation, respiratory rate, heart rate variability, diastolic pressure, systolic pressure, cardiac workload, hemoglobin level, glycemia.

[0086] As shown in figure 3, the method comprises at least the following steps:

[0087] At step SI, a video sequence of a skin surface of the user acquired from a camera is received.

[0088] Then, at step S2 a mesh 5 is applied on the frames 4 of the video sequence to define regions of interest 6 on said at least one portion of skin surface. Each region of interest encompasses (comprises) one or, preferably, multiple pixels of one frame. The skin surface of the user advantageously comprises at least part of the face 3 of the user; in this case the mesh is placed over the face. A dynamic mesh, configured for tracking regions of interest on a face, works by adapting to changes in the shape, position, and expression of the face over time.

[0089] In one embodiment, the mesh is a dynamic mesh applied on the frames using a pre-trained mesh generator machine learning model such that the face is tracked and regions of interest are consistent along the video sequence, the mesh generator machine learning model being trained on a dataset of face images labelled with a specific mesh. Notably, at first, an initialization mesh is placed over the face in the first frame of the video sequence. This initialization mesh consists of vertices and edges that outline key facial features such as the eyes, nose, mouth, and facial contours. Often, a pre-trained specific machine learning model, such as a facial landmark detector, is used to identify these key points and initialize the mesh. Then the initial mesh is adapted frame-by-frame, indeed as the video sequence progresses, the initialization mesh needs to adapt to changes in the facial expressions, head movements, and lighting conditions. This is achieved through feature tracking algorithms that follow the movement of the vertices from one frame to the next. Said “mesh generator” machine learning model may be for example a pre-trained machine learning model of the type of a 2D convolutional neural network, such as a ResNet. Advantageously, a ResNet may be employed for its capability to capture intricate spatial features across multiple layers. In other words, it as to beunderstood that multiple regions of interest are defined to cover different zones of the skin, so that for example one region of interest is defined and tracked along the video sequence (frame by frame) to so that it will comprise pixels associated to the representation of the nose of the user in all the successive frames.

[0090] During training, various loss functions may be considered alone or in combination to optimize model performance, including categorical cross-entropy to classify mesh points accurately and mean squared error to refine mesh placement relative to regions of interest. Optimizers like Adam or SGD (Stochastic Gradient Descent) with momentum may be utilized to efficiently navigate the model's parameter space and converge towards optimal mesh predictions. This comprehensive approach ensures the machine learning model effectively learns to apply the dynamic mesh, facilitating precise and consistent tracking of facial regions of interest in real-time video analysis scenarios.

[0091] Then, at step S3, for each frame and for each region of interest, a raw trace value may be computed based on the value(s) of the point(s) (i.e., pixels) in said region of interest. The raw trace value may be computed by filtering the values of the points of the region of interest, preferably by averaging the values of the points in the region of interest. Other methods of filtering may be used, e.g. using colour filtering, a machine learning model for segmentation or classification, thresholding, etc. As stated above the camera may be acquire images in RGB, so the values of the three RGB channels associated to each point of the region of interest may be used to compute the raw trace values for each channel.

[0092] Then, at step S4, for each region of interest 6, and over the video sequence (i.e., for each frame 4), a raw trace signal of the video sequence in said region of interest may be computed using said raw trace values. This raw trace signal may be the concatenation of the raw trace values, so to obtain a signal representative of the blood volume pulse as a function of time (i.e., during the time window of the video sequence). In the case of RGB images, a raw trace signal for each channel may be computed.

[0093] Then, at step S5, for each region of interest, a raw photoplethysmography (PPG) signal may be computed based on the raw trace signals. In this way, if for example, 11regions of interests had been identified frame-by-frame in the video sequence, the step S5 will produce 11 raw photoplethysmography (PPG) signals, one for each region of interest.

[0094] If only one channel is used for the video sequence, the raw PPG signals may be obtained by normalizing and / or further filtering the raw trace signals depending on the biosignal to obtain. For example, for heart rate, the signal may be bandpass filtered between 0,6 Hz and 4 Hz.

[0095] If several channels are used for the video sequence (e.g. when a RGB camera is used, 3 channels are available), the method may further comprise a step of computing, for each region of interest, combined raw trace signals by combining the raw trace signals of all channels of the video sequence, and a step of computing for each region of interest, raw PPG signals based on the combined raw trace signals. In other words, at step S5 may be configured to obtain the raw PPG signal for one region of interest by merging the raw trace signals, obtained for the three channels, associated to said region of interest. Raw trace signals of all channels may be combined by using appropriate models to better extract relevant information for the considered biosignal. Again, raw PPG signals may be obtained by normalizing and / or further filtering the raw trace signals based on the biosignal to obtain. Alternatively, the raw PPG signals may be obtained using only the raw trace signals obtained from the green channel.

[0096] The raw PPG signals may further be normalized and / or filtered before step S6.

[0097] According to one embodiment, step S6 may comprise obtaining, region of interest 6, a PPG quality signal (representative of the quality of the raw PPG signal) for each obtained raw PPG signal associated to said region of interest 6. This may be obtained by feeding each raw photoplethysmography (PPG) signal to the (pre-trained) signal analysis machine learning model. Details on the training of this signal analysis machine learning model will be provided below in the description of the devices and methods for training.

[0098] A PPG quality signal representative of the quality of the raw PPG signal may be obtained for a portion of the raw signal during a selected time window, for the integrality of the raw PPG signal (i.e., time window of the video sequence), or for a specific time point of the raw PPG signal as an instantaneous measurement.

[0099] According to one alternative embodiment, at step S6, each of the obtained raw PPG signals are segmented using the results obtained from a pre-trained (signal analysis) machine learning model configured to obtain a PPG quality signal representative of the quality of the raw PPG signal. For example, the pre-trained (signal analysis) machine learning model is a ID convolutional neural network (e.g. U-net). Figure 5 shows an example of raw PPG signal segmentation by a pre-trained (signal analysis) machine learning model (i.e., obtained using the PPG quality signal inferred by the signal analysis machine learning model). In this example, the (signal analysis) machine learning model outputs a binary value (0 or 1). Bad quality zones 7 (output value = 0) are identified in grey along the 30 seconds sequence shown in the figure. Therefore, by for example, simply multiplying these binary outputs to the raw PPG signal, is possible to segment the raw PPG signal and remove the low-quality signals.

[0100] In other examples, the (signal analysis) machine learning model may output continuous values between 0 and 1.

[0101] Then, at step S7, processed PPG signals may be computed for each region of interest based on the respective PPG quality signal and the associated raw PPG signal (i.e., PPG quality signal and the raw PPG signal computed for a same region of interest), so as to reduce contribution of low-quality raw PPG signal in said computed processed PPG signals.

[0102] In an example, the PPG quality signals may be thresholded (e.g. the PPG quality signal can only take values 0 or 1). In other words, even when the output of the signal analysis) machine learning is not binary, a threshold may be applied to the PPG quality signals and, for example, the value 1 may be associated to the PPG quality signals values superior to the threshold and the value 0 may be associated to the PPG quality signals values inferior to the threshold. The processed PPG signals may be computed by multiplying the obtained thresholded PPG quality signals and the corresponding PPG raw signals with each other. Other types of computations may be used to obtain the processed PPG signals, for example using a machine learning model trained to reconstruct a signal based on the PPG quality signal and the raw PPG signal, or using an adaptative filtering approach making use of the PPG quality signal and the associated raw PPG signal.

[0103] Then, at step S8, a global PPG signal of the skin surface of the user is computed based on the processed PPG signals. The global PPG signal may be computed by selecting the best quality processed PPG signal, or by combining best quality parts of the processed PPG signals, or using a machine learning model to predict the global PPG signal based on the processed PPG signals.

[0104] Finally, at step S9, said biosignal is computed based on said global PPG signal of the skin surface of the user.

[0105] In one example, the biosignal is heart rate, and the heart rate is obtained by frequential analysis of the global PPG signal.

[0106] It should be noted, that other metrics may be obtained using the system 100 or 200 or using a computer-implemented method 300 as described above, based on obtained biosignals, for example: irregular heartbeats events, atrial fibrillation events, cardiac workload, mental stress index, anxiety index, depression index, cardiovascular disease risk, heart attack risk, stroke risk, hypertension risk, type 2 diabetes risk, hypercholesterolemia risk, hyperglyceridemia risk, etc.

[0107] Figure 6 shows a comparison plot of heart rate obtained by a system according to the invention implementing a method according to the invention for obtaining heart rate using a RGB camera, and a reference certified medical device. These measurements were obtained during a clinical study at hospital emergencies on various different patients. Moreover, the measurements were carried out by physicians and nurses without specific formation. One can see that the invention provides very good match for heart rate measurement in such conditions.

[0108] According to one embodiment, the biosignal is the heart rate variability (HRV). The HRV may be computed by detecting the time at which each beat is detected / represented in the global PPG signal and compute their variability. For instance, a temporal estimation like the RMSSD (Root Mean Square of Successive Differences) may be calculated from the global PPG signal. This RMSSD value may be obtained by calculating each successive time difference between heartbeats in milliseconds (e.g., N time differences) present in the global PPG signal, squaring each successive timedifference, summing the N time differences, dividing the sum by N, then applying the square root on the sum divided by N. Many other temporal estimates of HRV exist like the SDNN (standard deviation of all NN intervals which are the time laps between successive R waves in the ECG signal), or the PNN50 (percentage of successive NN intervals differing by more than 50 ms). Frequential estimates also exist in which we analyze the spectral properties of the signal of the time differences between the heartbeats. Frequential estimates may include very low frequency (VEF) power, low frequency (EF) power, or high frequency (HF) power. Non-linear estimates also exist like SD1 (standard deviation 1), SD2 (standard deviation 2), SD1 / SD2, which are based on the Poincare Plot.

[0109] The above computer- implemented method may be implemented in systems 100 or 200.

[0110] The present disclosure also relates to a computer program product. The computer program product includes instructions which, when the program is run by a computer, cause the computer to automatically perform the steps of the method 300 according to one of the embodiments described above.

[0111] The computer program product enabling the execution of the method 300 described above can be written as computer programs, code segments, instructions, or any combination thereof, to individually or collectively instruct or configure the processor or the computer to function as a specific use machine or computer to perform the operations carried out by hardware components. In one example, the computer program comprises machine code directly executed by a processor or computer, such as machine code produced by a compiler. In another example, the computer program product comprises high-level code which is executed by a processor or computer using an interpreter. Programmers with ordinary skills in the art can easily write the instructions or software based on the functional schemes and flow charts illustrated in the drawings and corresponding descriptions in the specification, which disclose algorithms to perform the operations of the method as described above.

[0112] The present invention also pertains to a device for training a signal analysis machine leaning model.

[0113] The device may be configured to perform the training according to a first “labelled” embodiment which uses a labelled training dataset.

[0114] According to this first embodiment, the device for training comprises an input configured to receive a labelled training dataset comprising a plurality of labelled training samples associated to a plurality of subjects. Each labelled training sample comprises, for one subject, at least one labelled video sequence of at least one portion of skin surface of said subject, wherein each labelled frame of each labelled video sequence is labelled by defining regions of interest 6 on said at least one portion of skin surface and associating to each region of interest 6 a weight. The definition of the regions of interest 6 may be done using a dynamic mesh, as described above for the system of the present invention. Then a user may manually annotate the regions of interest 6 with a weight or the weight may be automatically calculated using a signal analysis algorithm.

[0115] The device for training comprises at least one processor configured to obtain said trained signal analysis machine learning model by training a machine learning architecture using said labelled training dataset.

[0116] In this case, the following steps are iteratively repeated till convergence during the training in order to parametrize the initialization parameter of a machine learning architecture, which may have been pretrained on generic data or not. The machine learning architecture may be a convolutional neural network, such as a U-Net. The machine learning architecture may comprise multiple encoder and decoder blocks, with each block containing convolutional layers, ReLU activation functions, batch normalization, and pooling layers.

[0117] The iteratively repeated steps on each labelled training sample are:• for each frame 4 of said at least one subject video sequence, apply a mesh 5 on the frame 4 to define regions of interest 6 on said at least one portion of skin surface, wherein each region of interest comprises at least one point and each point is associated to a value,• for each frame 4 of said video sequence and for each region of interest 6, compute a raw trace value based on at least said value of at the least one point in said region of interest,• for each region of interest 6: compute over the frames 4 of the at least one subject video sequence, a raw trace signal of the subject video sequence associated to said region of interest 6 using said raw trace values computed for said region of interest 6, compute a raw PPG signal based on said raw trace signal associated to said region of interest 6, obtain a PPG quality signal representative of the quality of the raw PPG signal associated to said region of interest 6 by feeding said raw photoplethysmography (PPG) signal as input to said machine learning architecture, compute a processed PPG signal based on said PPG quality signal and the raw PPG signal associated to said region of interest 6,• modify the machine learning architecture parameters to minimize a loss function configured to compare said processed PPG signal, for each region of interest, to reference processed PPG signals obtained from the labelled subject video sequence of the labelled training sample.

[0118] Each of the processing steps described for the training may be implemented according to the embodiments described for the system and method for obtaining a global PPG signal of the present invention.

[0119] Said reference processed PPG signals may be obtained by concatenating the weights of the labelled frames on each region of interest.

[0120] A loss function that may be employed is a combination of cross-entropy loss for segmentation and total variation loss to enforce continuity in the output. The optimization of the machine learning architecture may be achieved using gradient descent techniques.

[0121] According to an alternative embodiment for training, the device generates the signal analysis machine leaning model using a “global” training dataset, which comprises “real” PPG measurements, such as from an oximeter.

[0122] In this alternative embodiment, the device is configured to receive a global training dataset comprising a plurality of global training samples associated to a plurality of subjects. Each global training sample comprising data from a subject of the plurality of subject. Each global training sample comprises at least one subject video sequence of at least one portion of skin surface of said subject and at least one reference PPG signal (i.e., ground truth signal). The at least one video sequence and the at least one reference PPG signal are acquired in one same time window and preferably in a synchronous way. The reference PPG signal may be obtained from a pulse oximeter placed on a finger or a chest strap heart rate monitor on the subject, while the video sequence is acquired.

[0123] The device then may obtain the trained signal analysis machine leaning model using said global training dataset. The training comprises iterating till convergence, for each global training sample, the following steps:• for each frame 4 of said at least one subject video sequence, apply a mesh 5 on the frame 4 to define regions of interest 6 on said at least one portion of skin surface, wherein each region of interest comprises at least one point and each point is associated to a value,• for each frame 4 of said video sequence and for each region of interest 6, compute a raw trace value based on at least said value of at the least one point in said region of interest,• for each region of interest 6: compute over the frames 4 of the at least one subject video sequence, a raw trace signal of the subject video sequence associated to said region of interest 6 using said raw trace values computed for said region of interest 6, compute a raw PPG signal based on said raw trace signal associated to said region of interest 6,obtain a PPG quality signal representative of the quality of the raw PPG signal associated to said region of interest 6 by feeding said raw PPG signal as input to said machine learning architecture, compute a processed PPG signal based on said PPG quality signal and the raw PPG signal associated to said region of interest 6,• compute a global PPG signal of the at least one portion of the skin surface of the subject based on the processed PPG signals computed for said regions of interest 6,• modify the machine learning architecture parameters to minimize a loss function configured to compare said global PPG signal to the at least one reference PPG signal of the global training sample.

[0124] More in details according to this alternative embodiment, the device for training is configured to leveraging an attention mechanism (e.g. multi-head self-attention, selfattention CNN) to obtain the PPG quality signals.

[0125] The loss function for training the machine learning model may be one of or a combination of the following: a cross-correlation loss (e.g. maximizing the similarity between the global PPG signals and the reference PPG signals), a Fourier-domain loss (e.g., ensuring the frequency components of the global PPG signals match those of the ground truth- reference PPG signals), a smoothness loss (e.g., penalizing abrupt changes in the global PPG signals to maintain temporal continuity).

[0126] The present invention pertains as well to the computer- implemented methods, implemented by the device for training described above.EXAMPLES

[0127] Example 1: Extracting face blood volume variations into a Blood Volume Pulse (BVP) as the biosignal

[0128] Quality weights (i.e. PPG quality signals) are learned directly using an attention mechanism. The different BVP are still weighted averaged into the final BVP, but theweights are directly learned by the learning process whose objective is to make the weighted average BVP closer to the ground truth PPG signal. Thus here, the dataset doesn't need to be manually annotated, we just need to collect videos of patient's face with a reference PPG signal temporally synchronized (this is done by recording the signal of a CE marked accurate pulse oximeter placed on the finger of the user). The following provides details about the architecture of the neural network, the training procedure and the loss function to be used. It is observed that while full-face data extraction is generally robust in terms of error correction, under certain situations data from a certain semantic zone (shown in Figure 8) can give a better signal, generally in terms of signal-to-noise ratio. The idea of this method is to combine good quality temporal regions from all zones (i.e., processed PPG signal for all regions of interest) in order to construct a single zone- weighted blood- volume pulse (i.e., global PPG signal).

[0129] The model assumes that raw data are extracted on eleven regions of interest of the user’s face (the ten semantic zone of Figure 8 and the full-face). However, the model is designed to be agnostic to the number of regions of interest or their semantic signification, making its decision solely based on the signal observed in each region of interest. A schematic representation of the signal analysis machine learning model according to the present example.

[0130] Eet T 6matrjx ofrawja(aextracted over E frames, where Z represents the number of zones / regions of interest (11 in this case), and C = 3 is the number of colors in the raw data. Semantically, the first dimension of the matrix represents the regions of interest on the face, i.e., from where raw data (i.e. raw PPG signals) are extracted; the second dimension corresponds to blue-green-red colors from the images, while the third dimension represents the time axis. Therefore, combining raw data temporally implies inferring mixture coefficients for each timestamp along the time axis, so that the sum of these coefficients for a given timestamp over all zones is equal to 1 (see definition of Cj below).

[0131] The first step consists of extracting Z blood- volume pulses (i.e., processed PPG signal) from this matrix. The data T are thus subject to normalization, which involves mean-normalization of the data, followed by band pass filtering (0.66 - 3.00 Hz) and adetrending operation. This is followed by the Plane-Orthogonal-to-Skin (POS) method, a standard method for inferring the blood-volume pulse from color traces (i.e. raw trace signal). These operations result in a matrix S G RZxlxL representing candidate bloodvolume pulses (i.e., raw PPG signal) from Z zones.

[0132] The idea now is to determine mixture coefficients Cj G [0,l]lxlxL, j 6 { 1,2,...,Z} to infer the contribution of each candidate BVP. The model first infers feature vectors from the candidate signals by passing them through a sequence of dilated convolutional layers. Each dilated convolutional kernel is of size 3, with dilation = padding = 5, where 5 6 { 1,2,3,5,10}. Such a configuration ensures that the number of samples in the signal does not change after the convolutional operation, so that all feature vectors have the same size along the length dimension (i.e., L). Let F G >ZxDxL= {fj }Zj=i = M (S) be the D-channel feature representation of the Z semantic zones. This model employs self-attention to regress mixture coefficients Cj. Specifically, we use multi-head self-attention with H heads, so that

[0133] Here Cj is the residual vector used to infer the mixture coefficients Cj. Each attention operation is defined on query, key, and value matrices as follows.

[0134] The residual feature vector is now used to infer attention using the operation

[0135] where Cixi is a 1x1 convolutional kernel with D input channels and 1 output channel, and is a learnable parameter.

[0136] Finally, the zone- weighted BVP (i.e., global PPG signal) is calculated as

[0137] where O denotes element- wise matrix multiplication.

[0138] Most datasets include an observed - ground truth - photoplethysmography (PPG) signal associated with a synchronized recorded video. This ground truth PPG signal is registered using a medical device that is affixed to the user's finger, making it much more reliable than remote PPG. With this information, it is possible to encourage the model to find appropriate mixture coefficients, because the zone- weighted BVP s must obey certain properties, as determined by the ground truth PPG signals. The following loss functions taken alone or in combination may be used as training functions for the model.

[0139] Cross -correlation loss: As the ground truth PPG signal is usually obtained using an oximeter and recorded at one of hands' fingers, it shows a phase shift compared to the BVP obtained from the face. Hence, using a direct mean-squared error loss to train s with s would be not feasible. However, the phase shift is always advantageously a rather small angle and hence, a cross-correlation-based mean-squared error function can be used instead. The cross -correlation loss aims to maximize the cross-correlation between s and s within a small window, in which the phase shift lies. Concretely,

[0140] where [★] denotes the cross-correlation operation computed over a window [a, b]. The window [a, b] is chosen so that the two signals (the ground truth PPG signal s and the inferred global PPG signal s) will be in phase for some value in [a, b], and where the cross -correlation is maximized.

[0141] Fourier-domain loss: Another important aspect is training the model to produce s so that it has the correct frequencies as are present in s, disregarding the phase shift. Such a constraint is imposed using a Fourier-domain loss, by introducing a mean-squared error loss term between the power spectra of s and s as given by

[0142] where F(.) denotes the fast Fourier transform.

[0143] Smoothness loss: As the inference of the mixture coefficients is not directly supervised, it is beneficial to regularize their values by imposing a semantic constraint. Such a constraint is imagined by considering temporally neighboring values of mixture coefficients for a particular zone, which would realistically not be fluctuating too much between the neighborhood. It is imposed hence a smoothness constraint on these coefficients, over each zone, by imposing a Lasso penalty on differences along the time axis, as given by

[0144] Figure 9A and 9B show graphs, each associated to one region of interest or the full face, comparing PPG quality signal, ground truth and raw PPG signals. Figure 9 A shows, from top to bottom panel, the signal for the full-face and individual semantic zones of figure 8: the cheek right lower, cheek right upper, cheek left lower, cheek left upper and forehead right lower regions. Figure 9B shows, from top to bottom panel, results for other semantic zones of figure 8: the forehead left lower, forehead center lower, nose full, chin right and chin left regions. Each panel represents the PPG quality signal as the filled region associated to the left-hand side unitless y-axis. Each panel further represents the ground truth PPG signal (dot-dashed line) and raw PPG signal (solid line) associated to the right-hand side y-axis in arbitrary units. The x-axis is a time axis in seconds.

[0145] The ground truth PPG signal allows to help the model to finds appropriate mixture coefficients as observed by the raw PPG signal which tends towards the ground truth PPG signal. It is also observed that the full-face data extraction is the most robust in terms of error correction. Finally, it is clearly shown that the quality signal varies with time and is not the same for all the region of interest at a given time. Therefore, combining good quality temporal regions from all zones allows to construct a single zone- weighted bloodvolume pulse.

Claims

CLAIMS1. A device for training a signal analysis machine leaning model to be used for obtaining a global PPG signal of at least one portion of a skin surface of a user comprised in a video sequence, said device comprising: at least one input configured to receive a global training dataset comprising a plurality of global training samples associated to a plurality of subjects, each global training sample comprising, for one subject, at least one subject video sequence of at least one portion of skin surface of said subject and at least one reference photoplethysmography (PPG) signal, said at least one video sequence and said at least one reference PPG signal being acquired in one same time window; at least one processor configured to obtain said trained signal analysis machine learning model by training a machine learning architecture using said global training dataset, wherein training comprises iterating till convergence, for each global training sample, the following steps:• for each frame (4) of said at least one subject video sequence, apply a mesh (5) on the frame (4) to define regions of interest (6) on said at least one portion of skin surface, wherein each region of interest comprises at least one point and each point is associated to a value,• for each frame (4) of said video sequence and for each region of interest (6), compute a raw trace value based on at least said value of at the least one point in said region of interest,• for each region of interest (6): compute over the frames (4) of the at least one subject video sequence, a raw trace signal of the subject video sequence associated to said region of interest (6) using said raw trace values computed for said region of interest (6), compute a raw photoplethysmography (PPG) signal based on said raw trace signal associated to said region of interest (6), obtain a PPG quality signal representative of the quality of the raw PPG signal associated to said region of interest (6) by feeding said rawphotoplethysmography (PPG) signal as input to said machine learning architecture, compute a processed PPG signal based on said PPG quality signal and the raw PPG signal associated to said region of interest (6),• compute a global PPG signal of the at least one portion of the skin surface of the subject based on the processed PPG signals computed for said regions of interest (6),• modify the machine learning architecture parameters to minimize a loss function configured to compare said global PPG signal to the at least one reference PPG signal of the global training sample.

2. The device according to claim 1, wherein machine learning architecture is ID convolutional neural network.

3. A device for training a signal analysis machine leaning model to be used in a system (100; 200) for obtaining a global PPG signal of at least one portion of a skin surface of a user from a video sequence of said at least one portion of a skin surface of said user, said device comprising: at least one input configured to receive a labelled training dataset comprising a plurality of labelled training samples associated to a plurality of subjects, each labelled training sample comprising, for one subject, at least one labelled subject video sequence of at least one portion of skin surface of said subject, wherein each labelled frame of each labelled subject video sequence is labelled by defining regions of interest (6) on said at least one portion of skin surface and associating to each region of interest (6) a weight; at least one processor configured to obtain said trained signal analysis machine learning model by training a machine learning architecture using said labelled training dataset, wherein training comprises iterating till convergence, for each labelled training sample, the following steps:• for each frame (4) of said at least one subject video sequence, apply a mesh (5) on the frame (4) to define regions of interest (6) on said at least one portionof skin surface, wherein each region of interest comprises at least one point and each point is associated to a value,• for each frame (4) of said video sequence and for each region of interest (6), compute a raw trace value based on at least said value of at the least one point in said region of interest,• for each region of interest (6): compute over the frames (4) of the at least one subject video sequence, a raw trace signal of the subject video sequence associated to said region of interest (6) using said raw trace values computed for said region of interest (6), compute a raw photoplethysmography (PPG) signal based on said raw trace signal associated to said region of interest (6), obtain a PPG quality signal representative of the quality of the raw PPG signal associated to said region of interest (6) by feeding said raw photoplethysmography (PPG) signal as input to said machine learning architecture, compute a processed PPG signal based on said PPG quality signal and the raw PPG signal associated to said region of interest (6),• modify the machine learning architecture parameters to minimize a loss function configured to compare said processed PPG signal, for each region of interest, to reference processed PPG signals obtained from the labelled subject video sequence of the labelled training sample.

4. A system (100; 200) for obtaining a global PPG signal of at least one portion of a skin surface of a user from a video sequence of said at least one portion of a skin surface of said user using a signal analysis machine leaning model obtained from a device according to any one of claims 1 to 3, said system comprising: at least one input (110; 211) configured to receive the video sequence of said at least one portion of skin surface of the user (1; 2) acquired from a camera (111; 21 la), at least one processor (120; 212; 222) configured to: for each frame (4) of said video sequence, apply a mesh (5) on the frame (4) to define regions of interest (6) on said at least one portion of skin surface,wherein each region of interest comprises at least one point and each point is associated to a value,• for each frame (4) of said video sequence and for each region of interest (6), compute a raw trace value based on at least said value of at the least one point in said region of interest,• for each region of interest (6): compute over the frames (4) of the video sequence, a raw trace signal of the video sequence associated to said region of interest (6) using said raw trace values computed for the region of interest (6), compute a raw photoplethysmography (PPG) signal based on said raw trace signal associated to said region of interest (6), obtain a PPG quality signal representative of the quality of the raw PPG signal associated to said region of interest (6) using the signal analysis machine learning model feed in with said raw photoplethysmography (PPG) signal, compute a processed PPG signal based on the PPG quality signal and the raw PPG signal associated to said region of interest (6), so as to reduce contribution of low-quality raw PPG signals in said computed processed PPG signals,• compute a global PPG signal of the at least one portion of the skin surface of the user based on the processed PPG signals computed for said regions of interest (6), at least one output configured to provide said global PPG signal.

5. The system according to claim 4, wherein at least one portion of the skin surface of the user comprises at least part of a face (3) of the user.

6. The system according to claim 5, wherein the mesh (5) is a dynamic mesh applied on the frames (4) using a pre-trained mesh generator machine learning model such that the face is tracked and regions of interest (6) are consistent along the video sequence, the machine learning model being trained on a dataset of face images labelled with a specific mesh.

7. The system according to any one of claims 4 to 6, wherein the raw trace signals are computed by filtering the values of the points of the region of interest (6), preferably by averaging the values of the points in the region of interest (6).

8. The system according to any one of claims 4 to 7, wherein, for one region of interest (6), the PPG quality signal is thresholded, and the corresponding processed PPG signal is computed by multiplying the obtained thresholded PPG quality signal and the corresponding PPG raw signal with each other.

9. The system according to any one of claims 4 to 8, wherein, for one region of interest (6), the raw PPG signal is obtained by filtering the raw trace signal associated to said region of interest (6), said filtering being adapted to the biosignal to obtain.

10. The system according to any one of claims 4 to 9, wherein said at least one processor is further configured to compute at least one biosignal of said user based on said global PPG signal, and wherein said at least one output is further configured to output said at least one biosignal.

11. The system according to claim 10, wherein the biosignal is one of the following: heart rate, oxygen saturation, respiratory rate, heart rate variability, diastolic pressure, systolic pressure, cardiac workload, hemoglobin level, glycemia.

12. The system according to claim 11 , wherein the biosignal is heart rate, and the heart rate is computed using frequential analysis of the global PPG signal.

13. A computer-implemented method (300) for obtaining a global PPG signal of at least one portion of a skin surface of a user from a video sequence of said at least one portion of a skin surface of said user, said method comprising: receiving a video sequence of a skin surface of the user (SI), for each frame (4) of said video sequence, applying a mesh on the frame of the video sequence to define regions of interest (4) on said at least one portion of skin surface(S2), wherein each region of interest comprises at least one point and each point is associated to a value, for each frame (4) of said video sequence and for each region of interest (6), computing a raw trace value (S3) based on at least said value of at least one point in said region of interest, for each region of interest: o computing, over the frames (4) of the video sequence, a raw trace signal of the video sequence (S4) associated to said region of interest (6) using said raw trace values computed for the region of interest (6), o computing a raw PPG signal (S5) based on said raw trace signal associated to said region of interest (6), o obtaining a PPG quality signal representative of the quality of the raw PPG signal associated to said region of interest (6) using the signal analysis machine learning model feed in with said raw photoplethysmography (PPG) signal (S6), o computing a processed PPG signal (S7) based on the PPG quality signal and the raw PPG signal associated to said region of interest (6), computing a global PPG signal (S8) of the at least one portion of the skin surface of the user based on the processed PPG signals computed for said regions of interest (6).

14. The method according to claim 13, wherein said method further comprises computing said biosignal (S9) based on said global PPG signal of the skin surface of the user.

15. A non-transitory program storage device comprising instructions which, when executed by a computer, cause the computer to carry out the method according to any one of claims 13 to 14.

16. A computer program comprising software code adapted to perform a computer implemented method according to any one of claims 13 to 14.

Images