Non-contact vital sign detection method and device based on self-supervised learning

Abstract

The invention relates to the technical field of image processing, and discloses a non-contact vital sign detection method and device based on self-supervised learning. The method specifically comprises the following steps: generating a visual-text pair based on a visual sample and a text cue word, extracting visual features and text features by adopting a visual-language model, and aligning the visual features and the text features by using a contrast loss function; reconstructing a shielded video region in the visual sample based on the visual features and the text features, and optimizing the reconstructed video sample by using a reconstruction loss function; the predicted physiological signal is optimized using a frequency contrast loss function and a frequency ranking loss function. A self-supervised learning method is adopted, accurate estimation of the physiological indexes can be realized without using an external PPG signal as supervision, and the demand on computing resources is reduced; through the design of the model and the loss function, the accuracy of signal extraction and the generalization performance of the model are improved; no contact type sensor in any form needs to be worn, the economic cost is low, and the privacy is good.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a non-contact vital sign detection method and device based on self-supervised learning. Background Art

[0002] Physiological measurement refers to the quantitative or qualitative measurement and analysis of human physiological phenomena through scientific instruments and methods. Existing physiological measurement technologies have obvious dependence on data acquisition. It is usually necessary to capture synchronized physiological signals, such as PPG signals, through contact sensors such as wearables or finger clips while recording facial videos. This approach not only makes the data collection and annotation process complicated and time-consuming, but also the experience is not ideal for participants. In addition, in order to ensure the synchronization between video data and physiological signals, it is often necessary to develop a special system, which undoubtedly increases the economic cost of data collection. At the same time, due to possible errors in the labels, the accuracy of the final algorithm will also be affected. This reliance on contact sensors limits the ease of application and user-friendliness of the technology, which in turn affects its application and popularization in a wider range of scenarios. Summary of the invention

[0003] The present application provides a non-contact vital sign detection method based on self-supervised learning to solve the problems in the prior art, such as the need to capture synchronous physiological signals through contact sensors, the increased economic costs due to the need to develop a dedicated system, and the impact of possible errors in labels on the accuracy of the algorithm.

[0004] Correspondingly, the present application also provides a non-contact vital sign detection device based on self-supervised learning, an electronic device, and a computer-readable storage medium to ensure the implementation and application of the above method.

[0005] In order to solve the above technical problems, the present application discloses a non-contact vital sign detection method based on self-supervised learning, the method comprising:

[0006] Generate visual-text pairs based on visual samples and corresponding text prompt words; the visual samples include positive samples and negative samples, and the text prompt words are used to describe the relative proportion of signal frequencies in positive samples and negative samples;

[0007] A pre-trained visual-language model is used to extract the visual features of the video sample and the text features of the text prompt words, and a preset contrastive loss function is used to align the visual features and text features;

[0008] Reconstruct the masked video area in the visual sample based on the visual features and text features, and use the preset reconstruction loss function to optimize the reconstructed video sample;

[0009] Physiological signals are predicted from different visual samples and optimized using the preset frequency contrast loss function and frequency ranking loss function.

[0010] Preferably, before generating a visual-text pair based on the visual sample and the corresponding text prompt word, the method further comprises:

[0011] Based on a given video, positive samples are generated through a spatial enhancement method, and negative samples with physiological signals of different frequencies are generated through a preset frequency enhancement module.

[0012] Preferably, generating a visual-text pair based on a visual sample and a corresponding text prompt word comprises:

[0013] Horizontally splice the space-time graphs of positive samples and negative samples to obtain a comparative space-time graph;

[0014] Generate text prompt words according to the relative ratio of the frequency of physiological signal changes on the left and right sides of the contrasting spatiotemporal graph;

[0015] Combining contrastive spatiotemporal graphs with textual prompt words forms visual-text pairs.

[0016] Preferably, reconstructing the masked video area in the visual sample based on the visual features and the text features includes:

[0017] Masking a part of the area in the contrast space-time map to generate a mask contrast space-time map;

[0018] The visual features are input into the preset visual graph reconstruction module, and the masked video areas in the visual samples are reconstructed through the self-attention mechanism and the cross-attention mechanism.

[0019] Preferably, the contrastive loss function aligns visual features with textual features by enhancing feature similarity between positive sample pairs and distinguishing features between negative sample pairs;

[0020] Positive sample pairs consist of videos with the same physiological signal frequency and corresponding text descriptions;

[0021] Negative sample pairs consist of videos of different physiological signal frequencies and their corresponding text descriptions.

[0022] Preferably, the reconstruction loss function optimizes the reconstructed video samples by minimizing the difference between the reconstruction result and the comparison spatiotemporal graph.

[0023] Preferably, the frequency contrast loss function adopts the InfoNCE loss function;

[0024] The frequency sorting loss function is used to constrain the frequency relationship between the positive sample signal set and the negative sample signal set to be consistent with the label; the positive sample signal set is the set formed by the results predicted from the positive samples, and the negative sample signal set is the set formed by the results predicted from the negative samples.

[0025] The present application also discloses a non-contact vital sign detection device based on self-supervised learning, the device comprising:

[0026] A sample generation module is used to generate visual-text pairs based on visual samples and corresponding text prompt words; the visual samples include positive samples and negative samples, and the text prompt words are used to describe the relative proportion of signal frequencies in the positive samples and the negative samples;

[0027] A feature extraction module, for extracting visual features of video samples and text features of text prompt words using a pre-trained visual-language model, and aligning the visual features and text features using a preset contrast loss function;

[0028] A video reconstruction module is used to reconstruct the masked video area in the visual sample based on the visual features and text features, and optimize the reconstructed video sample using a preset reconstruction loss function;

[0029] The signal prediction module is used to predict physiological signals from different visual samples and optimize the physiological signals using the preset frequency contrast loss function and frequency sorting loss function.

[0030] The present application also discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, one or more methods described in the present application are implemented.

[0031] The present application also discloses a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, one or more methods described in the present application are implemented.

[0032] In this application, visual-text pairs are generated based on visual samples and corresponding text prompts, visual features and text features are extracted using a visual-language model, and the contrast loss function is used to align the visual features and text features; the masked video area in the visual sample is reconstructed based on the visual features and text features, and the reconstruction loss function is used to optimize the reconstructed video sample, and the frequency contrast loss function and the frequency sorting loss function are used to optimize the predicted physiological signals. Using the self-supervised learning method, accurate estimation of physiological indicators such as heart rate can be achieved without the need for external PPG signals as supervision, reducing the demand for computing resources. At the same time, through the design of the model and the loss function, the accuracy of the extraction of physiological signals (such as heart rate, respiratory rate and heart rate variability) and the generalization performance of the model are improved. In addition, the method in this application does not require the wearing of any form of contact sensor, which reduces the economic cost and protects the privacy rights and interests of users.

[0033] Additional aspects and advantages of the present application will be given in the following description, which will become apparent from the following description, or will be understood through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0035] Figure 1 A flowchart of a non-contact vital sign detection method based on self-supervised learning provided in an embodiment of the present application;

[0036] Figure 2 A schematic diagram of a visual-language model provided in an embodiment of the present application;

[0037] Figure 3 An overall framework diagram of the non-contact vital signs detection method provided in an embodiment of the present application;

[0038] Figure 4 A schematic diagram of generating a spatiotemporal graph of visual-text pairs and mask comparison provided in an embodiment of the present application;

[0039] Figure 5 A structural diagram of a visual image reconstruction module provided in an embodiment of the present application;

[0040] Figure 6 A schematic diagram of the structure of a non-contact vital sign detection device based on self-supervised learning provided in an embodiment of the present application;

[0041] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0042] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present application.

[0043] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0044] Those skilled in the art will appreciate that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art in the field to which the present invention belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0045] The solution provided in the embodiments of the present application can be executed by any electronic device, such as a terminal device or a server, wherein the server can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected via wired or wireless communication, and this application is not limited here. For the technical problems existing in the prior art, the non-contact vital signs detection method and device based on self-supervised learning provided in this application is intended to solve at least one of the technical problems of the prior art.

[0046] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0047] The present application embodiment provides a possible implementation method, such as Figure 1 As shown, a flowchart of a non-contact vital sign detection method based on self-supervised learning is provided, which can be executed by any electronic device, and optionally, can be executed on a server or terminal device.

[0048] like Figure 1 As shown in , the method may include the following steps:

[0049] Step 101, generating a visual-text pair based on a visual sample and a corresponding text prompt word; the visual sample includes a positive sample and a negative sample, and the text prompt word is used to describe the relative proportion of the signal frequency in the positive sample and the negative sample;

[0050] In the embodiment of the present application, a remote photoplethysmography signal (rPPG) is obtained from the collected facial video to measure key physiological indicators such as heart rate, heart rate variability and respiratory rate. The positive sample retains the rPPG signal frequency of the facial video, while the negative sample has an rPPG signal of a different frequency.

[0051] Step 102: Use a pre-trained visual-language model to extract visual features of the video sample and text features of the text prompt word, and use a preset contrast loss function to align the visual features and text features.

[0052] Step 103: reconstruct the masked video area in the visual sample based on the visual features and the text features, and optimize the reconstructed video sample using a preset reconstruction loss function.

[0053] Step 104 , predicting physiological signals from different visual samples, and optimizing the physiological signals using a preset frequency contrast loss function and a frequency sorting loss function.

[0054] The embodiment of the present application integrates visual-language models (VLMs) into the remote physiological measurement task, and estimates the remote photoplethysmography signal through self-supervised learning, thereby realizing non-contact measurement of key physiological indicators such as heart rate, heart rate variability and respiratory rate. Specifically, the embodiment of the present application adopts a multi-task learning framework, including a visual graph reconstruction task, a visual-text contrast learning task, and a frequency contrast and sorting task. These tasks are optimized through loss functions, including a text-guided visual graph reconstruction task to restore the obscured video area, and a visual-text contrast learning task to align visual and text features. In addition, a frequency contrast and sorting task is introduced to optimize the rPPG signal estimated from different video samples; the model is optimized end-to-end through reconstruction loss, visual-text contrast loss, frequency contrast loss and frequency sorting loss. Among them, the loss function is the above-mentioned reconstruction loss function L r , contrast loss function L vtc , frequency contrast loss function L fc And the frequency ranking loss function L fr A linear combination of:

[0055] L=L r +L vtc +L fc +L fr

[0056] The optimization process of each task in the multi-task learning framework can achieve accurate estimation of physiological indicators such as heart rate without the need for external PPG signals as supervision, greatly simplifying the monitoring process, especially in situations where skin contact measurement is not convenient.

[0057] In the embodiment of the present application, the need for synchronous recording of PPG signals is reduced through self-supervised learning, making remote physiological signal measurement technology more portable and practical. At the same time, through innovative module design and loss function, the accuracy of extraction of physiological signals (such as heart rate, respiratory rate and heart rate variability) and the generalization performance of the model are improved. In addition, the method implemented in the present application does not require wearing any form of contact sensor to provide users with a more private and comfortable monitoring method, making physiological monitoring almost imperceptible. This method not only improves user acceptance, but also broadens the scope of application of physiological monitoring technology, and opens up new paths for telemedicine services and personal health monitoring. In many fields such as home medical care, emergency medical services, and monitoring of public health events, it shows great application potential and practical value, and is expected to become an important force to promote relevant technological progress and innovation. Therefore, the method of self-supervised learning is adopted in the embodiment of the present application, without the need for external PPG signals as supervision, to achieve accurate estimation of physiological indicators such as heart rate, reduce the demand for computing resources, reduce economic costs, and protect the privacy rights and interests of users.

[0058] In an optional embodiment, before generating a visual-text pair based on the visual sample and the corresponding text prompt word, the method further includes:

[0059] Based on a given video, positive samples are generated through a spatial enhancement method, and negative samples with physiological signals of different frequencies are generated through a preset frequency enhancement module.

[0060] like Figure 3 As shown, for facial video x a , using spatial enhancement techniques such as image rotation (0°, 90°, 180° and 270°) and horizontal / vertical flipping to process the input facial video. These spatial enhancement methods do not change the image color information, only the pixel position, so the skin color change process caused by heartbeat in the facial video will not be disturbed. In each round of model training, two spatial enhancement mechanisms are randomly selected and applied to the video slices to generate positive samples X with the same frequency as the original video rPPG signal p .

[0061] A frequency enhancement (LFA) module is designed to generate negative samples with rPPG signals of different frequencies. Optionally, the pyramid structure of the frequency enhancement module includes a 3D convolutional layer, a 3D residual block, and a downsampling operation to generate multi-scale features that reflect the spatial details of the video at different scales. At each feature scale, the rPPG signal frequency of the feature is changed by the frequency modulation block, and the spatial dimension is unified by the upsampling operation. Finally, the video clip is reconstructed by the 3D convolutional layer to generate the negative sample X. n .

[0062] In an optional embodiment, if Figure 2 , generating visual-text pairs based on visual samples and corresponding text prompt words, including:

[0063] Horizontally splice the space-time graphs of positive samples and negative samples to obtain a comparative space-time graph;

[0064] Generate text prompt words according to the relative ratio of the frequency of physiological signal changes on the left and right sides of the contrasting spatiotemporal graph;

[0065] Combining contrastive spatiotemporal maps with textual cues to form frequency-guided visual-text pairs.

[0066] The text prompt words are used to describe the frequency ratio of the skin color of the sample changing over time.

[0067] Optionally, construct spatiotemporal maps (STMaps) as follows Figure 3 As shown, for the generated positive sample X p and negative samples X nFor each video frame in the video, multiple facial regions of interest (ROIs) are located, and the RGB three-channel pixel means of the ROIs are calculated respectively, and the calculated means are arranged in rows along the time dimension to form a spatiotemporal graph. That is, each row of the spatiotemporal graph is a skin color time variation signal of a certain facial ROI. Each column represents the skin color pixel value of a different ROI at the current moment. The spatiotemporal graph corresponding to the positive sample in the embodiment of the present application is denoted as M p , the space-time graph corresponding to the negative sample is denoted as M n .

[0068] like Figure 4 As shown, in the embodiment of the present application, the positive sample spatiotemporal graph M is horizontally spliced p and negative sample negative sample space-time graph M n Constructing Comparative Spatiotemporal Maps (C-STMaps) c , to show the skin color change process at different frequencies. Optionally, the positive sample spatiotemporal map M p The frequency is f a , negative sample negative sample space-time graph M n The frequency is 0.5×f a , for the positive sample space-time graph M p and negative sample space-time graph M n Merge and crop, the cropping method can be selected as center cropping, and then generate a comparative space-time map M c For a comparative space-time graph M c , the relative frequency ratio of skin color changes on the left and right sides (i.e., frequency ratio R) is specific, and then a text prompt word D is generated based on the ratio, such as "the frequency of horizontal color changes on the left / right side of the image is [ratio] times that on the right / left side." If the ratio is 1 / 2, the text prompt word D is "the frequency of horizontal color changes on the left / right side of the image is 1 / 2 times that on the right / left side." The contrasting spatiotemporal graph and text prompt words can be combined to form a visual-text pair to guide the model to understand frequency-related visual attributes.

[0069] In an optional embodiment, if Figure 3 As shown, the visual-language model includes a visual encoder and a text encoder;

[0070] Use a pre-trained visual-language model to extract visual features of video samples and text features of text prompts, including:

[0071] The contrastive spatiotemporal graph is input into the visual encoder to extract visual features based on the self-attention mechanism and feedforward neural network;

[0072] The text prompt words are input into the text encoder, and the BERT model is used to extract text features.

[0073] In the embodiment of the present application, the input of the visual encoder is the contrast space-time graph. The visual encoder is based on the visual transformer architecture, which converts the input contrast space-time graph M c The image is divided into multiple patches, and feature representations are generated for each patch through a self-attention mechanism and a feed-forward network. The features of these patches are then integrated to form a complete visual feature, which includes a special [CLS] tag that represents the global visual feature of the contrastive spatiotemporal map.

[0074] The input of the text encoder is the text prompt word. The text encoder uses the BERT model to receive the text prompt word and convert it into text features. The text encoder uses its transformer structure to understand the semantic information of the text. Similar to the visual encoder, the text encoder also outputs a [CLS] tag as the global text feature of the text prompt word.

[0075] In the feature extraction stage, the contrastive spatiotemporal graph and the corresponding textual cues are fed into the visual encoder and textual encoder, respectively. The visual and textual features output by the encoder are then used to calculate the similarity of visual-text pairs and optimized by the designed contrastive loss function. In addition, the global [CLS] tags extracted from the positive samples are fed into the constructed rPPG estimation layer for predicting the rPPG signal. The predicted rPPG signal is further used for frequency contrast and ranking tasks to optimize the model parameters.

[0076] In the embodiment of the present application, visual-text pairs can be used for pre-training to obtain a visual-language model with high accuracy. For example, Figure 2 Four groups of visual-text pairs are shown on the right. The first group includes "tomato" images and the corresponding text prompts "cherry tomatoes ripening on the vine, of different sizes, ranging from bright red to orange and green"; the second group includes "hair" images and the corresponding text prompts "long colorful hair gradating from green to blue to purple"; the third group includes "kitchen" images and the corresponding text prompts "In the past few months, we have been admiring colorful kitchens - from blue to green, from red to yellow"; the fourth group includes "sky" images and the corresponding text prompts "the color of the sky changes from gold to orange and then to blue, and the clouds move from the middle to the right". The above groups of visual-text pairs are input into the visual-language model, and text-guided visual image reconstruction can be achieved through visual-text contrast learning. In addition, the features extracted by the visual encoder can predict the rPPG signal, and the predicted rPPG signal can be optimized by calculating the frequency contrast loss.

[0077] In an optional embodiment, reconstructing the masked video area in the visual sample based on the visual features and the text features includes:

[0078] Masking a part of the area in the contrast space-time map to generate a mask contrast space-time map;

[0079] The visual features are input into the preset visual graph reconstruction module, and the masked video areas in the visual samples are reconstructed through the self-attention mechanism and the cross-attention mechanism.

[0080] like Figure 3 and Figure 4 As shown, each comparison space-time graph M c Divide into multiple non-overlapping patches, and randomly select some small patches in a certain proportion for masking (i.e., random masking) to generate a mask contrast spatiotemporal map (M-STMap) M o , these masked patches will serve as targets that the model needs to reconstruct.

[0081] like Figure 5 As shown in the figure, a text-guided visual map reconstruction (TVR) module is designed. The input of the TVR module is a concatenated sequence of visual features containing masked and unmasked patches (z o ). This module consists of three cross-attention submodules. In each submodule, a self-attention layer is first included, which enables the model to capture the visual context information of the masked patch area based on the unmasked area. Then, cross attention calculation is performed to integrate the semantic information of the text prompt word Interacting with visual features enables the model to reconstruct the masked patches using the frequency description in the text prompt words, enhancing the model's understanding of the frequency of skin color changes, thereby obtaining a joint representation Improve the ability to estimate rPPG signals in videos and use feed-forward layers to further improve After executing three such blocks, the patch is reconstructed and the output (m q ) is mapped back to the original RGB space. In addition, during the self-attention and cross-attention processes, layer normalization and residual connections are used to improve the learning ability of the model and avoid the gradient vanishing problem.

[0082] In an optional embodiment, the reconstruction loss function optimizes the reconstructed video samples by minimizing the difference between the reconstruction result and the comparison spatiotemporal image.

[0083] like Figure 5 As shown in Figure 2, the TVR module maps the features back to the original RGB space through a series of convolution and upsampling operations, completes the reconstruction of the mask area, and obtains the reconstructed feature m qThe reconstruction loss (Lr) is used to quantify the difference between the reconstruction result and the original C-STMaps, and by minimizing this difference, it ensures that the model can effectively restore the periodic law of skin color changes. Enhance the model's ability to capture and understand physiological signal characteristics.

[0084] The expression of the reconstruction loss function is:

[0085]

[0086] Where MSE(.) represents mean square error, m q Represents the reconstructed visual image, m c Represents the original contrasting spatiotemporal graph, with a total of N reconstructed visual graphs.

[0087] In an optional embodiment, the contrastive loss function achieves alignment of visual features and text features by enhancing feature similarity between positive sample pairs and distinguishing features between negative sample pairs;

[0088] Positive sample pairs consist of videos with the same physiological signal frequency and corresponding text descriptions;

[0089] Negative sample pairs consist of videos of different physiological signal frequencies and their corresponding text descriptions.

[0090] The visual-text contrast loss is calculated through the contrast loss function to achieve the alignment of visual features and text features. The expression of the contrast loss function is:

[0091]

[0092] in represents the cosine similarity between the visual [CLS] token and the textual [CLS] token embeddings, τ1 is a hyperparameter, and exp(.) represents the exponential function.

[0093] In an optional embodiment, the frequency contrast loss function adopts the InfoNCE loss function;

[0094] In the present application embodiment, Figure 3 As shown in the figure, the positive sample signal set is brought closer in the feature space, while it is pushed away from the negative sample signal set in the feature space. By optimizing the frequency distribution of positive and negative samples, the model is guided to focus on the frequency commonalities and differences of the positive and negative sample sets. The InfoNCE loss function commonly used in contrastive learning mechanism is used to define the frequency contrast loss:

[0095]

[0096] in Indicates signal and The mean square error between the power spectral density (PSD) of , M represents the number of negative samples. τ2 is a hyperparameter.

[0097] The frequency sorting loss function is used to constrain the frequency relationship between the positive sample signal set and the negative sample signal set to be consistent with the label; the positive sample signal set is the result obtained by predicting from the positive sample (positive signal Y n ), the negative sample signal set is the result obtained by predicting from the negative sample (negative signal Y p ) is the set formed by .

[0098] Considering that negative samples are generated according to a specific frequency ratio, the frequencies of these negative samples and positive samples should follow a certain size relationship (i.e., sorting). Constraining the predicted frequency size relationship of the positive and negative sample signal sets to be consistent with the labels can improve the accuracy of the model's estimated heart rate and other vital signs. The expression of the frequency sorting loss function is:

[0099]

[0100] in is a logical function s(.) is the predicted frequency level of the signal, l(.) is the frequency level label of the signal. Q is the sum of the number of positive and negative samples. q and c are any two signals in the set of positive and negative signals.

[0101] Based on the same principle as the method provided in the embodiment of the present application, the embodiment of the present application also provides a non-contact vital sign detection device based on self-supervised learning, such as Figure 6 As shown, the device comprises:

[0102] A sample generation module 601 is used to generate a visual-text pair based on a visual sample and a corresponding text prompt word; the visual sample includes a positive sample and a negative sample, and the text prompt word is used to describe the relative proportion of the signal frequency in the positive sample and the negative sample;

[0103] A feature extraction module 602 is used to extract visual features of video samples and text features of text prompt words 603 using a pre-trained visual-language model, and align the visual features and text features using a preset contrast loss function;

[0104] A video reconstruction module 604 is used to reconstruct the masked video area in the visual sample based on the visual features and the text features, and optimize the reconstructed video sample using a preset reconstruction loss function;

[0105] The signal prediction module 605 is used to predict the physiological signal from different visual samples and optimize the physiological signal using a preset frequency contrast loss function and a frequency sorting loss function.

[0106] In the embodiment of the present application, the need for synchronous recording of PPG signals is reduced through self-supervised learning, making remote physiological signal measurement technology more portable and practical. At the same time, through innovative module design and loss function, the accuracy of extraction of physiological signals (such as heart rate, respiratory rate and heart rate variability) and the generalization performance of the model are improved. In addition, the method implemented in the present application does not require wearing any form of contact sensor to provide users with a more private and comfortable monitoring method, making physiological monitoring almost imperceptible. This method not only improves user acceptance, but also broadens the scope of application of physiological monitoring technology, and opens up new paths for telemedicine services and personal health monitoring. In many fields such as home medical care, emergency medical services, and monitoring of public health events, it shows great application potential and practical value, and is expected to become an important force to promote relevant technological progress and innovation. Therefore, the method of self-supervised learning is adopted in the embodiment of the present application, without the need for external PPG signals as supervision, to achieve accurate estimation of physiological indicators such as heart rate, reduce the demand for computing resources, reduce economic costs, and protect the privacy rights and interests of users.

[0107] The non-contact vital sign detection device based on self-supervised learning provided in the embodiment of the present application can achieve Figures 1 to 5 To avoid repetition, the various processes implemented in the method embodiment are not described here.

[0108] The non-contact vital signs detection device based on self-supervised learning in the embodiments of the present application can execute the non-contact vital signs detection method based on self-supervised learning provided in the embodiments of the present application. The implementation principles are similar. The actions performed by each module and unit in the non-contact vital signs detection device based on self-supervised learning in each embodiment of the present application correspond to the steps in the non-contact vital signs detection method based on self-supervised learning in each embodiment of the present application. For the detailed functional description of each module of the non-contact vital signs detection device based on self-supervised learning, please refer to the description of the corresponding non-contact vital signs detection method based on self-supervised learning shown in the previous text, which will not be repeated here.

[0109] Based on the same principle as the method shown in the embodiment of the present application, the embodiment of the present application also provides an electronic device, which may include but is not limited to: a processor and a memory; a memory for storing a computer program; a processor for executing the non-contact vital sign detection method based on self-supervised learning shown in any optional embodiment of the present application by calling a computer program. Compared with the prior art, the non-contact vital sign detection method based on self-supervised learning provided by the present application adopts a self-supervised learning method, which can achieve accurate estimation of physiological indicators such as heart rate without the need for external PPG signals as supervision, thereby reducing the demand for computing resources. At the same time, through the design of the model and the loss function, the accuracy of the extraction of physiological signals (such as heart rate, respiratory rate and heart rate variability) and the generalization performance of the model are improved. In addition, it is not required to wear any form of contact sensor, which reduces the economic cost and protects the privacy rights and interests of users.

[0110] In an optional embodiment, an electronic device is also provided, such as Figure 7 As shown, Figure 7 The electronic device 700 shown may be a server, including: a processor 701 and a memory 703. The processor 701 and the memory 703 are connected, such as through a bus 702. Optionally, the electronic device 700 may further include a transceiver 704. It should be noted that in actual applications, the transceiver 704 is not limited to one, and the structure of the electronic device 700 does not constitute a limitation on the embodiments of the present application.

[0111] Processor 701 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 701 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.

[0112] The bus 702 may include a path to transmit information between the above components. The bus 702 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 702 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0113] The memory 703 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to these.

[0114] The memory 703 is used to store the application code for executing the solution of the present application, and the execution is controlled by the processor 701. The processor 701 is used to execute the application code stored in the memory 703 to implement the contents shown in the above method embodiment.

[0115] Among them, electronic devices include but are not limited to: mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 7 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0116] The server provided in this application can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal can be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected by wired or wireless communication, and this application does not limit this.

[0117] An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer can execute the corresponding content in the aforementioned method embodiment.

[0118] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0119] It should be noted that the computer-readable storage medium mentioned above in the present application can also be a computer-readable signal medium or a combination of a computer-readable storage medium and a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0120] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0121] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0122] According to one aspect of the present application, a computer program product or computer program is provided, the computer program product or computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the non-contact vital sign detection method and apparatus based on self-supervised learning provided in the above-mentioned various optional implementations.

[0123] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0124] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0125] The modules involved in the embodiments described in this application can be implemented by software or hardware. The name of the module does not limit the module itself in some cases. For example, the sample generation module can also be described as "a sample generation module for generating visual-text pairs based on visual samples and corresponding text prompt words".

[0126] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

Claims

1. A non-contact vital sign detection method based on self-supervised learning, characterized in that: The method comprises: Generate a visual-text pair based on a visual sample and a corresponding text prompt word; the visual sample includes a positive sample and a negative sample, and the text prompt word is used to describe the relative proportion of the signal frequency in the positive sample and the negative sample; Extracting visual features of the video sample and text features of the text prompt word using a pre-trained visual-language model, and aligning the visual features and the text features using a preset contrast loss function; Reconstructing the masked video area in the visual sample based on the visual features and the text features, and optimizing the reconstructed video sample using a preset reconstruction loss function; Physiological signals are predicted from different visual samples, and the physiological signals are optimized using a preset frequency contrast loss function and a frequency sorting loss function.

2. The non-contact vital sign detection method based on self-supervised learning according to claim 1, characterized in that: Before generating a visual-text pair based on the visual sample and the corresponding text prompt word, the method further includes: Based on a given video, the positive sample is generated by a spatial enhancement method, and the negative sample with physiological signals of different frequencies is generated by a preset frequency enhancement module.

3. The non-contact vital sign detection method based on self-supervised learning according to claim 2, characterized in that: The generating of a visual-text pair based on a visual sample and a corresponding text prompt word comprises: horizontally splicing the space-time graphs of the positive sample and the negative sample to obtain a comparative space-time graph; Generating the text prompt word according to the relative ratio of the frequency of the physiological signal changes on the left and right sides of the comparative spatiotemporal graph; The contrasting spatiotemporal graph and the textual prompt word are combined to form the visual-textual pair.

4. The non-contact vital sign detection method based on self-supervised learning according to claim 3 is characterized in that: The step of reconstructing the masked video area in the visual sample based on the visual feature and the text feature includes: Masking a part of the area in the comparison space-time map to generate a mask comparison space-time map; The visual features are input into a preset visual image reconstruction module, and the masked video area in the visual sample is reconstructed through a self-attention mechanism and a cross-attention mechanism.

5. The non-contact vital sign detection method based on self-supervised learning according to claim 1, characterized in that: The contrast loss function achieves alignment of the visual features and the text features by enhancing the feature similarity between the positive sample pairs and distinguishing the features of the negative sample pairs; The positive sample pair consists of a video with the same physiological signal frequency and a corresponding text description; The negative sample pairs consist of videos of different physiological signal frequencies and corresponding text descriptions.

6. The non-contact vital sign detection method based on self-supervised learning according to claim 1, characterized in that: The reconstruction loss function optimizes the reconstructed video samples by minimizing the difference between the reconstruction result and the contrast spatiotemporal graph.

7. The non-contact vital sign detection method based on self-supervised learning according to claim 1, characterized in that: The frequency contrast loss function adopts the InfoNCE loss function; The frequency sorting loss function is used to constrain the frequency size relationship between the positive sample signal set and the negative sample signal set to be consistent with the label; the positive sample signal set is a set formed by the results predicted from the positive samples, and the negative sample signal set is a set formed by the results predicted from the negative samples.

8. A non-contact vital sign detection device based on self-supervised learning, characterized in that: The device comprises: A sample generation module, used to generate a visual-text pair based on a visual sample and a corresponding text prompt word; the visual sample includes a positive sample and a negative sample, and the text prompt word is used to describe the relative proportion of the signal frequency in the positive sample and the negative sample; A feature extraction module, used to extract the visual features of the video sample and the text features of the text prompt word using a pre-trained visual-language model, and align the visual features and the text features using a preset contrast loss function; A video reconstruction module, used to reconstruct the masked video area in the visual sample based on the visual features and the text features, and optimize the reconstructed video sample using a preset reconstruction loss function; The signal prediction module is used to predict physiological signals from different visual samples and optimize the physiological signals using a preset frequency contrast loss function and a frequency sorting loss function.

9. An electronic device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images