Non-contact coronary heart disease assessment device based on facial video

Abstract

The application discloses a non-contact coronary heart disease evaluation device based on a face video and belongs to the technical field of intelligent medical treatment. It comprises a data extraction module, which is used for extracting a video pulse signal of a certain feature region in a target object video stream and a frame image corresponding to the video pulse signal as a training set, so as to train a first prediction model and a second prediction model; a judgment module is used for outputting an evaluation result according to the overall uncertainty judgment result of the first prediction model and the second prediction model; the application comprehensively considers two factors of pulse features and facial features, respectively evaluates the uncertainty of the prediction of two models, selects the prediction result of the smaller uncertainty as the final coronary heart disease detection result, more comprehensively evaluates the coronary heart disease related features, and makes the detection result more reliable.

Description

Technical Field

[0001] This invention belongs to the field of intelligent medical technology, specifically, it relates to a non-contact coronary heart disease assessment device based on facial video. Background Technology

[0002] Coronary atherosclerotic heart disease, or CAD for short, refers to heart disease caused by myocardial ischemia and hypoxia due to coronary atherosclerosis. CAD is a common disease among middle-aged and elderly people, and it can lead to complications such as myocardial infarction, sudden cardiac death, and stroke, seriously endangering the patient's life and health. Early diagnosis and treatment of CAD are crucial for improving treatment outcomes and prognosis. However, many CAD patients do not experience obvious symptoms in the early stages, and their lack of awareness of their condition can lead to delayed treatment.

[0003] Clinical guidelines recommend screening for coronary artery disease (CAD) in high-risk groups to improve the detection rate of asymptomatic CAD. Traditional screening methods include electrocardiograms (ECG) and coronary multi-slice CT angiography. However, these methods rely on specialized medical equipment and complex procedures, making them difficult to apply to routine screening of large populations. Recent studies have shown that CAD can cause pathological changes in the pulse wave, such as pulse rate arrhythmia and disappearance of dicrotic waves. Photoplethysmography (PPG) is a commonly used pulse wave detection method. It uses optical sensors to capture changes in subcutaneous blood volume in areas such as the fingertips and wrists as the pulse changes, enabling convenient pulse wave detection. Identifying pathological pulse wave characteristics using PPG technology can serve as a screening method for CAD. On the other hand, a research team from Fuwai Hospital published an article in the *European Heart Journal* indicating that CAD patients often exhibit typical facial features such as baldness on the forehead, deep eye bags, and pale lips. Establishing a mapping relationship between facial features and CAD through an artificial intelligence model could serve as another screening method for CAD. However, some patients with coronary heart disease have relatively distinct pulse characteristics, and others have relatively distinct facial features. These two screening methods only consider a subset of these characteristics, and their accuracy still has room for improvement.

[0004] With the rapid development of computer vision and imaging equipment, Video Photoplethysmography (VPPG) has become a cutting-edge research direction in the field of PPG. Facial skin color undergoes subtle changes with pulse. VPPG captures these changes using a regular camera and extracts the pulse wave using computer vision technology, enabling non-contact pulse wave detection based on facial video. However, both pulse wave-based and facial feature-based coronary heart disease screening methods have limitations; the features they incorporate are relatively limited, making it difficult to obtain reliable assessment results for coronary heart disease. Summary of the Invention

[0005] To address the limitations and inaccuracies of existing pulse wave-based coronary artery disease (CAD) assessment methods, the following embodiment provides a non-contact CAD assessment device based on facial video. This device comprehensively considers both pulse and facial features, feeding the pulse wave and facial image from the facial video into a neural network to construct two CAD prediction models. Based on this, the uncertainty of the two models is evaluated, and the prediction result with the lower uncertainty is selected as the final CAD detection result. Compared to existing CAD screening methods, this approach provides a more comprehensive assessment of CAD-related characteristics, making the detection results more reliable.

[0006] To solve the above problems, the present invention adopts the following technical solution.

[0007] The first aspect of the present invention provides a non-contact coronary heart disease assessment device based on facial video, comprising:

[0008] The data extraction module is used to extract the video pulse signal containing a certain feature region in the video stream of the target object and the frame image corresponding to the video pulse signal as a training set;

[0009] The first prediction model is obtained by training the video pulse signal as a training set.

[0010] The second prediction model is obtained by training the first frame image corresponding to the video pulse signal as a training set.

[0011] The judgment module is used to judge the overall uncertainty of the first prediction model and the second prediction model and output the evaluation result. When the overall uncertainty of the first prediction model is greater than the overall uncertainty of the second prediction model, the prediction result of the second prediction model is selected as the evaluation result. When the uncertainty of the second prediction model is greater than the uncertainty of the first prediction model, the prediction result of the first prediction model is selected as the evaluation result.

[0012] As an example, the judgment module also includes a quantization calculation module;

[0013] The quantization calculation module is used to quantify the average data uncertainty and model uncertainty of the first and second prediction models, and to calculate the overall uncertainty of the first and second prediction models based on the average data uncertainty and model uncertainty.

[0014] As an example, the quantization calculation module includes a first calculation unit and a second calculation unit;

[0015] The first computing unit is used to obtain the average data uncertainty of the first prediction model and the second prediction model. The average data uncertainty is obtained by adding an extra output branch to the output layer of the first prediction model and the second prediction model, and introducing the average data uncertainty parameter into the loss function of the first prediction model and the second prediction model. The model learns the average data uncertainty autonomously during the training process and obtains it through the output of the newly added branch.

[0016] The second calculation unit is used to obtain the model uncertainty of the first prediction model and the second prediction model. The model uncertainty is mainly obtained by measuring the weight changes of the first prediction model and the second prediction model when processing the same sample.

[0017] As an example, the first prediction model, after introducing the average data uncertainty parameter, yields a rewritten first loss function, the formula of which is:

[0018]

[0019] in The first prediction model's loss function does not include uncertainty parameters. It is the average data uncertainty; It is a probability vector The first in c One element, This is the probability vector output by the first prediction model. Here, is the mapping equation for the model, x is the input of the first prediction model, W1 is the weight of the first prediction model, and exp is the natural exponent operation. It is an element-wise addition.

[0020] The second prediction model, after introducing the average data uncertainty parameter, yields a rewritten second loss function, the formula of which is:

[0021]

[0022] in, The second prediction model is a loss function that does not include uncertainty parameters. W1 represents the average data uncertainty of the second prediction model; W2 represents the weights of the second prediction model.

[0023] As an example, it also includes an optimization module, which comprises a Bayesian unit, a sampling unit, and a training unit:

[0024] A Bayesian unit is used to replace the fixed values ​​of the weights of the first and second prediction models with probability distributions using a Bayesian network.

[0025] A sampling unit is used to add a sampling layer before each layer of the first and second prediction models, and to perform Monte Carlo sampling during training and testing.

[0026] The training unit updates the first and second prediction models based on the redetermined first and second loss functions, respectively. The second calculation unit then obtains the model uncertainties of the first and second prediction models and calculates their uncertainties.

[0027] As an example, the data extraction module includes:

[0028] The position detection unit is used to detect the position of the face in each frame of the video stream according to the face detection algorithm and obtain the facial feature points.

[0029] The tracking unit is used to counteract the rigid motion interference of the face by tracking the motion trajectory of facial feature points in the video stream over time, and to divide the cheek region based on the cheek periphery feature points among the facial feature points.

[0030] The signal extraction unit is used to extract the green channel information of the cheek video, calculate the average value of all pixels in each frame of the video stream, and obtain the skin color change signal.

[0031] The separation unit is used to filter the skin color change signal using a bandpass filter, thereby separating the video pulse wave signal.

[0032] A second aspect of the present invention provides an electronic device, including a processor, an input device, an output device, and a memory, wherein the processor, input device, output device, and memory are connected in sequence.

[0033] The memory is used to store computer programs, which include program instructions.

[0034] The processor is configured according to the program instructions to:

[0035] Extract the video pulse signal containing a certain feature region in the video stream of the target object and the frame image corresponding to the video pulse signal;

[0036] Based on the first prediction model trained using the video pulse signal as the training set, a first evaluation result is obtained.

[0037] A second evaluation result is obtained based on the second prediction model trained using the first frame image corresponding to the video pulse signal as the training set.

[0038] Based on the overall uncertainty judgment results of the first prediction model and the second prediction model, the evaluation result is output; when the overall uncertainty of the first prediction model is greater than the overall uncertainty of the second prediction model, the prediction result of the second prediction model is selected as the evaluation result; when the uncertainty of the second prediction model is greater than the uncertainty of the first prediction model, the prediction result of the first prediction model is selected as the evaluation result.

[0039] As an example, the overall uncertainty calculation method includes:

[0040] Quantitatively calculate the average data uncertainty and model uncertainty of the first and second prediction models;

[0041] Based on the average data uncertainty and model uncertainty of the first and second prediction models, calculate the overall uncertainty of the first and second prediction models respectively.

[0042] As an example, the average data uncertainty is obtained by adding an extra output branch to the output layer of the first and second prediction models, and introducing the average data uncertainty parameter into the loss function of the first and second prediction models. The model learns the average data uncertainty autonomously during training and obtains it through the newly added branch.

[0043] The second calculation unit is used to obtain the model uncertainty of the first prediction model and the second prediction model. The model uncertainty is mainly obtained by measuring the weight changes of the first prediction model and the second prediction model when processing the same sample.

[0044] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0045] This invention comprehensively considers both pulse and facial features, feeding the pulse wave from a face video and the face image into a neural network to construct two coronary heart disease prediction models. Based on this, the uncertainty of the predictions of the two models is evaluated, and the prediction result of the model with the smaller uncertainty is selected as the final coronary heart disease detection result. Compared with existing coronary heart disease screening methods, this invention provides a more comprehensive assessment of coronary heart disease-related features, and the detection results are more reliable. Attached Figure Description

[0046] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The drawings are used to provide a further understanding of the embodiments of this application and constitute a part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps. In the drawings:

[0047] Figure 1This is a block diagram of a non-contact coronary heart disease assessment device based on facial video provided in an embodiment of the present invention;

[0048] Figure 2 This is a flowchart of a non-contact coronary heart disease assessment method based on facial video provided in an embodiment of the present invention;

[0049] Figure 3 This is a block diagram of the first prediction model structure provided in an embodiment of the present invention;

[0050] Figure 4 The illustration shows a block diagram of an electronic device according to an embodiment of this application;

[0051] Figure 5 This is a block diagram of the second prediction model structure provided in an embodiment of the present invention;

[0052] Figure 6 This is a block diagram of the judgment logic of the judgment module provided in an embodiment of the present invention. Detailed Implementation

[0053] Hereinafter, exemplary embodiments according to this application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments of this application. It should be understood that this application is not limited to the exemplary embodiments described herein.

[0054] Exemplary device

[0055] The non-contact coronary heart disease assessment device based on facial video can include a computer, an input module, a communication module, and a display module in application scenarios. The computer is connected to the input module, the communication module, and the display module respectively. The communication module is used to connect to the electronic medical record system and the hospital's test database, or to the output end of the camera, so as to obtain the patient's medical record data.

[0056] See Figure 1 As shown, this example of a non-contact coronary artery disease assessment device based on facial video generally includes:

[0057] The data extraction module 20 is used to extract the video pulse signal containing a certain feature region in the video stream of the target object, and the corresponding frame image as a training set. The target object can be a patient with coronary heart disease in a hospital, or a person without coronary heart disease. The video pulse signal can be obtained from the patient with coronary heart disease using video photoplethysmography. The video pulse signal is used as training set one, and the first frame image corresponding to the video pulse signal is used as training set two.

[0058] As an example, the data extraction module includes:

[0059] The position detection unit 210 is used to detect the position of the face in each frame of the video stream according to the face detection algorithm and obtain the facial feature points.

[0060] Specifically, a 30-second video of a face is recorded using a camera; the Viola-Jones face detection algorithm is used to detect the face position in each frame of the video, and the Discriminative Response Map Fitting (DRM) algorithm is used to detect the position of 66 feature points of the face.

[0061] The interference removal unit 220 is used to cancel the rigid motion interference of the face based on the motion trajectory of the tracked facial feature points in the video stream over time; and to divide the cheek region based on the cheek peripheral feature points among the facial feature points.

[0062] Specifically, the Kanade-Lucas-Tomasi algorithm is used to track the motion trajectory of 66 facial feature points in the video over time to counteract the interference of rigid facial motion; then, based on the cheek periphery feature points among the 66 feature points obtained above, the cheek region is divided to shield interference from other regions.

[0063] The signal extraction unit 230 is used to extract the green channel information of the cheek video, calculate the average value of all pixels in each frame of the video, and establish a one-dimensional time change sequence, i.e., the skin color change signal.

[0064] The separation unit 240 is used to filter the skin color change signal with a bandpass filter to separate the video pulse wave signal.

[0065] Specifically, a bandpass filter with a passband of 0.8-4 Hz is used to filter the skin color change signal, and the video pulse wave is separated from it as a training set.

[0066] The first prediction model 40 is trained using the video pulse signal as the training set. This first prediction model can also be called the pulse-coronary heart disease prediction model. It should be noted that, since the input pulse wave is a one-dimensional time series, and the rhythm regularity of the pulse wave has a potential correlation with the risk of coronary heart disease, the main part of this model uses a long short-term memory network with temporal feature representation capabilities.

[0067] like Figure 3 As shown, as an example, the first prediction model mainly includes a segmentation layer, a long short-term memory module, a merging layer, and a fully connected module.

[0068] The segmentation method is used to divide the input video pulse wave into 5 segments on an average time axis, with each segment lasting 6 seconds and the segments not overlapping.

[0069] The Long Short-Term Memory (LSTM) module consists of two LSM layers. Each LSM layer contains hidden state units at 5 time steps. Each hidden state unit at each time step contains 180 network nodes, which extract the state features of a pulse segment. The activation function is ReLU.

[0070] The merging layer is used to combine the state features of the five time steps extracted from the long short-term memory layer into a single pulse time-series feature.

[0071] The fully connected module consists of three fully connected layers used to establish a mapping relationship between pulse timing features and coronary artery disease risk. The activation functions for the first two fully connected layers are ReLU, and the activation function for the last fully connected layer is Softmax.

[0072] In the specific implementation process, the loss function of the first prediction model is designed as the cross-entropy between the network prediction and the training label:

[0073]

[0074] in This represents the loss function of the first prediction model; Indicates the first Example of input training samples; It is the probability vector output by the first prediction model. The mapping equation for the model, As input to the model, (for the model weights). Indicates the first Example of a one-hot encoded label for a sample; Represents cross-entropy; Represents the number of training samples; log represents taking the natural logarithm. This represents the dot product of vectors.

[0075] Training set 1 is obtained from data extraction module 20. Facial pulse waves are input into the above model to obtain first prediction model 40. The first prediction model 40 is used to detect the target object and output the first detection result of coronary heart disease.

[0076] The second prediction model 30 is trained using the first frame image corresponding to the video pulse signal as the training set. This second prediction model can also be called the photo-coronary heart disease prediction model.

[0077] like Figure 5 As shown, the model structure of the second prediction model mainly includes three convolutional modules connected in sequence, and a fully connected module.

[0078] The first convolutional module includes a first convolutional layer, a second convolutional layer, and a max-pooling layer. The first convolutional layer has 64 kernels, each 3 pixels × 3 pixels in size, with each kernel stepping 1 pixel. Edges of the feature map are padded with mean, and the activation function is ReLU. The convolutional layer extracts feature maps from the image. The second convolutional layer has the same structure as the first. The max-pooling layer has a pooling window size of 3 pixels × 3 pixels, with each pooling window stepping 2 pixels, used to filter key features and avoid overfitting.

[0079] The fully connected module consists of an unfolded layer, a first fully connected layer, and a second fully connected layer. The unfolded layer expands the two-dimensional feature map into a one-dimensional feature vector, making it easier to feed into the subsequent fully connected layers. The two fully connected layers are used to establish a mapping relationship from image features to coronary heart disease risk. The activation function of the first fully connected layer is ReLU, and the activation function of the second is Softmax.

[0080] The data extraction module 20 obtains the training set 2, which is the first frame image in the face video. The first frame image is then input into the above model to obtain the second prediction model 30. The second prediction model 30 is used to detect the target object to be detected, and the second detection result of coronary heart disease is output.

[0081] As a preferred example, to avoid the influence of background areas in the image on the detection of coronary heart disease, the Viola-Jones face detection algorithm is used to detect face regions in the image, and the pixel values ​​of other regions are recorded as 0. Since the input is a three-channel image, a two-dimensional convolutional neural network with image feature description capabilities is selected for the main part of the model.

[0082] Furthermore, the loss function of the second prediction model is designed as the cross-entropy between the network's predicted values ​​and the training labels:

[0083] (2)

[0084] in This represents the loss function of the second prediction model; Indicates the first Example input sample; It is the probability vector output by the second prediction model (photo-coronary artery disease prediction model). The mapping equation for the model, As input to the model, (for the model weights). Indicates the first Example of a one-hot encoded label for a sample; Represents cross-entropy; Represents the number of training samples; log represents taking the natural logarithm. This represents the dot product of vectors.

[0085] The judgment module 50 is used to judge the overall uncertainty of the first prediction model and the second prediction model, and output the evaluation result. When the overall uncertainty of the first prediction model is greater than that of the second prediction model, the prediction result of the second prediction model is selected as the evaluation result; when the uncertainty of the second prediction model is greater than that of the first prediction model, the prediction result of the first prediction model is selected as the evaluation result. Compared with existing coronary heart disease screening methods, this method more comprehensively assesses coronary heart disease-related characteristics, and the detection results are more reliable.

[0086] It should be understood that the prediction results of both the first and second prediction models often contain uncertainties. The reliability of the model prediction results can be assessed by quantifying the prediction uncertainties of both. Model prediction uncertainty can be further divided into average data uncertainty and model uncertainty.

[0087] In one example, the judgment module further includes a quantization calculation module. The quantization calculation module is used to quantify and calculate the average data uncertainty and model uncertainty of the first prediction model and the second prediction model, and to calculate the overall uncertainty of the first prediction model and the second prediction model based on the average data uncertainty and model uncertainty.

[0088] Specifically, the quantization calculation module includes a first calculation unit and a second calculation unit.

[0089] The first computing unit is used to obtain the average data uncertainty of the first prediction model and the second prediction model. The average data uncertainty is obtained by adding an additional output branch to the output layer of the first prediction model and the second prediction model, and introducing the average data uncertainty parameter into the loss function of the first prediction model and the second prediction model. The model learns the average data uncertainty autonomously during the training process and obtains it through the output of the newly added branch.

[0090] The average data uncertainty is caused by inherent noise in the observed data (such as sensor error, data labeling error, etc.). The first calculation unit is used to obtain the average data uncertainty of the first prediction model and the second prediction model. The average data uncertainty is obtained by introducing uncertainty parameters into the loss functions of the first prediction model and the second prediction model, and the model learns it autonomously during the training process.

[0091] In one example, for the first prediction model, namely the pulse-coronary artery disease prediction model, the average data uncertainty is used as the additional branch output of the model, and the main output is the prediction of coronary artery disease. The first loss function for model training is... It needs to be rewritten as:

[0092] (3)

[0093] in, It is the loss function that does not consider the average data uncertainty, i.e., the cross-entropy described by formula (1). It is the average data uncertainty of the branch output of the first prediction model (pulse-coronary artery disease prediction model). It is the probabilities of coronary artery disease prediction output by the main pathway of the pulse-coronary artery disease prediction model. It is a probability vector The first in c One element, It is the output of the model ( The mapping equation for the first prediction model is... As input to the first prediction model, (where is the weight of the first prediction model), and exp is the natural exponentiation operation. It is an element-wise addition.

[0094] The principle of formula (3) is explained as follows: In the first term of the formula The error between the model's predictions and the labels is described. For training samples with high inherent noise, it is undesirable... A value that is too small can cause the model to overfit to noise. Therefore, by adding a denominator... To control This caused the overall loss Reasonable results can still be obtained under sufficiently convergent conditions. .therefore, The value of this term is proportional to the inherent noise of the data and can be used as a quantitative description of the average data uncertainty. However, using only the first term as the loss function can easily lead to... The value is too large and therefore This loses its practical significance. To avoid this situation, a second item is added. As a penalty. The announcement (3) does not require averaging the data uncertainty. The training labels, i.e. It can be obtained through unsupervised training.

[0095] Similarly, in another example, for the second prediction model, namely the photo-coronary artery disease prediction model, a second loss function is introduced after considering the average data uncertainty. It needs to be rewritten as:

[0096] (4)

[0097] in, It is the original loss function described by formula (2). It is the average data uncertainty of the branch output of the second prediction model (photo-coronary artery disease prediction model); These are the weights for the second prediction model.

[0098] In one example, the second computing unit is used to obtain the model uncertainty of the first prediction model and the second prediction model. The model uncertainty is mainly obtained by measuring the weight changes of the first prediction model and the second prediction model when processing the same sample.

[0099] The model uncertainty here stems from incomplete network training, preventing the model from correctly identifying unseen data. Currently, model uncertainty is primarily determined by measuring the weight changes of the network when processing the same sample.

[0100] The calculation of the model uncertainty of the first prediction model and the second prediction model also includes an optimization module, which includes a Bayesian unit, a sampling unit and a training unit.

[0101] Since the weights of traditional deterministic neural networks remain unchanged, Bayesian networks are needed for modeling. Bayesian networks replace the fixed values ​​of the weights in traditional neural networks with probability distributions; the training process of the weights essentially involves calculating their posterior probabilities. Where x and y are the input and output of the Bayesian network, respectively. It's network weight. However... Since an analytical solution cannot be obtained, a simple distribution needs to be constructed. To perform approximate fitting, thereby simplifying the complex... The problem is replaced by finding simple distribution parameters. The optimal solution problem. Monte Carlo variational inference is equivalent to using the Bernoulli distribution. To approximate the posterior distribution of a Bayesian network The specific steps are to add a dropout layer before the long short-term memory layer and fully connected layer of the first prediction model, and before the convolutional layer and fully connected layer of the second prediction model, and perform sampling operations (i.e. Monte Carlo sampling) during both training and testing.

[0102] After introducing Monte Carlo sampling, based on the redetermined first loss function (3) and second loss function (4), the first prediction model and the second prediction model are updated respectively. After the training of the two models is completed, their overall uncertainty is calculated respectively. Here, the overall uncertainty includes the model uncertainty and the average data uncertainty.

[0103] It should be noted that the average data uncertainty of the predicted model branch output in formulas (3) and (4) is the mean of the K values ​​obtained after K samplings. For better distinction, the average data uncertainty is changed to the average data uncertainty. The model has two outputs: one is the coronary heart disease detection result, and the other is the average data uncertainty. After introducing Monte Carlo sampling, assuming K samplings, the model will have K different outputs for a given input. The entropy of the mean of the K coronary heart disease prediction results (main output) is the model uncertainty, and the mean of the K average data uncertainties (branch output) is the average data uncertainty.

[0104] For the first prediction model, the overall uncertainty of the model is... The calculation formula is as follows:

[0105]

[0106] K It refers to the number of Monte Carlo samplings during the testing process; It is the first k The average data uncertainty of the pulse-coronary artery disease prediction model branch output after Monte Carlo sampling It is the first k The coronary artery disease prediction probability vector output by the main pathway of the pulse-coronary artery disease model after Monte Carlo sampling. It is the mapping equation of the model after Monte Carlo sampling. These are the input test samples. Indicates the weights of the model Perform Monta Carlo sampling that follows a Bernoulli distribution; Represents the calculation of probability distribution Entropy;

[0107] The first term of the formula The second term describes the average data uncertainty. The model uncertainty is described, therefore The overall prediction uncertainty of the first prediction model is described.

[0108] Similarly, for the overall uncertainty of the second prediction model The calculation formula is as follows:

[0109]

[0110] It is the first k The average data uncertainty of the output of the coronary artery disease model branch after the Monte Carlo sampling. It is the first kPhotographs after Monte Carlo sampling - Coronary artery disease prediction probability vector output by the main pathway of the coronary artery disease model It is the mapping equation of the model after Monte Carlo sampling;

[0111] The first term of the formula The second term describes the average data uncertainty. The model uncertainty is described, therefore The overall uncertainty of the predictions made by the second prediction model is described.

[0112] like Figure 6 As shown, a first prediction model (pulse-coronary artery disease prediction model) and a second prediction model (photograph-coronary artery disease prediction model) are used to predict the presence of coronary artery disease. The uncertainties of the prediction results of the two models are compared, and the one with lower uncertainty is selected as the final coronary artery disease detection result. Based on this strategy, a joint coronary artery disease prediction model is constructed, and the model structure is as follows. Figure 6 As shown in the figure. Y express Established; N represents Not true; & represents an AND gate, i.e., if If the result is positive, select the image-coronary artery disease prediction model; otherwise, select the pulse-coronary artery disease prediction model. U 1 represents the overall uncertainty of the first prediction model. U pulse , U 2 represents the overall uncertainty of the second prediction model. U photo .

[0113] Exemplary electronic devices

[0114] Below, for reference Figure 4 This application describes an electronic device according to embodiments thereof. The electronic device may be a mobile device itself, or a standalone device independent of it, which can communicate with the mobile device to receive collected input signals from it and send selected target decision actions to it.

[0115] Figure 4 A block diagram of an electronic device according to an embodiment of this application is illustrated.

[0116] like Figure 4 As shown, the electronic device 10 includes one or more processors 11 and memory 12.

[0117] The processor 11 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 10 to perform desired functions.

[0118] The memory 12 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 11 may execute the program instructions to implement the decision-making behavior and decision-making methods of the various embodiments of this application described above, and / or other desired functions.

[0119] In one example, electronic device 10 may further include input device 13 and output device 14, these components being interconnected via a bus system and / or other forms of connection mechanism (not shown). For example, input device 13 may include various devices such as on-board diagnostic (OBD) systems, unified diagnostic services (UDS), inertial measurement units (IMUs), cameras, lidar, millimeter-wave radar, ultrasonic radar, vehicle-to-everything (V2X) communication, etc. Input device 13 may also include, for example, a keyboard, mouse, etc. Output device 14 may include, for example, a display, speakers, printer, and communication networks and their connected remote output devices, etc.

[0120] Of course, for the sake of simplicity, Figure 4 Only some of the components of the electronic device 10 relevant to this application are shown in this illustration; components such as buses, input / output interfaces, etc., are omitted. In addition, the electronic device 10 may include any other suitable components depending on the specific application.

[0121] Controlling other components in the electronic device 10 to perform the desired functions includes:

[0122] S110: Extract the video pulse signal containing a certain feature region in the video stream of the target object and the frame image corresponding to the video pulse signal;

[0123] S120: Based on the first prediction model trained using the video pulse signal as a training set, obtain the first evaluation result;

[0124] S130: Based on the first prediction model trained using the first frame image corresponding to the video pulse signal as the training set, a second evaluation result is obtained;

[0125] S140: Based on the overall uncertainty judgment results of the first prediction model and the second prediction model, output the evaluation result; when the overall uncertainty of the first prediction model is greater than the overall uncertainty of the second prediction model, the prediction result of the second prediction model is selected as the evaluation result; when the uncertainty of the second prediction model is greater than the uncertainty of the first prediction model, the prediction result of the first prediction model is selected as the evaluation result.

[0126] It also includes S150: quantifying the average data uncertainty and model uncertainty of the first and second prediction models;

[0127] As an example, the overall uncertainty calculation method includes:

[0128] Quantitatively calculate the average data uncertainty and model uncertainty of the first and second prediction models;

[0129] Based on the average data uncertainty and model uncertainty of the first and second prediction models, calculate the overall uncertainty of the first and second prediction models respectively.

[0130] As an example, the average data uncertainty is obtained by adding an extra output branch to the output layer of the first and second prediction models, and introducing the average data uncertainty parameter into the loss function of the first and second prediction models. The model learns the average data uncertainty autonomously during training and obtains it through the newly added branch.

[0131] The second calculation unit is used to obtain the model uncertainties of the first and second prediction models. These model uncertainties are primarily obtained by measuring the weight changes of the first and second prediction models when processing the same sample. The specific methods for calculating the average data uncertainty and model uncertainty can be obtained through examples of exemplary devices.

[0132] Exemplary computer program products and computer-readable storage media

[0133] In addition to the methods and devices described above, embodiments of this application may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps in the decision-making methods according to various embodiments of this application as described in the "Exemplary Methods" section above.

[0134] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0135] Furthermore, embodiments of this application may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps in the decision-making methods according to various embodiments of this application described in the "Exemplary Methods" section above.

[0136] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0137] The basic principles of this application have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this application are merely examples and not limitations, and should not be considered as essential features of each embodiment of this application. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the application to the necessity of employing the aforementioned specific details for implementation.

[0138] The block diagrams of devices, apparatuses, devices, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0139] It should also be noted that in the apparatus, equipment, and methods of this application, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions of this application.

[0140] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this application. Therefore, this application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0141] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this application to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A non-contact coronary heart disease assessment device based on facial video, characterized in that, include: The data extraction module is used to extract the video pulse signal containing a certain feature region in the video stream of the target object and the frame image corresponding to the video pulse signal as a training set; The first prediction model is obtained by training the video pulse signal as a training set. The second prediction model is obtained by training the first frame image corresponding to the pulse signal as a training set. The judgment module is used to judge the overall uncertainty of the first prediction model and the second prediction model and output the evaluation result. When the overall uncertainty of the first prediction model is greater than the overall uncertainty of the second prediction model, the prediction result of the second prediction model is selected as the evaluation result. When the uncertainty of the second prediction model is greater than the uncertainty of the first prediction model, the prediction result of the first prediction model is selected as the evaluation result. For the first prediction model, the overall uncertainty of the model is... The calculation formula is as follows: K It refers to the number of Monte Carlo samplings during the testing process; It is the first k The average data uncertainty of the pulse-coronary artery disease prediction model branch output after Monte Carlo sampling It is the first k The coronary artery disease prediction probability vector output by the main pathway of the pulse-coronary artery disease model after Monte Carlo sampling. It is the mapping equation of the model after Monte Carlo sampling. These are the input test samples. Indicates the weights of the model Perform Monta Carlo sampling that follows a Bernoulli distribution; Represents the calculation of probability distribution Entropy; The first term of the formula The second term describes the average data uncertainty. The model uncertainty is described; Regarding the overall uncertainty of the second prediction model The calculation formula is as follows: It is the first k The average data uncertainty of the output of the coronary artery disease model branch after the Monte Carlo sampling. It is the first k Photographs after Monte Carlo sampling - Coronary artery disease prediction probability vector output by the main pathway of the coronary artery disease model It is the mapping equation of the model after Monte Carlo sampling; The first term of the formula The second term describes the average data uncertainty. The model uncertainty is described, where the average data uncertainty is caused by inherent noise in the observed data.

2. The non-contact coronary heart disease assessment device based on facial video according to claim 1, characterized in that, The judgment module includes a quantization calculation module, which includes a first calculation unit and a second calculation unit. The first computing unit is used to obtain the average data uncertainty of the first prediction model and the second prediction model. The average data uncertainty is obtained by adding an extra output branch to the output layer of the first prediction model and the second prediction model, and introducing the average data uncertainty parameter into the loss function of the first prediction model and the second prediction model. The model learns the average data uncertainty autonomously during the training process and obtains it through the output of the newly added branch. The second calculation unit is used to obtain the model uncertainty of the first prediction model and the second prediction model. The model uncertainty is obtained by measuring the weight changes of the first prediction model and the second prediction model when processing the same sample.

3. The non-contact coronary heart disease assessment device based on facial video according to claim 2, characterized in that: The first prediction model, after introducing the average data uncertainty parameter, yields a rewritten first loss function, the formula of which is: in The first prediction model's loss function does not include uncertainty parameters. It is the average data uncertainty; It is a probability vector The first in c One element, This is the probability vector output by the first prediction model. Here, is the mapping equation for the model, x is the input of the first prediction model, W1 is the weight of the first prediction model, and exp is the natural exponent operation. It is element-wise addition; The second prediction model, after introducing the average data uncertainty parameter, yields a rewritten second loss function, the formula of which is: in, The second prediction model is a loss function that does not include uncertainty parameters. W1 represents the average data uncertainty of the second prediction model; W2 represents the weights of the second prediction model.

4. The non-contact coronary heart disease assessment device based on facial video according to claim 3, characterized in that, It also includes an optimization module, which comprises a Bayesian unit, a sampling unit, and a training unit: A Bayesian unit is used to replace the fixed values ​​of the weights of the first and second prediction models with probability distributions using a Bayesian network. A sampling unit is used to add a sampling layer before each layer of the first and second prediction models, and to perform Monte Carlo sampling during training and testing. The training unit updates the first and second prediction models based on the redetermined first and second loss functions, respectively. The second calculation unit then obtains the model uncertainties of the first and second prediction models and calculates their uncertainties.

5. The non-contact coronary heart disease assessment device based on facial video according to claim 1, characterized in that, The data extraction module includes: The position detection unit is used to detect the position of the face in each frame of the video stream according to the face detection algorithm and obtain the facial feature points. The tracking unit is used to counteract the rigid motion interference of the face by tracking the motion trajectory of facial feature points in the video stream over time, and to divide the cheek region based on the cheek periphery feature points among the facial feature points. The signal extraction unit is used to extract the green channel information of the cheek video, calculate the average value of all pixels in each frame of the video stream, and obtain the skin color change signal. The separation unit is used to filter the skin color change signal using a bandpass filter, thereby separating the video pulse wave signal.

6. An electronic device, characterized in that, It includes a processor, an input device, an output device, and a memory, which are connected in sequence. The memory is used to store computer programs, the computer programs including program instructions; The processor is configured according to the program instructions to: Extract the video pulse signal containing a certain feature region in the video stream of the target object and the frame image corresponding to the video pulse signal; Based on the first prediction model trained using the video pulse signal as the training set, a first evaluation result is obtained. A second evaluation result is obtained based on the second prediction model trained using the first frame image corresponding to the pulse signal as the training set. Based on the overall uncertainty judgment results of the first prediction model and the second prediction model, the evaluation result is output; when the overall uncertainty of the first prediction model is greater than the overall uncertainty of the second prediction model, the prediction result of the second prediction model is selected as the evaluation result; when the uncertainty of the second prediction model is greater than the uncertainty of the first prediction model, the prediction result of the first prediction model is selected as the evaluation result. For the first prediction model, the overall uncertainty of the model is... The calculation formula is as follows: K It refers to the number of Monte Carlo samplings during the testing process; It is the first k The average data uncertainty of the pulse-coronary artery disease prediction model branch output after Monte Carlo sampling It is the first k The coronary artery disease prediction probability vector output by the main pathway of the pulse-coronary artery disease model after Monte Carlo sampling. It is the mapping equation of the model after Monte Carlo sampling. These are the input test samples. Indicates the weights of the model Perform Monta Carlo sampling that follows a Bernoulli distribution; Represents the calculation of probability distribution Entropy; The first term of the formula The second term describes the average data uncertainty. The model uncertainty is described; Regarding the overall uncertainty of the second prediction model The calculation formula is as follows: It is the first k The average data uncertainty of the output of the coronary artery disease model branch after the Monte Carlo sampling. It is the first k Photographs after Monte Carlo sampling - Coronary artery disease prediction probability vector output by the main pathway of the coronary artery disease model It is the mapping equation of the model after Monte Carlo sampling; The first term of the formula The second term describes the average data uncertainty. The model uncertainty is described, where the average data uncertainty is caused by inherent noise in the observed data.

7. An electronic device according to claim 6, characterized in that, The average data uncertainty is obtained by adding an extra output branch to the output layer of the first and second prediction models, and introducing the average data uncertainty parameter into the loss function of the first and second prediction models. The model learns the average data uncertainty autonomously during training and outputs it through the newly added branch. The model uncertainty is obtained by measuring the weight changes of the first and second prediction models when processing the same sample.

Images