Heart rate monitoring method and device based on multi-source data fusion

Abstract

The invention discloses a heart rate monitoring method and device based on multi-source data fusion. The method comprises the following steps: acquiring heart rate monitoring data information; the heart rate monitoring data information comprises video data information and microwave data information; performing feature extraction on the heart rate monitoring data information to obtain optimized feature information; training a preset heart rate monitoring model by using the optimized feature information to obtain an optimized heart rate monitoring model; and processing to-be-processed heart rate monitoring data information by using the optimized heart rate monitoring model to obtain a heart rate monitoring result. According to the method, the same data features are extracted from the multi-source data, the implementation process of multi-source data fusion is facilitated, the multi-mode data are converted into the same mode, the model is simplified through feature-level fusion, resource waste caused by data-level fusion is avoided, and the problems that the monitoring effect of a camera becomes poor in a dark environment, and the monitoring accuracy is high are solved. And the microwave data cannot be matched with the personnel identity in a multi-target environment.

Description

Technical Field

[0001] The present invention relates to the technical field of machine learning, and in particular, to a heart rate monitoring method and device for multi-source data fusion. Background Art

[0002] Heart rate is an important vital sign for detecting the health status of the human body, and the onset conditions of most diseases are directly or indirectly related to changes in physiological signals. Heart rate monitoring provides reliable diagnosis and real-time basis for doctors. In current hospital treatments, contact devices such as finger clip meters and electrocardiograms are mostly used for heart rate monitoring. However, contact devices not only have low comfort, but may also cause additional harm to burn patients and infants. Therefore, the present invention uses a non-contact method to monitor heart rate.

[0003] Technologies for non-contact heart rate monitoring mainly include radio frequency systems, fiber optic sensors, computer vision, and radar systems. However, the above monitoring technologies have their advantages and limitations. Among them, the signal emission direction of the radio frequency system is relatively divergent, and the multipath effect is relatively serious. The fiber optic sensing is greatly affected by the light source fluctuation during operation. Computer vision estimates vital signs through changes in skin reflection characteristics, and at the same time can use face recognition algorithms to determine the identity of the target person, and can match patient information during monitoring. The radar system using electromagnetic technology is sensitive to the perception of weak amplitude movements such as heartbeats due to its characteristics of high frequency and short wavelength.

[0004] Current research shows that in ippg (imaging photoplethysmogram) monitoring, when the target person is in a dark or smoky environment, it will affect the camera's acquisition of face information, and in severe cases, it will lead to the inability to obtain facial information; while in microwave monitoring, when multiple target persons in the detection range exchange positions, the corresponding relationship between the target person and the signal cannot be determined, resulting in errors in heart rate detection. Therefore, using cameras and radars to collect data simultaneously for heart rate monitoring can achieve the purpose of matching vital sign information and user identity. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a heart rate monitoring method and device for multi-source data fusion, which is used to combine the effective information in camera video data and radar echo signals for non-contact heart rate monitoring. The method extracts the effective information of multi-source data, through feature fusion, uses a feature selection algorithm to screen out features with higher importance, and uses a classifier optimized by the Artificial Bee Colony Algorithm (ABC) to complete heart rate classification, providing an effective solution for identity matching and monitoring in dark environments.

[0006] To solve the above technical problems, a first aspect of an embodiment of the present invention discloses a heart rate monitoring method for multi-source data fusion, and the method includes:

[0007] S1, obtaining heart rate monitoring data information; the heart rate monitoring data information includes video data information and microwave data information;

[0008] S2, performing feature extraction on the heart rate monitoring data information to obtain optimized feature information;

[0009] S3, using the optimized feature information to train a preset heart rate monitoring model to obtain an optimized heart rate monitoring model;

[0010] S4, using the optimized heart rate monitoring model to process the heart rate monitoring data information to be processed to obtain a heart rate monitoring result.

[0011] As an optional implementation manner, in the first aspect of an embodiment of the present invention, the performing feature extraction on the heart rate monitoring data information to obtain optimized feature information includes:

[0012] S21, preprocessing the heart rate monitoring data information to obtain preprocessed heart rate monitoring data information; the preprocessed heart rate monitoring data information includes preprocessed video data information and preprocessed microwave data information;

[0013] S22, performing feature extraction on the preprocessed video data information to obtain video feature information;

[0014] S23, performing feature extraction on the preprocessed microwave data information to obtain microwave data feature information;

[0015] S24, performing feature fusion on the video feature information and the microwave data feature information to obtain fusion feature information;

[0016] S25, performing data screening on the fusion feature information to obtain optimized feature information.

[0017] As an optional implementation manner, in the first aspect of an embodiment of the present invention, the preprocessing the heart rate monitoring data information to obtain preprocessed heart rate monitoring data information includes:

[0018] S211, preprocessing the video data information to obtain preprocessed video data information;

[0019] S212, preprocessing the microwave data information to obtain preprocessed microwave data information.

[0020] As an alternative implementation, in the first aspect of the embodiments of the present invention, the preprocessing of the video data information to obtain preprocessed video data information includes:

[0021] S2111, performing face detection on the video data information to obtain facial image information;

[0022] S2112, performing channel separation on the facial image information to obtain green channel data information;

[0023] S2113, performing normalization and filtering on the green channel data information to obtain preprocessed video data information.

[0024] As an alternative implementation, in the first aspect of the embodiments of the present invention, the preprocessing of the microwave data information to obtain preprocessed microwave data information includes:

[0025] S2121, performing data rearrangement on the microwave data information to obtain a data matrix;

[0026] S2122, processing the data matrix to obtain a spectrum matrix;

[0027] S2123, performing phase difference processing on the spectrum matrix to obtain preprocessed microwave data information.

[0028] As an alternative implementation, in the first aspect of the embodiments of the present invention, the feature extraction of the preprocessed video data information to obtain video feature information includes:

[0029] S221, performing time-domain feature extraction on the preprocessed video data information to obtain first video feature information;

[0030] S222, performing frequency-domain feature extraction on the preprocessed video data information to obtain second video feature information;

[0031] S223, performing multi-scale entropy feature extraction on the preprocessed video data information to obtain third video feature information;

[0032] S224, performing feature fusion on the first video feature information, the second video feature information, and the third video feature information to obtain video feature information.

[0033] As an alternative implementation, in the first aspect of the embodiments of the present invention, the data screening of the fusion feature information to obtain optimized feature information includes:

[0034] S251, performing correlation calculation on the fusion feature information to obtain correlation information;

[0035] S252. Screen the fused feature information according to the correlation information to obtain first fused feature information;

[0036] S253. Use an optimized support vector machine model to screen the first fused feature information to obtain optimized feature information.

[0037] In the second aspect of the embodiments of the present invention, a heart rate monitoring device for multi-source data fusion is disclosed. The device includes:

[0038] An information acquisition module, configured to acquire heart rate monitoring data information; the heart rate monitoring data information includes video data information and microwave data information;

[0039] A feature extraction module, configured to extract features from the heart rate monitoring data information to obtain optimized feature information;

[0040] A model training module, configured to use the optimized feature information to train a preset heart rate monitoring model to obtain an optimized heart rate monitoring model;

[0041] A heart rate monitoring module, configured to use the optimized heart rate monitoring model to process the heart rate monitoring data information to be processed to obtain a heart rate monitoring result.

[0042] As an optional implementation manner, in the second aspect of the embodiments of the present invention, the extracting features from the heart rate monitoring data information to obtain optimized feature information includes:

[0043] S21. Preprocess the heart rate monitoring data information to obtain preprocessed heart rate monitoring data information; the preprocessed heart rate monitoring data information includes preprocessed video data information and preprocessed microwave data information;

[0044] S22. Extract features from the preprocessed video data information to obtain video feature information;

[0045] S23. Extract features from the preprocessed microwave data information to obtain microwave data feature information;

[0046] S24. Perform feature fusion on the video feature information and the microwave data feature information to obtain fused feature information;

[0047] S25. Screen the fused feature information to obtain optimized feature information.

[0048] As an optional implementation manner, in the second aspect of the embodiments of the present invention, the preprocessing the heart rate monitoring data information to obtain preprocessed heart rate monitoring data information includes:

[0049] S211. Preprocess the video data information to obtain preprocessed video data information;

[0050] S212. Preprocess the microwave data information to obtain preprocessed microwave data information.

[0051] As an optional implementation manner, in the second aspect of the embodiments of the present invention, the preprocessing the video data information to obtain preprocessed video data information includes:

[0052] S2111. Perform face detection on the video data information to obtain facial image information;

[0053] S2112. Perform channel separation on the facial image information to obtain green channel data information;

[0054] S2113. Perform normalization and filtering on the green channel data information to obtain preprocessed video data information.

[0055] As an optional implementation manner, in the second aspect of the embodiments of the present invention, the preprocessing the microwave data information to obtain preprocessed microwave data information includes:

[0056] S2121. Rearrange the microwave data information to obtain a data matrix;

[0057] S2122. Process the data matrix to obtain a spectrum matrix;

[0058] S2123. Perform phase difference processing on the spectrum matrix to obtain preprocessed microwave data information.

[0059] As an optional implementation manner, in the second aspect of the embodiments of the present invention, the extracting features from the preprocessed video data information to obtain video feature information includes:

[0060] S221. Extract time-domain features from the preprocessed video data information to obtain first video feature information;

[0061] S222. Extract frequency-domain features from the preprocessed video data information to obtain second video feature information;

[0062] S223. Extract multi-scale entropy features from the preprocessed video data information to obtain third video feature information;

[0063] S224. Perform feature fusion on the first video feature information, the second video feature information, and the third video feature information to obtain video feature information.

[0064] As an alternative embodiment, in the second aspect of the embodiments of the present invention, the data screening of the fusion feature information to obtain optimized feature information includes:

[0065] S251, performing correlation calculation on the fusion feature information to obtain correlation information;

[0066] S252, screening the fusion feature information according to the correlation information to obtain first fusion feature information;

[0067] S253, using an optimized support vector machine model to perform data screening on the first fusion feature information to obtain optimized feature information.

[0068] The third aspect of the present invention discloses another heart rate monitoring device for multi-source data fusion, and the device includes:

[0069] A memory storing executable program code;

[0070] A processor coupled to the memory;

[0071] The processor calls the executable program code stored in the memory and executes some or all of the steps in the heart rate monitoring method for multi-source data fusion disclosed in the first aspect of the embodiments of the present invention.

[0072] The fourth aspect of the present invention discloses a computer-readable storage medium, and the computer-readable storage medium stores computer instructions, which are used to execute some or all of the steps in the heart rate monitoring method for multi-source data fusion disclosed in the first aspect of the embodiments of the present invention when called.

[0073] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0074] (1) In the data preprocessing part of the method of the present invention, face extraction is performed on video data and converted into one-dimensional signal data, enhancing the consistency of multi-source data; data rearrangement is performed on radar echo signals (microwave signals), and target phase signals are extracted to highlight the data validity, realizing data enhancement.

[0075] (2) The method of the present invention extracts the same data features from multi-source data, facilitating the implementation process of multi-source data fusion, converting multi-modal data into the same modality, simplifying the model through feature-level fusion, and avoiding resource waste caused by data-level fusion. Description of the Drawings

[0076] To more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0077] Figure 1 It is a schematic flowchart of a heart rate monitoring method for multi-source data fusion disclosed in an embodiment of the present invention;

[0078] Figure 2 It is a data preprocessing flowchart disclosed in an embodiment of the present invention;

[0079] Figure 3 It is a flowchart of a hybrid feature selection algorithm disclosed in an embodiment of the present invention;

[0080] Figure 4 It is a flowchart of classifier optimization disclosed in an embodiment of the present invention;

[0081] Figure 5 It is a schematic structural diagram of a heart rate monitoring device for multi-source data fusion disclosed in an embodiment of the present invention;

[0082] Figure 6 It is a schematic structural diagram of another heart rate monitoring device for multi-source data fusion disclosed in an embodiment of the present invention. Detailed implementation manners

[0083] To enable those skilled in the art to better understand the solutions of the present invention, the following clearly and completely describes the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0084] The terms "first", "second", etc. in the specification and claims of the present invention and the above drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, device, product, or equipment that includes a series of steps or units is not limited to the listed steps or units, but optionally further includes steps or units not listed, or optionally further includes other steps or units inherent to these processes, methods, products, or equipment.

[0085] References herein to "embodiments" mean that the particular features, structures, or characteristics described in connection with the embodiments can be included in at least one embodiment of the invention. The phrase appears in various places in the specification and does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will explicitly and implicitly understand that the embodiments described herein can be combined with other embodiments.

[0086] The present invention discloses a heart rate monitoring method and device for multi-source data fusion. The method includes obtaining heart rate monitoring data information; the heart rate monitoring data information includes video data information and microwave data information; extracting features from the heart rate monitoring data information to obtain optimized feature information; using the optimized feature information to train a preset heart rate monitoring model to obtain an optimized heart rate monitoring model; using the optimized heart rate monitoring model to process the heart rate monitoring data information to be processed to obtain a heart rate monitoring result. The method of the present invention extracts the same data features from multi-source data, facilitates the implementation process of multi-source data fusion, converts multi-modal data into the same modality, simplifies the model through feature-level fusion, avoids resource waste caused by data-level fusion, and solves the problems that the monitoring effect of the camera deteriorates in a dark environment and the microwave data cannot match the identity of a person in a multi-target environment. The following will be described in detail respectively.

[0087] Embodiment 1

[0088] Please refer to Figure 1 , Figure 1 which is a schematic flow chart of a heart rate monitoring method for multi-source data fusion disclosed in an embodiment of the present invention. Among them, Figure 1 The described method for detecting the signal frequency band of an unmanned aerial vehicle is applied to the technical field of machine learning, and the embodiment of the present invention does not make a limitation. As Figure 1 shown, the heart rate monitoring method for multi-source data fusion may include the following operations:

[0089] S1. Obtain heart rate monitoring data information; the heart rate monitoring data information includes video data information and microwave data information;

[0090] The video data information comes from camera video data, and the microwave data information comes from radar echo signals;

[0091] S2. Extract features from the heart rate monitoring data information to obtain optimized feature information;

[0092] S3. Use the optimized feature information to train a preset heart rate monitoring model to obtain an optimized heart rate monitoring model;

[0093] S4. Use the optimized heart rate monitoring model to process the heart rate monitoring data information to be processed to obtain a heart rate monitoring result.

[0094] Optionally, the feature extraction of the heart rate monitoring data information to obtain optimized feature information includes:

[0095] S21. Preprocess the heart rate monitoring data information to obtain preprocessed heart rate monitoring data information; the preprocessed heart rate monitoring data information includes preprocessed video data information and preprocessed microwave data information;

[0096] S22. Extract features from the preprocessed video data information to obtain video feature information;

[0097] S23. Extract features from the preprocessed microwave data information to obtain microwave data feature information;

[0098] S24. Perform feature fusion on the video feature information and the microwave data feature information to obtain fusion feature information;

[0099] S25. Screen the fusion feature information to obtain optimized feature information.

[0100] Optionally, the preprocessing of the heart rate monitoring data information to obtain preprocessed heart rate monitoring data information includes:

[0101] S211. Preprocess the video data information to obtain preprocessed video data information;

[0102] S212. Preprocess the microwave data information to obtain preprocessed microwave data information.

[0103] Optionally, the preprocessing of the video data information to obtain preprocessed video data information includes:

[0104] S2111. Perform face detection on the video data information to obtain facial image information;

[0105] The method of face detection is: obtain left video data and right video data, input the left video data and the right video data into a feature point fusion network model respectively to obtain first feature information and second feature information, perform fusion on the first feature information and the second feature information to obtain fusion feature information, and input the fusion feature information into a face detection network to obtain facial image information;

[0106] The feature point fusion network model is Voxel - RCNN, which includes: a 3D backbone network, a 2D backbone network, followed by a Region Proposal Network (RPN), a Voxel RoI Pool, and a detection subnet for bounding box optimization. Voxel - RCNN, based on VoxelNet, first divides the initial point cloud into regular voxels and uses the 3D backbone network for feature extraction. Then, the 3D feature volume is converted into a Bird's Eye View (BEV) form, and region proposals are generated using the 2D backbone network and RPN from the BEV perspective; the Voxel RoI Pool directly extracts RoI features from the 3D features. Finally, the RoI features are used for further bounding box optimization at the detection head.

[0107] The face detection network includes an encoder - bottleneck - decoder structure. The encoder uses a residual network as the backbone network to extract image features, the CSPNet network and the SPP network are used as the bottleneck (neck) networks to enhance the model's feature extraction ability, and the decoder is composed of stacked transposed convolution networks.

[0108] S2112. Channel - separate the facial image information to obtain green - channel data information.

[0109] S2113. Normalize and filter the green - channel data information to obtain pre - processed video data information.

[0110] Normalization is an existing technology in the art, and this embodiment does not limit it.

[0111] Filtering uses a Butterworth band - pass filter with a frequency range of 0.8 - 3 Hz.

[0112] Optionally, the pre - processing of the microwave data information to obtain pre - processed microwave data information includes:

[0113] S2121. Rearrange the microwave data information to obtain a data matrix.

[0114] The microwave data information is rearranged into a data matrix MS[M, N]. The rearrangement method is as follows:

[0115] Segment the microwave data information to obtain M segments of signals.

[0116] The mean value of the amplitude is calculated as:

[0117]

[0118] where A i is the amplitude of the i - th segment of the signal. The relative standard deviation of the amplitude is:

[0119]

[0120] where SA is the standard deviation of the amplitude.

[0121] The mean of the center frequency is:

[0122]

[0123] where f i is the center frequency of the i-th segment of the signal. The relative standard deviation of the center frequency is

[0124]

[0125] where S f is the standard deviation of the center frequency.

[0126] The mean of the amplitude, the relative standard deviation of the amplitude, the standard deviation of the amplitude, the mean of the center frequency, the relative standard deviation of the center frequency, and the standard deviation of the center frequency constitute the elements of each row of the matrix, i.e., N = 6;

[0127] S2122, process the data matrix to obtain a spectrum matrix;

[0128] The method is to perform an FFT transform;

[0129] S2123, perform phase difference processing on the spectrum matrix to obtain preprocessed microwave data information.

[0130] Optionally, extracting features from the preprocessed video data information to obtain video feature information includes:

[0131] S221, perform time-domain feature extraction on the preprocessed video data information to obtain first video feature information;

[0132] S222, perform frequency-domain feature extraction on the preprocessed video data information to obtain second video feature information;

[0133] S223, perform multi-scale entropy feature extraction on the preprocessed video data information to obtain third video feature information;

[0134] Time-domain features have great advantages in representing signal amplitude and time scale. Frequency-domain analysis converts the heart rate signal with amplitude varying over time into a heart rate power spectrum varying with frequency, and converts the time-domain signals of each frequency band into frequency-domain signals through fast Fourier transform. Entropy represents the overall characteristics of the information source on average and can evaluate the irregularity and complexity of the heart rate signal.

[0135] S224, perform feature fusion on the first video feature information, the second video feature information, and the third video feature information to obtain video feature information.

[0136] The fusion method is as follows: Input the vectors X and Y to be fused.

[0137] 1. Initialization

[0138] Obtain the autocovariance matrix ∑ of sample X 11 , the covariance matrix ∑ of Y 22 and the cross-covariance matrix ∑ of X and Y 12 and ∑ 21 ; is the between-class matrix, is the within-class matrix;

[0139] 2. According to the optimization objective, construct the Lagrangian equation

[0140]

[0141] where Q1 = ∑ 11 , Q2 = ∑ 22 , β, λ, θ are coefficient constants obtained from multiple experiments, and are not limited in this invention.

[0142] The optimization objective is: Minimize such that

[0143] 3. Take the partial derivatives of W x and W y to transform the optimization problem into the solution of eigenvalues

[0144] 4. Apply the eigen-decomposition method to obtain the matrix composed of the eigenvectors corresponding to the non-zero vectors and the transformation matrix W of X x .

[0145] 5. According to obtain the transformation matrix W of Y y .

[0146] 6. Substitute the transformation matrix into equation to obtain the final fusion feature Σ k .

[0147] Fuse the first video feature information and the second video feature information to obtain the fourth video feature information, and fuse the fourth video feature information and the third video feature information to obtain the video feature information.

[0148] Optionally, the data screening of the fusion feature information to obtain the optimized feature information includes:

[0149] S251, perform correlation calculation on the fusion feature information to obtain correlation information;

[0150] S252. According to the correlation information, screen the fused feature information to obtain the first fused feature information;

[0151] Preset a correlation information threshold, and remove the fused feature information with a correlation information less than the threshold to obtain the first fused feature information;

[0152] S253. Use an optimized support vector machine model to screen the first fused feature information to obtain optimized feature information.

[0153] The optimized support vector machine model is:

[0154] Suppose in R n space, the training data set is defined as T = {(x i , y i ) | i = 1, 2,..., m}, where x i is the input and y i ∈{+1, -1} is the corresponding output. They have a total of m training samples, each sample has n attributes, among which m1 samples belong to the positive class and m2 samples belong to the negative class, and are represented by matrix A and matrix B respectively.

[0155] Suppose C p and C n clusters are obtained from the positive class and the negative class respectively, that is

[0156]

[0157] In the formula: c i , i = 1, 2,..., 6 are penalty parameters; ξ, η are slack variables; E + and E - energy factors of the hyperplane; and are the covariance matrices corresponding to the i - th cluster and the j - th cluster in the 2 classes respectively, i = 1, 2,..., C P , j = 1, 2,..., C n .

[0158] Substitute the equality constraint condition into the objective function formula (1) to get:

[0159]

[0160] Derive w + and b + in (3) respectively, and obtain from the KKT condition:

[0161]

[0162] Rearranging Eqs. (4) and (5) into matrix form gives:

[0163]

[0164] Let \(P = [A\ e + , Q = [B\ e - , and Then the solution can be expressed as

[0165]

[0166] where: I is the identity matrix of appropriate dimension.

[0167] The optimized support vector machine model in this embodiment determines the class label of a new sample through the following decision function:

[0168]

[0169] It can be seen that the method of the present invention performs face extraction on video data in the data preprocessing part and converts it into one-dimensional signal data, enhancing the consistency of multi-source data; rearranges the radar echo signal (microwave signal) data, extracts the target phase signal to highlight the data validity, and realizes data enhancement. The method of the present invention extracts the same data features from multi-source data, facilitating the implementation process of multi-source data fusion, converting multi-modal data into the same modality, simplifying the model through feature-level fusion, and avoiding resource waste caused by data-level fusion. The method of the present invention combines the mRMR algorithm and the SVM_RFE algorithm in the feature selection part to obtain high-quality features with higher importance after double screening. In the classifier optimization part of the method of the present invention, in order to reduce the degree of empirical decision-making of manual operations, an intelligent optimization algorithm is added, and the optimal parameters are selected through accuracy to complete the optimization of the classifier.

[0170] It can be seen that the method of the present invention performs face extraction on video data in the data preprocessing part and converts it into one-dimensional signal data, enhancing the consistency of multi-source data; rearranges the radar echo signal (microwave signal) data, extracts the target phase signal to highlight the data validity, and realizes data enhancement. The method of the present invention extracts the same data features from multi-source data, facilitating the implementation process of multi-source data fusion, converting multi-modal data into the same modality, simplifying the model through feature-level fusion, and avoiding resource waste caused by data-level fusion. The method of the present invention combines the mRMR algorithm and the SVM_RFE algorithm in the feature selection part to obtain high-quality features with higher importance after double screening. In the classifier optimization part of the method of the present invention, in order to reduce the degree of empirical decision-making of manual operations, an intelligent optimization algorithm is added, and the optimal parameters are selected through accuracy to complete the optimization of the classifier.

[0171] Embodiment 2

[0172] A non-contact heart rate monitoring method based on multi-source data fusion proposed by the present invention. The monitoring method includes microwave data and video data preprocessing, feature extraction and fusion, feature selection, and classifier optimization parts, and comprises the following steps:

[0173] In the video data processing part, the RetinaFace face detection algorithm is used to extract the face from the video data to obtain the facial image of the target person. The three-channel data of the RGB image is separated, and the green-channel data rich in human blood vessel change information is extracted therefrom and normalized to complete the extraction of volume wave data. A Butterworth band-pass filter with a frequency of 0.8 - 3 Hz is used to preliminarily screen the heart rate signal to obtain the preprocessed video data. In the microwave data preprocessing part, the microwave data is first rearranged into a data matrix MS[M, N]. The P-point FFT is calculated for each column of the matrix MS[M, N] to obtain the range-dimensional FFT spectrum matrix MS[M, N]. The range cell with the largest amplitude is selected and the corresponding phase signal is extracted. The complete phase change is obtained through phase unwrapping, and the heart rate signal is enhanced through phase difference to obtain the preprocessed microwave signal. Figure 2 It is the data preprocessing flowchart disclosed in the embodiment of the present invention.

[0174] When extracting features, time-domain features, frequency-domain features, and non-linear features, i.e., multi-scale entropy features, are respectively extracted from the preprocessed microwave signal and video data. Time-domain features have great advantages in representing signal amplitude and time scale. The mean value reflects the change in heart rate frequency, and the standard deviation reflects the degree of dispersion of the heart rate signal in the frequency domain. Frequency-domain analysis converts the heart rate signal whose amplitude changes with time into a heartbeat power spectrum that changes with frequency, and converts the time-domain signal of each frequency band into a frequency-domain signal through fast Fourier transform. Entropy represents the overall characteristics of the information source on average and can evaluate the irregularity and complexity of the heart rate signal. After feature extraction, the fusion of multi-source data is achieved through feature concatenation. Figure 3 It is the flowchart of the hybrid feature selection algorithm disclosed in the embodiment of the present invention.

[0175] In the feature selection part, it is completed by combining the mRMR and SVM-RFE algorithms. The mRMR selects the features with the greatest correlation with the sample classification category following the principle of maximum correlation, and evaluates the redundancy degree between features through the principle of minimum redundancy; the SVM-RFE algorithm constructs a sorting criterion for features by the absolute value of the weight of each dimension in the support vector machine hyperplane, and recursively eliminates the signal feature with the smallest weight in the feature set one by one.

[0176] In the part of optimizing the classifier, the classifier adopted by the present invention is SVM. However, during the application of SVM, the penalty factor and the parameters of the radial basis kernel function need to be set manually, and the empirical input limits the accuracy of the classifier to a certain extent. Therefore, the present invention uses the ABC algorithm with prominent convergence characteristics to find the optimal parameter values. Figure 4 It is the flowchart for optimizing the classifier disclosed in the embodiment of the present invention.

[0177] The ABC algorithm realizes it by leading bees to search for nectar sources, saving the nectar sources with large fitness values, following bees to calculate the fitness values to save the most nectar sources. When the number of consecutive non-updated times of the nectar source reaches the limit, the scout bees generate new solutions using chaotic mapping, record the optimal nectar source and continuously loop this process. When the number of loops reaches the predefined number of times, the optimal parameters are output. After the parameters obtained after the ABC algorithm runs to completion are input into the SVM classifier, the classification result of the optimized classifier is obtained.

[0178] It can be seen that the method of the present invention extracts faces from video data in the data preprocessing part and converts them into one-dimensional signal data, enhancing the consistency of multi-source data; rearranges the data of radar echo signals (microwave signals) and extracts the target phase signals to highlight the data validity, realizing data enhancement. The method of the present invention extracts the same data features from multi-source data, facilitating the implementation process of multi-source data fusion, converting multi-modal data into the same modality, simplifying the model through feature-level fusion, and avoiding the resource waste caused by data-level fusion. The method of the present invention combines the mRMR algorithm and the SVM_RFE algorithm in the feature selection part to obtain high-quality features with higher importance after double screening. In the part of optimizing the classifier, in order to reduce the degree of empirical determination of manual operations, an intelligent optimization algorithm is added, and the optimal parameters are selected through accuracy to complete the optimization of the classifier.

[0179] Embodiment III

[0180] Please refer to Figure 5 , Figure 5 It is the schematic flowchart of a heart rate monitoring device for multi-source data fusion disclosed in the embodiment of the present invention. Among them, Figure 5 The described drone signal frequency band detection device is applied to the field of machine learning technology, which is not limited in the embodiment of the present invention. As Figure 5 shown, the heart rate monitoring device for multi-source data fusion may include the following operations:

[0181] S301, an information acquisition module, is used to acquire heart rate monitoring data information; the heart rate monitoring data information includes video data information and microwave data information;

[0182] S302, a feature extraction module, is used to extract features from the heart rate monitoring data information to obtain optimized feature information;

[0183] S303, a model training module, configured to train a preset heart rate monitoring model by using the optimized feature information to obtain an optimized heart rate monitoring model;

[0184] S304, a heart rate monitoring module, configured to process the heart rate monitoring data information to be processed by using the optimized heart rate monitoring model to obtain a heart rate monitoring result.

[0185] Embodiment Four

[0186] Please refer to Figure 6 , Figure 6 which is a schematic flowchart of another heart rate monitoring device with multi-source data fusion disclosed in the embodiments of the present invention. Among them, Figure 6 the described drone signal frequency band detection device is applied to the field of machine learning technology, which is not limited in the embodiments of the present invention. As Figure 6 shown, the heart rate monitoring device with multi-source data fusion may include the following operations:

[0187] a memory 401 storing executable program code;

[0188] a processor 402 coupled to the memory 401;

[0189] The processor 402 calls the executable program code stored in the memory 401 and is configured to execute the steps in the heart rate monitoring method with multi-source data fusion described in Embodiment One and Embodiment Two.

[0190] Embodiment Five

[0191] The embodiments of the present invention disclose a computer-readable storage medium storing a computer program for electronic data exchange, wherein the computer program enables a computer to execute the steps in the heart rate monitoring method with multi-source data fusion described in Embodiment One and Embodiment Two.

[0192] The above-described device embodiments are merely illustrative. The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed to multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative efforts.

[0193] Through the specific descriptions of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the essence of the above technical solution, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, and the storage medium includes read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc memories, magnetic disk memories, tape memories, or any other medium that can be used to carry or store data and is computer-readable.

[0194] Finally, it should be noted that: What is disclosed in an embodiment of a multi-source data fusion heart rate monitoring method and device of the present invention is only a preferred embodiment of the present invention, and is only used to illustrate the technical solution of the present invention, rather than limiting it; Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A heart rate monitoring method using multi-source data fusion, characterized in that: The method comprises: S1, obtaining heart rate monitoring data information; the heart rate monitoring data information includes video data information and microwave data information; S2, extracting features from the heart rate monitoring data information to obtain optimized feature information; S3, using the optimized feature information to train a preset heart rate monitoring model to obtain an optimized heart rate monitoring model; S4, using the optimized heart rate monitoring model to process the heart rate monitoring data information to be processed to obtain a heart rate monitoring result.

2. The heart rate monitoring method of multi-source data fusion according to claim 1, characterized in that: The step of extracting features from the heart rate monitoring data information to obtain optimized feature information includes: S21, preprocessing the heart rate monitoring data information to obtain preprocessed heart rate monitoring data information; the preprocessed heart rate monitoring data information includes preprocessed video data information and preprocessed microwave data information; S22, extracting features from the pre-processed video data information to obtain video feature information; S23, performing feature extraction on the pre-processed microwave data information to obtain microwave data feature information; S24, performing feature fusion on the video feature information and the microwave data feature information to obtain fused feature information; S25, performing data screening on the fused feature information to obtain optimized feature information.

3. The heart rate monitoring method of multi-source data fusion according to claim 2, characterized in that: The preprocessing of the heart rate monitoring data information to obtain preprocessed heart rate monitoring data information includes: S211, preprocessing the video data information to obtain preprocessed video data information; S212, preprocessing the microwave data information to obtain preprocessed microwave data information.

4. The heart rate monitoring method of multi-source data fusion according to claim 3, characterized in that: The preprocessing of the video data information to obtain preprocessed video data information includes: S2111, performing face detection on the video data information to obtain facial image information; S2112, performing channel separation on the facial image information to obtain green channel data information; S2113, normalize and filter the green channel data information to obtain pre-processed video data information.

5. The heart rate monitoring method of multi-source data fusion according to claim 3, characterized in that: The preprocessing of the microwave data information to obtain preprocessed microwave data information includes: S2121, rearrange the microwave data information to obtain a data matrix; S2122, processing the data matrix to obtain a frequency spectrum matrix; S2123, performing phase difference processing on the frequency spectrum matrix to obtain pre-processed microwave data information.

6. The heart rate monitoring method of multi-source data fusion according to claim 2, characterized in that: The step of extracting features from the pre-processed video data information to obtain video feature information includes: S221, extracting time domain features from the pre-processed video data information to obtain first video feature information; S222, extracting frequency domain features from the pre-processed video data information to obtain second video feature information; S223, performing multi-scale entropy feature extraction on the pre-processed video data information to obtain third video feature information; S224: Perform feature fusion on the first video feature information, the second video feature information, and the third video feature information to obtain video feature information.

7. The heart rate monitoring method of multi-source data fusion according to claim 2, characterized in that: The step of screening the fused feature information to obtain optimized feature information includes: S251, performing correlation calculation on the fused feature information to obtain correlation information; S252, screening the fused feature information according to the correlation information to obtain first fused feature information; S253: Using an optimized support vector machine model, perform data screening on the first fused feature information to obtain optimized feature information.

8. A heart rate monitoring device with multi-source data fusion, characterized in that: The device comprises: An information acquisition module, used to acquire heart rate monitoring data information; the heart rate monitoring data information includes video data information and microwave data information; A feature extraction module, used to extract features from the heart rate monitoring data information to obtain optimized feature information; A model training module, used to train a preset heart rate monitoring model using the optimized feature information to obtain an optimized heart rate monitoring model; The heart rate monitoring module is used to process the heart rate monitoring data information to be processed using the optimized heart rate monitoring model to obtain a heart rate monitoring result.

9. A heart rate monitoring device with multi-source data fusion, characterized in that: The device comprises: A memory storing executable program code; a processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the heart rate monitoring method of multi-source data fusion as described in any one of claims 1-7.

10. A computer storable medium, characterized in that: The computer storable medium stores computer instructions, and when the computer instructions are called, they are used to execute the heart rate monitoring method of multi-source data fusion according to any one of claims 1 to 7.

Images