A non-contact user state monitoring system and method

Abstract

The application discloses a kind of non-contact user state monitoring system and method, the system includes body temperature monitoring module, heart rate monitoring module, video acquisition module, control module, signal processing module, storage module, communication module, display module, power module, alarm module and mobile terminal;Utilize signal processing module, process body temperature data information, heart rate data information and video data information, obtain user state monitoring result, and send user state monitoring result to control module and communication module. Utilize communication module, send user state monitoring result to mobile terminal;Utilize control module, send user state monitoring result to display module and alarm module;Utilize display module, display user state monitoring result;Utilize alarm module, alarm processing is carried out to user state monitoring result. The method of the application can protect the privacy of user, avoid the risk of cross infection, reduce the operation of monitoring instrument, and high safety.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of user state monitoring, in particular to a non-contact user state monitoring system and method. BACKGROUND

[0002] Most of the currently commonly used user state monitoring devices are contact type, the measuring device directly contacts with the body part of the person, and the monitoring is mainly artificial, and the monitoring method has problems of heavy nursing burden, low safety and low management efficiency. The non-contact millimeter wave radar and time-frequency monitoring data have obvious advantages, can protect the privacy of the user, do not contact with the user, greatly facilitate the people who need long-term continuous state monitoring, avoid the risk of cross infection, reduce the operation of the monitoring instrument, and are particularly suitable for state monitoring of the elderly, children, mental patients and large-area burn patients. Therefore, it is of great significance to study the non-contact user state monitoring method. SUMMARY

[0003] The technical problem to be solved by the present application is to provide a non-contact user state monitoring system and method, which uses non-contact body temperature monitoring, heart rate monitoring and video acquisition method to obtain user body temperature data, heart rate data, blood pressure data and action recognition results, processes the body temperature data, heart rate data, blood pressure data and action recognition results, and obtains user state monitoring results.

[0004] To solve the above technical problems, the first aspect of the embodiment of the present application discloses a non-contact user state monitoring system, the system comprises a body temperature monitoring module, a heart rate monitoring module, a video acquisition module, a control module, a signal processing module, a storage module, a communication module, a display module, a power module, an alarm module and a mobile terminal.

[0005] The control module is data-connected with the body temperature monitoring module, the heart rate monitoring module, the video acquisition module, the signal processing module, the storage module, the display module, the power module and the alarm module, and is used for system control.

[0006] The body temperature monitoring module is data-connected with the control module, and is used for collecting user body temperature data information.

[0007] The heart rate monitoring module is data-connected with the control module, and is used for collecting user heart rate data information.

[0008] The video acquisition module is data-connected with the control module, and is used for collecting user video data information.

[0009] The signal processing module is in data connection with the control module and the communication module, and is configured to process the body temperature data information, the heart rate data information and the video data information to obtain a user state monitoring result, and send the user state monitoring result to the control module and the communication module.

[0010] The communication module is in data connection with the signal processing module and the mobile terminal, and is configured to send the user state monitoring result to the mobile terminal.

[0011] The display module is in data connection with the control module, and is configured to display the user state monitoring result.

[0012] The power module is in data connection with the control module, and is configured to supply power for the user state monitoring system.

[0013] The alarm module is in data connection with the control module, and is configured to alarm according to the user state monitoring result.

[0014] The storage module is in data connection with the control module, and is configured to store data.

[0015] The mobile terminal is in data connection with the communication module, and is configured to receive and process the user state monitoring result.

[0016] The non-contact user state monitoring method applied to the non-contact user state monitoring system comprises the following steps:

[0017] S1, collecting body temperature data information of a user by using a body temperature monitoring module, and sending the body temperature data information to a signal processing module after buffering;

[0018] S2, collecting heart rate data information of the user by using a heart rate monitoring module, and sending the heart rate data information to the signal processing module after buffering;

[0019] S3, collecting video data information of the user by using a video acquisition module, and sending the video data information to the signal processing module after buffering;

[0020] S4, processing the body temperature data information, the heart rate data information and the video data information by using the signal processing module to obtain a user state monitoring result, and sending the user state monitoring result to a control module and a communication module; the user state monitoring result comprises a physiological parameter recognition result and a motion recognition result, and comprises:

[0021] S41, processing the body temperature data information, the heart rate data information and the video data information to obtain a physiological parameter recognition result, which comprises:

[0022] S411, feature extraction is conducted on the body temperature data information to obtain body temperature feature parameter information;

[0023] S412, the heart rate data information is processed to obtain heart rate feature parameter information;

[0024] S413, the video data information is processed to obtain blood pressure feature parameter information;

[0025] S414, the body temperature feature parameter information, the heart rate feature parameter information and the blood pressure feature parameter information are fused to obtain fused feature parameter information;

[0026] S415, the fused feature parameter information is used to train a preset physiological parameter recognition model to obtain an optimized physiological parameter recognition model;

[0027] S416, the optimized physiological parameter recognition model is used to process the to-be-processed fused feature parameter information to obtain a physiological parameter recognition result;

[0028] S42, the video data information is processed to obtain an action recognition result

[0029] S5, the communication module is used to send the user state monitoring result to a mobile terminal;

[0030] S6, the control module is used to send the user state monitoring result to a display module and an alarm module;

[0031] S7, the display module is used to display the user state monitoring result;

[0032] S8, the alarm module is used to alarm the user state monitoring result.

[0033] As an optional implementation, in the embodiment of the present application, the body temperature monitoring module comprises an infrared temperature sensor, which detects the body temperature of the user through the infrared energy radiated by the user.

[0034] The heart rate monitoring module comprises a radar sensor, which measures the heart rate of the user by receiving the reflected electromagnetic wave of the user.

[0035] As an optional implementation, in the embodiment of the present application, the feature extraction is conducted on the body temperature data information to obtain body temperature feature parameter information, which comprises:

[0036] S4111, short-time Fourier transform is conducted on the body temperature data information to obtain frequency domain feature information;

[0037] S4112, the preset denoising network model is used to process the frequency domain feature information, and body temperature feature parameter information is obtained.

[0038] As an optional implementation, in the embodiment of the present application, the processing of the heart rate data information to obtain heart rate feature parameter information comprises:

[0039] S4121, the heart rate data information is processed to obtain first feature parameter information, second feature parameter information and third feature parameter information.

[0040] The calculation formula of the first feature parameter information is:

[0041] S1=20log 10 |X(k)| 2

[0042] wherein, S1 is the first feature parameter information, x(n) is the heart rate data information, N is the length of x(n), x(n)=[x1, x2, …, xN-1] is the i-th sample point in x(n), i=1, 2, …, N-1. N-1 i

[0043] The calculation formula of the second feature parameter information is:

[0044]

[0045] wherein, j represents an imaginary unit, φ(x k ) represents the instantaneous phase of x k , S k is the k-th value in the second feature parameter information S2, x k , k=1, 2, …, N-1 is the k-th sample point in x(n).

[0046] The calculation method of the third feature parameter information is:

[0047]

[0048] wherein S3 is the third feature parameter information.

[0049] S4122, the first feature parameter information, the second feature parameter information and the third feature parameter information are integrated to obtain heart rate feature parameter information.

[0050] As an optional implementation, in the embodiment of the present application, the integration of the first feature parameter information, the second feature parameter information and the third feature parameter information to obtain heart rate feature parameter information comprises: ​​

[0051] Integrating the first feature parameter information, the second feature parameter information and the third feature parameter information by using a feature integration model to obtain heart rate feature parameter information;

[0052] The feature integration model expression is:

[0053] t=WQ+b

[0054] Wherein, t is heart rate feature parameter information, W is weight information, b is bias information, W and b are set by experiment, and Q is input, Indicates a tensor product operation.

[0055] As an optional implementation, in the embodiment of the application, the processing of the video data information to obtain blood pressure feature parameter information comprises:

[0056] S4131, pre-processing the video data information to obtain pre-processed video data information;

[0057] S4132, extracting features from the pre-processed video data information to obtain video feature parameter information;

[0058] S4133, training a preset blood pressure measurement model by using the video feature parameter information to obtain an optimized blood pressure measurement model;

[0059] S4134, processing the video feature parameter information to be processed by using the optimized blood pressure measurement model to obtain blood pressure feature parameter information.

[0060] As an optional implementation, in the embodiment of the application, the processing of the video data information to obtain an action recognition result comprises:

[0061] S421, processing the video data information to obtain video frame image information;

[0062] S422, performing human body detection on the video frame image information to obtain human body image information;

[0063] S423, extracting joint points from the human body image information to obtain joint data information;

[0064] S424, performing time dimension stacking processing on the joint data information to obtain joint point sequence data information;

[0065] S425, training a preset action recognition model by using the joint point sequence data information to obtain an optimized action recognition model;

[0066] S426, using the optimized action recognition model, processing the node sequence data information to be processed to obtain an action recognition result.

[0067] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0068] The present application provides a non-contact user state monitoring system and scheme, provides non-contact millimeter wave radar, temperature, video monitoring data, and monitors the state of the user to obtain physiological parameter recognition results and action recognition results, and uses an alarm system for comprehensive decision and alarm. The method of the present application can protect the privacy of the user, does not contact the user, greatly facilitates the population that needs to be continuously monitored for a long time, avoids the risk of cross infection, reduces the operation of the monitoring instrument, is high in safety, and improves management efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0069] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor.

[0070] Figure 1 is a structural schematic diagram of a non-contact user state monitoring system disclosed by the embodiments of the present application;

[0071] Figure 2 is a flowchart of a non-contact user state monitoring method disclosed by the embodiments of the present application. DETAILED DESCRIPTION

[0072] In order to make the personnel in the technical field better understand the present application scheme, the technical solutions in the embodiments of the present application will be described clearly and completely in the following with reference to the drawings of the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0073] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish different objects, not to describe a specific order. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, device, product or equipment including a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units not listed, or optionally includes other steps or units inherent to the process, method, product or equipment.

[0074] Reference to an "embodiment" herein means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase that the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are they necessarily mutually exclusive of other embodiments. It is explicitly contemplated that embodiments described herein can be combined with each other in their various permutations and combinations.

[0075] The application discloses a kind of non-contact user state monitoring system and method, the system includes body temperature monitoring module, heart rate monitoring module, video acquisition module, control module, signal processing module, storage module, communication module, display module, power module, alarm module and mobile terminal;With signal processing module, body temperature data information, heart rate data information and video data information are handled, and user state monitoring result is obtained, and user state monitoring result is sent to control module and communication module. With communication module, user state monitoring result is sent to mobile terminal;With control module, user state monitoring result is sent to display module and alarm module;With display module, user state monitoring result is displayed;With alarm module, user state monitoring result is alarm processing.The method of the application can protect the privacy of user, avoid the risk of cross infection, reduce the monitoring instrument operation, and high safety.The following are described in detail.

[0076] Embodiment one

[0077] Please refer to Figure 1 , Figure 1 It is a kind of non-contact user state monitoring system structure schematic diagram disclosed by the embodiment of the application. Among them, Figure 1 The non-contact user state monitoring system described is applied to household state monitoring technical field, and the embodiment of the application is not limited. Figure 1 As shown in the figure, the non-contact user state monitoring system includes body temperature monitoring module, heart rate monitoring module, video acquisition module, control module, signal processing module, storage module, communication module, display module, power module, alarm module and mobile terminal;

[0078] The control module is connected with the body temperature monitoring module, the heart rate monitoring module, the video acquisition module, the signal processing module, the storage module, the display module, the power module and the alarm module data, for system control;

[0079] The body temperature monitoring module is connected with the control module data, for gathering user's body temperature data information;

[0080] The heart rate monitoring module is connected with the control module data, for gathering user's heart rate data information;

[0081] The video acquisition module is in data connection with the control module, and is configured to acquire video data information of the user.

[0082] The signal processing module is in data connection with the control module and the communication module, and is configured to process the body temperature data information, the heart rate data information and the video data information to obtain a user state monitoring result, and send the user state monitoring result to the control module and the communication module.

[0083] The communication module is in data connection with the signal processing module and the mobile terminal, and is configured to send the user state monitoring result to the mobile terminal.

[0084] The display module is in data connection with the control module, and is configured to display the user state monitoring result.

[0085] The power module is in data connection with the control module, and is configured to supply power for the user state monitoring system.

[0086] The alarm module is in data connection with the control module, and is configured to alarm according to the user state monitoring result.

[0087] The storage module is in data connection with the control module, and is configured to store data.

[0088] The mobile terminal is in data connection with the communication module, and is configured to receive and process the user state monitoring result.

[0089] Optionally, the body temperature monitoring module comprises an LU90614 infrared temperature sensor, which detects the body temperature of the user through infrared energy radiated by the user, and has the characteristics of fast response speed, non-contact, high accuracy and wide application field.

[0090] The heart rate monitoring module comprises an R60ABH1 radar sensor, which measures the heart rate of the user by receiving reflected electromagnetic waves of the user.

[0091] Specifically, the radar transmission frequency band is 60GHz electromagnetic wave, the part of the human body to be measured emits electromagnetic wave, the millimeter wave radar module receives the electromagnetic wave signal reflected by the human body, and the electromagnetic wave signal is demodulated, and then signal processing such as signal amplification, filtering and AD conversion is performed. The MCU of the millimeter wave radar receives the demodulated electromagnetic wave signal, and calculates the demodulated electromagnetic wave signal to obtain the heart rate of the user.

[0092] In this embodiment, the storage module is a DDR3 storage module.

[0093] In this embodiment, the video acquisition module is a camera placed indoors, which needs to be adjusted in angle to face the user to be monitored when in use.

[0094] The control module is an FPGA development board, and an AX7035 of ALINX is selected.

[0095] Optionally, the signal processing module can be completed by an upper computer or a dedicated DSP signal processing board.

[0096] The communication module is completed by a wireless communication module or other communication modules, and the embodiment is not limited.

[0097] The display module and the power module are general technologies in the field.

[0098] It can be seen that the present application provides a non-contact user state monitoring system and scheme, provides non-contact millimeter wave radar, temperature, video monitoring data, monitors the state of the user, obtains physiological parameter recognition results and action recognition results, and comprehensively decides and alarms by using an alarm system. The method can protect the privacy of the user, does not contact the user, greatly facilitates the population that needs to continuously monitor the state for a long time, avoids the risk of cross infection, reduces the operation of the monitoring instrument, is high in safety, and improves the management efficiency.

[0099] Embodiment two

[0100] Please refer to Figure 2 , Figure 2 is a flowchart of a non-contact user state monitoring method disclosed by the embodiment of the present application. Wherein, Figure 2 The non-contact user state monitoring method described is applied to the field of user state monitoring technology, and the embodiment of the present application is not limited. As Figure 2 shown, the non-contact user state monitoring method comprises:

[0101] S1, collecting user body temperature data information by using a body temperature monitoring module, and sending the body temperature data information to a signal processing module after buffering;

[0102] S2, collecting user heart rate data information by using a heart rate monitoring module, and sending the heart rate data information to the signal processing module after buffering;

[0103] S3, collecting user video data information by using a video acquisition module, and sending the video data information to the signal processing module after buffering;

[0104] S4, processing the body temperature data information, the heart rate data information and the video data information by using the signal processing module, obtaining a user state monitoring result, and sending the user state monitoring result to a control module and a communication module; the user state monitoring result comprises physiological parameter recognition results and action recognition results;

[0105] S5, sending the user state monitoring result to a mobile terminal by using the communication module;

[0106] The guardian may not be present beside the monitored user at times, at which time the user state monitoring result can be received on the mobile terminal.

[0107] S6, sending the user state monitoring result to a display module and an alarm module by using the control module;

[0108] S7, displaying the user state monitoring result by using the display module;

[0109] The display module displays the physiological parameter recognition result and the action recognition result for people to view and perform necessary processing.

[0110] S8, performing alarm processing on the user state monitoring result by using the alarm module.

[0111] The alarm module receives the physiological parameter recognition result and the action recognition result, processes the physiological parameter recognition result and the action recognition result by using the trained neural network model based on Bi-GRU and double attention mechanism, obtains a physiological parameter recognition confusion matrix and an action recognition confusion matrix, processes the physiological parameter recognition confusion matrix and the action recognition confusion matrix, and obtains a comprehensive evaluation result including four levels of normal, relatively dangerous, dangerous, and critical, and prompts by using different indicator lights and alarm sounds.

[0112] Optionally, the user state monitoring result obtained by processing the body temperature data information, the heart rate data information, and the video data information by using the signal processing module includes:

[0113] S41, processing the body temperature data information, the heart rate data information, and the video data information to obtain a physiological parameter recognition result;

[0114] S42, processing the video data information to obtain an action recognition result.

[0115] Optionally, the physiological parameter recognition result obtained by processing the body temperature data information, the heart rate data information, and the video data information includes:

[0116] S411, performing feature extraction on the body temperature data information to obtain body temperature feature parameter information;

[0117] S412, processing the heart rate data information to obtain heart rate feature parameter information;

[0118] S413, processing the video data information to obtain blood pressure feature parameter information;

[0119] S414, performing feature fusion on the body temperature feature parameter information, the heart rate feature parameter information and the blood pressure feature parameter information to obtain fused feature parameter information;

[0120] Optionally, the fusion method is as follows:

[0121] Obtain the clustering feature information X and the clustering feature information Y to be fused; X is an n'×m' matrix, Y is a p×m' matrix, m' represents the sample number, n' and p represent the dimensions of two features, the two matrices are projected to 1 dimension for linear representation, and the projection vectors a1 and b1 are corresponded respectively, so that the feature matrix after projection becomes:

[0122]

[0123] The purpose is to maximize the correlation coefficient between X' and Y', so as to obtain the projection vectors a1 and b1 when the correlation coefficient is maximum, that is:

[0124]

[0125] The data is standardized before projection, the purpose of standardization is to make the mean value of the data as 0 and the variance as 1, and the following can be obtained:

[0126] cov(X',Y')=cov(a1 T X,b1 T Y)=E(<a1 T X,b1 T Y>)=E((a1 T X)(b1 T Y) T )=a1 T E(XY T )b1

[0127]

[0128] D(X)=cov(X,X)=E(X T X)

[0129] D(Y)=cov(Y,Y)=E(Y T Y)

[0130] cov(X,Y)=E(XY T ),cov(Y,X)=E(YX T )

[0131] S XX =cov(X,X),then the solution target is converted into:

[0132]

[0133] Step1: Calculate the variance S of X, Y XX and the covariance S YY of XY and YX XY = S YX T ;

[0134] Step2: Calculate the matrix

[0135] Step3: Solve the singular value of M', get the maximum singular value and its front and rear singular vectors u, v;

[0136] Step4: The projection vectors a1 and b1 of X and Y are respectively:

[0137]

[0138] Step5: The fusion feature vector Z of X and Y is

[0139] According to the above method, the body temperature feature parameter information and the heart rate feature parameter information are fused to obtain first fusion information; the first fusion information and the blood pressure feature parameter information are fused to obtain fusion feature parameter information.

[0140] S415, using the fusion feature parameter information, training a preset physiological parameter recognition model to obtain an optimized physiological parameter recognition model;

[0141] The preset physiological parameter recognition model is a convolutional neural network model based on global weighting, and the model is a global weighting structure;

[0142] The global weighting structure includes adding a global weighting module to the convolutional layer of the convolutional neural network model;

[0143] The global weighting module uses an adaptive threshold to eliminate noise of the convolutional neural network model;

[0144] The convolution kernel of the convolutional neural network model based on global weighting uses a feature superposition method for convolution operation;

[0145] The loss function L of the convolutional neural network model based on global weighting is:

[0146]

[0147] Wherein, the λ parameter is used to control the degree of inter-class dispersion, x i is the predicted value of the current neuron output, c yi is the true sample label value, is the weight of the neuron, b j bias is the bias of the neuron, n is the number of neurons, and m is the number of features of the input sample.

[0148] S416, using the optimized physiological parameter identification model, processing the fusion feature parameter information to be processed to obtain a physiological parameter identification result. The physiological parameter identification result is four levels of excellent, good, medium and poor.

[0149] Optionally, the feature extraction of the body temperature data information to obtain the body temperature feature parameter information comprises:

[0150] S4111, performing short-time Fourier transform on the body temperature data information to obtain frequency domain feature information;

[0151] S4112, using a preset denoising network model to process the frequency domain feature information to obtain the body temperature feature parameter information.

[0152] Denoising network model: using STFT (short-time Fourier transform) to convert the signal to be processed into time-frequency domain, using a window length of 256 samples, an overlap rate of 75%, and using a Hamming window. By discarding the frequency samples corresponding to the negative frequency, the length of the frequency spectrum vector can be reduced to 129. A full convolutional network composed of 16 convolutional layers is defined, the first layer has a convolution kernel size of 9x8, and there are 18 convolution kernels. The second to 13th convolutional layers are repeated four times, each containing three layers, with convolution kernel widths of 5, 9 and 9, respectively, and the number of convolution kernels is 64. The last convolutional layer has a convolution kernel width of 129 and only one convolution kernel. In this network, convolution is only performed in the frequency dimension direction, and all layers except the first layer have a convolution kernel width of 1 along the time dimension. After the convolutional layer, a batch normalization layer and a ReLU activation function layer are used, and finally a regression layer is used for output.

[0153] A convolutional neural network is established to complete the mapping F of the STFT spectrum X(m, k) of the signal to be processed and the STFT spectrum S(m, k) of the clean signal to be processed, achieving the purpose of denoising. The input prediction variable signal and the network target signal are the amplitude spectrum of the received noisy signal to be processed and the clean signal to be processed, respectively. The goal of the network is to learn the mapping F so that the network outputs the amplitude spectrum of the noisy signal to be processed. The regression network uses the prediction variable input to minimize the mean square error between its output and the input target, so the loss function of the denoising network model can be expressed as

[0154]

[0155] Where B represents the batch size, M and K represent the total number of STFT time frames and frequency points, respectively.

[0156] Optionally, the processing of the heart rate data information to obtain heart rate feature parameter information comprises:

[0157] S4121, processing the heart rate data information to obtain first feature parameter information, second feature parameter information and third feature parameter information;

[0158] The calculation formula of the first feature parameter information is:

[0159] S1=20log 10 |X(k)| 2

[0160] wherein, S1 is the first feature parameter information, x(n) is the heart rate data information, N is the length of x(n), x(n)=[x0, x1, x2, …, x N-1 ], x i i=0, 1, 2, …, N-1 is the i-th sample point in x(n);

[0161] The calculation formula of the second feature parameter information is:

[0162]

[0163] wherein, j represents an imaginary unit, φ(x k ) represents the instantaneous phase of x k , S k k is the k-th in the second feature parameter information S2, x k k=1, 2, …, N-1 is the k-th sample point in x(n);

[0164] The calculation method of the third feature parameter information is:

[0165]

[0166] wherein S3 is the third feature parameter information;

[0167] S4122, integrating the first feature parameter information, the second feature parameter information and the third feature parameter information to obtain heart rate feature parameter information.

[0168] Optionally, the integration of the first feature parameter information, the second feature parameter information and the third feature parameter information to obtain heart rate feature parameter information comprises:

[0169] integrating the first feature parameter information, the second feature parameter information and the third feature parameter information by using a feature integration model to obtain heart rate feature parameter information;

[0170] The feature integration model expression is:

[0171] t = WQ + b

[0172] Wherein, t is a heart rate feature parameter information, W is a weight information, b is a bias information, W and b are set by experiment, Q is an input, Indicates a tensor product operation.

[0173] Optionally, the video data information is processed to obtain blood pressure feature parameter information, comprising:

[0174] S4131, pre-processing the video data information to obtain pre-processed video data information;

[0175] Including extracting an imaging photoplethysmography (IPPG) signal from the video data information, and performing a sliding average processing on the IPPG signal to obtain a smoothed IPPG signal;

[0176] Removing outliers from the smoothed IPPG signal to obtain a first smoothed IPPG signal;

[0177] Processing the first smoothed IPPG signal to remove signal segments with abnormal heart rate, to obtain a second smoothed IPPG signal;

[0178] Processing the second smoothed IPPG signal to remove signal segments with serious discontinuity, to obtain a third smoothed IPPG signal;

[0179] Performing autocorrelation calculation on the third smoothed IPPG signal, and removing signal segments with high changes according to the autocorrelation value to obtain the pre-processed video data information;

[0180] S4132, extracting features from the pre-processed video data information to obtain video feature parameter information;

[0181] S41321, processing the pre-processed video data information using a fuzzy parameter calculation model to obtain fuzzy parameter information;

[0182] The fuzzy parameter calculation model expression is:

[0183]

[0184] Wherein, x(t) is pre-processed video data information, A x (θ,τ) is fuzzy parameter information, t is a time variable, τ is a displacement variable, θ is a frequency variable corresponding to τ, and * indicates taking a conjugate.

[0185] S41322, performing Hadamard product calculation on the fuzzy parameter information and a preset kernel function to obtain an optimized kernel function;

[0186] The optimization kernel function expression is:

[0187] k(θ,τ)=A x (θ,τ)g(θ,τ)

[0188] wherein k(θ,τ) is an optimization kernel function, g(θ,τ) is a preset kernel function, a=0.001, β=50;

[0189] S41323, processing the pre-processed video data information and the optimization kernel function to obtain video feature parameter information;

[0190] The video feature parameter information calculation formula is:

[0191]

[0192] wherein C x is video feature parameter information, the integral range is -∞~∞, u and t are time variables, τ is a time shift variable, Ω is a frequency variable corresponding to u, and θ is a frequency variable corresponding to τ.

[0193] S4133, training a preset blood pressure measurement model using the video feature parameter information to obtain an optimized blood pressure measurement model;

[0194] The preset blood pressure measurement model is a CTPN model;

[0195] The CTPN model includes a VGG16 convolution model, a bidirectional LSTM model, and a fully connected layer;

[0196] The video feature parameter information is input into the VGG16 convolution model; the VGG16 network includes 13 convolution layers, 5 maximum pooling layers, and 3 fully connected layers;

[0197] A two-dimensional convolution kernel w∈R 3×3 is used in the convolution layer to obtain a feature matrix C n ;

[0198]

[0199] wherein n represents the number of convolution operations, m represents the number of convolution kernels, p i represents the i-th obtained feature matrix, f represents a nonlinear activation function, · represents corresponding operations of shared weights of the convolution kernel and the feature matrix, w represents weights of the convolution kernel, and b represents a bias value, R 3×3 represents a 3×3 real matrix;

[0200] The method using maximum pooling in the pooling layer is characterized by the formula for feature extraction as follows:

[0201] p u =Max 2×2 [C n ]

[0202] wherein u represents the number of times of pooling, Max 2×2 represents the operation method of maximum pooling with a size of 2*2 matrix, p u is the extracted feature; after the convolution and pooling operations, the data stream after the reshape processing is input into the bidirectional LSTM model to obtain a feature vector with a time sequence attribute, which is expressed by the formula as follows:

[0203] o t =g(V st +V’ sT′ )

[0204]

[0205] wherein s t represents the output of the forward time sequence at t moment, s’ t represents the output of the reverse time sequence at t moment, U Xt represents the initial input of the forward time sequence, represents the initial input of the reverse time sequence, represents the input of the forward time sequence at the previous moment, represents the input of the reverse time sequence at the next moment, o t represents the output at t moment, after the feature extraction of the time sequence model, the spatial+sequence feature vector is input into the RPN network, and the feature vector set is classified by the softmax to obtain positive feedback and negative feedback classification, and the other is used to calculate the boundary value regression offset of the feature vector set to obtain accurate measurement results.

[0206] S4134, processing the video feature parameter information to be processed by using the optimized blood pressure measurement model to obtain blood pressure feature parameter information.

[0207] Optionally, the processing of the video data information to obtain the action recognition result comprises:

[0208] S421, processing the video data information to obtain video frame image information;

[0209] S422, performing human body detection on the video frame image information to obtain human body image information;

[0210] S423, performing joint point extraction on the human body image information to obtain joint data information;

[0211] S424, stack the joint data information along the time dimension to obtain joint node sequence data information;

[0212] S425, train a preset action recognition model using the joint node sequence data information to obtain an optimized action recognition model;

[0213] The action recognition model is a multi-scale time convolution model, which includes a multi-scale feature extraction and a multi-scale feature fusion. The feature extraction part is composed of a multi-scale pyramid convolution module, which extracts different time scale features on multiple branches to obtain multi-scale feature maps on channels. The multi-scale feature fusion module is composed of three steps: first, use the SEWeight module to extract the multi-scale feature weight matrix on different channels; then use Softmax to normalize the multi-scale feature weight matrix to ensure that the importance of each feature channel is reasonably considered to produce more effective feature fusion results; finally, the multi-scale feature maps on different channels are multiplied by the channel feature weight matrix element by element, so that the multi-scale features with higher weights are more prominent, and the multi-scale features with lower weights are weakened.

[0214] S426, process the joint node sequence data information to be processed using the optimized action recognition model to obtain an action recognition result. The action recognition result includes 16 action categories such as standing, walking, sitting, falling, etc.

[0215] It can be seen that the present application proposes a non-contact user state monitoring system and scheme, provides non-contact millimeter wave radar, temperature, video monitoring data, and monitors the state of the user to obtain physiological parameter recognition results and action recognition results, and uses an alarm system for comprehensive decision and alarm. The method of the present application can protect the privacy of the user, does not contact the user, greatly facilitates the population that needs to be continuously monitored for a long time, avoids the risk of cross infection, reduces the operation of the monitoring instrument, has high safety, and improves management efficiency.

[0216] The device embodiments described above are only schematic, and the modules illustrated as separate components can or can not be physically separated, and the components illustrated as modules can or can not be physical modules, i.e. they can be located in one place or distributed on multiple network modules. Part or all of the modules can be selected according to actual needs to achieve the purpose of the present embodiment scheme. Those skilled in the art can understand and implement it without creative labor.

[0217] Those skilled in the art can clearly understand the implementation of the various embodiments by means of software and the necessary general hardware platform through the above specific description of the embodiments, and of course, the embodiments can also be implemented by hardware. Based on such understanding, the above technical solutions, essentially or in other words, the part of the prior art that contributes to the present application can be embodied in the form of a software product. The computer software product can be stored in a computer readable storage medium, including a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disk storage, a magnetic disk storage, a magnetic tape storage, or any other computer readable medium that can be used to carry or store data.

[0218] Finally, it should be noted that: the non-contact user state monitoring system and method disclosed by the embodiments of the present application are only the preferred embodiments of the present application, and are used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that; the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A non-contact user state monitoring system, characterized by, The system comprises a body temperature monitoring module, a heart rate monitoring module, a video acquisition module, a control module, a signal processing module, a storage module, a communication module, a display module, a power module, an alarm module and a mobile terminal; The control module is in data connection with the body temperature monitoring module, the heart rate monitoring module, the video acquisition module, the signal processing module, the storage module, the display module, the power module and the alarm module, and is configured to perform system control; The body temperature monitoring module is in data connection with the control module, and is configured to acquire body temperature data information of a user; The heart rate monitoring module is in data connection with the control module, and is configured to acquire heart rate data information of a user; The video acquisition module is in data connection with the control module, and is configured to acquire video data information of a user; The signal processing module is in data connection with the control module and the communication module, and is configured to process the body temperature data information, the heart rate data information and the video data information to obtain a user state monitoring result, and send the user state monitoring result to the control module and the communication module; The communication module is in data connection with the signal processing module and the mobile terminal, and is configured to send the user state monitoring result to the mobile terminal; The display module is in data connection with the control module, and is configured to display the user state monitoring result; The power module is in data connection with the control module, and is configured to supply power to the user state monitoring system; The alarm module is in data connection with the control module, and is configured to alarm according to the user state monitoring result; The storage module is in data connection with the control module, and is configured to store data; The mobile terminal is in data connection with the communication module, and is configured to receive and process the user state monitoring result. The steps of the non-contact user state monitoring method applied to the non-contact user state monitoring system comprise: S1, acquiring body temperature data information of a user by using a body temperature monitoring module, and sending the body temperature data information to a signal processing module after buffering; S2, acquiring heart rate data information of a user by using a heart rate monitoring module, and sending the heart rate data information to the signal processing module after buffering; S3, acquiring video data information of a user by using a video acquisition module, and sending the video data information to the signal processing module after buffering; S4, processing the body temperature data information, the heart rate data information and the video data information by using a signal processing module to obtain a user state monitoring result, and sending the user state monitoring result to a control module and a communication module; the user state monitoring result comprises a physiological parameter recognition result and a motion recognition result, and comprises: S41, processing the body temperature data information, the heart rate data information and the video data information to obtain a physiological parameter recognition result, which comprises: S411, extracting features from the body temperature data information to obtain body temperature feature parameter information; S412, processing the heart rate data information to obtain heart rate feature parameter information, which comprises: S4121, processing the heart rate data information to obtain first feature parameter information, second feature parameter information and third feature parameter information; The calculation formula of the first feature parameter information is: wherein, , is a first feature parameter information, is a heart rate data information, N is a length of, , is a first sample in, i the first feature parameter information. The calculation formula of the second feature parameter information is: In the formula, j denotes the imaginary unit, denotes the instantaneous phase of is second characteristic parameter information, is the first k sample point in The calculation method of the third feature parameter information is: wherein is the third feature parameter information; S4122, integrating the first feature parameter information, the second feature parameter information and the third feature parameter information to obtain heart rate feature parameter information, including: Integrating the first feature parameter information, the second feature parameter information and the third feature parameter information by using a feature integration model to obtain heart rate feature parameter information; The feature integration model expression is: wherein, t is heart rate feature parameter information, W is weight information, b is bias information, W and b from an experimental setup, Q is an input, , denotes a tensor product operation; S413, processing the video data information to obtain blood pressure feature parameter information; S414, performing feature fusion on the body temperature feature parameter information, the heart rate feature parameter information and the blood pressure feature parameter information to obtain fusion feature parameter information; S415, training a preset physiological parameter recognition model by using the fusion feature parameter information to obtain an optimized physiological parameter recognition model; S416, processing the fusion feature parameter information to be processed by using the optimized physiological parameter recognition model to obtain a physiological parameter recognition result; S42, processing the video data information to obtain an action recognition result S5, sending the user state monitoring result to a mobile terminal by using the communication module; S6, sending the user state monitoring result to a display module and an alarm module by using the control module; S7, displaying the user state monitoring result by using the display module; S8, performing alarm processing on the user state monitoring result by using the alarm module.

2. The non-contact user state monitoring system of claim 1, wherein, The body temperature monitoring module includes an infrared temperature measurement sensor, which detects the body temperature of the user through the infrared energy radiated by the user; The heart rate monitoring module includes a radar sensor, which measures the heart rate of the user by receiving the reflected electromagnetic wave of the user.

3. The non-contact user state monitoring system of claim 1, wherein, The feature extraction on the body temperature data information to obtain body temperature feature parameter information includes: S4111, performing short-time Fourier transform on the body temperature data information to obtain frequency domain feature information; S4112, processing the frequency domain feature information by using a preset denoising network model to obtain body temperature feature parameter information.

4. The non-contact user state monitoring system of claim 1, wherein, The processing of the video data information to obtain blood pressure feature parameter information includes: S4131, preprocessing the video data information to obtain preprocessed video data information; S4132, performing feature extraction on the preprocessed video data information to obtain video feature parameter information; S4133, training a preset blood pressure measurement model by using the video feature parameter information to obtain an optimized blood pressure measurement model; S4134, processing the video feature parameter information to be processed by using the optimized blood pressure measurement model to obtain blood pressure feature parameter information.

5. The non-contact user state monitoring system of claim 1, wherein, The processing of the video data information to obtain an action recognition result includes: S421, processing the video data information to obtain video frame image information; S422, human body detection is carried out on the video frame image information, and human body image information is obtained; S423, joint point extraction is carried out on the human body image information, and joint data information is obtained; S424, the joint data information is stacked along the time dimension, and joint sequence data information is obtained; S425, the joint sequence data information is used to train a preset motion recognition model, and an optimized motion recognition model is obtained; S426, the optimized motion recognition model is used to process the joint sequence data information to be processed, and a motion recognition result is obtained.

Images