Human factor intelligence-based vital sign signal measurement method and apparatus, and device

Abstract

Embodiments of the present description provide a human factor intelligence-based vital sign signal measurement method and apparatus, and a device. The method comprises: acquiring individual feature representation data of a measured object, environmental feature representation data of the environment where the measured object is located, and vital sign spectrum data of the measured object, wherein the individual feature representation data and the environmental feature representation data have a differential impact on the vital sign spectrum data; and performing signal value prediction on the basis of the individual feature representation data, the environmental feature representation data and the vital sign spectrum data to obtain a vital sign signal value with the differential impact removed, so that the accuracy of vital sign signal measurement can be improved.

Description

Method, device and equipment for measuring vital sign signals based on human intelligence

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to the Chinese patent application filed with the Patent Office of China on December 22, 2023, with application number 202311792773.3 and invention name “Method, device and equipment for measuring vital signs signals based on human intelligence”, and the Chinese patent application filed with the Patent Office of China on December 25, 2023, with application number 202311801230.3 and invention name “A method for detecting vital signs based on human intelligence and related equipment”, the entire contents of which are incorporated by reference into this application. Technical Field

[0003] The embodiments in this specification relate to the field of computer technology, and specifically to a method, device, and apparatus for measuring vital sign signals based on human intelligence. Background Art

[0004] Vital sign signals are a set of medical parameters that can describe a person's health status and body functions, which may include heart rate, respiratory rate, body temperature, and blood pressure.

[0005] In related technologies, when measuring vital sign signals, contact sensors such as electrocardiogram monitors or millimeter-wave radars can usually be used for measurement. However, the use of contact sensors for measurement may interfere with the vital sign signals of the person being measured and requires regular calibration and maintenance, which has certain limitations. The measurement of vital sign signals based on millimeter-wave radar is easily interfered with, and its measurement accuracy needs to be improved. Summary of the Invention

[0006] This application proposes a method, device and equipment for measuring vital sign signals based on human factors intelligence to improve the accuracy of vital sign signal measurement.

[0007] An embodiment of the present specification provides a method for measuring vital sign signals based on human factors intelligence, the method comprising: obtaining individual feature representation data of a measured object, environmental feature representation data of the environment in which the measured object is located, and vital sign spectrum data of the measured object; wherein the individual feature representation data and the environmental feature representation data have differential effects on the vital sign spectrum data; and performing signal value prediction based on the individual feature representation data, the environmental feature representation data, and the vital sign spectrum data to obtain a vital sign signal value after removing the differential effects.

[0008] An embodiment of the present specification provides a vital sign signal measurement device based on human factors intelligence, the device comprising: a feature data acquisition module for acquiring individual feature representation data of a measured object, environmental feature representation data of the environment in which the measured object is located, and vital sign spectrum data of the measured object; wherein the individual feature representation data and the environmental feature representation data have differential effects on the vital sign spectrum data; and a vital sign determination module for performing signal value prediction based on the individual feature representation data, the environmental feature representation data, and the vital sign spectrum data to obtain a vital sign signal value after removing the differential effects.

[0009] An embodiment of this specification provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the measurement method described in any of the above embodiments when executing the computer program.

[0010] The embodiments of this specification also provide an edge computing device, including a memory, a processor, and a communication interface. The memory stores a computer program, and the processor implements the above-mentioned vital sign signal measurement method when executing the computer program.

[0011] An embodiment of this specification provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the measurement method described in any of the above embodiments is implemented.

[0012] The multiple implementation methods provided in this specification obtain individual feature representation data of the measured object, environmental feature representation data of the environment in which the measured object is located, and vital sign spectrum data of the measured object, and perform signal value prediction based on the individual feature representation data, environmental feature representation data, and vital sign spectrum data to obtain vital sign signal values ​​after removing the differential effects of the individual feature representation data on the vital sign spectrum data and removing the differential effects of the environmental feature representation data on the vital sign spectrum data. In this way, the accuracy of vital sign signal measurement can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] FIG1a is a schematic diagram of a vital sign signal measurement system provided in an embodiment of this specification;

[0014] FIG1b is a schematic diagram of a vital sign signal measurement system provided in an embodiment of this specification;

[0015] FIG2 is a flow chart of a method for measuring vital signs signals according to an embodiment of the present disclosure;

[0016] FIG3 is a flow chart of a method for acquiring individual feature representation data according to an embodiment of this specification;

[0017] FIG4 is a flow chart of a method for acquiring environmental characteristic representation data according to an embodiment of the present disclosure;

[0018] FIG5 is a flow chart of a method for acquiring vital sign spectrum data according to an embodiment of the present disclosure;

[0019] FIG6 is a schematic flow chart of a method for determining a target distance unit according to an embodiment of the present disclosure;

[0020] FIG7 a is a schematic flow chart of a method for measuring vital sign signals according to an embodiment of the present disclosure;

[0021] FIG7 b is a flow chart of a method for acquiring environmental characteristic representation data provided in an embodiment of this specification;

[0022] FIG8a is a schematic diagram of a flow chart of a target classification model training method provided in an embodiment of this specification;

[0023] FIG8b is a flow chart of a training sample construction method provided in an embodiment of this specification;

[0024] FIG9 is a schematic diagram of a computer device provided in accordance with an embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] In order to help those skilled in the art better understand the solutions in this specification, the following will clearly and completely describe the technical solutions in the embodiments of this specification in conjunction with the drawings in the embodiments of this specification. Obviously, the embodiments described are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this specification.

[0026] In some related technologies, when measuring vital sign signals, contact sensors such as electrocardiogram monitors or millimeter-wave radars are usually used for measurement. However, the use of contact sensors for measurement may interfere with the vital sign signals of the person being measured and requires regular calibration and maintenance, which has certain limitations. The measurement of vital sign signals based on millimeter-wave radar is easily subject to interference, and its measurement accuracy needs to be improved.

[0027] Therefore, when measuring vital sign signals based on millimeter wave radar, it is necessary to provide a vital sign signal measurement method based on human intelligence, that is, to provide a vital sign signal measurement method that varies from person to person, so as to measure vital sign signals of different objects under test. First, the individual feature representation data of the object under test, the environmental feature representation data of the environment in which the object under test is located, and the vital sign spectrum data of the object under test are obtained. Then, the signal value is predicted based on the individual feature representation data, the environmental feature representation data, and the vital sign spectrum data to obtain the vital sign signal value without the difference influence of the individual feature representation data and the environmental feature representation data on the vital sign spectrum data. In this way, the accuracy of measuring the vital sign signal of the object under test can be improved.

[0028] 1a is a schematic diagram illustrating a scenario example of a vital sign signal measurement system based on human factors intelligence according to an embodiment of this specification. The vital sign signal measurement system 100 may include a first measurement device 110 , a video signal acquisition device 120 , and a second measurement device 130 .

[0029] Specifically, the first measuring device 110 can transmit and receive frequency modulated continuous wave radar signals, so that a digital mixing signal can be determined based on the transmission and reception of the frequency modulated continuous wave radar signals, and further the vital sign spectrum data of the measured object 140 can be obtained based on the digital mixing signal.

[0030] Specifically, the video signal acquisition device 120 may include a depth camera or a multi-eye matching camera. For example, the multi-eye matching camera may be a binocular matching camera. The video signal acquisition device 120 may capture the environment in which the subject 140 is located using the depth camera or the multi-eye matching camera to obtain a video signal, so that the biometric data of the subject and the environmental data of the environment in which the subject is located can be obtained based on the video signal, thereby enabling the individual feature representation data of the subject to be obtained based on the biometric data, and enabling the environmental feature representation data of the environment in which the subject is located to be obtained based on the environmental data.

[0031] Specifically, the second measuring device 130 can be a measuring device different from the first measuring device 110. For example, the second measuring device 130 can be a contact measuring device, or in other words, a contact sensor. As an example, the second measuring device 130 can be any one of a mechanical measuring device and a biosignal measuring device. The mechanical measuring device can be a respiratory belt sensor. For example, the respiratory belt sensor can be fixed to the thorax using an elastic belt or a chest belt, and measure the respiratory rate by detecting the expansion and contraction changes of the elastic belt or chest belt. The mechanical measuring device can also be a respiratory quality sensor. For example, the respiratory quality sensor can use a pressure sensor or a mass sensor to measure the flow rate and volume of respiratory gas, and calculate the respiratory rate by analyzing the characteristics of the respiratory gas. The biosignal measuring device can be an electrocardiogram (ECG) monitoring device or a blood oxygen saturation (SpO2) monitoring device. ECG monitoring devices can collect ECG signals using electrodes attached to the skin to calculate heart rate; SpO2 monitoring devices can measure the blood oxygen saturation in the blood using a photoelectric sensor to calculate heart rate and respiratory rate. In this way, the true value for training the target classification model can be obtained through the second measuring device.

[0032] As an example, the first measurement device 110 can be connected to the video signal acquisition device 120 and the second measurement device 130. The first measurement device 110 can be deployed with a target classification model to determine the vital sign signal value. The target classification model deployed on the first measurement device 110 is trained based on vital sign spectrum data, individual feature representation data, environmental feature representation data, and true values ​​at the same historical moment.

[0033] As another example, please refer to Figure 1b, which is a schematic diagram of a scenario example of a vital sign signal measurement system provided in an embodiment of this specification. The vital sign signal measurement system 100 may also include a data processing device 150, which can be connected to the first measurement device 110, the video signal acquisition device 120, and the second measurement device 130. The data processing device 150 can be deployed with a target classification model to determine the vital sign signal value through the target classification model. The target classification model deployed on the data processing device 150 is obtained by model training based on the vital sign spectrum data, individual feature representation data, environmental feature representation data and true value at the same historical moment. Exemplarily, the data processing device 150 can be a server or a terminal.

[0034] The embodiment of this specification provides a method for measuring vital signs signals based on human intelligence. Please refer to Figure 2. Figure 2 is a flow chart of a method for measuring vital signs signals based on human intelligence provided by this embodiment. This embodiment provides method operation steps such as the flow chart, but more or fewer operation steps may be included based on conventional or non-creative labor. The sequence of steps listed in the embodiment is only one execution method among many step execution sequences and does not represent the only execution sequence. When the system or server product in practice is executed, it can be executed sequentially or in parallel according to the method shown in the embodiment (for example, a parallel processor or a multi-threaded processing environment). The vital signs signal measurement method can be applied to a first measuring device or a data processing device in a vital signs signal measurement system. Specifically, as shown in Figure 2, the vital signs signal measurement method may include the following steps.

[0035] Step S210: Acquire individual characteristic representation data of the measured object, environmental characteristic representation data of the measured object's environment, and vital sign spectrum data of the measured object; wherein the individual characteristic representation data and environmental characteristic representation data have different effects on the vital sign spectrum data.

[0036] In some cases, it is possible to determine the presence of vital sign spectrum data corresponding to the subject being measured. Furthermore, it is also possible to determine the presence of individual characteristic representation data for the subject being measured, as well as the presence of environmental characteristic representation data for the subject's environment, so that the vital sign signal value of the subject being measured can be predicted based on the individual characteristic representation data, the environmental characteristic representation data, and the vital sign spectrum data.

[0037] The vital sign spectrum data may be determined based on a millimeter wave radar. For example, the first measuring device may be equipped with a millimeter wave radar, so that the vital sign spectrum data may be determined based on the millimeter wave radar of the first measuring device.

[0038] Specifically, frequency modulated continuous wave radar signals transmitted and received by the first measuring device via a millimeter wave radar can be mixed to generate a digital mixed signal, and the vital sign spectrum data of the measured object can be determined based on the digital mixed signal. This digital mixed signal is equivalent to the radio frequency reflection signal in the subsequent second embodiment, both of which are obtained by the millimeter wave radar. The vital sign spectrum data of the measured object is equivalent to the first characteristic data in the second embodiment. For example, the first measuring device can generate a frequency modulated continuous wave radar signal via a millimeter wave radar transmitter. The frequency of the frequency modulated continuous wave radar signal can vary over time, for example, increasing or decreasing over time within a specified time period. The transmitted frequency modulated continuous wave radar signal can be reflected by the measured object and returned to the first measuring device. The first measuring device can receive the reflected frequency modulated continuous wave radar signal via a millimeter wave radar receiver and mix the transmitted frequency modulated continuous wave radar signal with the received frequency modulated continuous wave radar signal to generate a mixed signal. The mixed signal, also known as an intermediate frequency signal or a beat signal, can facilitate the subsequent determination of the vital sign spectrum data of the measured object.

[0039] Exemplarily, the frequency modulated continuous wave radar signal may be a chirp signal.

[0040] Exemplarily, analog-to-digital conversion may be performed on the mixing signal to obtain a digital mixing signal, so that subsequent spectrum estimation-related operations can be performed based on the digital mixing signal.

[0041] For example, after determining the digital mixing signal, spectrum estimation-related operations can be performed based on the digital mixing signal to determine vital sign spectrum data. The spectrum estimation-related operations can include at least one of distance dimension determination, processing unit determination, phase extraction, phase unwrapping, phase differentiation, bandpass filtering, and spectrum estimation.

[0042] Among them, the distance dimension determination (Range FFT) is the distance information determined based on the Fast Fourier Transfor (FFT) of the digital mixing signal, or in other words, the Range curve determined based on the FFT of the digital mixing signal, that is, the spectrum of the distance dimension determined based on the FFT of the digital mixing signal, so that multiple distance units can be determined. The processing unit determination (Range bin tracking) is to determine the distance unit corresponding to the object under test from multiple distance units. Phase extraction (Extract Phase), that is, to extract the phase of the distance unit corresponding to the object under test. Phase unwrapping (Phase Unwrapping) is used to perform phase unwrapping to obtain a phase signal. Phase difference (Phase Difference) is used to enhance the unwrapped phase signal and reduce the existing phase drift. Bandpass filtering is used to filter the corresponding phases in the phase signal using a bandpass filter to distinguish different vital sign signals, or to filter the frequencies formed by the corresponding phase changes in the phase signal to distinguish them, thereby facilitating the subsequent determination of different vital sign signals. Spectral estimation involves performing an FFT transform on the obtained phase signal to obtain the vital sign spectrum data of the subject being measured, so that the vital sign signal of the subject can be determined based on the vital sign spectrum data of the subject being measured.

[0043] The individual characteristic representation data may be data that can describe the object being measured, or in other words, may be data used to identify the object being measured.

[0044] The environmental feature representation data may be data that can describe the environment in which the object under test is located, or in other words, may be data used to identify the environment in which the object under test is located.

[0045] The differential impact of individual feature representation data on vital sign spectrum data can refer to the impact of individual differences of the measured subject on the vital sign spectrum data. For example, individual differences of the measured subject can refer to the measured subject's identity, age, gender, skin type, etc. For example, the difference between the frequency-hopping continuous wave radar signals reflected by oily skin and dry skin may be relatively large, and the resulting vital sign spectrum data will also be relatively different. Different skin types have different differential impacts on vital sign spectrum data, which means that the individual feature representation data of the measured subject has a differential impact on the vital sign spectrum data.

[0046] The differential impact of environmental feature representation data on vital sign spectrum data can refer to the impact of environmental differences in the environment of the measured object on the vital sign spectrum data. For example, environmental differences in the environment of the measured object can refer to differences in lighting, brightness, temperature, and humidity in the environment of the measured object. For example, the difference between the frequency-hopping continuous wave radar signal reflected by the measured object in a high-humidity environment and the frequency-hopping continuous wave radar signal reflected in a low-humidity environment is relatively large. In a high-humidity environment, the performance of the frequency-hopping continuous wave radar signal may be attenuated, which may in turn affect the determination of the measured object's vital sign signal based on the vital sign spectrum data.

[0047] Step S220: performing signal value prediction based on the individual feature representation data, the environmental feature representation data, and the vital sign spectrum data to obtain the vital sign signal value with the difference effect removed.

[0048] In the above embodiment, by obtaining the individual characteristic representation data of the measured object, the environmental characteristic representation data of the environment in which the measured object is located, and the vital sign spectrum data of the measured object, and performing signal value prediction based on the individual characteristic representation data, the environmental characteristic representation data, and the vital sign spectrum data, the differential influence of the individual characteristic representation data and the environmental characteristic representation data on the vital sign spectrum data is removed, and the corresponding vital sign signal value is obtained. In this way, the accuracy of the vital sign signal measurement of the measured object can be improved.

[0049] In some embodiments, referring to FIG. 3 , obtaining individual characteristic representation data of the measured object may include the following steps.

[0050] Step S310: Acquire biometric data of the object under test based on the video signal; wherein the video signal is obtained by photographing the environment in which the object under test is located.

[0051] In some cases, the environment in which the object is located can be photographed and video signals can be collected using video signal acquisition equipment.

[0052] Specifically, the biometric data of the subject can be obtained based on the video signal captured by the video signal acquisition device. For example, the biometric data of the subject can be any one of the facial data, iris data, retinal data, and eye pattern data of the subject.

[0053] As an example, when acquiring facial data based on a video signal captured by a video signal acquisition device, the video signal can be preprocessed. For example, preprocessing operations such as noise removal and contrast enhancement can be performed on the video signal to enhance the quality of the video signal. After the video signal is preprocessed, facial region recognition can be performed on the preprocessed video signal. When the facial region is detected, facial features can be further extracted. For example, facial features or facial data such as the facial contour, eyes, mouth, and nose, and geometric features of the face, texture features, etc. can be extracted to facilitate subsequent individual recognition of the subject based on this facial data.

[0054] As an example, the video signal acquisition device can be a designated device capable of performing any one of iris recognition, retina recognition, and eye pattern recognition, so that the iris data, retina data, or eye pattern data of the subject can be obtained based on the video signal captured by the designated video signal acquisition device.

[0055] Step S320: Based on the biometric data in the video signal, perform individual identification on the subject to obtain individual feature representation data.

[0056] Specifically, biometric comparison data may be pre-stored, and individual identification results may be obtained based on the comparison of the biometric data with the pre-stored biometric comparison data, and individual feature representation data may be determined based on the individual identification results.

[0057] The individual identification result may include at least one of the identity identifier, gender characteristics, age characteristics, and skin type characteristics of the subject. For example, the identity identifier may refer to identification information used to uniquely identify the subject.

[0058] In some cases, the individual recognition result can be a high-dimensional sparse feature vector. In order to facilitate the subsequent prediction of the vital sign signal value of the subject, the individual recognition result can be embedded (Embedding) into a low-dimensional dense feature vector of a specified dimension to obtain individual feature representation data. In this way, in the process of predicting the vital sign signal value based on the individual feature representation data, the amount of calculation and storage space can be reduced, and the prediction efficiency can be improved.

[0059] High-dimensional sparse feature vectors can be used to process some unstructured data. For example, the individual identification result of a subject can be represented as a high-dimensional sparse feature vector. Low-dimensional dense feature vectors, also known as individual feature representation data, can refer to the individual feature representation data obtained after embedding the individual identification result. They can retain the important features and information of the individual identification result as a high-dimensional sparse feature vector, while reducing the dimensionality and complexity of the individual identification result.

[0060] In the above embodiment, the biometric data of the measured object is obtained based on the video signal, and the measured object is individually identified based on the biometric data to obtain individual feature representation data. In this way, the individual feature representation data used to identify the measured object can be quickly obtained, which facilitates the subsequent removal of the difference influence of the individual feature representation data on the vital sign spectrum data, thereby improving the accuracy of the vital sign signal value.

[0061] In some embodiments, referring to FIG. 4 , the environmental characteristic representation data of the environment in which the measured object is located can be obtained through the following steps.

[0062] Step S410: Acquire environmental data of the environment in which the object under test is located based on the video signal; wherein the video signal is obtained by photographing the environment in which the object under test is located.

[0063] Specifically, environmental data of the environment in which the object under test is located can be obtained based on the video signal captured by the video signal acquisition device. Exemplarily, the video signal can refer to a video image, and the video image can be preprocessed, for example, by performing denoising, contrast enhancement, or other preprocessing to enhance image quality. The environment in which the object under test is located in the preprocessed video image can then be processed and analyzed to extract environmental data of the environment in which the object under test is located in the video image. Exemplarily, the environmental data can include at least one of temperature, humidity, weather, wind speed, and light intensity.

[0064] Exemplarily, the video signal may carry at least one of the following information: shooting time, shooting location, longitude and latitude, and altitude. Extracting environmental data of the environment in which the object is located may be performed by estimating and determining at least one of the shooting time, shooting location, longitude and latitude, and altitude based on the video image. The shooting time may include any of the following seasons: spring, summer, autumn, and winter; any of the following: daytime and nighttime; and any hour or minute of the 24 hours of a day. The shooting location may be indoors, outdoors, or in a semi-open space. Semi-open spaces may include balconies or terraces, corridors or aisles, rooftops or rooftops, greenhouses or greenhouses, and the like.

[0065] Step S420: extracting features based on the environmental data in the video signal to obtain environmental feature representation data; wherein the environmental feature representation data includes at least one of humidity features, temperature features, weather features, wind speed features, and light intensity.

[0066] In some cases, environmental data can be a high-dimensional sparse feature vector. In order to facilitate the subsequent prediction of the vital sign signal value of the measured object, the environmental data can be embedded (Embedding) into a low-dimensional dense feature vector of a specified dimension to obtain environmental feature representation data. In this way, in the process of predicting the vital sign signal value based on the environmental feature representation data, the amount of calculation and storage space can be reduced, and the prediction efficiency can be improved.

[0067] Environmental data can be represented as a high-dimensional sparse feature vector. Environmental feature representation data can refer to the environmental data obtained after embedding. It can retain the important features and information of the environmental data as a high-dimensional sparse feature vector, while reducing the dimensionality and complexity of the environmental data.

[0068] In the above embodiment, environmental data of the environment in which the object under test is located is obtained based on the video signal, and feature extraction is performed based on the environmental data to obtain environmental feature representation data. In this way, environmental feature representation data for the environment in which the object under test is located can be quickly obtained, which facilitates the subsequent removal of the differential impact of the environmental feature representation data on the vital sign spectrum data, thereby improving the accuracy of the vital sign signal value.

[0069] In some embodiments, referring to FIG. 5 , the vital sign spectrum data of the subject can be acquired through the following steps.

[0070] Step S510: Acquire a video signal and a digital mixed signal of a first measuring device; wherein the video signal is obtained by photographing the environment in which the measured object is located; and the digital mixed signal is determined based on the transmission and reception of a frequency modulated continuous wave radar signal by the first measuring device.

[0071] In some cases, the vital sign spectrum data of the subject can be determined based on the digital mixing signal and the video signal. Determining the vital sign spectrum data based on the digital mixing signal combined with the video signal can improve the accuracy of the vital sign spectrum data.

[0072] Specifically, the environment in which the object under test is located can be photographed by a video signal acquisition device to obtain a video signal, and at the same time, a digital mixed signal determined by the first measuring device transmitting and receiving a frequency-modulated continuous wave radar signal through a millimeter-wave radar can be obtained, so as to determine the vital sign spectrum data of the object under test through the video signal and the digital mixed signal.

[0073] For example, the video signal may have a shooting time, and the digital mixing signal may have corresponding time information. The shooting time of the video signal and the time information corresponding to the digital mixing signal are consistent, or in other words, the video signal and the digital mixing signal are aligned in time.

[0074] Step S520: determining an initial distance unit of the measured object relative to the first measuring device based on the digital mixing signal.

[0075] Specifically, the initial range bin can be determined based on a range dimension determination (Range FFT) operation and a processing unit determination (Range bin tracking) operation. For example, a Fast Fourier Transform (FFT) can be performed on the digital mixed signal to determine a spectrum of the range dimension, thereby obtaining multiple range bins. A search can then be performed within the multiple range bins to determine the range bin corresponding to the object being measured, which serves as the initial range bin.

[0076] Step S530: Correcting the initial distance unit based on the video signal to obtain the target distance unit.

[0077] Specifically, after determining the initial distance unit corresponding to the object under test, the initial distance unit can be corrected based on the video signal to obtain the target distance unit corresponding to the object under test, so that phase extraction can be performed based on the target distance unit to determine the phase signal, and then the vital sign spectrum data of the object under test can be determined based on the phase signal.

[0078] Step S540: Determine vital sign spectrum data based on the target distance unit.

[0079] For example, after the initial distance unit is corrected to the target distance unit according to the video signal, a phase extraction operation may be performed on the target distance unit to obtain the phase corresponding to the measured object.

[0080] For example, the digital mixing signal can be determined by transmitting and receiving a chirp signal through a millimeter-wave radar. As an example, in a frame, or in a frame period, multiple chirp signals can be continuously transmitted by the millimeter-wave radar. The frame period can refer to a complete transmission and reception period of a chirp signal. For example, when a frame period is 50 milliseconds and the duration of a chirp signal is 50 microseconds, the number of chirp signals transmitted in a frame is 1000.

[0081] For example, a mixed signal obtained by mixing the transmitted chirp signal and the received reflected chirp signal can be subjected to analog-to-digital conversion. During the analog-to-digital conversion, the mixed signal can be sampled based on a specified number of samples to obtain a digital mixed signal. The specified number of samples is the number of samples within a chirp signal. The digital mixed signal can have a corresponding bandwidth.

[0082] As an example, for a digital mixed signal, an initial range unit is determined based on a range dimension determination (Range FFT) operation and a processing unit determination (Range bin tracking) operation. The initial range unit is then corrected based on a video signal to obtain a target range unit. A phase extraction (Extract Phase) operation is then performed on the target range unit to determine the phase of the target range unit corresponding to the measured object. This process can be repeated. In this way, the change of the target range unit of the measured object with the number of frames can be determined, that is, the change of the phase of the measured object with time can be determined. Subsequently, a phase unwrapping operation can be performed to obtain a phase signal of the measured object. A phase difference operation can be used to enhance the unwrapped phase signal and reduce the existing phase drift. A bandpass filtering operation can be used to filter the corresponding phase in the phase signal for differentiation, or in other words, the frequency formed by the corresponding phase change in the phase signal is filtered for differentiation, to facilitate the subsequent determination of different vital sign signals. The obtained phase signal can be FFT transformed based on a spectrum estimation operation to obtain vital sign spectrum data of the measured object.

[0083] Exemplarily, vital sign spectrum data can be obtained by embedding the results of a spectrum estimation operation. In other words, the vital sign spectrum data is a low-dimensional dense feature vector with a specified dimension. Exemplarily, the vital sign spectrum data can include at least information about the number of transmitted waves, information about the number of transmitted wave samples, and bandwidth information. The number of transmitted waves can be the number of chirp signals transmitted in a frame, the number of transmitted wave samples can be a specified number of samples within a chirp signal, and the bandwidth can refer to the bandwidth after bandpass filtering.

[0084] In the above embodiment, a video signal is obtained by photographing the environment in which the measured object is located. At the same time, a digital mixing signal is determined based on the transmission and reception of a frequency-modulated continuous wave radar signal by the first measuring device. Then, an initial distance unit of the measured object relative to the first measuring device is determined based on the digital mixing signal. The initial distance unit is corrected based on the video signal to obtain a target distance unit. The vital sign spectrum data is determined based on the target distance unit. In this way, the initial distance unit in the process of determining the vital sign spectrum data of the measured object can be corrected based on the video signal, thereby improving the accuracy of the measured object in measuring the vital sign signals.

[0085] In some embodiments, referring to FIG. 6 , correcting the initial distance unit based on the video signal to obtain the target distance unit may include the following steps.

[0086] Step S610: Detect the distance between the first measuring device and the measured object based on the video signal to obtain a video detection distance.

[0087] Step S620: Correct the initial distance unit based on the video detection distance to obtain the target distance unit.

[0088] Specifically, the judgment can be made based on the comparison between the video detection distance and the initial distance unit. If the difference between the video detection distance and the initial distance unit is greater than a specified difference threshold, the initial distance unit can be corrected based on the video detection distance. As an example, correcting the initial distance unit based on the video detection distance can mean that when the video detection distance is greater than the specified difference threshold, a plurality of distance units obtained based on the spectrum of the distance dimension are searched again in the plurality of distance units to determine the distance unit corresponding to the object being measured as the target distance unit. As another example, correcting the initial distance unit based on the video detection distance can mean that when the video detection distance is greater than the specified difference threshold, the initial distance unit is adjusted based on the video detection distance to obtain the target distance unit.

[0089] In the above embodiment, it is possible to determine the video detection distance between the first measuring device and the object under measurement based on the video signal, and to correct the initial distance unit in the process of determining the vital sign spectrum data of the object under measurement based on the video detection distance, thereby improving the accuracy of the object under measurement when measuring the vital sign signal.

[0090] In some embodiments, referring to FIG. 7 a , signal value prediction based on individual feature representation data, environmental feature representation data, and vital sign spectrum data to obtain vital sign signal values ​​with difference effects removed may include the following steps.

[0091] Step S710a: performing feature combination on the individual feature representation data, the environmental feature representation data and the vital sign spectrum data to obtain a feature combination result.

[0092] Specifically, the individual feature representation data, the environmental feature representation data, and the vital sign spectrum data can be feature-joined so that the individual feature representation data, the environmental feature representation data, and the vital sign spectrum data can be converted into a feature-joined result suitable for model training. For example, if the individual feature representation data, the environmental feature representation data, and the vital sign spectrum data are all low-dimensional dense feature vectors of a specified dimension, the individual feature representation data, the environmental feature representation data, and the vital sign spectrum data are vector-concatenated, and the resulting concatenated vector is used as the feature-joined result.

[0093] Step S720a: performing signal value prediction based on the feature combination result to obtain a vital sign signal value; wherein the vital sign signal value includes at least one of heart rate and respiratory rate.

[0094] For example, the signal value prediction can be performed based on the feature combination result to obtain the heart rate. The signal value prediction can be performed based on the feature combination result to obtain the respiratory rate. The signal value prediction can also be performed based on the feature combination result to obtain the heart rate and respiratory rate.

[0095] As an example, the phases in the phase signals corresponding to the heart rate and respiratory rate are usually different, or in other words, the frequencies formed by the phase changes in the phase signals corresponding to the heart rate and respiratory rate are usually different. In other words, the bandwidth information in the vital signs spectrum data corresponding to the heart rate and respiratory rate is different. In this way, the heart rate or respiratory rate can be obtained by predicting the signal value based on the vital signs spectrum data obtained after bandpass filtering.

[0096] For example, the signal value may be predicted based on different confidence indicators.

[0097] In the above embodiment, by performing feature combination on individual feature representation data, environmental feature representation data and vital sign spectrum data, and performing signal value prediction based on the feature combination result, a vital sign signal value including at least one of the heart rate and respiratory rate is obtained. In this way, the difference influence of the individual feature representation data and the environmental feature representation data on the vital sign spectrum data can be removed, and the accuracy of the vital sign signal measurement of the measured object can be improved.

[0098] In some embodiments, feature combination is performed on individual feature representation data, environmental feature representation data, and vital sign spectrum data to obtain a feature combination result, which may include: splicing the individual feature representation data, environmental feature representation data, and vital sign spectrum data to obtain a feature combination result.

[0099] Specifically, individual feature representation data, environmental feature representation data and vital sign spectrum data can all be specified dimensions. When performing feature combination on the individual feature representation data, environmental feature representation data and vital sign spectrum data, the individual feature representation data, environmental feature representation data and vital sign spectrum data can be concatenated based on the specified dimensions to obtain feature combination results, so that the vital sign signal value can be predicted based on the feature combination results.

[0100] In the above embodiment, by splicing individual feature representation data and environmental feature representation data into vital sign spectrum data, when measuring the vital sign signals of the subject, the differential effects of the individual feature representation data and environmental feature representation data on the vital sign spectrum data are taken into account and removed, thereby improving the accuracy of the vital sign signal measurement.

[0101] In some embodiments, referring to FIG. 7 b , obtaining individual characteristic representation data of the measured object and environmental characteristic representation data of the environment in which the measured object is located may include the following steps.

[0102] S710b: Acquire a video signal obtained by shooting the environment in which the measured object is located.

[0103] S720b: Determine environmental data and individual recognition results of the measured object based on the video signal.

[0104] S730b: Mapping the environmental data to obtain environmental feature representation data of a specified dimension.

[0105] S740b: Mapping the individual recognition results to obtain individual feature representation data of a specified dimension.

[0106] Specifically, a video signal captured by a video signal acquisition device targeting the environment of the subject under test can be obtained, and biometric data of the subject under test can be obtained based on the video signal. Exemplarily, the acquired biometric data can be facial data of the subject under test. For example, facial region recognition can be performed on the subject under test based on the video signal. When the facial region is detected, facial features such as the facial contour, eyes, mouth, and nose are further extracted to obtain facial data. Individual identification can also be performed based on the facial data to obtain an individual identification result for the subject under test, including at least one of an identity identifier, gender feature, age feature, and skin type feature. The individual identification result can then be mapped to obtain individual feature representation data of a specified dimension.

[0107] Specifically, a video signal captured by a video signal acquisition device for the environment in which the object under test is located can be obtained, and environmental data of the environment in which the object under test is located can be obtained based on the video signal. Exemplarily, the environmental data obtained may include at least one of temperature, humidity, weather, wind speed, and light intensity. Exemplarily, the video signal may have at least one of the information such as shooting time, shooting location, longitude and latitude, and altitude. Exemplarily, at least one of the information such as shooting time, shooting location, longitude and latitude, and altitude is used to determine the environmental data of the environment in which the object under test is located. Then, the environmental data can be mapped to obtain environmental feature representation data of a specified dimension.

[0108] The designated dimensions of the environmental feature representation data and the individual feature representation data are the same.

[0109] For example, the vital sign spectrum data may be of a specified dimension. When the environment data and the individual recognition result are mapped to environment feature representation data and individual feature representation data of the specified dimension, respectively, the specified dimension may be determined according to the specified dimension of the vital sign spectrum data.

[0110] In some embodiments, the vital sign signal value may be output by a target classification model. Referring to FIG8 a , the target classification model may be obtained by the following training method.

[0111] S810a: Construct a training sample set; wherein the training sample set includes multiple training samples, and the training samples include historical vital sign spectrum data, historical individual feature representation data, and historical environmental feature representation data; the labels of the training samples adopt the true values ​​of historical vital sign signals.

[0112] S820a: Perform model training on the initial classification model according to the training samples and labels to obtain a target classification model; wherein the initial classification model is built based on any one of the VGG, EfficientNet, and ResNet model structures.

[0113] Specifically, after constructing the training sample set, the historical individual feature representation data, historical environmental feature representation data and historical vital sign spectrum data in the training sample set can be input into the initial classification model for modeling. The historical individual feature representation data, historical environmental feature representation data and historical vital sign spectrum data are feature-joined by the initial classification model to obtain the historical feature joint result. Prediction is performed based on the historical feature joint result to obtain the predicted vital sign signal. Furthermore, the initial classification model corresponds to a loss function, and the input of the initial classification model corresponds to a label. The label and the predicted vital sign signal are brought into the loss function to determine the model loss value, and the parameters of the initial classification model are updated based on the determined model loss value, and so on, until the model training stopping condition is met to obtain the target classification model. Among them, the model training stopping condition can be that the model loss value tends to converge, or that the training rounds reach a preset number of rounds.

[0114] In the above implementation, training samples are constructed using historical vital sign spectrum data, historical individual feature representation data, and historical environmental feature representation data. In the process of training the initial classification model using the training samples and the true values ​​of historical vital sign signals, the initial classification model gradually learns the ability to remove the differential effects of individual feature representation data and environmental feature representation data on vital sign spectrum data, thereby obtaining a target classification model, thereby obtaining accurate vital sign signal values.

[0115] In some implementations, referring to FIG. 8 b , training samples may be constructed in the following manner.

[0116] S810b: Determine historical vital sign spectrum data based on historical digital mixed signals collected by the first measuring device at historical moments for the measured object or the non-measured object.

[0117] S820b: Acquire historical video signals collected at historical moments.

[0118] S830b: Determine historical individual feature representation data and historical environment feature representation data based on the historical video signal.

[0119] Specifically, because model training requires training samples, historical digital mixing signals determined by the first measuring device transmitting and receiving frequency-modulated continuous-wave radar signals via millimeter-wave radar are obtained at multiple historical moments prior to model training. Based on these historical digital mixing signals, the historical initial distance unit of the measured object or non-measured object relative to the first measuring device is determined. The distance between the first measuring device and the measured object or non-measured object is then detected based on historical video signals to obtain a historical video-detected distance. The historical initial distance unit is corrected based on the historical video-detected distance to obtain a historical target distance unit, and historical vital sign spectrum data is determined based on the historical target distance unit.

[0120] Specifically, the biometric data of the measured object or the non-measured object can be obtained based on the historical video signal, and the measured object or the non-measured object can be individually identified based on the biometric data in the historical video signal to obtain historical individual feature representation data.

[0121] Specifically, environmental data of the environment in which the measured object or the non-measured object is located can be obtained based on the historical video signal, and features can be extracted based on the environmental data in the historical video signal to obtain historical environmental feature representation data.

[0122] Specifically, the true value of the historical vital sign signal can be determined based on the collection of the measured object or the non-measured object by the second measuring device, so as to use the true value of the historical vital sign signal as a label for model training.

[0123] At this point, training samples input to the initial classification model can be constructed based on the historical vital sign spectrum data, the historical individual feature representation data, and the historical environmental feature representation data.

[0124] It should be noted that the historical vital sign spectrum data, historical individual feature representation data, historical environmental feature representation data, and historical vital sign signal true values ​​are time-aligned. In other words, the historical vital sign spectrum data, historical individual feature representation data, historical environmental feature representation data, and historical vital sign signal true values ​​are all determined at the same historical moment in time among multiple historical moments.

[0125] In the above implementation, historical vital sign spectrum data, historical individual feature representation data, and historical environmental feature representation data are used to construct training samples, providing a data basis for training the initial classification model.

[0126] The embodiments of this specification provide a method for measuring vital sign signals based on human intelligence, which can be applied to a first measuring device or a data processing device in a vital sign signal measurement system. The method can include the following steps.

[0127] Step S901: Acquire historical vital sign spectrum data; wherein the historical vital sign spectrum data is collected and determined by a first measuring device at a historical moment for a measured object or a non-measured object.

[0128] Step S903: Acquire historical video signals collected at historical moments.

[0129] Step S905: Acquire historical vital sign signal true values; wherein, the historical vital sign signal true values ​​are collected and determined by the second measuring device at historical moments for the measured object or the non-measured object.

[0130] Step S907: Determine historical individual feature representation data and historical environment feature representation data based on the historical video signal.

[0131] Step S909: constructing training samples based on historical individual feature representation data, historical environmental feature representation data, and historical vital sign spectrum data.

[0132] It should be noted that a training sample set is composed of multiple training samples. The training samples include historical vital sign spectrum data, historical individual feature representation data, and historical environmental feature representation data; the labels of the training samples use the true values ​​of historical vital sign signals.

[0133] Step S911: Perform model training on the initial classification model according to the training samples and labels to obtain a target classification model.

[0134] Step S913: Acquire individual characteristic representation data of the measured object, environmental characteristic representation data of the measured object's environment, and vital sign spectrum data of the measured object; wherein the individual characteristic representation data and environmental characteristic representation data have different effects on the vital sign spectrum data.

[0135] Specifically, the individual characteristic representation data of the object under test is obtained in the following manner, including: obtaining the biometric data of the object under test based on a video signal; wherein the video signal is obtained by photographing the environment in which the object under test is located; based on the biometric data in the video signal, the object under test is individually identified to obtain the individual characteristic representation data.

[0136] Specifically, the biometric data of the subject is any one of facial data, iris data, retinal data, and eye pattern data;

[0137] Specifically, the individual feature representation data includes at least one of the gender feature, age feature, and skin type feature of the subject.

[0138] Specifically, environmental feature representation data of the environment in which the object under test is located is obtained in the following manner, including: obtaining environmental data of the environment in which the object under test is located based on a video signal; wherein the video signal is obtained by photographing the environment in which the object under test is located; performing feature extraction based on the environmental data in the video signal to obtain environmental feature representation data; wherein the environmental feature representation data includes at least one of humidity features, temperature features, weather features, and wind speed features.

[0139] Specifically, the vital sign spectrum data of the measured object is obtained in the following manner, including: obtaining a video signal and a digital mixing signal of a first measuring device; wherein the video signal is obtained by photographing the environment in which the measured object is located; the digital mixing signal is determined based on the transmission and reception of a frequency-modulated continuous wave radar signal by the first measuring device; an initial distance unit of the measured object relative to the first measuring device is determined based on the digital mixing signal; the initial distance unit is corrected based on the video signal to obtain a target distance unit; and the vital sign spectrum data is determined based on the target distance unit.

[0140] Specifically, the initial distance unit is corrected based on the video signal to obtain the target distance unit, including: detecting the distance between the first measuring device and the measured object based on the video signal to obtain the video detection distance; and correcting the initial distance unit based on the video detection distance to obtain the target distance unit.

[0141] Step S915: performing signal value prediction based on the individual feature representation data, the environmental feature representation data, and the vital sign spectrum data to obtain the vital sign signal value with the difference effect removed.

[0142] Specifically, the vital sign signal value is output by the target classification model.

[0143] Specifically, feature combination can be performed on individual feature representation data, environmental feature representation data and vital sign spectrum data to obtain feature combination results; signal value prediction can be performed based on the feature combination results to obtain vital sign signal values; wherein the vital sign signal values ​​include at least one of heart rate and respiratory rate.

[0144] It can be understood that in the various implementations of this specification, the size of the serial number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the implementation methods of this specification.

[0145] The embodiments of this specification provide a vital sign signal measurement device based on human factors intelligence. The vital sign signal measurement device can be applied to a first measurement device or a data processing device in a vital sign signal measurement system. The measurement device can include a feature data acquisition module and a vital sign determination module.

[0146] A feature data acquisition module is used to obtain individual feature representation data of the measured object, environmental feature representation data of the measured object's environment, and vital sign spectrum data of the measured object; wherein the individual feature representation data and the environmental feature representation data have different effects on the vital sign spectrum data;

[0147] The vital sign determination module is used to predict the signal value based on the individual feature representation data, the environmental feature representation data, and the vital sign spectrum data to obtain the vital sign signal value without the influence of the difference.

[0148] In some embodiments, the feature data acquisition module is further configured to: acquire biometric data of the subject under test based on a video signal; wherein the video signal is obtained by filming the subject's environment; and perform individual identification of the subject under test based on the biometric data in the video signal to obtain individual feature representation data. The biometric data of the subject under test may be any one of facial data, iris data, retinal data, and eye pattern data; and the individual feature representation data may include at least one of the subject's gender, age, and skin type.

[0149] In some embodiments, the feature data acquisition module is further used to: acquire environmental data of the environment in which the object under test is located based on the video signal; wherein the video signal is obtained by photographing the environment in which the object under test is located; perform feature extraction based on the environmental data in the video signal to obtain environmental feature representation data; wherein the environmental feature representation data includes at least one of humidity features, temperature features, weather features, and wind speed features.

[0150] In some embodiments, the feature data acquisition module is further used to: acquire a video signal and a digital mixing signal of a first measuring device; wherein the video signal is obtained by photographing the environment in which the object to be measured is located; the digital mixing signal is determined based on the transmission and reception of frequency-modulated continuous wave radar signals by the first measuring device through a millimeter-wave radar; determine an initial distance unit of the object to be measured relative to the first measuring device based on the digital mixing signal; correct the initial distance unit based on the video signal to obtain a target distance unit; and determine vital sign spectrum data based on the target distance unit.

[0151] In some embodiments, the feature data acquisition module is further used to: detect the distance between the first measuring device and the measured object based on the video signal to obtain the video detection distance; and correct the initial distance unit based on the video detection distance to obtain the target distance unit.

[0152] In some embodiments, the vital sign determination module is further used to: perform feature combination on individual feature representation data, environmental feature representation data and vital sign spectrum data to obtain a feature combination result; perform signal value prediction based on the feature combination result to obtain a vital sign signal value; wherein the vital sign signal value includes at least one of heart rate and respiratory rate.

[0153] In some embodiments, the vital sign determination module is further configured to: perform splicing processing on the individual feature representation data, the environmental feature representation data, and the vital sign spectrum data to obtain a feature combination result.

[0154] In some embodiments, the feature data acquisition module is also used to: obtain a video signal obtained by shooting the environment in which the object to be measured is located; determine the environmental data and the individual identification result of the object to be measured based on the video signal; map the environmental data to obtain environmental feature representation data of a specified dimension; map the individual identification result to obtain individual feature representation data of a specified dimension.

[0155] In some embodiments, the vital sign signal value is output through a target classification model. The measurement device may further include a model training module. The model training module is used to: construct a training sample set; wherein the training sample set includes multiple training samples, and the training samples include historical vital sign spectrum data, historical individual feature representation data, and historical environmental feature representation data; the labels of the training samples are based on the true values ​​of the historical vital sign signals; the initial classification model is trained based on the training samples and labels to obtain a target classification model; wherein the initial classification model is built based on any one of the VGG, EfficientNet, and ResNet model structures.

[0156] In some embodiments, the model training module is further used to construct training samples in the following manner: determine historical vital sign spectrum data based on historical digital mixed signals collected by the first measuring device at historical moments for the measured object or non-measured object; obtain historical video signals collected at historical moments; and determine historical individual feature representation data and historical environment feature representation data based on the historical video signals.

[0157] The specific functions and effects achieved by the measuring device can be explained in conjunction with other embodiments of this specification and will not be repeated here. Each module in the measuring device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in the computer device in the form of software so that the processor can call and execute the operations corresponding to each of the above modules.

[0158] The embodiments of this specification also provide a computer device, including a memory and a processor. The memory stores a computer program, and the processor implements the vital sign signal measurement method in the above embodiment when executing the computer program.

[0159] In this embodiment, please refer to Figure 9. The computer device may be a terminal, and its internal structure diagram may be as shown in Figure 9. The computer device includes a processor, a memory, and a communication interface connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, an operator network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, it implements the vital sign signal measurement method in any of the above examples.

[0160] An embodiment of this specification also provides an edge computing device, including a memory, a processor, and a communication interface, wherein the memory stores a computer program, and when the processor executes the computer program, the vital sign signal measurement method in any of the above examples is implemented.

[0161] Example 2

[0162] To address the issues of environmental influences on vital sign detection signals and the large differences in reflected signals between different people when using millimeter-wave radar for vital sign detection, the present invention combines millimeter-wave radar with remote photoelectric plethysmography to achieve more accurate vital sign detection. The implementation process of a vital sign detection method based on human-factor intelligence provided by the present invention may include the following steps:

[0163] Step one, obtain the radio frequency reflection signal and continuous multi-frame image data about the target object. The detection principle of millimeter wave radar is to detect the phase change in a specific distance interval caused by the tiny vibration of the target. The principle of remote photoplethysmography (rPPG) is that the blood flow caused by the beating of the heart will cause subtle brightness changes on the skin, and the reflected ambient light is used to measure the subtle brightness changes of the skin, thereby realizing the detection of vital signs. In an embodiment of the present invention, a more accurate vital sign signal is obtained by combining millimeter wave radar with rPPG technology. That is, the radio frequency reflection signal required for the millimeter wave radar detection method is obtained, and the continuous multi-frame image data (i.e., video data) required for rPPG is obtained to generate the vital sign signal.

[0164] The RF reflection signal can be obtained by transmitting a target pulse signal to a target object using a millimeter-wave radar and collecting the initial reflection signal after the target pulse signal is reflected by the target object. The obtained initial reflection signal is then preprocessed to obtain the RF reflection signal. Preprocessing can include noise reduction and filtering. Continuous multi-frame image data can be captured using a video recording device. The RF reflection signal and the continuous multi-frame image data are intercepted to obtain the desired RF reflection signal and multi-frame image data. For example, a total of one minute of RF reflection signal and one minute of multi-frame image data are collected. The start time of the RF reflection signal acquisition coincides with the start time of the multi-frame image data acquisition. If 30 seconds of data is required to generate a vital sign signal, since the RF reflection signal and the multi-frame image data acquisition begin at the same time, the collected 1-minute RF reflection signal can be intercepted, retaining the RF reflection signal for the first 30 seconds. Similarly, the multi-frame image data can also be retained for the first 30 seconds. This results in an RF reflection signal and multi-frame image data with the same acquisition start and end times.

[0165] In step 2, first feature data is obtained based on the RF reflection signal. Furthermore, second feature data is obtained based on multiple frames of image data. Specifically, for the first feature data, a range curve can be extracted from the collected RF reflection signal. This can be accomplished by performing a fast Fourier transform (FFT) on the collected RF reflection signal. Multiple range bins (range bins) are then determined within the range curve. The millimeter-wave radar transmits a target pulse signal to the target object, denoted as s(t), and the target pulse signal received at the receiver is denoted as r(t). The mixed and filtered intermediate frequency signal received from the object at a distance R is R(t). The FFT signal in step 1 is the intermediate frequency signal R(t). Reflections from objects at different distances have different frequencies. Therefore, the FFT is performed on the intermediate frequency signal, and the millimeter-wave radar duration is divided into different range bins, referred to as range bins. To measure small-scale changes in the target object, the phase variation of the intermediate frequency signal R(t) within the range bins needs to be measured. Therefore, after determining multiple range bins, it is necessary to determine the phases at the multiple range bins, so as to construct a phase signal based on the phases at the multiple range bins. The first feature data is obtained based on the phase signal. The vibration signal, i.e., the heartbeat signal or the breathing signal, is obtained by continuously extracting the phase at the range bin m within a preset time. The first feature data can be implemented in the form of a spectrum estimation result of the vibration signal. The first feature data can be specifically implemented in the form of a K×M×N matrix. Wherein, K is the number of transmitted waves in a frame (if the frame period is 50ms and Tc is 50us, then K is 1000), M is the number of samples in a pulse, and N is the bandwidth (due to the different characteristics of heart rate and breathing rate, the bandwidth value of the heart rate is different from the bandwidth value of the breathing rate).

[0166] As for the second feature data, it can be extracted by intercepting the image data within the region of interest (ROI). Specifically, the region of interest in multiple frames of image data can be determined, and the region of interest includes the face area, etc. The determination of the region of interest can be achieved by a target detection algorithm (such as a face detection algorithm), etc. Then, the multiple frames of image data are intercepted according to the determined region of interest to obtain corresponding multiple region of interest image data. Then, the multiple region of interest image data are converted from RGB encoded data to YUV encoded data to obtain corresponding multiple region of interest chromaticity data. Finally, the second feature data is obtained based on the multiple region of interest chromaticity data. Specifically, the remote photoplethysmography signal can be extracted from the obtained multiple region of interest chromaticity data. Thus, the second feature data is obtained based on the remote photoplethysmography signal.

[0167] Step 3: Fusing the first feature data and the second feature data to obtain fused features. Feature fusion can be achieved by feature-level fusion, weighted fusion, feature concatenation, feature superposition, feature selection, feature crossover, and other methods.

[0168] Step 4: Obtain the target subject's vital sign signals based on the fused features. Specifically, a neural network model can be used to obtain the vital sign signals. Specifically, the fused features can be input into a pre-trained target neural network model. The target neural network model then outputs the vital sign signals.

[0169] In some embodiments, the target neural network model can be implemented as a vision transformer network model (referred to as a ViT network model). The training method of the target neural network model may include: detecting the target to be detected, and obtaining a radio frequency reflection signal, continuous multi-frame image data, and a vital sign signal about the target to be detected. The vital sign signal can be obtained by collecting the target to be detected through a vital sign acquisition device (such as a medical instrument, etc.). Afterwards, the vital sign signal is aligned with the radio frequency reflection signal to obtain a vital sign true value signal, and the radio frequency reflection signal is feature fused with the multi-frame image data to obtain a vital sign feature signal. Before fusion, the features of the radio frequency reflection signal and the features of the multi-frame image data can be extracted separately and feature fused. The specific steps are the same as the above-mentioned steps of fusing the first feature data and the second feature data to obtain a fused feature, and will not be repeated here.

[0170] A training dataset is then constructed based on the vital sign feature signals and the true vital sign signals. The initial model is then iteratively trained based on the training dataset. Training is concluded when the initial model meets preset conditions, resulting in the target neural network model. The preset conditions can be set as model convergence or accuracy reaching a preset threshold.

[0171] In a specific embodiment, the human-intelligence-based vital sign detection method provided by the present invention is divided into three main steps. The first two steps, in no particular order, are: acquiring millimeter-wave radar signal features and rPPG signal features. The third step is to input these two acquired features into a trained ViT network model to obtain an output vital sign signal.

[0172] In the step of acquiring the millimeter wave radar signal characteristics, the specific implementation steps include:

[0173] (1) Range FFT: Perform fast Fourier transform on the collected RF reflection signal to obtain the Range curve.

[0174] (2) Range bin tracking: The range of the target can be determined by the approximate position relationship between the radar and the human body. The range bin corresponding to the target is obtained by searching for the maximum value within the range.

[0175] (3) Extract Phase: Extract the phase at the target Range bin.

[0176] Then, steps (1) to (3) are executed cyclically. The frame period is 50ms, that is, the phase of the target is extracted once in each frame period. If the radial distance between the target and the distance changes, it is necessary to obtain the current range bin according to the range bin tracking algorithm, and then extract the phase. After cyclically transmitting N frames, the target phase can be obtained as the value changes with the number of frames. It can also be regarded as the relationship between the target phase and time, which is recorded as the vibration signal x(t).

[0177] (4) Phase Unwrapping: Since the phase value is between [-π, π], it needs to be unwrapped to obtain the actual displacement curve. Therefore, whenever the phase difference between consecutive values ​​is greater than / less than ±π, phase unwrapping is performed by subtracting 2π from the phase.

[0178] (5) Phase Difference: Perform a phase difference operation on the unwrapped phase by subtracting consecutive phase values. This helps enhance the heartbeat signal and eliminate any phase drift.

[0179] (6) Bandpass Filtering: Based on the difference in heart rate and respiratory rate, the phase value is filtered using a bandpass filter to distinguish them.

[0180] (7) Spectral Estimation: Range estimation. For respiratory frequency, the phase signal belonging to the respiratory frequency identified above can be FFTed, and the corresponding respiratory frequency within N frames can be obtained based on the peak value and its harmonic characteristics. The respiratory frequency within a period of time is recorded, and the respiratory frequency at that time is judged based on different confidence indicators, and the relationship between the respiratory frequency and time is output. For heart rate, the phase signal belonging to the heart rate identified above can be filtered first. The purpose is to reduce the impact of the relative position movement of the human body on the heart rate measurement. (Because the measurement of heart rate is based on the distance difference caused by the small movement of the heart contraction and relaxation, which causes phase changes, according to the micro-Doppler principle, when the human body swings significantly, it will affect its accuracy.) Here, the samples are divided, a threshold is set to determine whether it meets the range of heart rate changes, and data under stable conditions are selected for range estimation. That is, the phase signal belonging to the heart rate after filtering is FFTed, and the corresponding heart rate within N frames can be obtained based on the peak value and its harmonic characteristics. Record heart rate over a period of time, determine the heart rate at that time based on different confidence indicators, and output the relationship between heart rate and time. The recording process can introduce an autocorrelation mechanism to improve the accuracy of the output.

[0181] From this, we can determine the relationship between respiratory rate and heart rate over time. We can then use long-sequence supervision to enhance the model's feature perception. This can be achieved through an encoder-decoder architecture, where the encoder can extract the features of the millimeter-wave radar signal. The encoder and decoder architecture can use LSTM, with the encoder output representing the millimeter-wave radar signal features. Millimeter-wave radar signal features are represented by a matrix of K*M*N, where K is the number of transmitted waves in a frame, M is the number of samples in a chirp, and N is the bandwidth.

[0182] To acquire rPPG signal features, face recognition is required to obtain the facial ROI region. The ROI region is a multi-frame facial image (RGB image). A YUV image is then generated based on these multi-frame facial images to better represent the luminance information. The YUV image contains six channels of data: R, G, B, Y, U, and V. From these six channels, the CHROM signal, POS (Plane-Orthogonal-to-Skin) signal, and filtered signal are extracted. This step is implemented by aligning the face across different frames (i.e., multiple frames of facial image data) based on detected landmarks. The facial region is then divided into n ROI blocks, R1, R2, ..., and Rn. The average color value for each color channel within each block is calculated. The average color values ​​for each channel at the same block location but across different frames are concatenated into a sequence: R1, G1, B1, R2, G2, B2, ..., Rn, Gn, Bn. Sequences from the same color channel are connected into a map (R, G, and B) of size N x L, where N = n. In addition, the RGB color space is converted to the YUV color space (Y, U, and V). The CHROM algorithm and POS algorithm are then used to process the corresponding CHROM signal and POS signal. And filtered by a Butterworth bandpass filter or other filters to obtain the filtered signal. Finally, the different combined signals are cascaded into four spatio-temporal maps (STMap) as rPPG signal features. The four STMaps include: CHROM-STMap, POS-STMap, Filtered-STMap, and Original-STMap. The matrix of STMap is expressed as: B*S*T, where B represents the frame rate, S is the number of ROI blocks, and T is the time length.

[0183] The specific implementation process of the third step mentioned above is to fuse the K*M*N millimeter-wave radar signal features and the B*S*T rPPG signal features. The fusion method can be to flatten the signal to a one-dimensional dimension and pass it through a fully connected layer to a fixed dimension (for example: 1024). The millimeter-wave radar signal features are then converted into a 1024-dimensional vector. The rPPG signal features are then converted into four 1024-dimensional vectors. After splicing, the resulting five 1024-dimensional vectors serve as the input to the ViT neural network model. The ViT neural network model then outputs the final vital sign signal.

[0184] Corresponding to the above-mentioned human-based intelligence-based vital sign detection method, an embodiment of the present invention provides a human-based intelligence-based vital sign detection device, which includes: an acquisition module, a feature extraction module, a feature fusion module, and a processing module.

[0185] The acquisition module is used to acquire radio frequency reflection signals and continuous multi-frame image data about the target object.

[0186] The feature extraction module is configured to obtain first feature data based on the radio frequency reflection signal and second feature data based on multiple frames of image data.

[0187] The feature fusion module is used to perform feature fusion on the first feature data and the second feature data to obtain a fused feature.

[0188] The processing module is used to obtain the vital sign signal of the target object based on the fusion feature.

[0189] The human intelligence-based vital sign detection device provided in the embodiments of the present application can be used to implement the technical solutions of the above-mentioned method embodiments of this specification. Its implementation principles and technical effects can be further referred to the relevant descriptions in the method embodiments.

[0190] Corresponding to the above-mentioned human-based intelligence-based vital sign detection method, an embodiment of the present invention provides an electronic device. The components of this electronic device may include, but are not limited to, one or more processors, a communication interface, and a memory, and a communication bus connecting different system components (including the memory, communication interface, and processor). For details, please refer to the relevant content of the computer device shown in Figure 9 of Example 1.

[0191] The embodiments of this specification also provide a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a computer, the computer executes the vital sign signal measurement method in the above-mentioned embodiment 1, or the vital sign detection method based on human factors intelligence in embodiment 2.

[0192] The embodiments of this specification also provide a computer program product including instructions, which, when executed by a computer, causes the computer to execute the vital sign signal measurement method in the above-mentioned embodiment 1, or the vital sign detection method based on human factors intelligence in embodiment 2.

Claims

1. A method for measuring vital sign signals based on human intelligence, characterized in that: The method comprises: Acquire individual characteristic representation data of the measured object, environmental characteristic representation data of the environment in which the measured object is located, and vital sign spectrum data of the measured object; wherein the individual characteristic representation data and the environmental characteristic representation data have different effects on the vital sign spectrum data; Signal value prediction is performed based on the individual feature representation data, the environmental feature representation data, and the vital sign spectrum data to obtain the vital sign signal value with the difference influence removed.

2. The measuring method according to claim 1, characterized in that: The individual characteristic representation data of the measured object is obtained by the following method, including: Acquiring biometric data of the object under test based on a video signal; wherein the video signal is obtained by photographing the environment in which the object under test is located; Based on the biometric data in the video signal, the object under test is individually identified to obtain the individual feature representation data.

3. The measuring method according to claim 2, characterized in that: The biometric data of the subject being measured is any one of facial data, iris data, retinal data, and eye pattern data; The individual characteristic representation data includes at least one of the gender characteristic, age characteristic, and skin type characteristic of the subject.

4. The measuring method according to claim 1, characterized in that: The environmental characteristic representation data of the environment in which the measured object is located is obtained by the following method, including: Acquiring environmental data of the environment in which the measured object is located based on the video signal; wherein the video signal is obtained by photographing the environment in which the measured object is located; Feature extraction is performed based on the environmental data in the video signal to obtain the environmental feature representation data; wherein the environmental feature representation data includes at least one of humidity features, temperature features, weather features, and wind speed features.

5. The measuring method according to claim 1, characterized in that: The spectrum data of the vital signs of the measured object is obtained based on the digital mixing signal of the first measuring device.

6. The measuring method according to claim 5, characterized in that: The spectrum data of the vital signs of the measured object is obtained based on the digital mixing signal of the first measuring device, including: Acquire a video signal and a digital mixing signal of a first measuring device; wherein the video signal is obtained by photographing the environment in which the measured object is located; and the digital mixing signal is determined based on the transmission and reception of a frequency modulated continuous wave radar signal by the first measuring device through a millimeter wave radar; Determine an initial distance unit where the measured object is located relative to the first measuring device based on the digital mixing signal; Correcting the initial distance unit based on the video signal to obtain a target distance unit; The vital sign spectrum data is determined based on the target distance unit.

7. The measuring method according to claim 6, characterized in that: The step of correcting the initial distance unit based on the video signal to obtain a target distance unit includes: Detecting the distance between the first measuring device and the measured object based on the video signal to obtain a video detection distance; The initial distance unit is corrected based on the video detection distance to obtain the target distance unit.

8. The measuring method according to claim 5, characterized in that: The signal value prediction is performed based on the individual feature representation data, the environmental feature representation data, and the vital sign spectrum data to obtain the vital sign signal value after removing the difference influence, including: Acquire continuous multi-frame image data; Acquire second feature data based on the continuous multiple frames of image data; Performing feature fusion on the vital sign spectrum data of the measured object acquired based on the digital mixing signal of the first measuring device and the second feature data to obtain a fused feature; Based on the individual feature representation data, the environmental feature representation data, and the fusion feature, a vital sign signal value is predicted to obtain a vital sign signal value with the difference effect removed.

9. The measuring method according to claim 8, characterized in that: The multiple frames of image data are RGB encoded data, and obtaining the second feature data based on the multiple frames of image data includes: Determine a region of interest in the multiple frames of image data, wherein the region of interest includes a face region; Intercepting the multiple frames of image data according to the determined focus area to obtain corresponding multiple focus area image data; Convert the plurality of focus area image data from RGB encoded data to YUV encoded data to obtain corresponding plurality of focus area chromaticity data; The second feature data is obtained based on the plurality of focus area chromaticity data.

10. The measuring method according to claim 9, characterized in that: The obtaining the second feature data based on the plurality of focus area chromaticity data comprises: extracting a remote photoplethysmography signal from the plurality of region of interest chromaticity data; The second characteristic data is obtained based on the remote photoplethysmography signal.

11. The measuring method according to claim 1, characterized in that: The signal value prediction is performed based on the individual feature representation data, the environmental feature representation data, and the vital sign spectrum data to obtain the vital sign signal value after removing the difference influence, including: Performing feature combination on the individual feature representation data, the environment feature representation data and the vital sign spectrum data to obtain a feature combination result; Signal value prediction is performed according to the feature combination result to obtain the vital sign signal value; wherein the vital sign signal value includes at least one of heart rate and respiratory rate.

12. The measuring method according to claim 11, characterized in that: The step of combining the individual feature representation data, the environment feature representation data and the vital sign spectrum data to obtain a feature combination result includes: The individual feature representation data, the environmental feature representation data and the vital sign spectrum data are spliced ​​to obtain the feature combination result.

13. The measuring method according to claim 1, characterized in that: The step of obtaining individual characteristic representation data of the measured object and environmental characteristic representation data of the environment in which the measured object is located comprises: Acquire a video signal obtained by shooting the environment in which the measured object is located; Determine environmental data and individual identification results of the measured object based on the video signal; Mapping the environmental data to obtain environmental feature representation data of a specified dimension; Mapping is performed on the individual recognition result to obtain the individual feature representation data of the specified dimension.

14. The measuring method according to claim 1, characterized in that: The vital sign signal value is output by a target classification model; the target classification model is obtained by the following training method: Constructing a training sample set; wherein the training sample set includes a plurality of training samples, and the training samples include historical vital sign spectrum data, historical individual feature representation data, and historical environmental feature representation data; the labels of the training samples adopt the true values ​​of historical vital sign signals; The initial classification model is trained according to the training samples and the labels to obtain the target classification model.

15. The measuring method according to claim 14, characterized in that: The training samples are constructed in the following way: Determine the historical vital sign spectrum data based on the historical digital mixed frequency signal collected by the first measuring device at a historical moment for the measured object or the non-measured object; Acquire a historical video signal collected at the historical moment; Determining the historical individual feature representation data and the historical environment feature representation data according to the historical video signal; The true value of the historical vital sign signal collected by the second measuring device at the historical moment for the measured object or the non-measured object is used as a label.

16. The measuring method according to claim 15, characterized in that: The second measuring device is a measuring device different from the first measuring device; wherein the second measuring device is any one of a mechanical measuring device and a biological signal measuring device.

17. A vital sign signal measuring device based on human factors intelligence, characterized in that: The device comprises: A feature data acquisition module, used to acquire individual feature representation data of the measured object, environmental feature representation data of the environment in which the measured object is located, and vital sign spectrum data of the measured object; wherein the individual feature representation data and the environmental feature representation data have different effects on the vital sign spectrum data; The vital sign determination module is used to predict the signal value based on the individual feature representation data, the environmental feature representation data, and the vital sign spectrum data to obtain the vital sign signal value without the influence of the difference.

18. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the vital sign signal measurement method according to any one of claims 1 to 16 is implemented.

19. An edge computing device, comprising a memory, a processor and a communication interface, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the vital sign signal measurement method according to any one of claims 1 to 16 is implemented.

20. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the vital sign signal measurement method according to any one of claims 1 to 16 is implemented.

Images