Heart surface multi-point joint collection and multi-modal fusion system, method and device

Abstract

The present disclosure provides a kind of heart surface multi-point joint acquisition and multimodal fusion system, method and equipment, belong to heart state monitoring technical field, the system includes: heart shock signal sensor is used to be measured at multiple acquisition positions of thoracic cavity and obtains multiple initial heart shock signals, each acquisition position corresponds one initial heart shock signal;First signal processing module is used to preprocess multiple initial heart shock signals, and obtains multiple target heart shock signals;Radar signal sensor is used to emit radar signal and collect the initial radar echo signal returned by measured thoracic cavity;Second signal processing module is used to preprocess initial radar echo signal, and obtains face array radar echo signal;Data processing module is used to determine the target physiological state information for reflecting measured thoracic cavity according to multiple target heart shock signals and face array radar echo signal. More accurate heart activity information can be provided, thereby improving the accuracy of diagnosis.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of heart state monitoring, and in particular to a heart surface multi-point joint acquisition and multi-modal fusion system, method and device. BACKGROUND

[0002] With the rapid development of the medical and health field, heart health monitoring technology is increasingly valued. As one of the main causes of death worldwide, early diagnosis and real-time monitoring of heart disease is crucial to improving treatment effectiveness and patient quality of life.

[0003] Traditional heart monitoring methods, such as electrocardiogram (ECG), although play an important role in clinical diagnosis, ECG often only collects heart electricity, but the heart structure is complex, in addition to the heart electricity signal, the mechanical vibration of heart activity is also important for the evaluation of heart health status, but the mechanical vibration of heart activity is greatly affected by the external environment, the signal is weak and not easy to analyze. SUMMARY

[0004] The present disclosure provides a heart surface multi-point joint acquisition and multi-modal fusion system, method and device; which can reduce the influence caused by single weak signal or noise interference, so that the key features obtained by final analysis are more accurate and content is more abundant, and the accuracy of diagnosis is improved.

[0005] The technical solution of the present disclosure is implemented as follows:

[0006] In a first aspect, the present disclosure provides a heart surface multi-point joint acquisition and multi-modal fusion system, which comprises: a heart vibration signal sensor, a first signal processing module connected with the heart vibration signal sensor, a radar signal sensor, a second signal processing module connected with the radar signal sensor, and a data processing module connected with the first signal processing module and the second signal processing module; the heart vibration signal sensor is configured to acquire a plurality of initial heart vibration signals at a plurality of acquisition positions of a measured chest cavity, each acquisition position corresponding to an initial heart vibration signal; the first signal processing module is configured to preprocess the plurality of initial heart vibration signals to obtain a plurality of target heart vibration signals; the radar signal sensor is configured to emit a radar signal and acquire an initial radar echo signal returned by the measured chest cavity; the second signal processing module is configured to preprocess the initial radar echo signal to obtain a planar array radar echo signal; and the data processing module is configured to determine target physiological state information reflecting the measured chest cavity according to the plurality of target heart vibration signals and the planar array radar echo signal.

[0007] In a second aspect, the present disclosure provides a heart surface multi-point joint acquisition and multi-modal fusion method, comprising: acquiring a plurality of initial heart shock signals at a plurality of acquisition positions of a measured chest cavity, each acquisition position corresponding to an initial heart shock signal; preprocessing the plurality of initial heart shock signals to obtain a plurality of target heart shock signals; transmitting a radar signal and acquiring an initial radar echo signal returned by the measured chest cavity; preprocessing the initial radar echo signal to obtain a surface array radar echo signal; and determining target physiological state information reflecting the measured chest cavity according to the plurality of target heart shock signals and the surface array radar echo signal.

[0008] In a third aspect, the present disclosure provides an electronic device, comprising a processor, a memory, and a program or instruction stored on the memory and executable on the processor, wherein the program or instruction is executed by the processor to implement the steps of the heart surface multi-point joint acquisition and multi-modal fusion method according to the second aspect.

[0009] In a fourth aspect, the present disclosure provides a computer-readable storage medium, wherein the computer-readable storage medium stores a program or instruction, and the program or instruction is executed by a processor to implement the steps of the heart surface multi-point joint acquisition and multi-modal fusion method according to the second aspect.

[0010] In a fifth aspect, the present disclosure provides a computer program product, wherein the computer program product comprises a computer program or instruction, and when the computer program product is executed on a processor, the processor executes the computer program or instruction to implement the steps of the heart surface multi-point joint acquisition and multi-modal fusion method according to the second aspect.

[0011] In a sixth aspect, the present disclosure provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is configured to execute a program or instruction to implement the heart surface multi-point joint acquisition and multi-modal fusion method according to the second aspect.

[0012] The disclosure provides a heart surface multi-point joint acquisition and multi-modal fusion system, which comprises a heart shock signal sensor, a first signal processing module connected with the heart shock signal sensor, a radar signal sensor, a second signal processing module connected with the radar signal sensor, and a data processing module connected with the first signal processing module and the second signal processing module; the heart shock signal sensor is configured to acquire a plurality of initial heart shock signals at a plurality of acquisition positions of a measured chest cavity, and each acquisition position corresponds to an initial heart shock signal; the first signal processing module is configured to preprocess the plurality of initial heart shock signals to obtain a plurality of target heart shock signals; the radar signal sensor is configured to emit a radar signal and acquire an initial radar echo signal returned by the measured chest cavity; the second signal processing module is configured to preprocess the initial radar echo signal to obtain a planar array radar echo signal; and the data processing module is configured to determine target physiological state information reflecting the measured chest cavity according to the plurality of target heart shock signals and the planar array radar echo signal. The physiological state information reflecting the heart activity included in the target heart shock signal and the planar array radar echo signal is fused, the fused signal can be enhanced in intensity, and the influence caused by weak single signal or noise interference can be reduced, so that the final target physiological state information can more accurately indicate the heart activity, and the diagnostic accuracy is improved. BRIEF DESCRIPTION OF DRAWINGS

[0013] Figure 1 FIG. 1 is a structural schematic diagram of a heart surface multi-point joint acquisition and multi-modal fusion system provided by the disclosure;

[0014] Figure 2 FIG. 3 is a schematic diagram of an acquisition position provided by the disclosure;

[0015] Figure 3 FIG. 5 is a schematic diagram of an echo detection principle based on a multiple-input multiple-output millimeter wave radar provided by the disclosure;

[0016] Figure 4 FIG. 7 is a structural schematic diagram of a heart surface multi-point joint acquisition and multi-modal fusion system provided by the disclosure;

[0017] Figure 5 FIG. 9 is a feature point schematic diagram provided by the disclosure;

[0018] Figure 6 FIG. 11 is a flow schematic diagram of a heart surface multi-point joint acquisition and multi-modal fusion method provided by the disclosure;

[0019] Figure 7 FIG. 13 is a hardware structure schematic diagram of an electronic device provided by the disclosure. DETAILED DESCRIPTION

[0020] The technical solutions in the embodiments of the present application will be clearly described below with reference to the drawings in the present disclosure. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art belong to the scope of protection of the present application.

[0021] The terms "first", "second", and the like in the specification of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the present disclosure can be implemented in an order other than that illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally a category and do not limit the number of objects, for example, the first object can be one or more.

[0022] Figure 1 A heart surface multi-point joint acquisition and multi-modal fusion system is shown in the present disclosure. As shown in the figure, the heart surface multi-point joint acquisition and multi-modal fusion system 10 includes a heart shock signal sensor 101, a first signal processing module 102 connected with the heart shock signal sensor 101, a radar signal sensor 103, a second signal processing module 104 connected with the radar signal sensor 103, and a data processing module 105 connected with the first signal processing module 102 and the second signal processing module 104. Figure 1

[0023] The heart shock signal sensor 101 is configured to acquire a plurality of initial heart shock signals at a plurality of acquisition positions of a measured chest cavity. The first signal processing module 102 is configured to preprocess the plurality of initial heart shock signals to obtain a plurality of target heart shock signals. The radar signal sensor 103 is configured to emit a radar signal and acquire an initial radar echo signal returned by the measured chest cavity. The second signal processing module 104 is configured to preprocess the initial radar echo signal to obtain a planar array radar echo signal. The data processing module 105 is configured to determine target physiological state information reflecting the measured chest cavity according to the plurality of target heart shock signals and the planar array radar echo signal.

[0024] The heart shock signal sensor 101 is configured to acquire a plurality of initial heart shock signals at a plurality of acquisition positions of a measured chest cavity.

[0025] ​Cardiac vibration signals typically refer to the minute vibrations synchronized with the heartbeat caused by the contraction of the heart and the impact of blood on large blood vessels. Based on this, in this disclosure, the cardiac vibration signal sensor 101 can be implemented as a vibration sensor for measuring vibration-related information of an object (such as amplitude, frequency, acceleration, etc.), for example, a piezoelectric crystal accelerometer, a piezoelectric crystal displacement sensor, a microelectromechanical system (MEMS) accelerometer, or a MEMS displacement sensor. Taking a piezoelectric crystal accelerometer or a MEMS accelerometer as an example, it can capture the linear acceleration components of cardiac vibrations. These signals reflect the minute body vibrations caused by the physical movement of the heart. By analyzing these signals, information related to cardiac activity can be extracted, such as peak values ​​(corresponding to the closing and opening of the heart valves), signal amplitude, energy, area, and time intervals. Meanwhile, a piezoelectric crystal displacement sensor or a MEMS displacement sensor can capture the amplitude of cardiac vibrations. Since the chest cavity being measured is divided into multiple acquisition locations, the number of cardiac vibration signal sensors 101 is consistent with the acquisition locations. During implementation, each vibration sensor is attached to the corresponding acquisition location of the chest cavity being measured, thereby acquiring the initial cardiac vibration signal at each acquisition location.

[0026] Since vibration signals at different locations in the chest cavity can reflect different cardiac physiological activities, in order to ensure that the initial cardiac vibration signals collected can more comprehensively reflect cardiac activity, multiple acquisition locations can be set at certain intervals to cover the entire chest cavity being tested.

[0027] However, having too many acquisition points requires filtering out the cardiac signals that best represent cardiac physiological activity from multiple initial cardiac seismic signals, increasing the processing workload. Therefore, to ensure that the acquired cardiac seismic signals reflect more cardiac activity while reducing the number of acquisition points, thus saving hardware resources and minimizing user discomfort, a new approach is needed.

[0028] In some embodiments, since the mechanical vibrations generated by the physiological activities of the human heart mainly originate from the opening and closing movements of the four valves within the heart, therefore, as Figure 2 The diagram shows the set acquisition locations, which include: the mitral valve area M, located at the point of strongest apical impulse in the 5th intercostal space, 0.5-1.0 cm medial to the left midclavicular line; the pulmonary valve area P, located at the 2nd intercostal space on the left sternal border; the aortic valve area A, located at the 2nd intercostal space on the right sternal border; the second aortic valve area E, located at the 3rd intercostal space on the left sternal border; and the tricuspid valve area T, located at the lower left sternal border, i.e., between the 4th and 5th intercostal spaces on the left sternal border.

[0029] The sampling location can be determined based on the height and weight of the subject; alternatively, an initial radar echo signal of a certain duration can be collected first, and the shape and size of the heart in the chest cavity can be determined based on the initial radar echo signal, and then a more reasonable sampling location can be set based on the shape and size.

[0030] Since the acquired initial heart shock signals usually include noise, the first signal processing module 102 is configured to preprocess the plurality of initial heart shock signals to obtain a plurality of target heart shock signals.

[0031] In some embodiments, the preprocessing includes filtering and baseline drift removal, and the first signal processing module 102 is specifically configured to filter and remove baseline drift from the plurality of initial heart shock signals to obtain the plurality of target heart shock signals.

[0032] The noise in the original signal is mostly from the power frequency interference generated by the power supply when the system is working, and part of the noise is from the bioelectric signals of the human body such as electromyographic signals. Since the vibration frequency generated by the human heart activity is mainly in the interval of 0.05 Hz-2 Hz, a Butterworth band-pass filter can be used to filter out signals with frequencies not in the range of 0.05 Hz-2 Hz to remove noise signals unrelated to heart activity.

[0033] Baseline drift refers to slow-changing non-periodic trends appearing in the signal. These trends are not directly caused by heart activity, but are caused by signal changes from other factors. The essence of baseline drift is to superimpose a direct current component and a low-frequency component on the original signal, which may affect the accuracy of the signal and the subsequent data processing results.

[0034] The causes of baseline drift usually include: the interaction between heart activity and breathing pattern (i.e. cardiopulmonary coupling) can introduce breathing-related low-frequency fluctuations in the initial heart shock signal, leading to baseline drift; random motion caused by body posture changes or muscle activity can cause additional fluctuations in the initial heart shock signal, which in turn causes baseline drift.

[0035] The purpose of removing baseline drift is to remove non-physiological, non-interesting low-frequency trend components in the signal. These components are not directly caused by heart activity, but are caused by other factors such as cardiopulmonary coupling and random body motion of the human body. In order to eliminate the influence of these baseline drifts, various methods can be used, such as detrend function, wavelet transform, zero point calibration, empirical mode decomposition, etc. But because the detrend function is a simple and effective method to remove the trend item in the signal, it can subtract a best-fitting straight line (linear trend) or higher-order polynomial trend from the signal, so that the mean of the signal is close to zero. In practice, this method is particularly suitable for removing linear or segmented linear trend items.

[0036] The radar signal sensor 103 is configured to emit a radar signal and collect an initial radar echo signal returned by the measured chest cavity.

[0037] The radar signal sensor 103 transmits a frequency-modulated continuous wave to the measured chest cavity and receives the initial radar echo signal reflected back, thereby capturing the slight vibration of the measured chest cavity caused by the beating heart. The radar signal sensor 103 receives the initial radar echo signal through a plurality of virtual transceiving channels composed of a plurality of transmitting antennas and a plurality of receiving antennas, and the radar signal is transmitted in a time division multiple access (TDMA) mode, so that only one pair of transceiving antennas is working at each time of transmitting and receiving the radar signal, and the radiation area of the radar signal sensor 103 covers the entire measured chest cavity. The initial radar signal includes information such as the speed, acceleration, phase, and vibration direction of the vibration.

[0038] Similar to the above-mentioned heart vibration signal, noise will also be included in the collected initial radar echo signal. Therefore, the second signal processing module 104 is configured to preprocess the initial radar echo signal to obtain a planar array radar echo signal.

[0039] In some embodiments, the preprocessing includes filtering processing, static background removal, and sidelobe interference removal. The second signal processing module 104 is specifically configured to perform filtering processing, static background removal, and sidelobe interference removal on the initial radar echo signal to obtain the planar array radar echo signal.

[0040] The filtering processing can refer to the filtering processing of the initial heart vibration signal described above, which will not be described here again.

[0041] The static background mainly comes from the radar system itself and fixed objects in the surrounding environment. In order to remove these interferences, a hybrid analog and digital compensation technique can be used to suppress leakage and static clutter. For example, a sliding average filtering algorithm can be used to remove part of the static background; a frequency domain filter can also be used to remove the static background. For example, by designing a band-stop filter, the signal in a specific frequency range can be suppressed, thereby removing the interference of the static background.

[0042] Sidelobe interference refers to the scattering of radar signals in non-target directions. These scattered signals may interfere with the detection of target signals. In order to remove the sidelobe interference, a sidelobe suppression technique can be used, such as using a window function to reduce the sidelobe level; using the characteristics of Radon transform to distinguish the main axis of the target from the sidelobe. When the main axis and the sidelobe have different directions, the peak points are in different positions in the Radon transform domain, which can be distinguished in the Radon transform domain; integrating along the azimuth and range to find the sidelobe, using local mean filtering to eliminate its influence, etc.

[0043] The data processing module 105 determines target physiological state information reflecting the measured chest cavity according to the plurality of target heart vibration signals and the planar array radar echo signal.

[0044] Specifically, since the target heart vibration signal and the surface array radar echo signal can both collect the vibration signal of the to-be-measured chest cavity, the characteristics included in both the target heart vibration signal and the surface array radar echo signal are fused, and the fused signal can enhance the strength of the signal and reduce the influence caused by a single weak signal or noise interference.

[0045] Since the radiation surface of the radar signal sensor 103 is the entire to-be-measured chest cavity, the usual analysis is to fuse and analyze the radar echo signals returned by the entire to-be-measured chest cavity, but the precision of this analysis method is not enough, and there may be a large difference between the two signals reflecting the vibration of different regions. The direct fusion result may not accurately reflect the actual heart activity. Therefore, the heart vibration signal sensor 101 has a fixed collection position on the to-be-measured chest cavity, so that the radar echo signal and the target heart vibration signal are the heart activity signals collected at the same position.

[0046] In some embodiments, the radar signal sensor 103 includes a radar antenna array, and the data processing module 105 is specifically configured to: determine a reference point corresponding to the center of the radar antenna array and mapped to the to-be-measured chest cavity; determine, from the surface array radar echo signal, a multi-point micro-motion radar echo signal at each collection position according to the distance between the reference point and each collection position and a radar transmission distance, the radar transmission distance being the distance between the radar signal sensor 103 and the to-be-measured chest cavity; and determine a target physiological state of the to-be-measured chest cavity according to the target heart vibration signal at each collection position and the corresponding multi-point micro-motion radar echo signal.

[0047] The radar antenna array is a system composed of multiple antenna elements (also known as array elements), which are arranged in a certain geometric pattern and work together to achieve a specific radar function. Each antenna element in the radar antenna array can be arranged as needed, such as a linear array arranged in a linear manner, a planar array arranged in a plane (such as a rectangular array, a circular array, etc.), and a three-dimensional array distributed in a certain space (such as a spherical array). The radar antenna array in the present disclosure is a planar array. Radar antenna arrays with different arrangements have different radiation regions, and the center of the radiation region is the center of the radar antenna array.

[0048] Specifically, as Figure 3 The echo detection principle diagram based on the multiple-input multiple-output millimeter wave radar is shown, and the coordinates of the five collection positions in the to-be-measured chest cavity planar diagram corresponding to the three-dimensional to-be-measured chest cavity are respectively denoted as . The specific coordinate values are , , , , , , determine the reference point of the radar antenna center mapping to the measured chest cavity, that is, the position of the reference point is at the position where the azimuth angle and the elevation angle of the radar are both 0, such as taking the collection position A as the reference point, denoted as . The transmission distance of the reference point, that is, the collection position A to the radar, is denoted as , and the azimuth angle and the elevation angle of the collection position P, the collection position T, the collection position E, and the collection position M relative to the radar are calculated, respectively denoted as , , , , wherein: , and , . According to the azimuth angle and the elevation angle of each collection position, the multi-point micro-motion radar echo signal at each collection position is identified from the radar surface array radar echo signal.

[0049] In this embodiment, each multi-point micro-motion radar echo signal and the corresponding target heart vibration signal reflect the heart activity at the same collection position, the matching degree of the two signals is higher, the fusion accuracy is higher, and the target physiological state information obtained for reflecting the heart activity is also more accurate.

[0050] In some embodiments, as shown in Figure 4 , the heart surface multi-point joint collection and multi-modal fusion system 10 further comprises: an electrocardiogram signal sensor 106, a third signal processing module 107 connected with the electrocardiogram signal sensor 106; the electrocardiogram signal sensor 106 is configured to collect an initial electrocardiogram signal of the measured chest cavity; the third signal processing module 107 is configured to preprocess the initial electrocardiogram signal to obtain a target electrocardiogram signal; and the data processing module 105 is specifically configured to determine target physiological state information for reflecting the measured chest cavity according to the target electrocardiogram signal, the plurality of target heart vibration signals, and the plurality of multi-point micro-motion radar echo signals.

[0051] The beating of the heart is controlled by the electrical signals generated by the heart pacemaker cells. These electrical signals are generated as the heart excites different parts, forming a series of electrical signals. The electrocardiogram signal sensor 106 is used to collect the electrical signals of the heart activity.

[0052] Since the collected initial electrocardiogram signal is weak and includes noise, the initial electrocardiogram signal needs to be preprocessed to obtain a target electrocardiogram signal with reduced noise.

[0053] In some embodiments, the preprocessing includes amplification processing, filtering processing, and baseline drift removal; and the third signal processing module 107 is specifically configured to perform amplification processing, filtering processing, and baseline drift removal on the initial electrocardiogram signal to obtain the target electrocardiogram signal.

[0054] The initial heart electric signal is amplified by a signal amplifier, and then is subjected to filtering processing and baseline drift removal. The filtering processing and baseline drift removal can refer to the above description of the initial heart vibration signal processing process, and will not be described here.

[0055] The features in the target heart electric signal, the plurality of target heart vibration signals and the plurality of multi-point micro-motion radar echo signals include mechanical activity features and electrical activity features, and thus contain more abundant heart activity information.

[0056] The acquired signals include various features, but the aortic valve opening and closing mark the start and end of ventricular contraction and diastole, and the mitral valve opening and closing reflect the blood flow dynamics between the atrium and the ventricle. Therefore, the four feature points of aortic valve opening, aortic valve closing, mitral valve opening and mitral valve closing can best reflect the physiological activity of the heart. By monitoring these feature points, the mechanical function and blood flow dynamics of the heart can be effectively evaluated.

[0057] Therefore, in some embodiments, the data processing module 105 is configured to determine the aortic valve opening and closing feature points and the mitral valve opening and closing feature points at each collection position according to the target heart electric signal, the plurality of target heart vibration signals and the plurality of multi-point micro-motion radar echo signals.

[0058] Specifically, in some embodiments, the data processing module 105 is specifically configured to: determine a unit heart electric signal of a cardiac cycle, a unit radar echo signal at each collection position and a unit heart vibration signal at each collection position according to the target heart electric signal, the plurality of target heart vibration signals and the plurality of multi-point micro-motion radar echo signals; divide the unit radar echo signal and the unit heart vibration signal at each collection position into a first sub-wave and a second sub-wave according to time points corresponding to R peaks and T peaks in the unit heart electric signal; in the first sub-wave, determine a maximum value in a first preset time length before a first reference origin as a mitral valve closing feature point, and determine a maximum value in a second preset time length after the first reference origin as an aortic valve opening feature point, the first reference origin being a minimum point in a first experience time range after the R peak; in the second sub-wave, determine a maximum value in a third preset time length before a second reference origin as an aortic valve closing feature point, and determine a minimum value in a fourth preset time length after the second reference origin as a mitral valve opening feature point, the second reference origin being a maximum point in a second experience time range after the T peak.

[0059] Each of the target ECG signal, the target heart shock signal and the multi-point micro-motion radar echo signal is composed of signals of a plurality of cardiac cycles (the entire time period from the beginning of one heartbeat to the beginning of the next heartbeat), each of which is substantially the same, and thus extracting the target physiological state information from the target ECG signal, the target heart shock signal and the multi-point micro-motion radar echo signal can be simplified to extracting the target physiological state information from the signals of one cardiac cycle of each of the signals.

[0060] In the target ECG signal, the R peak is an identifier of ventricular depolarization, i.e., an identifier of ventricular contraction, and a cardiac cycle can be determined according to the R peak, which is about 1 s, and thus 0.5 s before and after the R peak can be taken as a cardiac cycle. Since the target ECG signal, the multi-point micro-motion radar echo signal and the target heart shock signal are synchronously collected, the multi-point micro-motion radar echo signal and the target heart shock signal are divided into a plurality of cardiac cycles according to the start time point and the end time point of each cardiac cycle of the target ECG signal.

[0061] Since the signals of one cardiac cycle may not effectively represent the actual state of heart activity due to noise and system reasons, the signals of one cardiac cycle can be fused according to the signals of a plurality of cardiac cycles, and the specific fusion can be weighted average, direct superposition, etc. For example, the ECG signals of 10 consecutive cardiac cycles in the target ECG signal are weighted and summed to obtain a unit ECG signal; the heart shock signals of 10 consecutive cardiac cycles in the target heart shock signal are weighted and summed to obtain a unit heart shock signal; and the radar echo signals of 10 consecutive cardiac cycles in the multi-point micro-motion radar echo signal are weighted and summed to obtain a unit radar echo signal. In this way, the unit ECG signal, the unit radar echo signal and the unit heart shock signal to be analyzed represent the actual heart activity.

[0062] The unit radar echo signal and the unit heart shock signal can be divided into a first sub-wave and a second sub-wave according to the contraction of the heart, the first sub-wave representing the contraction of the heart, and the second sub-wave representing the diastole of the heart. In the target ECG signal, the T peak is an identifier of ventricular repolarization, i.e., an identifier of ventricular diastole, and thus the first sub-wave and the second sub-wave are divided according to the R peak and the T peak. Since the time interval between the T peak and the R peak is fixed, the T peak can also be determined according to the R peak.

[0063] The division of the first sub-wave and the second sub-wave first needs to determine the R peak, the maximum value in the unit electrocardiogram signal corresponding to the R peak, and the extreme value point in the preset time period after the R peak corresponding to the T peak. According to the time points corresponding to the R peak and the T peak, the first sub-wave and the second sub-wave are divided. Specifically, according to the experimental value, the time point corresponding to the R peak is taken as the origin, the wave corresponding to the first time period after the first time period range is the first sub-wave, the wave corresponding to the second time period after the R peak corresponding to the time point is taken as the origin, the second time period range is the second sub-wave, or the T peak is taken as the origin, the wave corresponding to the second time period after the third time period is the second sub-wave, and the third time period is greater than the second time period.

[0064] After the first sub-wave and the second sub-wave are divided, the feature points need to be determined. The first sub-wave includes the mitral valve closing feature point and the aortic valve opening feature point, and the second sub-wave includes the aortic valve closing feature point and the mitral valve opening feature point. Specifically, according to the experiment, the first preset time period and the second preset time period are determined, the maximum value in the first preset time period before the first reference origin is the mitral valve closing feature point, and the maximum value in the second preset time period after the first reference origin is the aortic valve opening feature point. According to the experiment, the third preset time period and the fourth preset time period are determined, the maximum value in the third preset time period before the second reference origin is the aortic valve closing feature point, and the minimum value in the fourth preset time period after the second reference origin is the mitral valve opening feature point.

[0065] Exemplarily, as shown in Figure 5 , it is a schematic diagram for exemplarily determining the feature points. The R peak and the T peak are determined from the target electrocardiogram, S1 represents the first sub-wave, S2 represents the second sub-wave, the first experience time period range is 125ms to 155ms after the R peak, the second experience time period range is 135ms to 175ms after the T peak, the first reference origin is ICP, the second reference origin is IRP, according to the first reference origin, the maximum value is determined as the aortic valve opening feature point AO and the mitral valve closing feature point MC, and according to the second reference origin, the maximum value is determined as the aortic valve closing feature point AC and the minimum value is determined as the mitral valve opening feature point MO.

[0066] It should be noted that the first sub-wave and the second sub-wave described above represent the first sub-wave and the second sub-wave in the unit radar echo signal, the first sub-wave and the second sub-wave in the unit radar echo signal. And the above description is based on one unit radar echo signal, unit heart shock signal and unit electrocardiogram signal, in the actual acquisition process, multiple unit radar echo signals, unit heart shock signals and unit electrocardiogram signals can be obtained, and the processing process of each unit radar echo signal, unit heart shock signal and unit electrocardiogram signal is the same.

[0067] Fusing the aortic valve opening and closing feature points and the mitral valve opening and closing feature points at each collection position to obtain the target physiological state information. Specifically, at each collection position, the aortic valve opening and closing feature points and the mitral valve opening and closing feature points of the unit heart shock signal and the aortic valve opening and closing feature points and the mitral valve opening and closing feature points of the unit radar echo signal are obtained, and the corresponding feature points are fused (such as weighted summation), such as fusing the aortic valve opening feature points of the unit heart shock signal and the aortic valve opening feature points of the unit radar echo signal, fusing the mitral valve opening feature points of the unit heart shock signal and the mitral valve opening feature points of the unit radar echo signal, etc. Each fused feature point after fusion is the target physiological state information.

[0068] It should be noted that the waveforms of the same feature points of the unit radar echo signal and the corresponding unit heart shock signal may be opposite, and the waveforms need to be flipped to make the waveforms consistent before fusion.

[0069] In this way, four feature points that can best reflect the heart activity are extracted from the complex radar echo signal, heart shock signal and electrocardiogram signal, and because multiple collection positions are set, four feature points can be obtained at each collection position. By comparing and analyzing the feature points at each position, a more accurate heart state evaluation result is obtained.

[0070] In some embodiments, the data processing module 105 is specifically configured to determine first physiological state information according to the plurality of target heart shock signals; and determine second physiological state information according to the face array radar echo signal. The first physiological state information and the second physiological state information are heart state information other than the target physiological state information.

[0071] Since the radar echo signal is monitored by a non-contact method, the vibration caused by breathing and body shaking is much larger than the vibration caused by heart activity. Therefore, some subtle vibrations in the heart cannot be monitored by the radar signal sensor 103. However, the heart shock signal is collected by a contact method, and the chest to be measured is usually static. Therefore, the heart shock sensor 101 can monitor some subtle vibrations of the heart, such as the slight vibration of the human body caused by the heart pumping activity. Therefore, the first physiological state information reflecting some slight vibrations of the heart activity is analyzed from the target heart shock signal.

[0072] In addition, the radar echo signal also includes some heart activity information that is not in the heart shock signal, such as the four-dimensional heart mechanical activity signal that can be extracted by the radar signal sensor 103. This signal not only includes the time dimension, but also possibly includes the spatial dimension. Therefore, the second physiological state information not included in the target heart shock signal is analyzed from the face array radar echo signal.

[0073] Thus, the unique features of the target heart vibration signal and the surface array radar echo signal can also be extracted, and compared with a single sensor, more abundant heart activity information can be provided, so that the key features obtained by the final analysis are more accurate and rich in content, and the diagnostic accuracy is improved.

[0074] Since the radar signal sensor 103 is non-contact, the monitored vibration signal may not be accurate. The heart vibration signal sensor 101 is contact, has less noise, and the detected vibration signal is more accurate. However, the radar signal sensor 103 has the advantage of non-contact. In order to meet the non-contact and accuracy at the same time, in some embodiments, the working parameters of the radar signal sensor 103 are adjusted according to the difference between the vibration signal collected by the heart vibration signal sensor 101 and the vibration signal collected by the radar signal sensor 103, so that the vibration signals collected by the heart vibration signal sensor 101 and the radar signal sensor 103 at the same collection position are finally adjusted to be less than the difference threshold. Thus, the accuracy of the vibration signal collected by the radar signal sensor 103 is improved.

[0075] For the hardware setting of the heart surface multi-point joint acquisition and multi-modal fusion system provided by the present disclosure, since the radar signal is collected by a non-contact method, and the electrocardio signal and the heart vibration signal both require a sensor in direct contact with the human body, in the hardware setting, the heart vibration signal sensor 101, the first signal processing module 102, the electrocardio signal sensor 106, and the third signal processing module 107 can be integrated together and made by using flexible printed circuit board technology. This technology is a printed circuit with high reliability and excellent flexibility, which is made of polyester film or polyimide as a substrate. By embedding circuit design on a bendable and thin plastic sheet, a large number of precise components can be embedded in a narrow and limited space, thereby forming a bendable and flexible circuit. This kind of circuit can be bent and folded at will (convenient for conforming to the irregular chest surface), has light weight, small volume, good heat dissipation, and easy installation. The physical size of the integrated together is actually 3 cm long, 3 cm wide, and 1.5 cm high. This size can more conveniently and easily place the sensor on the chest surface of the subject while ensuring the realization of all functions of the system, and can effectively reduce the additional vibration caused by the shaking of the sensor itself, and improve the authenticity and accuracy of the measurement results.

[0076] The radar sensor 103 and the second signal processing module 104 can be set as an independent hardware module. The first signal processing module 102, the second signal processing module 104, and the third signal processing module 107 can be connected by wire or wirelessly, and the data processing module 105 can be a terminal (such as a computer).

[0077] Exemplarily, the sampling frequency of the radar signal sensor 103 is set to 5MHz, the sampling depth is 16bit, the radar starting frequency is 77GHz, the ending frequency is 81GHz, and the effective bandwidth is 4GHz. The actual experimental data shows that under this parameter, the mechanical movement generated by the physiological activity of the heart can be recorded more accurately.

[0078] The mechanical vibration generated by the physiological activity of the human heart is about 20-40mg, and the time difference between the two adjacent feature points is about 25ms. Therefore, the sampling rate of the heart shock signal sensor 101 needs to be set to 250Hz, and the sampling depth is set to 20Bit, so as to ensure that the feature point information is recorded with high enough accuracy.

[0079] The frequency of the normal human heart mechanical vibration is about 1Hz-2Hz, and the QRS complex time limit is 0.06-0.10s, and the longest is not more than 0.11s. Therefore, the sampling rate of the electrocardiogram signal sensor 106 is set to 125Hz, so as to accurately record the QRS complex information while reducing the data transmission amount.

[0080] The system power consumption mainly comes from each module, so the current size of each module under normal working condition can be used to estimate the total power consumption of the whole system. For example, when the system works under 3.7V power supply voltage, the total power consumption is about 0.74W, and in the case of using a 700mAh battery, the total endurance of the system is about 3.5h under continuous data acquisition.

[0081] Based on the same design concept as the above-mentioned heart surface multi-point joint acquisition and multi-modal fusion system, the present disclosure also provides a heart surface multi-point joint acquisition and multi-modal fusion method, as shown in Figure 6 The heart surface multi-point joint acquisition and multi-modal fusion method includes the following steps S601-S605.

[0082] In step S601, a plurality of initial heart shock signals are acquired at a plurality of acquisition positions of the measured chest cavity. Each acquisition position corresponds to an initial heart shock signal.

[0083] In step S602, the plurality of initial heart shock signals are preprocessed to obtain a plurality of target heart shock signals.

[0084] In step S603, the radar signal is transmitted and the initial radar echo signal returned by the measured chest cavity is acquired.

[0085] In step S604, the initial radar echo signal is preprocessed to obtain a planar array radar echo signal.

[0086] In step S605, target physiological state information of the measured chest cavity is determined according to the plurality of target heart vibration signals and the surface array radar echo signals.

[0087] In some embodiments, the above step S605 of determining the target physiological state information of the measured chest cavity according to the plurality of target heart vibration signals and the surface array radar echo signals comprises: determining a reference point of the measured chest cavity to which the center of the radar antenna array is mapped; determining, from the surface array radar echo signals, a multi-point micro-motion radar echo signal at each collection position according to a distance between the reference point and each collection position and a radar transmission distance, the radar transmission distance being a distance between the radar signal sensor 103 and the measured chest cavity; and determining the target physiological state information of the measured chest cavity according to the target heart vibration signal and the multi-point micro-motion radar echo signal at each collection position.

[0088] In some embodiments, the method further comprises: collecting an initial electrocardiogram signal of the measured chest cavity; and preprocessing the initial electrocardiogram signal to obtain a target electrocardiogram signal. The step S605 of determining the target physiological state information of the measured chest cavity according to the plurality of target heart vibration signals and the surface array radar echo signals comprises: determining the target physiological state information of the measured chest cavity according to the target electrocardiogram signal, the plurality of target heart vibration signals, and the surface array radar echo signals.

[0089] In some embodiments, the step S605 of determining the target physiological state information of the measured chest cavity according to the plurality of target heart vibration signals and the surface array radar echo signals comprises: determining, according to the target electrocardiogram signal, the plurality of target heart vibration signals, and the plurality of multi-point micro-motion radar echo signals, an aortic valve opening and closing feature point and a mitral valve opening and closing feature point at each collection position; and fusing the aortic valve opening and closing feature point and the mitral valve opening and closing feature point at each collection position to obtain the target physiological state information.

[0090] In some embodiments, the step S605 determines the target physiological state information reflecting the measured chest cavity according to the plurality of target heart vibration signals and the surface array radar echo signals, including: determining a unit electrocardiogram signal of a cardiac cycle, a unit radar echo signal at each collection position and a unit heart vibration signal according to the target electrocardiogram signal, the plurality of target heart vibration signals and the plurality of multi-point micro-motion radar echo signals; dividing the unit radar echo signal and the unit heart vibration signal at each collection position into a first sub-wave and a second sub-wave according to time points corresponding to R peaks and T peaks in the unit electrocardiogram signal; in the first sub-wave, determining a maximum value in a first preset time length before a first reference origin as a mitral valve closing feature point, and determining a maximum value in a second preset time length after the first reference origin as an aortic valve opening feature point, the first reference origin being a minimum value point in a first experience time length range after the R peak; in the second sub-wave, determining a maximum value in a third preset time length before a second reference origin as an aortic valve closing feature point, and determining a minimum value in a fourth preset time length after the second reference origin as a mitral valve opening feature point, the second reference origin being a maximum value point in a second experience time length range after the T peak.

[0091] In some embodiments, the plurality of collection positions are: a mitral valve area, a pulmonary valve area, an aortic valve area, a second aortic valve area, and a tricuspid valve area.

[0092] In some embodiments, the step S605 determines the target physiological state information reflecting the measured chest cavity according to the plurality of target heart vibration signals and the surface array radar echo signals, including: determining first physiological state information according to the plurality of target heart vibration signals; determining second physiological state information according to the surface array radar echo signals, the first physiological state information and the second physiological state information being cardiac state information other than the target physiological state information.

[0093] In the embodiments of the present application, the technical effects of the above-mentioned heart surface multi-point joint collection and multi-modal fusion method embodiments can refer to the above-mentioned heart surface multi-point joint collection and multi-modal fusion system. To avoid repetition, it will not be described here.

[0094] Please refer to Figure 7 which shows a hardware structure schematic diagram of an electronic device provided by an example embodiment of the present disclosure. In some examples, the electronic device can be at least one of a smart phone, a smart watch, a desktop computer, a laptop computer, a virtual reality terminal, an augmented reality terminal, a wireless terminal, and a laptop computer. The electronic device has a communication function and can access a wired network or a wireless network. The electronic device can generally refer to one of a plurality of terminals, and those skilled in the art can know that the number of the above-mentioned terminals can be more or less. It can be understood that the electronic device undertakes the calculation and processing work of the technical solutions of the present disclosure, and the present disclosure does not limit this.

[0095] As Figure 7 indicated, the electronic device in the present disclosure can include one or more of the following components: a processor 710 and a memory 720.

[0096] Optionally, the processor 710 connects various parts within the entire electronic device by means of various interfaces and lines, performs various functions of the electronic device and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 720, and calling data stored in the memory 720. Optionally, the processor 710 can be implemented in at least one of the hardware forms of digital signal processing (DSP), field-programmable gate array (FPGA), programmable logic array (PLA). The processor 710 can integrate one or several combinations of central processing unit (CPU), graphics processing unit (GPU), neural-network processing unit (NPU) and baseband chip, etc. Among them, the CPU mainly processes operating systems, user interfaces and application programs, etc.; the GPU is responsible for rendering and drawing the content to be displayed on the touch display screen; the NPU is used to realize the artificial intelligence (AI) function; the baseband chip is used to process wireless communication. It can be understood that the above-mentioned baseband chip can also not be integrated into the processor 710, but be realized by a separate chip.

[0097] The memory 720 can include random access memory (RAM) and can also include read-only memory (ROM). Optionally, the memory 720 includes a non-transitory computer-readable storage medium. The memory 720 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 720 can include a program storage area and a data storage area, wherein the program storage area can store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playing function, an image playing function, etc.), instructions for implementing the above various method embodiments, etc.; the data storage area can store data created according to the use of the electronic device, etc.

[0098] In addition, those skilled in the art can understand that the structure of the electronic device shown in the above-mentioned drawings does not constitute a limitation on the electronic device, and the electronic device can include more or fewer components than the drawings, or combine certain components, or different component arrangements. For example, the electronic device also includes a display screen, a camera assembly, a microphone, a speaker, a radio frequency circuit, an input unit, a sensor (such as an acceleration sensor, an angular velocity sensor, a light sensor, etc.), an audio circuit, a WiFi module, a power supply, a Bluetooth module, and the like, which will not be described here.

[0099] The present disclosure also provides a computer-readable storage medium storing at least one instruction for being executed by a processor to implement the cardiac surface multi-point joint acquisition and multi-modal fusion method according to various embodiments.

[0100] The present disclosure also provides a computer program product including computer instructions stored in a computer-readable storage medium; a processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to cause the electronic device to perform the cardiac surface multi-point joint acquisition and multi-modal fusion method according to various embodiments.

[0101] The present disclosure also provides a chip including a processor and a communication interface, the communication interface and the processor being coupled, the processor being configured to run programs or instructions to implement various processes of the cardiac surface multi-point joint acquisition and multi-modal fusion method according to various embodiments, and achieve the same technical effects. To avoid repetition, details will not be described here.

[0102] It should be understood that the chip mentioned in the embodiments of the present disclosure can also be referred to as a system-level chip, a system chip, a chip system, or a system-on-chip, etc.

[0103] In several embodiments provided by the present disclosure, it should be understood that the disclosed system, device, server and method can be implemented by other means. For example, the above-mentioned device embodiments are only schematic, and the division of the units is only a logical function division, and actual implementation can be in another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units or components shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0104] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, i.e. may be located in one place, or may be distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0105] In addition, each functional unit in various embodiments of the present disclosure can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0106] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0107] Those skilled in the art should realize that in one or more of the above examples, the functions described in the present disclosure can be realized by hardware, software, firmware or any combination thereof. When realized by software, these functions can be stored in a computer readable medium or transmitted as one or more instructions or codes on a computer readable medium. The computer readable medium includes computer storage medium and communication medium, wherein the communication medium includes any medium that facilitates the transmission of computer programs from one place to another. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0108] It should be noted that the technical solutions described in the present disclosure can be combined arbitrarily without conflict.

[0109] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical range disclosed in the present application, which should be covered within the protection scope of the present application.

Claims

1. A heart surface multi-point joint acquisition and multi-modal fusion system, characterized in that, The heart surface multi-point joint acquisition and multi-modal fusion system comprises a heart shock signal sensor, a first signal processing module connected with the heart shock signal sensor, a radar signal sensor, a second signal processing module connected with the radar signal sensor, a data processing module connected with the first signal processing module and the second signal processing module, and an electrocardio signal sensor; The heart shock signal sensor is configured to acquire a plurality of initial heart shock signals at a plurality of acquisition positions of a measured chest cavity, each acquisition position corresponding to an initial heart shock signal; The first signal processing module is configured to pre-process the plurality of initial heart shock signals to obtain a plurality of target heart shock signals; The radar signal sensor is configured to emit a radar signal and acquire an initial radar echo signal returned by the measured chest cavity; The second signal processing module is configured to pre-process the initial radar echo signal to obtain a planar array radar echo signal; The electrocardio signal sensor is configured to acquire an initial electrocardio signal of the measured chest cavity; The data processing module is configured to determine, according to a target electrocardio signal, the plurality of target heart shock signals and a plurality of multi-point micro-motion radar echo signals, an aortic valve opening and closing feature point and a mitral valve opening and closing feature point at each acquisition position, the target electrocardio signal being obtained by pre-processing the initial electrocardio signal; Fusing the aortic valve opening and closing feature point and the mitral valve opening and closing feature point at each acquisition position, target physiological state information is obtained.

2. The system according to claim 1, wherein, The radar signal sensor comprises a radar antenna array and the data processing module is specifically configured to: determine a reference point corresponding to the center of the radar antenna array on the measured chest cavity; determine, according to the distance between the reference point and each acquisition position and a radar transmission distance, a multi-point micro-motion radar echo signal at each acquisition position from the planar array radar echo signal, the radar transmission distance being the distance between the radar signal sensor and the measured chest cavity; determine, according to the target heart shock signal and the multi-point micro-motion radar echo signal at each acquisition position, target physiological state information reflecting the measured chest cavity.

3. The system according to claim 2, wherein, The heart surface multi-point joint acquisition and multi-modal fusion system further comprises a third signal processing module connected with the electrocardio signal sensor; The third signal processing module is configured to pre-process the initial electrocardio signal to obtain a target electrocardio signal; The data processing module is specifically configured to determine, according to the target electrocardio signal, the plurality of target heart shock signals and a plurality of multi-point micro-motion radar echo signals, target physiological state information reflecting the measured chest cavity.

4. The system of claim 1, wherein, The data processing module is specifically configured to: determine, according to the target electrocardio signal, the plurality of target heart shock signals and a plurality of multi-point micro-motion radar echo signals, a unit electrocardio signal of a cardiac cycle, a unit radar echo signal at each acquisition position and a unit heart shock signal; divide the unit radar echo signal and the unit heart shock signal at each acquisition position into a first sub-wave and a second sub-wave according to the time points corresponding to R peaks and T peaks in the unit electrocardio signal; In the first sub-wave, a maximum value within a first preset time length before a first reference origin is determined as a mitral valve closing feature point, and a maximum value within a second preset time length after the first reference origin is determined as an aortic valve opening feature point, the first reference origin being a minimum value point within a first experience time length range after the R peak; In the second sub-wave, a maximum value within a third preset time length before a second reference origin is determined as an aortic valve closing feature point, and a minimum value within a fourth preset time length after the second reference origin is determined as a mitral valve opening feature point, the second reference origin being a maximum value point within a second experience time length range after the T peak.

5. The system according to any one of claims 1 to 4, characterized in that, The plurality of collection positions are: a mitral valve area, a pulmonary valve area, an aortic valve area, a second aortic valve area, and a tricuspid valve area.

6. The system according to any one of claims 1 to 4, characterized in that, The data processing module is specifically configured to determine first physiological state information according to the plurality of target heart vibration signals; Second physiological state information is determined according to the surface array radar echo signal, and the first physiological state information and the second physiological state information are cardiac state information other than the target physiological state information.

7. A method for combined acquisition of multiple points on the surface of a heart and multi-modal fusion, characterized in that, The method comprises: A plurality of initial heart vibration signals are collected at a plurality of collection positions of a measured chest cavity, each collection position corresponding to an initial heart vibration signal; The plurality of initial heart vibration signals are preprocessed to obtain a plurality of target heart vibration signals; A radar signal is emitted, and an initial radar echo signal returned by the measured chest cavity is collected; The initial radar echo signal is preprocessed to obtain a surface array radar echo signal; An initial electrocardiogram signal of the measured chest cavity is collected, and the initial electrocardiogram signal is preprocessed to obtain a target electrocardiogram signal; According to the target electrocardiogram signal, the plurality of target heart vibration signals, and the plurality of multi-point micro-motion radar echo signals, aortic valve opening and closing feature points and mitral valve opening and closing feature points at each collection position are determined; The aortic valve opening and closing feature points and the mitral valve opening and closing feature points at each collection position are fused to obtain target physiological state information.

8. An electronic device, comprising: The device comprises a processor, a memory, and a program or instruction stored on the memory and executable on the processor, and the program or instruction is executed by the processor to implement the steps of the cardiac body surface multi-point joint collection and multi-modal fusion method of claim 7.

9. A computer-readable storage medium, characterized in that, The readable storage medium stores a program or instruction, and the program or instruction is executed by the processor to implement the steps of the cardiac body surface multi-point joint collection and multi-modal fusion method of claim 7.

Images