Fall detection method and system based on integration of millimeter wave radar and vision sensors

Abstract

The application discloses a fall detection method and system based on integration of millimeter wave radar and visual sensor. The method comprises: collecting multi-modal data for a target, wherein the multi-modal data comprises a video stream obtained by using a visual sensor and point cloud data obtained by using a millimeter wave radar; extracting visual high-level semantic features from the video stream and millimeter wave features containing space-time information from the point cloud data; weighting the visual high-level semantic features and the millimeter wave features to obtain a weighted feature vector, and then obtaining fusion features containing cross-modal information by processing the weighted feature vector; and predicting a fall event based on the fusion features. The application can enhance the applicability of the fall detection system in a wider range of scenarios through multi-modal fusion.

Description

Technical Field

[0001] This invention relates to the field of Internet of Things (IoT) technology, and more specifically, to a fall detection method and system based on the integration of millimeter-wave radar and visual sensors. Background Technology

[0002] Falls among older adults are a significant global health issue, with many incidents resulting in serious injury, hospitalization, or death. Developing efficient fall detection systems is crucial to mitigating these adverse events and their associated economic burden. The main trends in fall detection systems can be categorized into wearable and non-wearable systems based on the invasiveness of the device. For wearable systems, researchers have explored various sensors, such as accelerometers, gyroscopes, and corresponding smart devices, for developing fall detection systems. These systems have demonstrated satisfactory performance due to their ability to detect subtle movements. However, the requirements for wearing and regular charging pose inconveniences for older adults, who often find wearable devices restrictive and are prone to forgetting to wear or charge them. These issues significantly limit the availability and widespread adoption of wearable fall detection systems. Therefore, this paper primarily discusses non-wearable fall detection systems.

[0003] For non-wearable fall detection systems, the most widely discussed technologies are vision-based systems and environmental systems (primarily radio frequency-based systems). For vision-based fall detection systems, the input data mainly consists of images and videos captured by cameras. These systems utilize camera sensors to collect information about body posture, shape, inactivity patterns, and movement. Commonly used camera sensors include RGB cameras, depth cameras, thermal imaging sensors, and combinations thereof.

[0004] In existing research, many approaches utilize the visual information captured by the aforementioned sensors to monitor subjects' head trajectory, changes in body shape, or body posture to achieve accurate fall detection. For example, see the references ( H. Ramirez, S. A. Velastin, I. Meza, E. Fabregas, D. Makris, and G. Farias, "Fall detection and activity recognition using human skeleton features,” Ieee Access, vol. 9, pp. 33 532–33 542, 2021. The research focuses on using human skeletal features combined with machine learning algorithms (such as random forests, support vector machines, multilayer perceptrons, and k-nearest neighbors) to detect falls. This method achieves an average accuracy of 98.5%. In contrast, the literature (… C.-J. Chong, W.-H. Tan, Y. C. Chang, M. F. N. Batcha, and E. Karuppiah, "Visual based fall detection with reduced complexity horprasert segmentation using superpixel,” in 2015 IEEE 12th International Conference on Networking, Sensing and Control. IEEE, 2015, pp. 462–467.The study employed a Horpraset algorithm for background subtraction and utilized a simple linear iterative clustering technique to create superpixel clusters for segmentation, achieving a sensitivity of 66.7% and a specificity of 80%. Despite the excellent performance of vision sensor-based systems, they typically face several major limiting factors, including line-of-sight (LOS) requirements and specific lighting conditions. Occlusion and low-light environments significantly weaken the robustness of the system.

[0005] Environmental sensor-based systems define posture, movement, and human presence by monitoring changes in signals within a specific region of interest (ROI). These sensors include infrared (IR) sensors, acoustic sensors, Wi-Fi devices, and radar systems. In recent years, with the rapid development of wireless sensing technologies, radio frequency (RF) sensor-based systems have received increasing attention.

[0006] RF technology detects falls by tracking fluctuations in RF signals or wireless channel state information (such as WiFi and Bluetooth), as human movement causes abnormal changes in RF signals. To detect falls, for example, signals from multi-antenna frequency-modulated continuous wave (FMCW) radio are analyzed to extract complex spatial and temporal features, and then trained using convolutional neural networks (CNNs). Another example is a fall prevention system based on FMCW radar, which continuously measures the distance between the radar and the surrounding environment, analyzing the relationship between human movement and radar frequency changes to predict falls. Other approaches utilize wireless channel state information to detect fall-related activities, leveraging rapid changes in WiFi or Bluetooth signals for fall detection.

[0007] RF-based fall detection systems are favored for their wide applicability and non-invasiveness. However, these systems are limited to specific areas of interest (ROI) and are susceptible to environmental changes. For example, there are two main problems: wireless networks are typically deployed within the limited area of ​​a home, resulting in coverage limitations; and since environmental sensors detect falls by analyzing RF signal fluctuations caused by a target, environmental anomalies can significantly interfere with system performance, i.e., there is an environment dependency problem.

[0008] Analysis shows that visual and radio frequency technologies have been extensively studied for fall detection systems based on a single sensor. However, due to the different sensing characteristics of these two types of sensors, each technology has its own advantages and disadvantages.

[0009] Vision-based fall detection systems utilize computer vision technology to detect fall events by analyzing video images. However, these systems are highly dependent on lighting conditions. Insufficient light or drastic changes in lighting can significantly impact the system's detection accuracy, thereby reducing its robustness and usability. Furthermore, because vision-based systems make decisions based on input images, they become completely blind and ineffective in occluded scenarios. However, due to privacy concerns, occlusion is quite common in home environments (such as bathrooms and shower rooms).

[0010] RF-based fall detection systems identify falls by transmitting and receiving electromagnetic signals and analyzing changes in reflected signals. However, due to bandwidth limitations, RF signals have limited resolution, making it difficult to accurately represent the human body. Furthermore, RF systems are susceptible to environmental interference because they sense the human body by analyzing reflected electromagnetic waves in the environment. During propagation, various reflections, refractions, and scattering phenomena caused by multipath effects exacerbate the sensitivity of RF systems to the environment. Moreover, the attenuation characteristics of RF signals limit their sensing range and predictive performance in scenarios requiring long-distance sensing.

[0011] In summary, accurate fall detection systems are crucial for addressing falls among the elderly. Since falls can occur anytime, anywhere, a comprehensive system with high applicability across a wide range of scenarios and seamless 24 / 7 operation is needed. However, existing fall detection systems are typically built on single-modal sensors, which inevitably suffer from the inherent limitations of single-modal systems, significantly restricting their application scenarios. Summary of the Invention

[0012] The purpose of this invention is to overcome the shortcomings of the prior art and provide a fall detection method and system based on the integration of millimeter-wave radar and visual sensors.

[0013] According to a first aspect of the present invention, a fall detection method based on the integration of millimeter-wave radar and a visual sensor is provided. The method includes the following steps: For the target, multimodal data is collected, including video streams obtained using a visual sensor and point cloud data obtained using millimeter-wave radar; Extract high-level visual semantic features from the video stream and extract millimeter-wave features containing spatiotemporal information from the point cloud data; The high-level visual semantic features and the millimeter-wave features are weighted to obtain a weighted feature vector, and then the weighted feature vector is processed to obtain a fusion feature containing cross-modal information; Fall events are predicted based on the fused features.

[0014] According to a second aspect of the present invention, a fall detection system based on the integration of millimeter-wave radar and a visual sensor is provided. The system includes a data acquisition module, a single-modal processing module, a multi-modal processing module, and a fall detection module, wherein: the data acquisition module is used to acquire multi-modal data for a target, the multi-modal data including a video stream obtained using a visual sensor and point cloud data obtained using millimeter-wave radar; the single-modal processing module is used to extract high-level visual semantic features from the video stream and millimeter-wave features containing spatiotemporal information from the point cloud data; the multi-modal processing module is used to weight the high-level visual semantic features and the millimeter-wave features to obtain a weighted feature vector, and then obtain a fused feature containing cross-modal information by processing the weighted feature vector; the fall detection module is used to predict fall events based on the fused features.

[0015] Compared with existing technologies, the advantages of this invention lie in the fact that the fall detection scheme based on the integration of millimeter-wave radar and visual sensors takes into account the unique characteristics of visual and RF sensors. By fusing these two modalities, it fully utilizes their complementarity, allowing the advantages of each modality to compensate for the shortcomings of the other. This multimodal fusion enhances the system's applicability in a wider range of scenarios.

[0016] Other features and advantages of the invention will become clear from the following detailed description of exemplary embodiments of the invention with reference to the accompanying drawings. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments of the invention and, together with their description, serve to explain the principles of the invention.

[0018] Figure 1 This is an overall block diagram of a fall detection system based on the integration of millimeter-wave radar and visual sensors according to an embodiment of the present invention; Figure 2 This is a schematic diagram of visual modal processing according to an embodiment of the present invention; Figure 3 This is a schematic diagram of millimeter-wave mode processing according to an embodiment of the present invention; Figure 4 This is a schematic diagram of a multimodal fusion module according to an embodiment of the present invention; Figure 5 This is a flowchart of a fall detection method based on the integration of millimeter-wave radar and visual sensors according to an embodiment of the present invention. Detailed Implementation

[0019] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the invention.

[0020] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the invention or its application or use.

[0021] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0022] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0023] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0024] To overcome the limitations of single-modal fall detection systems, this invention proposes a novel multimodal fall detection system, or VRFfall, which seamlessly integrates millimeter-wave radar and camera technologies. As a system with high generalization capabilities, VRFfall supports both multimodal and single-modal inputs, each with its own independent feature extraction pipeline. VRFfall employs a cross-modal knowledge transfer design, leveraging fusion knowledge from other modalities to enhance the performance of single-modal inputs. Furthermore, to ensure optimal fusion decisions under modal differences, VRFfall integrates an adaptive modal quality assessment module (MQAM), which dynamically evaluates and fuses features from both modalities.

[0025] To overcome the limitations of single-modal methods and improve the overall performance and robustness of the system, this invention designs a multimodal fall detection system integrating visual and millimeter-wave modalities, achieving 24 / 7 continuous fall detection. In summary, the provided fall detection system based on the integration of millimeter-wave radar and visual sensors includes a data acquisition module, a single-modal processing module, a multimodal processing module, and a fall detection module. Specifically: the data acquisition module collects multimodal data for a target, including video streams obtained using a visual sensor and point cloud data obtained using millimeter-wave radar; the single-modal processing module extracts high-level visual semantic features from the video stream and millimeter-wave features containing spatiotemporal information from the point cloud data; the multimodal processing module weights the high-level visual semantic features and the millimeter-wave features to obtain a weighted feature vector, and then processes the weighted feature vector to obtain a fused feature containing cross-modal information; the fall detection module predicts fall events based on the fused feature.

[0026] Specifically, in combination Figure 1 As shown, the data acquisition module integrates two complementary sensors: a commercial camera and a millimeter-wave radar. The camera continuously captures a video stream, visually recording dynamic changes in the scene. Simultaneously, the millimeter-wave radar actively emits and receives electromagnetic waves reflected from objects in the environment, converting them into precise point cloud data through signal processing technology. This point cloud data contains rich spatial depth information about the scene. To ensure data consistency and correlation, the video stream and point cloud data are synchronized in time.

[0027] The single-modal processing module focuses on extracting valuable high-level spatiotemporal features from a single data source, comprising a Vision Pipeline for processing RGB data and an RF Pipeline for point cloud data. For RGB data, the VGG-16 model is used for feature extraction, among other things. VGG-16, with its powerful representation learning capabilities, can automatically capture high-level semantic features in images. For point cloud data generated by millimeter-wave radar, a customized data processing workflow is employed, including denoising and smoothing. This approach effectively suppresses noise interference while preserving crucial spatial geometric information, resulting in low-noise and smooth millimeter-wave features.

[0028] The multimodal processing module (or multimodal fusion module) is responsible for deeply fusing high-level features from different modalities, including an adaptive modality quality assessment unit and a cross-modal knowledge sharing unit. The adaptive modality quality assessment unit adaptively weights features according to their importance, achieving optimal matching and integration of image features and millimeter-wave features in the feature space. The multimodal features generated by this fusion contain rich visual information and possess precise spatial localization capabilities. Finally, these fused features are input into the fall detection network for accurate and rapid fall event prediction.

[0029] The following sections will describe specific embodiments of the single-modal processing module and the multi-modal processing module.

[0030] I. Single-modal processing module For the single-modal processing module, in order to extract refined and semantically meaningful high-level features from the millimeter-wave modality and the visual modality, an independent feature extraction pipeline was implemented for each modality, namely the visual pipeline (or visual module) and the RF pipeline (or RF module or millimeter-wave module).

[0031] See Figure 2 As shown, the visual pipeline mainly includes the processes of high-level semantic feature extraction and visual temporal feature generation.

[0032] 1. High-level semantic feature extraction For high-level semantic feature extraction, a pre-trained VGG-16 model can be used. After removing its final classification layer, it can be used to extract features from RGB images with a resolution of, for example, 640×480 pixels. The VGG-16 model captures rich hierarchical representations through multiple convolutional and pooling layers. For example, the generated feature map has a size of 512×7×7, encoding different features at various locations and scales in the image. To reduce feature dimensionality and computational cost, a global average pooling layer can be applied to efficiently compress each image frame into a concise 512-dimensional feature vector.

[0033] 2. Generation of visual temporal features After obtaining 512-dimensional features, two fully connected (FC) layers can be set up to further compress the spatial features into a 128-dimensional representation, preparing for subsequent temporal feature extraction. For temporal feature extraction, a gated recurrent unit (GRU) is employed. The GRU block manages long-term dependencies in sequence data. The 128-dimensional features of T frames (T being the number of frames in a sample) are continuously input into the GRU block, which then generates 128-dimensional high-level visual semantic features, integrating a precise representation of historical context and current input.

[0034] See Figure 3 As shown, the RF pipeline mainly includes point cloud generation and denoising, and spatiotemporal feature extraction processes.

[0035] 1. Point cloud generation and denoising For frequency modulated continuous wave (FMCW) millimeter-wave radar, the transmitted signal can be represented as: (1) in, Indicates signal power. It is the phase offset constant. , and These represent the starting frequency, the duration of the linear frequency modulated pulse, and the bandwidth, respectively. By comparing the transmitted signal with the distance... The received signals reflected from and received by surrounding objects are mixed. ,in It is the speed of light, which can be used to obtain intermediate frequency (IF) signals: (2) in, It is the amplitude of the IF signal.

[0036] For example, using the TI IWR6843AOPEVM millimeter-wave radar equipped with 3 transmit (Tx) and 4 receive (Rx) channels, the raw data acquired via an analog-to-digital converter (ADC) is processed using advanced signal processing.

[0037] The process of generating 3D point clouds from millimeter-wave radar data involves several advanced signal processing steps. First, a range FFT (Fast Fourier Transform) is applied to ADC samples from the TX-RX pair to generate a range spectrum. Next, static target signals that remain unchanged across multiple consecutive frames are eliminated using a high-speed inter-frame differencing method, removing static clutter (such as reflections from walls or the ground). Then, Capon beamforming is used to generate a range-azimuth heatmap on a fine azimuth grid, while simultaneously estimating the covariance matrix based on the received antenna (RX) data to ensure better angular resolution. Finally, a constant false alarm rate (CFAR) algorithm is employed to detect targets in the range-azimuth heatmap. Common CFAR parameters include the number of training units, the number of guard units, and the false alarm probability (typically 1). or These parameters can be adjusted based on the intensity of ambient noise and the desired intensity of the target to optimize detection performance. For the detected target point, Capon beamforming is applied again to generate a one-dimensional elevation spectrum, and the maximum peak value is selected as the elevation estimate. Finally, Doppler spectrum analysis is performed using a virtual antenna array combined with chirp sequence to estimate the target's radial velocity, ultimately generating a 3D point cloud containing three-dimensional position and velocity information.

[0038] After obtaining the point cloud data, the mean values ​​of the x, y, and z coordinates of each frame of point cloud data are calculated. If no points are detected in a frame (i.e., the point count is 0), the mean value of the previous frame is used as a substitute. Then, Z-score normalization is applied to the coordinates in T frames. Outliers exceeding a threshold of 3 are identified and smoothed by averaging the coordinates of adjacent frames. Finally, the data is further smoothed using a sliding window of size Ts (empirically set to 5).

[0039] 2. Spatiotemporal Feature Extraction To deeply extract and fuse the spatiotemporal features of point clouds, a series of feature extraction processes were adopted. First, point cloud data of T consecutive frames (T being the number of frames in a sample) that have undergone fine signal processing were input into a network consisting of four residual blocks to enhance the model's ability to capture spatial information and improve the richness and robustness of feature representation.

[0040] After the spatial features are fully extracted, these feature sequences are further input into a GRU block to effectively capture the temporal dependencies in the sequence data. At this stage, the GRU block integrates and refines the spatial features between consecutive frames in time, thereby generating a feature representation that contains spatial details and temporal dynamic information.

[0041] After processing by the GRU block, a refined RF feature vector of 128 dimensions is finally obtained. Through this process, complex RF data is transformed into a feature representation that is easy to process and rich in spatiotemporal information.

[0042] II. Multimodal Processing Module For multimodal fusion, adaptive modal quality assessment is used to dynamically evaluate the quality of multimodal data and promote cross-modal knowledge sharing.

[0043] Combination Figure 4 As shown, the multimodal processing module includes adaptive modal quality assessment and cross-modal knowledge sharing processes.

[0044] 1. Adaptive Modal Quality Assessment (MQAM) To achieve adaptive evaluation of different modal qualities, a two-layer fully connected (FC) network is used in MQAM to receive 128-dimensional features generated by the visual pipeline and the RF pipeline respectively, and to perform weighted processing on these features.

[0045] Specifically, two 128-dimensional features are concatenated into a 256-dimensional feature vector. This concatenated feature vector is then weighted through a two-layer fully connected (FC) network with learned weights, generating two 128-dimensional weighted vectors. These two weighted vectors correspond to the original two 128-dimensional features, reflecting the model's evaluation of the quality of each modality feature. The weight learning process is based on a large amount of training data containing visual and radio frequency (RF) information of varying quality levels. Therefore, the model can adjust the weights according to the input quality, achieving adaptive modality quality assessment.

[0046] 2. Cross-modal knowledge sharing Cross-modal knowledge sharing is used to further fuse information from two modalities by processing weighted feature vectors and extracting cross-modal features useful for fall detection tasks.

[0047] Specifically, the weighted feature vector (the result of multiplying two 128-dimensional features by their corresponding weights) is fed into two subsequent fully connected (FC) layers. These FC layers fuse features from different modalities through linear transformations, forming higher-level feature representations that include cross-modal information. During the fusion process, useful information is transferred and shared between different modalities. For example, visual information may provide direct cues about human posture and movement, while radio frequency (RF) information can provide supplementary insights about human motion trajectory and speed. Through cross-modal knowledge sharing, MQAM gains a more comprehensive understanding of fall events, thereby improving the accuracy and robustness of predictions.

[0048] Accordingly, based on the above system, the present invention also provides a fall detection method based on the integration of millimeter-wave radar and a visual sensor. See also ​ As shown, the method includes the following steps: Step S1, for the target, collect multimodal data, the multimodal data including video stream obtained using a visual sensor and point cloud data obtained using millimeter-wave radar; Step S2, extract high-level visual semantic features from the video stream and extract millimeter-wave features containing spatiotemporal information from the point cloud data; Step S3, weight the high-level visual semantic features and the millimeter-wave features to obtain a weighted feature vector, and then obtain a fusion feature containing cross-modal information by processing the weighted feature vector; Step S4, predict fall events based on the fusion feature.

[0049] It should be noted that, without departing from the spirit of this invention, those skilled in the art can make appropriate changes or modifications to the above embodiments. For example, depth information can be added to the visual modal input. The MQAM evaluation module can be replaced with another framework (such as an attention mechanism). Furthermore, the specific structure of the model involved, such as the number of fully connected layers, the number of convolutional layers, and the size of the convolutional kernels, can be set according to actual needs.

[0050] To further verify the effectiveness of this invention, experimental verification was conducted. The verification process utilized a deep learning model developed on the PyTorch platform. Each convolutional layer incorporated a ResBlock with a kernel size of 3 followed by a ReLU activation function to introduce non-linearity. The model was trained using an NVIDIA GeForce RTX 4090 D GPU with a learning rate of 0.001, a batch size of 32, and a maximum of 240 epochs. Cross-entropy loss was chosen as the loss function and optimized using the Adam optimizer.

[0051] This invention was extensively evaluated on datasets collected from 20 volunteers under two environments and three conditions. The results show that the invention achieves excellent performance in fall detection, with accuracy, precision, recall, specificity, and F1 score of 0.9502, 0.9514, 0.9716, 0.9125, and 0.9457, respectively. The overall performance is shown in Table 1 below. Experimental results demonstrate that the system of this invention performs excellently under various environments and conditions, exhibiting high performance and superior generalization ability, showcasing the potential of a 24 / 7 continuous fall detection system.

[0052] Table 1: Overall Performance of the Invention

[0053] In summary, this invention simultaneously supports millimeter-wave radar, visual multimodal, and single-modal inputs, and designs independent feature extraction pipelines for each modality. Furthermore, an Adaptive Modal Quality Assessment (MQAM) module is designed for dynamically evaluating and fusing features from two modalities, ensuring optimal fusion decisions under modal differences. By leveraging cross-modal knowledge transfer, knowledge from another modality is fused, enhancing the performance of single-modal inputs and improving generalization capabilities.

[0054] In summary, given the high incidence and serious consequences of falls among the elderly, this invention expands the applicable scenarios of traditional single-modal fall detection systems through multimodal fusion design, demonstrating its high performance and excellent generalization ability under various environments and conditions. It achieves 24 / 7 monitoring of fall events and can be applied to the field of intelligent health monitoring, with the potential to become a standard configuration in this field.

[0055] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0056] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0057] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0058] The computer program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, Python, etc., and conventional procedural programming languages ​​such as "C" or similar languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions. This electronic circuitry can execute the computer-readable program instructions to implement various aspects of the invention.

[0059] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0060] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0061] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0062] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It will be known to those skilled in the art that implementation in hardware, implementation in software, and implementation using a combination of software and hardware are equivalent.

[0063] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of the invention is defined by the appended claims.

Claims

1. A fall detection method based on the integration of millimeter-wave radar and visual sensors, comprising the following steps: For the target, multimodal data is collected, including video streams obtained using a visual sensor and point cloud data obtained using millimeter-wave radar; Extract high-level visual semantic features from the video stream and extract millimeter-wave features containing spatiotemporal information from the point cloud data; The high-level visual semantic features and the millimeter-wave features are weighted to obtain a weighted feature vector, and then the weighted feature vector is processed to obtain a fusion feature containing cross-modal information; Fall events are predicted based on the fused features.

2. A fall detection system based on the integration of millimeter-wave radar and a visual sensor, comprising a data acquisition module, a single-modal processing module, a multi-modal processing module, and a fall detection module, wherein: The data acquisition module is used to collect multimodal data for the target, including video streams obtained using a visual sensor and point cloud data obtained using millimeter-wave radar. The single-modal processing module is used to extract high-level visual semantic features from the video stream and millimeter-wave features containing spatiotemporal information from the point cloud data; The multimodal processing module is used to weight the visual high-level semantic features and the millimeter-wave features to obtain a weighted feature vector, and then to process the weighted feature vector to obtain a fusion feature containing cross-modal information; The fall detection module is used to predict fall events based on the fused features.

3. The system according to claim 2, characterized in that, The high-level visual semantic features are obtained according to the following steps: Using a pre-trained VGG-16 model, after removing the final classification layer, features are extracted from the input video stream to obtain a spatial feature map; The spatial feature map is compressed using a fully connected layer to obtain a compressed spatial feature map; For the compressed spatial feature map, the long-term dependencies in the data are captured by a gated recursive unit to obtain the high-level visual semantic features.

4. The system according to claim 2, characterized in that, The point cloud data is obtained according to the following steps: For the target, obtain millimeter-wave radar data; For the millimeter-wave radar data, a range spectrum is generated using range FFT, and static target signals that remain unchanged in multiple consecutive frames are eliminated by the inter-frame difference method to remove static clutter. Using Capon beamforming technology, range-azimuth heatmaps are generated on a fine azimuth grid, and the covariance matrix is ​​estimated based on received antenna data; A constant false alarm rate algorithm is used to detect targets in the range-azimuth heatmap; For the detected target point, Capon beamforming technology is applied again to generate a one-dimensional elevation spectrum, and the maximum peak value is selected as the elevation angle estimate. By combining a virtual antenna array with continuous linear frequency modulation for Doppler spectrum analysis, the radial velocity of the target is estimated, and finally 3D point cloud data containing three-dimensional position and velocity information is generated.

5. The system according to claim 2, characterized in that, The millimeter-wave feature containing spatiotemporal information is obtained according to the following steps: The point cloud data is input into a network composed of multiple residual blocks to extract spatial feature sequences; The spatial feature sequence is input into a gated recursive unit to capture the temporal dependence in the sequence data and generate a feature representation containing spatial details and temporal dynamic information as the millimeter-wave feature containing spatiotemporal information.

6. The system according to claim 2, characterized in that, The multimodal processing module obtains the weighted feature vector according to the following steps: The high-level visual semantic features and the millimeter-wave features are concatenated to obtain a concatenated feature vector. For the feature vector of the connection, a weighted vector is generated by applying the learned weights through a first fully connected network. The weighted feature vector is obtained by weighting the high-level visual semantic features and the millimeter-wave features using the weighted vector.

7. The system according to claim 6, characterized in that, The fusion feature is obtained according to the following steps: The weighted feature vector is input into a second fully connected network to fuse features from different modalities through linear transformation, thereby obtaining the fused feature containing cross-modal information.

8. The system according to claim 7, characterized in that, Both the first fully connected network and the second fully connected network contain two fully connected layers.

9. The system according to claim 5, characterized in that, The number of residual blocks is set to 4.

10. A computer-readable storage medium having a computer program stored thereon, wherein, When the computer program is executed by the processor, it implements the steps of the method according to claim 1.

Images