Gesture recognition intelligent nursing bed system based on deep learning and control method

Abstract

The invention discloses a gesture recognition intelligent nursing bed system based on deep learning and a control method, and the system comprises a nursing bed body, an upper computer and a lower computer. The upper computer comprises an image acquisition module, a gesture recognition main control module and a physiological monitoring module; the lower computer comprises a control module and an execution mechanism; the method comprises the steps of gesture image acquisition and preprocessing; carrying out gesture recognition by using a YOLOv12 deep learning model; mapping and transmitting gesture instructions; adjusting the posture of the nursing bed; and monitoring and transmitting physiological parameters. Gesture control, posture adjustment and real-time physiological monitoring are achieved through deep learning, the convenience, safety and intelligent level of the nursing process are improved, and the diversified requirements of nursing are met.

Description

Technical Field

[0001] The present invention relates to a gesture recognition intelligent nursing bed system based on deep learning and a control method, belonging to the technical field of human-computer interaction. Background Art

[0002] With the accelerating aging of the global population, the demand for high-quality nursing services among the elderly and disabled has increased significantly. This demand places a heavy burden on families and society, especially for long-term bedridden patients, who are unable to adjust their posture and are prone to complications such as bedsores and muscle atrophy.

[0003] Nursing beds are crucial for addressing these challenges. Their core function is to relieve stress, improve blood circulation, and prevent complications by adjusting bed posture (such as back lift, leg lift, and side-to-side turning). However, current nursing beds on the market generally suffer from the following deficiencies: They are limited in functionality, with most offering only basic mechanical adjustments and lacking intelligent features; they are poorly interactive, with traditional beds relying on manual operation or simple remote controls, making them difficult for users, especially those with limited mobility, to operate independently; and they lack comfort and personalization, with existing devices unable to adjust to user needs in real time and lacking health monitoring capabilities, failing to meet the diverse demands of modern nursing care. Summary of the Invention

[0004] In response to the problems existing in the above-mentioned prior art, the present invention provides a gesture recognition intelligent nursing bed system and control method based on deep learning. The system and method can use deep learning to realize gesture control, posture adjustment and real-time physiological monitoring, thereby improving the convenience, safety and intelligence level of the nursing process and meeting the diverse needs of nursing.

[0005] To achieve the above-mentioned object, the present invention provides a gesture recognition intelligent nursing bed system based on deep learning, comprising a nursing bed body, an upper computer and a lower computer, wherein the upper computer comprises an image acquisition module, a gesture recognition main control module and a physiological monitoring module; the lower computer comprises a control module and an actuator;

[0006] The image acquisition module uses a camera to collect user gesture image data in real time to provide input for the system;

[0007] The gesture recognition master module is based on an improved YOLOv12 deep learning model and runs on the Jetson OrinNano platform, and is used to detect gestures on image data and output recognition results;

[0008] The physiological monitoring module uses millimeter wave radar to collect the user's physiological parameters and transmits the data through the wireless communication module;

[0009] The control module is used to receive control instructions sent by the host computer, analyze and generate drive signals;

[0010] The actuator includes an electric hydraulic rod for performing physical adjustment actions of the nursing bed.

[0011] A control method for a gesture recognition intelligent nursing bed system based on deep learning includes the following steps:

[0012] Step 1: Use a high-resolution camera to collect user gesture image data in real time and preprocess the image to generate standardized image data that meets the input requirements of the improved YOLOv12 deep learning model;

[0013] Step 2: Input the preprocessed image into the improved YOLOv12 deep learning model to perform gesture recognition and output the gesture category and its corresponding confidence score;

[0014] Step 3: Based on the identified gesture categories, the improved YOLOv12 deep learning model generates corresponding nursing bed control instructions through predefined mapping relationships and transmits the instructions to the control module using the Jetson Orin Nano platform;

[0015] Step 4: After receiving the command, the control module drives the actuator to adjust the physical posture of the nursing bed to enable the user's back to rise and fall, turn left and right, and lift the legs;

[0016] Step 5: Use the millimeter-wave radar sensor to monitor the user's physiological parameters in real time, including heart rate and respiratory rate, and transmit the data to a mobile terminal or the cloud through a wireless communication module to achieve remote monitoring of the user's physical condition.

[0017] Furthermore, the specific process of step 1 is as follows:

[0018] 1-1. Set the camera frame rate to no less than 30 frames per second and the resolution to no less than 1280×720 pixels;

[0019] 1-2. Scale the image to the set pixel value and normalize the pixel value so that its range falls within the [0,1] interval. The normalization formula is as follows:

[0020]

[0021] Among them, I represents the original pixel value, I max and I min are the minimum and maximum values ​​of the image pixel values, I norm is the normalized pixel value;

[0022] 1-3. Enhance the normalized image. The enhancement formula is as follows:

[0023] I enhance =α·I norm +β;

[0024] Where α is the contrast gain factor, which can be 1.2, and β is the brightness adjustment factor, which can be 0.1. Through the above processing, the recognizability of gesture features under complex lighting conditions is ensured.

[0025] Furthermore, the improvements of the improved YOLOv12 deep learning model in step 2 are: replacing the standard convolution module in A2C2f with full-dimensional dynamic convolution ODconv, and significantly enhancing the expressive power of feature extraction by introducing a dynamic weighting mechanism in the spatial dimension, channel dimension, convolution kernel dimension and filter dimension; introducing the ECA attention mechanism at the end of the backbone network, generating channel attention weights through global average pooling and lightweight 1D convolution, dynamically enhancing gesture-related feature expression, suppressing irrelevant background noise, and further improving the extraction accuracy of subtle gesture features; embedding the Diverse Branch Block module in the detection head, fusing multi-scale features through a multi-branch structure design, and using convolution kernels of different sizes (for example, 1x1, 3x3, 5x5) and diversified branch paths to capture detailed information of small targets (such as gestures), significantly improving the detection accuracy of small targets, and effectively solving the problem of gesture detection in complex backgrounds; optimizing the feature weights of the neck network through the adaptive spatial feature fusion ASFF module to reduce background interference;

[0026] The feature fusion formula of the ASFF module is as follows: Y = α·X1+β·X2+γ·X3; where X1, X2, and X3 are the input features of gesture images of different scales collected by the camera, and α, β, and γ are calculated through the Softmax function to calculate the adaptive weights: α, β, γ = Softmax(Conv 1×1 (X1,X2,X3));

[0027] The improved YOLOv12 deep learning model enhances gesture feature extraction through the efficient channel attention ECA mechanism, and the calculation formula is: c =σ(Conv1D(GAP(x))); where w c is the channel weight, σ is the Sigmoid activation function, GAP is the global average pooling, and Conv1D is the one-dimensional convolution operation;

[0028] The improved YOLOv12 model is adapted to the Jetson Orin Nano platform by optimizing its Backbone and Neck structures. The recognition results include eight types of gestures and confidence levels. The confidence level ranges from 0 to 1, reflecting the reliability of the detection results. The threshold is set to 0.7, and results below this value are filtered out to reduce false positives.

[0029] Furthermore, the working process of the A2C2f module of the improved YOLOv12 deep learning model is as follows:

[0030] The input feature map F is processed by the initial convolution layer and the basic feature map F0 is generated after A2C2f feature extraction;

[0031] The regional attention module divides the feature map F0 into multiple regions and calculates the attention weight A for each region. i =σ(W a ·F i +b a ), where W a and b a is a learnable parameter, σ is the activation function, F i Represents the i-th regional feature map after the feature map F0 is divided into multiple regions. The regional attention module divides the entire feature map F0 into several sub-regions, each sub-region corresponds to an F i , where i is the index of the region, F i represents the feature subset of F0 in a specific area, F i It is a subset of F0, and F0 generates multiple F by mesh division. i , satisfying F0=concat(F1,F2,…,F n )(if there is no overlap), where n is the number of regions, F i The number of channels C′ is equal to the number of channels of F0, and the spatial size h i ×w i It is a sub-region of F0; then the feature map F of each region i Multiply it with the corresponding attention weight to get the weighted feature map F i ′=A i ·F i , then all weighted feature maps are fused to obtain the enhanced feature map F enhanced =∑ i F i '; The enhanced feature map F enhanced With the global feature map F global Fusion is performed to form the final feature representation F final =α·F enhanced +(1-α)·F global, where α is the fusion weight; through this regional attention mechanism, the A2C2f module can dynamically adjust the weights of different regions in the feature map, so as to better capture the global context information and local detail features of the target, and improve the detection accuracy and robustness of the model in gesture recognition.

[0032] The working process of the improved R-ELAN of the YOLOv12 deep learning model is as follows: a residual connection from input to output is introduced inside each block. Combined with the scaling technology, this design optimizes the gradient flow by providing a direct propagation path for the gradient, thereby alleviating the problem of gradient disappearance or explosion, and significantly improving the training stability of the model. The formula is expressed as: f out =f in +β·Block(F in ), where f in represents the input feature, f out represents the output feature, and β represents the scaling factor;

[0033] The neck network of the improved YOLOv12 deep learning model fuses the multi-scale feature maps output by the backbone network through FPN and PANet, and embeds a regional attention module to enhance feature representation. The detection head of YOLOv12 converts the multi-scale feature maps of the neck network into high-precision target classification and bounding box prediction through multi-scale feature fusion and localization operations. The working process is as follows: the detection head upsamples and concatenates the multi-scale feature maps output by the neck network, and then further processes the fused feature maps through the convolution module. This process is repeated at different scales to ensure that both low-level and high-level features contribute to the final detection result. The processed feature maps are input into the detection output layer to complete the classification prediction and localization regression of multi-category targets.

[0034] The improved YOLOv12 deep learning model has been optimized through pruning and quantization. The pruning process removes redundant connections by calculating the L1 norm of each layer's weights. The specific implementation process is as follows:

[0035] For each layer in the model, calculate the absolute value of each weight. The L1 norm is essentially the sum of the absolute values ​​of the weights.

[0036] According to the preset pruning ratio, the number of weights that need to be pruned is determined. Specifically, the kth minimum value of the absolute value of the weight is found as the threshold T, where k is calculated according to the pruning ratio. All weights with absolute values ​​less than T are reset to zero to achieve pruning. This process is expressed by the following formula:

[0037] T = percentile(|W|,p);

[0038] W′=w⊙(|W|>T);

[0039] Where W is the weight matrix, p is the pruning ratio, and ⊙ represents element-by-element multiplication;

[0040] Quantization converts floating-point weights FP32 to integer type INT8, reducing the number of parameters by about 30% compared to the original. Its core is to map floating-point numbers to the integer range. For the conversion from FP32 to INT8, the quantization formula is as follows:

[0041]

[0042] Among them, scale is the scaling factor used to map the dynamic range of floating-point numbers to the range of INT8, and zero_point is the zero-point offset used to ensure that the floating-point zero value is accurately represented after quantization.

[0043] Furthermore, the process of step 3 is as follows: after the improved YOLOv12 deep learning model performs gesture category recognition, the system generates corresponding nursing bed control instructions through a predefined mapping table. For example, the "four-finger" gesture is mapped to the "raise back" instruction, and the "fist" gesture is mapped to the "raise leg" instruction. The mapping relationship is stored in the local SQLite database of Jetson Orin Nano, and the instruction is transmitted to the lower computer through the serial port in JSON format. The transmission data format is as follows: Command = {"gesture":"four","action":"raise back","confidence":0.94};

[0044] When the confidence level is ≥0.7, the system generates instructions according to the mapping table (such as "raise the head of the bed" corresponds to the instruction code "001"), and the instructions are transmitted to the control module through the serial port in JSON format. The serial port communication uses a baud rate of 115200bps to ensure the real-time and reliability of instruction transmission.

[0045] Furthermore, the process of step 4 is as follows: the lower computer uses an STM32F103C8T6 microcontroller, which parses the JSON data after receiving the command and generates a pulse width modulation (PWM) signal to control the electric hydraulic rod. For example, the PWM duty cycle corresponding to the "raise back" command is 70%, which drives the hydraulic rod to raise the bed back to 45 degrees. The adjustment angle is calculated by the following formula:

[0046]

[0047] Where h is the stroke of the hydraulic rod, L is the length of the bed back, and θ is the adjustment angle. After the action is completed, the lower computer feeds back the execution status to the upper computer via the serial port to ensure the accuracy of the action.

[0048] Furthermore, the process of step 5 is as follows:

[0049] 5-1. The system uses the R24BBD1 millimeter-wave radar to monitor the user's heart rate and respiratory rate in real time. The radar operates at a frequency of 24 GHz and a sampling frequency of 10 Hz. The system extracts chest micro-vibration signals through the micro-Doppler effect. The Doppler frequency shift calculation formula is: Among them, f d is the Doppler frequency shift, v is the vibration velocity, is the radar wavelength;

[0050] 5-2. The monitoring data is transmitted to the ThingsCloud platform via the ESP32 module using the Wi-Fi protocol. The data format is as follows: Data = {"heartrate":80,"resprate":16,"timestamp":"2025-03-15T10:00:00"};

[0051] 5-3. If an abnormality is detected (such as heart rate >120 beats / minute or <50 beats / minute), the system will broadcast an alarm "Attention, heart rate is abnormal" through the JQ8900-16P voice module and push a notification to the user's mobile terminal.

[0052] The present invention uses a high-resolution camera to collect the user's gesture image data in real time, and preprocesses the image to generate standardized image data that meets the input requirements of the improved YOLOv12 deep learning model; the preprocessed image is input into the improved YOLOv12 deep learning model to perform gesture recognition and output the gesture category; based on the recognized gesture category, the improved YOLOv12 deep learning model generates corresponding nursing bed control instructions through a predefined mapping relationship, and transmits the instructions to the control module using the Jetson Orin Nano platform; after receiving the instructions, the control module drives the actuator to adjust the physical posture of the nursing bed to enable the user to raise and lower his back, turn left and right, and lift his legs; a millimeter-wave radar sensor is used to monitor the user's physiological parameters, including heart rate and respiratory rate, in real time, and the data is transmitted to a mobile terminal or the cloud through a wireless communication module to achieve remote monitoring of the user's physical condition. Compared with the existing technology, the present invention uses YOLOv12 gesture recognition technology, allowing users to operate the nursing bed through simple gestures, improving convenience and hygiene, and is particularly suitable for people with limited mobility; based on the powerful computing power of the Jetson Orin Nano platform, real-time gesture recognition and posture adjustment are achieved, ensuring low latency and high performance of the system; combined with millimeter-wave radar to monitor heart rate and respiratory rate, remote data management is supported, and the safety and intelligence of nursing are improved; the optimized YOLOv12 model strikes a balance between accuracy and speed, adapts to the resource limitations of the embedded platform, and has high practicality. The present invention performs gesture recognition based on the Jetson Orin Nano edge platform and the improved YOLOv12 deep learning model, and uses deep learning technology to achieve gesture control, posture adjustment, and real-time physiological monitoring, significantly improving the convenience, safety, and intelligence of the nursing process. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 It is a schematic diagram of the system structure of the present invention;

[0054] Figure 2 It is a flowchart of the method of the present invention;

[0055] Figure 3 This is a schematic diagram of the improved YOLOv12 deep learning model structure of the present invention;

[0056] Figure 4 is a schematic diagram of gesture recognition according to an embodiment of the present invention;

[0057] Figure 5 It is a schematic diagram of the interface of an embodiment of the present invention. DETAILED DESCRIPTION

[0058] The present invention will be further described below with reference to the accompanying drawings.

[0059] like Figure 1 As shown, a gesture recognition intelligent nursing bed system based on deep learning includes a nursing bed body, an upper computer and a lower computer, wherein the upper computer includes an image acquisition module, a gesture recognition main control module and a physiological monitoring module; the lower computer includes a control module and an actuator;

[0060] The image acquisition module uses a camera to collect user gesture image data in real time to provide input for the system;

[0061] The gesture recognition master module is based on an improved YOLOv12 deep learning model and runs on the Jetson OrinNano platform, and is used to detect gestures on image data and output recognition results;

[0062] The physiological monitoring module uses millimeter wave radar to collect the user's physiological parameters and transmits the data through the wireless communication module;

[0063] The control module is used to receive control instructions sent by the host computer, analyze and generate drive signals;

[0064] The actuator includes an electric hydraulic rod for performing physical adjustment actions of the nursing bed.

[0065] like Figure 2 As shown, a control method for a gesture recognition intelligent nursing bed system based on deep learning includes the following steps:

[0066] Step 1: Use a high-resolution camera to collect user gesture image data in real time and preprocess the image to generate standardized image data that meets the input requirements of the improved YOLOv12 deep learning model;

[0067] Step 2: Input the preprocessed image into the improved YOLOv12 deep learning model to perform gesture recognition and output the gesture category and its corresponding confidence score;

[0068] The improved YOLOv12 model optimizes its Backbone and Neck structures to adapt to the Jetson Orin Nano platform. Recognition results include eight gesture categories and a confidence level (ranging from 0 to 1). The confidence level reflects the reliability of the detection result. The threshold is set at 0.7; results below this value are filtered to reduce false positives.

[0069] Step 3: Based on the identified gesture categories, the improved YOLOv12 deep learning model generates corresponding nursing bed control instructions through predefined mapping relationships and transmits the instructions to the control module using the Jetson Orin Nano platform;

[0070] When the confidence level is ≥0.7, the system generates instructions according to the mapping table (e.g., “raise the head of the bed” corresponds to the instruction code “001”), and the instructions are transmitted to the control module in JSON format through the serial port (baud rate 115200bps) to ensure real-time performance.

[0071] Step 4: After receiving the command, the control module drives the actuator to adjust the physical posture of the nursing bed to enable the user's back to rise and fall, turn left and right, and lift the legs;

[0072] Step 5: Use the millimeter-wave radar sensor to monitor the user's physiological parameters in real time, including heart rate and respiratory rate, and transmit the data to a mobile terminal or the cloud through a wireless communication module to achieve remote monitoring of the user's physical condition.

[0073] like Figure 3 As shown in the figure, the improvements of the improved YOLOv12 deep learning model are: replacing the standard convolution module in A2C2f with full-dimensional dynamic convolution ODconv, and introducing a dynamic weighting mechanism in the spatial dimension, channel dimension, convolution kernel dimension and filter dimension, which significantly enhances the expression ability of feature extraction; introducing the ECA attention mechanism at the end of the backbone network, generating channel attention weights through global average pooling and lightweight 1D convolution, dynamically enhancing gesture-related feature expression, suppressing irrelevant background noise, and further improving the extraction accuracy of subtle gesture features; embedding the DiverseBranch Block module in the detection head, fusing multi-scale features through multi-branch structure design, and using convolution kernels of different sizes (for example, 1x1, 3x3, 5x5) and diversified branch paths to capture detailed information of small targets (such as gestures), significantly improving the detection accuracy of small targets, and effectively solving the detection problem of gestures in complex backgrounds; optimizing the feature weights of the neck network through the adaptive spatial feature fusion ASFF module to reduce background interference;

[0074] The feature fusion formula of the ASFF module is as follows: γ=α·X1+β·X2+γ·X3; where X1, X2, and X3 are the input features of gesture images of different scales collected by the camera, and α, β, and γ are calculated through the Softmax function to calculate the adaptive weights: α, β, γ=Softmax(Conv 1×1 (X1,X2,X3));

[0075] The improved YOLOv12 deep learning model enhances gesture feature extraction through the efficient channel attention ECA mechanism, and the calculation formula is: c =σ(Conv1D(GAP(x))); where w c is the channel weight, σ is the Sigmoid activation function, GAP is the global average pooling, and Conv1D is the one-dimensional convolution operation.

[0076] Example: The Jetson Orin Nano main control board is used as the main control module. Its core processing unit integrates a 6-core ARMCortex-A78AE 64-bit CPU and a GPU designed based on the Ampere architecture, with 1024 CUDA cores and a 40TOPs NPU. It runs the Ubuntu 20.04 system and is responsible for gesture recognition and data processing. The main control board supports 8GB of LPDDR5 memory and M.2 SSD solid-state drive storage to meet the needs of model operation and data caching. The camera has a resolution of 1280×720 and a frame rate of 30 frames per second. It is equipped with an infrared fill light and supports night mode. It is connected to the Jetson Orin Nano via a USB 3.0 interface with a transmission bandwidth of 5Gbps. The millimeter-wave radar is model R24BBD1, 24GHz frequency band, with a detection range of 0.5-2 meters and power consumption of less than 1W. It communicates with ESP32 via the I2C interface.

[0077] The control module of the lower computer is STM32F103C8T6, with a main frequency of 72MHz, 64KB Flash and 20KBSRAM. It receives instructions through the serial port and outputs PWM signals. The electric hydraulic rod is powered by 24V, has a thrust of 500N, a stroke of 100mm, supports 0°-60° angle adjustment, and a response time of less than 0.1 second. The voice module model is JQ8900-16P, supports 16Mbit storage, and can record multiple voice prompts, such as "Back lifting completed" and "Abnormal heart rate", with an adjustable volume range of 10-30dB.

[0078] The Jetson Orin Nano communicates with the camera via USB and with the STM32F103C8T6 via a UART serial port (baud rate 115200bps). The millimeter-wave radar is connected to the ESP32 via I2C, and the ESP32 interacts with the cloud via Wi-Fi. The power module regulates the external 24V input to 5V and 3.3V, respectively, to power the main control board and peripherals. The hardware layout adopts a double-layer PCB design, and the main control board reserves 10 GPIO interfaces to support functional expansion. During the integration process, signal integrity is tested with an oscilloscope to ensure stable communication.

[0079] Hardware debugging: The camera was connected to the Jetson Orin Nano via a USB interface, and the video stream transmission delay was tested to be less than 20ms. The STM32F103C8T6 was connected to the hydraulic rod via a relay circuit, and an oscilloscope was used to verify the PWM signal waveform, showing a duty cycle error of less than 1%. The millimeter-wave radar communicated with the ESP32 via I2C, and the data transmission rate was tested to reach 10Hz.

[0080] Software debugging: Using PyCharm to debug the YOLOv12 model in Ubuntu 20.04, we verified an inference speed of 50 frames per second. We used STM32CubeIDE to test the control logic, achieving a command response time of less than 0.5 seconds. We also checked data uploads on the ESP32 through the serial port monitor, ensuring that cloud synchronization latency was less than 1 second.

[0081] The system is designed with multiple fault-tolerant logic: if the camera signal is interrupted, it automatically switches to pause mode and a voice prompt of "camera failure" is given; if physiological data is abnormal for 10 consecutive seconds, a "please check your health status" alarm is triggered; if serial port communication fails, the STM32F103C8T6 enters safe mode and stops all actions.

[0082] After the system boots up, the Jetson Orin Nano initializes the camera and YOLOv12 model and enters the gesture recognition loop. Recognition results are sent in JSON format via the serial port to the STM32F103C8T6, which interprets the commands and drives the hydraulic levers to perform the movements. Simultaneously, the ESP32 reads millimeter-wave radar data every second and uploads it to the cloud. If an abnormal physiological parameter is detected, the system activates the voice module to broadcast an alarm and pushes a message to the user terminal via MQTT. The software utilizes a multi-threaded design, allowing image processing, physiological monitoring, and command transmission to run in parallel, ensuring a system response time of less than 0.5 seconds.

[0083] Mapping of control instructions: The system defines 8 types of gestures and their corresponding actions, namely: "four fingers" corresponds to raising the back, with an angle range of 0° to 60°; "fist" corresponds to raising the leg, with an angle range of 0° to 45°; "five fingers open" corresponds to turning to the left, with an angle range of 0° to 30°; "rock gesture" corresponds to turning to the right, with an angle range of 0° to 30°; "two fingers" corresponds to lowering the back, with an angle range of 60° to 0°; "three fingers" corresponds to lowering the legs, with an angle range of 45° to 0°; "OK gesture" corresponds to pausing the current action; and "waving" corresponds to resetting to the initial state.

[0084] PWM signal generation: STM32 generates PWM signals according to the instructions, and the duty cycle is proportional to the adjustment angle. For example, the relationship between the starting angle θ and the duty cycle D is: When θ = 45°, D = 75%. The PWM signal frequency is set to 1kHz with an accuracy of 0.1% to ensure smooth movement of the hydraulic rod. Limit protection is added to the control logic. If the angle exceeds the set range, the PWM signal automatically resets to zero and stops the action.

[0085] Millimeter-wave radar physiological monitoring implementation process: The radar transmits a 24GHz continuous wave signal, receives the reflected signal, and calculates the heart rate and respiratory rate through the micro-Doppler effect. The frequency offset formula is: Where λ = 0.0125 m, v is extracted from the reflected signal using a fast Fourier transform (FFT) with a sampling window of 1 second. Heart rate is calculated by detecting the peak value in the 0.8-2 Hz frequency band, and respiratory rate is calculated by detecting the peak value in the 0.2-0.5 Hz frequency band.

[0086] Anomaly Detection: The normal heart rate range is set to 60-100 beats / minute and the respiratory rate to 12-20 beats / minute. If the heart rate or respiratory rate exceeds the range, the system records the abnormal timestamp and triggers the alarm logic. The anomaly detection algorithm uses a sliding window averaging method with a window size of 5 seconds to reduce instantaneous noise interference;

[0087] After 72 hours of continuous operation, the system maintained a gesture recognition accuracy of over 97%, the hydraulic lever operated without any faults, and the physiological data transmission success rate reached 99.8%. During the test, the system operated normally even when subjected to simulated power fluctuations (±10%), demonstrating its strong robustness. Figure 4 As shown, (a1) to (a3) ​​are before raising the back, back raising gesture recognition and after raising the back, (b1) to (b3) are before flattening the back, back flattening gesture recognition and after flattening the back, (c1) to (c3) are before raising the legs, leg raising gesture recognition and after raising the legs, (d1) to (d3) are before lowering the legs, leg lowering gesture recognition and after lowering the legs, (e1) to (e3) are before turning left, left turning gesture recognition and after turning left, (f1) to (f3) are before turning right, right turning gesture recognition and after turning right.

[0088] Through the ThingsCloud platform, users can view heart rate, respiratory rate and bed angle in real time, and support manual control function. Figure 5 As shown, (a) is the ThingsCloud mobile interface, and (b) is the ThingsCloud cloud interface. The interface adopts an intuitive design, and abnormal data is highlighted in red with a timestamp.

Claims

1. A gesture recognition intelligent nursing bed system based on deep learning, including a nursing bed body, characterized in that: It also includes a host computer and a slave computer, wherein the host computer includes an image acquisition module, a gesture recognition main control module and a physiological monitoring module; the slave computer includes a control module and an actuator; The image acquisition module uses a camera to collect user gesture image data in real time to provide input for the system; The gesture recognition master module is based on an improved YOLOv12 deep learning model and runs on the Jetson OrinNano platform, and is used to detect gestures on image data and output recognition results; The physiological monitoring module uses millimeter wave radar to collect the user's physiological parameters and transmits the data through the wireless communication module; The control module is used to receive control instructions sent by the host computer, analyze and generate drive signals; The actuator includes an electric hydraulic rod for performing physical adjustment actions of the nursing bed.

2. A control method for a gesture recognition intelligent nursing bed system based on deep learning, characterized in that: The steps include: Step 1: Use a high-resolution camera to collect user gesture image data in real time and preprocess the image to generate standardized image data that meets the input requirements of the improved YOLOv12 deep learning model; Step 2: Input the preprocessed image into the improved YOLOv12 deep learning model to perform gesture recognition and output the gesture category and its corresponding confidence score; Step 3: Based on the identified gesture categories, the improved YOLOv12 deep learning model generates corresponding nursing bed control instructions through predefined mapping relationships and transmits the instructions to the control module using the Jetson Orin Nano platform; Step 4: After receiving the command, the control module drives the actuator to adjust the physical posture of the nursing bed to enable the user's back to rise and fall, turn left and right, and lift the legs; Step 5: Use the millimeter-wave radar sensor to monitor the user's physiological parameters in real time, and transmit the data to the mobile terminal or the cloud through the wireless communication module to achieve remote monitoring of the user's physical condition.

3. The control method of the gesture recognition intelligent nursing bed system based on deep learning according to claim 2 is characterized in that: The specific process of step 1 is as follows: 1-1. Set the camera frame rate to no less than 30 frames per second and the resolution to no less than 1280×720 pixels; 1-2. Scale the image to the set pixel value and normalize the pixel value so that its range falls within the [0,1] interval. The normalization formula is as follows: Among them, I represents the original pixel value, I max and I min are the minimum and maximum values ​​of the image pixel values, I norm is the normalized pixel value; 1-3. Enhance the normalized image. The enhancement formula is as follows: I enhance =α·I norm +b; Among them, α is the contrast gain factor and β is the brightness adjustment factor.

4. The control method of the gesture recognition intelligent nursing bed system based on deep learning according to claim 2 is characterized in that: The improvements of the improved YOLOv12 deep learning model in step 2 are: replacing the standard convolution module in A2C2f with full-dimensional dynamic convolution ODconv, introducing a dynamic weighting mechanism in the spatial dimension, channel dimension, convolution kernel dimension and filter dimension; introducing the ECA attention mechanism at the end of the backbone network, generating channel attention weights through global average pooling and lightweight 1D convolution, dynamically enhancing gesture-related feature expressions and suppressing irrelevant background noise; embedding the Diverse Branch Block module in the detection head, fusing multi-scale features through a multi-branch structure design, and using convolution kernels of different sizes and diverse branch paths to capture detailed information of small targets; and optimizing the feature weights of the neck network through the adaptive spatial feature fusion ASFF module to reduce background interference. The feature fusion formula of the ASFF module is as follows: Y = α·X1+β·X2+γ·X3; where X1, X2, and X3 are the input features of gesture images of different scales collected by the camera, and α, β, and γ are calculated through the Softmax function to calculate the adaptive weights: α, β, γ = Softmax(Conv 1×1 (X1,X2,X3)); The improved YOLOv12 deep learning model enhances gesture feature extraction through the efficient channel attention ECA mechanism, and the calculation formula is: c =σ(Conv1D(GAP(x))); where w c is the channel weight, σ is the Sigmoid activation function, GAP is the global average pooling, and Conv1D is the one-dimensional convolution operation; The improved YOLOv12 model is adapted to the Jetson Orin Nano platform by optimizing its Backbone and Neck structures. The recognition results include eight types of gestures and confidence levels. The confidence level ranges from 0 to 1, reflecting the reliability of the detection results. The threshold is set to 0.7, and results below this value are filtered out to reduce false positives.

5. The control method of the gesture recognition intelligent nursing bed system based on deep learning according to claim 4 is characterized in that: The working process of the A2C2f module of the improved YOLOv12 deep learning model is as follows: The input feature map F is processed by the initial convolution layer and the basic feature map F0 is generated after A2C2f feature extraction; The regional attention module divides the feature map F0 into multiple regions and calculates the attention weight A for each region. i =σ(W a ·F i +b a ), where W a and b a is a learnable parameter, σ is the activation function, F i Represents the i-th regional feature map after the feature map F0 is divided into multiple regions; the regional attention module divides the entire feature map F0 into several sub-regions, each sub-region corresponds to an F i ; i is the index of the region, F i Represents the feature subset of F0 in a specific area; F i It is a subset of F0, and F0 generates multiple F by mesh division. i , satisfying F0=concat(F1,F2,…,F n ), where n is the number of regions, F i The number of channels C′ is equal to the number of channels of F0, and the spatial size h i ×w i It is a sub-region of F0; then the feature map F of each region i Multiply it with the corresponding attention weight to get the weighted feature map F i v=A i ·F i , then all weighted feature maps are fused to obtain the enhanced feature map F enhanced =∑ i F i '; The enhanced feature map F enhanced With the global feature map F global Fusion is performed to form the final feature representation F final =α·F enhanced +(1-α)·F global , where α is the fusion weight; The working process of R-ELAN included in the improved YOLOv12 deep learning model is as follows: a residual connection from input to output is introduced within each block, combined with scaling technology, the formula is expressed as: F out =F in +β·Block(F in ), where F in represents the input features, F out represents the output feature, and β represents the scaling factor; The improved YOLOv12 deep learning model's neck network fuses the multi-scale feature maps output by the backbone network through FPN and PANet, and embeds a regional attention module to enhance feature representation. The YOLOv12 detection head converts the multi-scale feature maps of the neck network into high-precision object classification and bounding box prediction through multi-scale feature fusion and localization operations. The working process is as follows: the detection head upsamples and concatenates the multi-scale feature maps output by the neck network, and then further processes the fused feature maps through the convolution module. This process is repeated at different scales. The processed feature maps are input to the detection output layer to complete the classification prediction and localization regression of multi-category objects. The improved YOLOv12 deep learning model has been optimized through pruning and quantization. The pruning process removes redundant connections by calculating the L1 norm of each layer's weights. The specific implementation process is as follows: For each layer in the model, calculate the absolute value of each weight. The L1 norm is essentially the sum of the absolute values ​​of the weights. According to the preset pruning ratio, the number of weights that need to be pruned is determined. Specifically, the kth minimum value of the absolute value of the weight is found as the threshold T, where k is calculated according to the pruning ratio. All weights with absolute values ​​less than T are reset to zero to achieve pruning. This process is expressed by the following formula: T = percentile(|W|,p); W′=W⊙(|W|>T); Where W is the weight matrix, p is the pruning ratio, and ⊙ represents element-by-element multiplication; Quantization converts floating-point weights FP32 to integer type INT8. Its core is to map floating-point numbers to integer ranges. For the conversion from FP32 to INT8, the quantization formula is as follows: Where scale is the scaling factor and zero_point is the zero-point offset, which is used to ensure that floating-point zero values ​​are accurately represented after quantization.

6. The control method of the gesture recognition intelligent nursing bed system based on deep learning according to claim 2 is characterized in that: The process of step 3 is as follows: After the improved YOLOv12 deep learning model performs gesture category recognition, the system generates corresponding nursing bed control instructions through a predefined mapping table and stores the mapping relationship in the local SQLite database of Jetson Orin Nano. The instructions are transmitted to the lower computer through the serial port in JSON format. The transmission data format is as follows: Command="{"gesture":"four","action":"raiseback","confidence":0.94}; When the confidence level is ≥0.7, the system generates instructions based on the mapping table, and the instructions are transmitted to the control module through the serial port in JSON format to ensure real-time performance.

7. The control method of the gesture recognition intelligent nursing bed system based on deep learning according to claim 2, characterized in that: The process of step 4 is as follows: the lower computer uses an STM32F103C8T6 microcontroller, parses the JSON data after receiving the instruction, generates a pulse width modulation (PWM) signal to control the electric hydraulic rod, and the adjustment angle is calculated by the following formula: Where h is the stroke of the hydraulic rod, L is the length of the bed back, and θ is the adjustment angle.

8. The control method of the gesture recognition intelligent nursing bed system based on deep learning according to claim 2, characterized in that: The process of step 5 is as follows: 5-1. The system uses the R24BBD1 millimeter-wave radar to monitor the user's heart rate and respiratory rate in real time, and extracts the chest micro-vibration signal through the micro-Doppler effect. The Doppler frequency shift calculation formula is: Among them, f d is the Doppler frequency shift, v is the vibration velocity, is the radar wavelength; 5-2. The monitoring data is transmitted to the ThingsCloud platform via the ESP32 module using the Wi-Fi protocol. The data format is as follows: Data = {"heartrate":80,"resprate":16,"timestamp":"2025-03-15T10:00:00"}; 5-3. If an abnormality is detected, the system will broadcast an alarm "Attention, abnormal heart rate" through the JQ8900-16P voice module and push a notification to the user's mobile terminal.

Images