Newborn abnormal motion pattern recognition method based on deep learning

Abstract

The invention discloses a newborn abnormal motion pattern recognition method based on deep learning, and relates to the technical field of computer vision, and the method comprises the following steps: synchronously collecting multi-mode motion data of a newborn in a supine state, the multi-modal motion data comprises a visible light video, an infrared thermal imaging video and a pressure distribution signal from a sensor array below the mattress; according to the invention, through collaborative acquisition and registration of multi-modal data, the interference of package shielding and body position change on video analysis is effectively overcome; through fusion of pressure sensing features and enhanced micro-motion visual features, comprehensive and stable characterization of neonatal micro and specific motion is realized. By introducing a development stage self-adaptive mechanism, the recognition precision and generalization ability of the newborns of different month ages are remarkably improved, and therefore an automatic and high-reliability abnormal movement recognition means fitting the physiological characteristics of the newborns is provided for clinic.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, specifically to a method for recognizing abnormal movement patterns in newborns based on deep learning. Background Technology

[0002] Early assessment of neonatal neuromotor development is crucial for the timely detection of potential risks such as brain injury and metabolic diseases. Among these assessments, the observation and analysis of spontaneous whole-body movement patterns in newborns is an important non-invasive clinical method. Traditionally, this has relied on visual observation and subjective judgment by experienced physicians, which suffers from inefficiency, inconsistent standards, and difficulty in achieving long-term continuous monitoring.

[0003] With the development of computer vision and artificial intelligence technologies, automatic motion assessment methods based on video analytics have emerged. These methods capture videos of newborns' movements using cameras, extract key point information using pose estimation algorithms, and then classify and identify them using time-series models. In addition, some studies have attempted to introduce wearable sensors or mattress pressure sensors to collect and analyze motion data separately.

[0004] However, existing technologies still have significant limitations. First, pure video analysis methods are susceptible to interference from occlusion by wrapping objects, changes in lighting, and irregular body position changes in newborns, leading to inaccurate posture estimation and unstable feature extraction. Second, contact-based wearable sensors may interfere with the infant's natural state, while single-type pressure sensors are insufficient to provide detailed limb movement information. Finally, existing methods are mostly considered as general behavior recognition tasks, failing to fully consider the significant physiological characteristics of newborns, such as small amplitude and specific frequency of movements, as well as dynamic development with corrected gestational age, resulting in insufficient generalization ability and clinical applicability of the models. Summary of the Invention

[0005] Based on this, the purpose of this invention is to provide a deep learning-based method for recognizing abnormal movement patterns in newborns, in order to solve the technical problems in the prior art, such as unstable video analysis caused by newborns being swaddled, changes in body position and light interference, incomplete data representation caused by the limitations of sensor use, and insufficient model recognition accuracy, robustness and clinical applicability caused by not fully considering the subtlety of their movements, frequency specificity and developmental stages.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a method for recognizing abnormal movement patterns in newborns based on deep learning, comprising the following steps: synchronously acquiring multimodal movement data of newborns in a supine position, wherein the multimodal movement data includes visible light video, infrared thermal imaging video, and pressure distribution signals from a sensor array under the mattress; based on the pressure distribution signals, spatial registration and dynamic region extraction are performed on the visible light video and infrared thermal imaging video, and the temporal trajectory of joint points with the newborn's trunk midline as the reference is extracted; the temporal trajectory of joint points is fused with the features extracted from the pressure distribution signals and input into a neural network model; wherein the neural network model integrates a micro-motion feature enhancement module and a developmental stage adaptive classification module; the micro-motion feature enhancement module is used to enhance the effective motion components in the temporal trajectory of joint points based on prior knowledge of the newborn's movement frequency band; the developmental stage adaptive classification module is used to select the corresponding classification weights for recognition based on the newborn's corrected gestational age information corresponding to the input data; and outputting the recognition result of the newborn having an abnormal movement pattern.

[0007] The present invention is further configured such that the spatial registration and dynamic region extraction of the video based on the pressure distribution signal specifically includes: calculating the position of the newborn's trunk midline and core pressure point according to the pressure distribution signal; aligning the synchronously acquired visible light video frames and infrared thermal imaging video frames with the newborn's trunk midline and core pressure point as reference; and dynamically cropping the region of interest containing the newborn's subject according to the aligned video frame sequence.

[0008] The present invention is further configured such that the extraction of the joint point temporal trajectory based on the newborn's trunk midline specifically involves: using the newborn's trunk midline as the longitudinal reference axis and a straight line passing through the core pressure point of the shoulder region and perpendicular to the longitudinal reference axis as the transverse reference axis, establishing a planar reference coordinate system; and transforming the joint point coordinates detected in each video frame to the planar reference coordinate system to form a temporal trajectory.

[0009] The present invention is further configured such that the micro-motion feature enhancement module operates by: performing frequency domain decomposition on the displacement signal in each coordinate direction of the joint time-series trajectory; extracting and reconstructing signal components whose frequencies are within the preset typical spontaneous motion frequency band of newborns.

[0010] The present invention is further configured such that the developmental stage adaptive classification module includes multiple sub-classifiers associated with different corrected gestational age intervals; the step of selecting corresponding classification weights for identification based on the newborn's corrected gestational age information specifically involves: mapping the input corrected gestational age value into a weight vector through a gating function, so as to weight and combine the outputs of the multiple sub-classifiers.

[0011] The present invention is further configured to include the following steps before extracting the timing trajectory of the joints: analyzing the stability and periodicity of the pressure distribution signal to determine whether the newborn is in a continuous supine resting state; and performing subsequent video analysis and recognition steps only when it is determined that the newborn is in the continuous supine resting state.

[0012] The present invention is further configured to include the steps of: associating the identification result with the newborn's electronic health record, and generating a motor development tracking report based on multiple consecutive identification results. In summary, the present invention has the following main beneficial effects: This invention effectively overcomes the interference of swaddling and body position changes on video analysis through multimodal data collaborative acquisition and registration; by fusing pressure sensing features with enhanced micro-motion visual features, it achieves a comprehensive and stable representation of the subtle and specific movements of newborns; by introducing a developmental stage adaptive mechanism, it significantly improves the recognition accuracy and generalization ability of newborns of different ages, thus providing clinical practice with an automated, highly reliable, and physiologically appropriate method for identifying abnormal movements in newborns. Attached Figure Description

[0013] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a diagram illustrating a clinical application scenario of the present invention. Detailed Implementation

[0014] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0015] The overall structure according to the present invention is as follows, such as Figures 1-2 The embodiments are illustrated below.

[0016] I. Data Acquisition and Preprocessing Step S0: State Determination and Triggering. The system continuously monitors the raw pressure distribution signal from the sensor array. It calculates the variance and periodicity of this signal within a preset time window (e.g., 60 seconds) (e.g., through autocorrelation function analysis). If the variance is below a first threshold (indicating stable body posture) and a significant periodic component consistent with the neonatal respiratory rhythm is detected, the neonate is determined to be in a "continuous supine resting state." Only when this condition is met will the system trigger the complete data acquisition and analysis process of subsequent steps S1 to S5, ensuring the quality of the input data.

[0017] Step S1: Synchronous acquisition of multimodal data.

[0018] In practice, a high-resolution fiber optic grating sensor array or piezoelectric film sensor array is laid in a rectangular grid pattern under the mattress of the neonatal incubator or crib. This array continuously collects time-series data on the pressure distribution exerted by the newborn's body on the mattress at a sampling frequency of no less than 10Hz, generating a pressure distribution map sequence.

[0019] Simultaneously, a visible light camera and an infrared thermal imaging camera are fixedly deployed directly above the bed. The two cameras are synchronized via hardware or software timestamps to ensure synchronization with the pressure data acquisition clock. Upon triggering, the system simultaneously records a duration (e.g., 90 seconds) of visible light video (RGB format) and infrared thermal imaging video (thermal radiation intensity matrix format). This yields time-aligned three-source data streams: pressure signal, RGB video, and infrared video.

[0020] Step S2: Collaborative preprocessing and feature extraction.

[0021] S21: Spatial reference benchmark calculation based on pressure signals. For a pressure distribution map sequence obtained in step S1, its time-averaged pressure distribution map is first calculated. In this average map, the main areas of pressure concentration are located using image processing techniques (such as thresholding and morphological operations). By calculating the second-order central moments of the pressure regions, the trunk midline (usually a straight line) representing the orientation of the newborn's trunk is fitted. At the same time, the core pressure points (i.e., local pressure maxima) corresponding to the main load-bearing areas such as the shoulders and hips are identified.

[0022] S22: Spatial registration and dynamic ROI extraction of video data. The torso midline and core pressure points (especially shoulder points) calculated in S21 are used as spatial references. For each RGB and infrared video frame at a synchronization time stamp, the following operations are performed: First, the coordinates of the pressure sensing plane (mattress plane) are mapped to the image plane of each video using pre-calibrated camera parameters. Then, using the torso midline direction and core pressure point positions mapped to the image plane as references, affine transformations are performed on the RGB and infrared frames to achieve pixel-level spatial registration between them. Next, a rectangular region is dynamically defined as the region of interest, centered on the torso midline in the registered image. The size of this region can be adaptively adjusted according to a preset ratio to the torso length to ensure that the newborn subject is always fully contained and irrelevant background is cropped out to the maximum extent.

[0023] S23: Joint Temporal Trajectory Extraction Based on a Reference Coordinate System. Within the ROI of each frame of the registered RGB video obtained in S22, a pre-trained neonatal pose estimation model (such as a model adapted based on HRNet or OpenPose architecture) is used to detect the pixel coordinates of multiple predetermined joints, including the head, neck, shoulder, elbow, wrist, hip, knee, and ankle. Subsequently, a planar reference coordinate system is established with the trunk midline as the vertical axis (Y-axis) and a straight line passing through the core pressure point of the shoulder and perpendicular to the vertical axis as the horizontal axis (X-axis). The origin of this coordinate system can be set at the intersection of the two axes. The detected joint pixel coordinates are calculated in the planar reference coordinate system by inverse affine transformation and coordinate transformation. This process is repeated for all video frames to generate the displacement temporal trajectory of each joint in the X and Y directions.

[0024] S24: Enhancement and Extraction of Micro-motion Feature Components. For the displacement time-series signal (denoted as s(t)) of each joint point in each coordinate axis direction obtained in S23, prior-based filtering enhancement is implemented. Specifically, the Empirical Mode Decomposition (IMF) method is used to process s(t), obtaining a series of intrinsic mode functions (IMFs). Typical spontaneous movements of newborns (such as limb shaking and arm extension) typically range in frequency from 0.1Hz to 2Hz. IMF components with center frequencies falling within this preset frequency band are selected and reconstructed into a new signal s'(t). This s'(t) is the micro-motion feature component, which enhances motion information relevant to the assessment and suppresses high-frequency noise (such as sensor noise) and low-frequency drift (such as slow overall movement).

[0025] Meanwhile, pressure change characteristics are extracted from the original pressure distribution time series data in step S1, such as the overall pressure center trajectory, the variance of pressure changes in each quadrant, and time series statistics such as the symmetry index of pressure distribution.

[0026] II. Construction and Training of Neural Network Models Step S3: Construct a neural network model that incorporates prior physiological knowledge of newborns.

[0027] This model can be implemented using PyTorch or TensorFlow frameworks, and its structure is designed as follows: Input layer: Receives two parts of data. The first part is the micro-motion feature components of all joints after processing by S24 (multi-channel time-series signal). The second part is the pressure change features extracted by S24 (multi-dimensional time-series or statistical features).

[0028] Multi-source feature fusion coding layer (S31): This layer first uses a one-dimensional convolutional layer to process the multi-channel micro-motion temporal signals separately, extracting their high-order spatiotemporal features. Simultaneously, a fully connected layer is used to encode the pressure change features. Then, the feature vectors from both are concatenated along their feature dimensions and fused and dimensionality-reduced through a fully connected layer, outputting a unified fused feature vector.

[0029] Developmental Stage Adaptive Gating Layer (S32): This layer receives neonatal corrected gestational age information (in weeks, e.g., 38 weeks) from the fused feature vector and external input. A gestational age routing function is predefined within the layer. This function discretizes continuous corrected gestational age values ​​and maps them to several predefined developmental stage intervals (e.g., interval 1: 34-36 weeks, interval 2: 37-40 weeks, interval 3: 41-44 weeks). For each interval, independent sub-network channels (e.g., different groups of fully connected layers) are set up in parallel within the model. The routing function calculates a weight vector based on the input corrected gestational age. The layers are weighted according to their corresponding gestational age intervals, with the highest weights for the intervals and lower weights for others (e.g., generated using a softmax function). Ultimately, the output of this layer is a weighted sum of the outputs of each sub-network channel. ,in , , These are the transformation functions for each sub-channel.

[0030] Anomaly Pattern Recognition Layer (S33): This layer consists of multiple sub-classifiers, which can be associated with or independently configured with the sub-network channels in S32. It receives the output of the gating layer. In this embodiment, this layer uses a softmax classifier, outputting a probability distribution vector. , respectively, represent the probability of belonging to normal movement or various abnormal movements (such as "monotonous GMs", "spasmodic-synchronous GMs", etc.).

[0031] Step S4: Model training.

[0032] A large number of neonatal multimodal data samples, labeled by clinical experts (labeling motor category and corrected gestational age), were collected to form training, validation, and test sets. During training, the predicted category was calculated using forward propagation, and the loss value was calculated using the cross-entropy loss function. All parameters of the model (including the parameters of each sub-network channel and gating function) were iteratively optimized using a backpropagation algorithm (such as the Adam optimizer) until the model's performance on the validation set stabilized. The training process explicitly required the model to learn the differences in motor patterns at different developmental stages.

[0033] III. Model Reasoning and Post-processing Step S5: Abnormal motion pattern identification.

[0034] For newborns to be evaluated, first ensure they are in a suitable condition (this can be determined through step S0 or confirmed by a nurse). Then, execute steps S1 to S24 to obtain their micromotor feature components and pressure change features, and acquire their corrected gestational age. Input the features and gestational age into the neural network model trained in step S4. After forward propagation, the model outputs a probability distribution from the abnormal pattern recognition layer. The category with the highest probability is taken as the recognition result. Simultaneously, a confidence threshold (e.g., 0.7) can be set; when the highest probability falls below this threshold, a "result uncertain" message is output.

[0035] Step S6: Result association and report generation.

[0036] The identification results (category, confidence level), corresponding timestamps, and newborn's unique identifier obtained from S5 are automatically written into the newborn's electronic health record via the hospital information system interface. The system periodically (e.g., daily or weekly) retrieves the historical identification results of the same newborn, plots the trend of its motor pattern category over time / developmental age, and automatically generates a structured motor development tracking report. The report may include trend analysis, risk warning levels (e.g., low, medium, high), etc., for clinicians' reference.

[0037] This invention details a complete and operable deep learning-based automatic identification system for abnormal neonatal movement patterns. Starting with simultaneous acquisition of multimodal data (pressure, visible light, infrared), the system creatively utilizes pressure signals to achieve spatial registration and stable feature extraction from video, and establishes a trunk reference coordinate system tailored to the supine position of newborns. By performing micro-motion enhancement on the movement trajectory based on prior frequency bands, it effectively focuses on assessing relevant movement components. The core of the system lies in constructing a neural network model that integrates developmental prior knowledge. This model, through a unique developmental stage adaptive gating mechanism, can dynamically adjust the identification logic according to the corrected gestational age, thereby achieving accurate discrimination of neonatal movement patterns at different developmental stages. Ultimately, the system not only outputs immediate identification results but also links to electronic health records and generates longitudinal developmental tracking reports. The entire implementation process is logically rigorous, forming a closed loop from data quality control, feature engineering, model design to clinical decision support, fully realizing the practicality, innovation, and reproducibility of the method.

[0038] Although embodiments of the present invention have been shown and described, these specific embodiments are merely explanations of the invention and are not intended to limit it. The specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. After reading this specification, those skilled in the art may make modifications, substitutions, and variations to the embodiments as needed without departing from the principles and spirit of the invention, but such modifications, substitutions, and variations are protected by patent law as long as they are within the scope of the claims of the present invention.

Claims

1. A method for recognizing abnormal movement patterns in newborns based on deep learning, characterized in that, Includes the following steps: Simultaneously acquire multimodal motion data of newborns in a supine position. The multimodal motion data includes visible light video, infrared thermal imaging video, and pressure distribution signals from a sensor array under the mattress. Based on the pressure distribution signal, spatial registration and dynamic region extraction are performed on the visible light video and infrared thermal imaging video, and the temporal trajectory of the joint points based on the central axis of the newborn's trunk is extracted. The temporal trajectory of the joint points is fused with the features extracted from the pressure distribution signal and then input into the neural network model; The neural network model integrates a micro-motion feature enhancement module and a developmental stage adaptive classification module. The micro-motion feature enhancement module is used to enhance the effective motion components in the temporal trajectory of the joint points based on prior knowledge of neonatal motion frequency bands. The developmental stage adaptive classification module is used to select the corresponding classification weight for identification based on the newborn corrected gestational age information corresponding to the input data. Output the identification results of the abnormal movement pattern of the newborn.

2. The method for recognizing abnormal movement patterns in newborns based on deep learning according to claim 1, characterized in that, The spatial registration and dynamic region extraction of video based on pressure distribution signals specifically includes: The positions of the neonatal trunk midline and core pressure points are calculated based on the pressure distribution signal. Using the neonatal trunk midline and core pressure point as references, the synchronously acquired visible light video frames and infrared thermal imaging video frames are aligned. Based on the aligned video frame sequence, the region of interest containing the newborn subject is dynamically cropped.

3. The method for recognizing abnormal movement patterns in newborns based on deep learning according to claim 1, characterized in that, The extraction of the temporal trajectory of joint points based on the midline of the newborn's trunk is specifically as follows: A planar reference coordinate system is established using the central axis of the newborn's torso as the longitudinal reference axis and a straight line passing through the core pressure point of the shoulder area and perpendicular to the longitudinal reference axis as the transverse reference axis. The coordinates of the key points detected in each video frame are transformed into the planar reference coordinate system to form a time-series trajectory.

4. The method for recognizing abnormal movement patterns in newborns based on deep learning according to claim 1, characterized in that, The micro-motion feature enhancement module operates as follows: Frequency domain decomposition is performed on the displacement signal in each coordinate direction of the time-series trajectory of the joint point; Extract and reconstruct signal components whose frequencies fall within the preset frequency band of typical spontaneous movements in newborns.

5. The method for recognizing abnormal movement patterns in newborns based on deep learning according to claim 1, characterized in that, The developmental stage adaptive classification module includes multiple sub-classifiers associated with different corrected gestational age intervals; The process of selecting corresponding classification weights based on the newborn's corrected gestational age information for identification is as follows: A gating function maps the input corrected gestational age value to a weight vector, which is then used to weight and combine the outputs of the multiple sub-classifiers.

6. The method for recognizing abnormal movement patterns in newborns based on deep learning according to claim 1, characterized in that, Before extracting the temporal trajectory of the joints, the following steps are also included: Analyze the stability and periodicity of the pressure distribution signal to determine whether the newborn is in a continuous supine resting state; Subsequent video analysis and recognition steps are performed only when the patient is determined to be in the sustained supine resting state.

7. A method for recognizing abnormal movement patterns in newborns based on deep learning according to any one of claims 1 to 6, characterized in that, It also includes the following steps: The identification results are associated with the newborn's electronic health record, and a motor development tracking report is generated based on multiple consecutive identification results.

Images