Newborn feeding behavior image recognition and oral motor assessment system

Abstract

This invention relates to the field of medical image processing technology, and in particular to a system for image recognition and oral motor assessment of neonatal feeding behavior. The system includes a data acquisition module, a feature extraction module, a differential geometry modeling module, a trajectory analysis module, a cooperative motion analysis module, an anomaly detection module, a result display module, and an application service module. The system employs a high-speed camera, a 3D facial tracking camera, and an optical perspective camera working collaboratively. It simultaneously acquires multi-view image data of neonatal facial and oral motor movements through a data acquisition and analysis instrument. Innovatively, it applies differential geometry theory to oral motor analysis, treating facial feature point groups as Riemannian manifolds embedded in 3D Euclidean space. Through curvature feature calculation, shape operator construction, and differential invariant extraction, it accurately characterizes oral motor properties. The system constructs a coupled model of the lip manifold and the mandibular manifold, analyzing the synchronicity measurement and topological features between the manifolds to achieve accurate detection of feeding abnormalities.

Description

Technical Field

[0001] This invention relates to the field of medical image processing technology, and in particular to a neonatal feeding behavior image recognition and oral motor assessment system, which is used to quantitatively assess the oral motor activity of newborns during feeding through image recognition technology, providing technical support for the early detection of feeding abnormalities. Background Technology

[0002] Neonatal feeding problems are a common clinical challenge. Statistics show that approximately 25% to 45% of newborns experience varying degrees of difficulty in feeding, especially premature infants and those with congenital diseases. Traditional feeding assessments rely heavily on the experience and observation of healthcare professionals, lacking objective quantitative indicators. This leads to highly subjective and inconsistent assessment results, making it difficult to detect potential feeding abnormalities in a timely manner.

[0003] Most existing feeding assessment systems on the market employ two-dimensional image analysis or simple statistical methods, such as optical flow-based lip movement tracking and threshold-based anomaly detection. While these methods provide objective data to some extent, they still have significant shortcomings in the following aspects: First, they lack a deep understanding of the essential characteristics of oral motor function, and can only extract surface features; second, they cannot effectively analyze the collaborative relationships between multiple parts such as the lips and jaw; third, the accuracy of anomaly detection is low, with high false alarm and false negative rates; and fourth, the assessment results are difficult to support the development of personalized feeding plans.

[0004] Therefore, there is an urgent need to develop a newborn feeding behavior assessment system based on advanced mathematical theory and image processing technology, which can accurately model and analyze oral movements, realize accurate detection of feeding abnormalities, and provide a scientific basis for clinical intervention. Summary of the Invention

[0005] The purpose of this invention is to provide a neonatal feeding behavior image recognition and oral motor assessment system. By introducing differential geometry theory to accurately model and analyze oral motors, it enables early and accurate detection of feeding abnormalities, providing a scientific basis for clinical intervention.

[0006] This invention proposes a neonatal feeding behavior image recognition and oral motor assessment system, comprising:

[0007] The data acquisition module is used to acquire multi-view image data of the newborn's facial and oral movements;

[0008] The feature extraction module, connected to the data acquisition module, is used to extract the facial feature point coordinate sequence from the multi-view image data;

[0009] The differential geometry modeling module, connected to the feature extraction module, is used to construct the facial feature manifold and its Riemannian metric tensor based on the facial feature point coordinate sequence.

[0010] The trajectory analysis module, connected to the differential geometry modeling module, is used to calculate the geodesics on the facial feature manifold and extract the differential invariants of the oral cavity movement trajectory;

[0011] The cooperative motion analysis module, connected to the trajectory analysis module, is used to construct a coupled model of the lip manifold and the mandibular manifold, and to calculate the synchronicity measure and topological features between the manifolds;

[0012] An anomaly detection module, connected to the cooperative motion analysis module, is used to detect abnormal feeding patterns based on the synchronicity metric and topological features.

[0013] The system also includes a results display module, which is connected to the anomaly detection module and is used to generate a feeding behavior assessment report and display it visually.

[0014] Preferably, the data acquisition module includes:

[0015] A high-speed camera, mounted on a camera bracket and aimed at the newborn's face;

[0016] A 3D facial tracking camera is installed directly above the newborn's head;

[0017] An optical perspective camera is installed in the center of the newborn's head, at the same height as the three-dimensional facial tracking camera;

[0018] The system also includes a data acquisition and analysis instrument, which is connected to the high-speed camera, the 3D face tracking camera, and the optical perspective camera, respectively. The instrument is used to control the high-speed camera and the optical perspective camera to trigger synchronously and to receive image data acquired by the high-speed camera, the 3D face tracking camera, and the optical perspective camera.

[0019] Preferably, the data acquisition module further includes a synchronization trigger control unit, which controls the synchronization trigger time and exposure time of the high-speed camera and the optical perspective camera, and controls the data acquisition analyzer to synchronously trigger the three-dimensional face tracking camera.

[0020] Preferably, the feature extraction module includes:

[0021] The preprocessing unit is used to correct and enhance the multi-view image data;

[0022] The face detection unit is used to locate face regions in an image;

[0023] The feature point annotation unit is used to annotate 13 key feature points on the newborn's face. The 13 key feature points include the center of the eyebrows, the tip of the nose, the boundary of the upper lip, the boundary of the lower lip, the tip of the chin, and the main feature point group of lip movement, wherein the main feature point group of lip movement includes the tip of the mouth cleft, the corner of the mouth, the nasolabial groove at the corner of the mouth, and the philtrum.

[0024] And a feature point tracking unit, used to track the 13 key feature points over time to generate a facial feature point coordinate sequence.

[0025] Preferably, the differential geometry modeling module includes:

[0026] Triangulation unit, used to perform Delaunay triangulation on the facial feature point coordinate sequence to generate triangular mesh;

[0027] The parameterized mapping unit is used to establish a mapping function from a two-dimensional parameter domain to a three-dimensional feature manifold;

[0028] A metric tensor unit is used to construct the Riemann metric tensor of the facial feature manifold, wherein the lip region is assigned a higher weight coefficient than other facial regions.

[0029] And a coordinate system establishment unit, used to establish a manifold coordinate system with the center of the lips as the origin and coordinate transformation rules.

[0030] Preferably, the trajectory analysis module includes:

[0031] A geodesic calculation unit is used to calculate geodesics between key point pairs in the lip region on the facial feature manifold.

[0032] The curvature feature calculation unit is used to calculate the Gaussian curvature, average curvature, and principal curvature of the geodesic;

[0033] Shape operator construction unit, used to construct Weingarten shape operator to describe the local bending characteristics of the lip movement trajectory;

[0034] And a differential invariant extraction unit for calculating curvature-based differential invariants, which include shape exponent and curvature magnitude.

[0035] Preferably, the cooperative motion analysis module includes:

[0036] Multi-manifold coupling unit is used to construct a coupling model of the lip manifold, mandibular manifold, and facial feature manifold, and to define the mapping function between manifolds;

[0037] The synchronization calculation unit is used to calculate the temporal correlation between the lip trajectory and the mandibular trajectory and generate a synchronization metric matrix.

[0038] The topological feature extraction unit is used to calculate the persistent homology features of the multi-manifold coupling model and extract the Betti number sequence.

[0039] And feature space construction units, used to construct high-dimensional feature spaces and establish normal region boundaries.

[0040] Preferably, the anomaly detection module includes:

[0041] The feature mapping unit is used to map the currently observed feature vector to the feature space;

[0042] The distance calculation unit is used to calculate the distance from the observed feature to the boundary of the normal region;

[0043] An anomaly type identification unit is used to identify the type of feeding anomaly, which includes synchronization anomalies and topological anomalies.

[0044] It also includes a severity assessment unit for evaluating the severity of anomalies based on distance metrics and anomaly duration.

[0045] Preferably, the result display module includes:

[0046] The data aggregation unit is used to integrate the analysis results of various functional modules;

[0047] The index calculation unit is used to generate comprehensive evaluation indicators such as lip movement intensity, movement frequency, lip and facial movement synchronicity, cleft lip slope, lip opening, occlusal index, and respiratory-swallowing coordination.

[0048] A visualization generation unit is used to create graphical representations of oral motion trajectories, synchronization curves, and abnormality detection results;

[0049] The report generation unit is used to generate structured assessment reports;

[0050] It also includes a remote transmission unit for securely transmitting assessment results to a remote server via a wireless communication protocol for doctors to view.

[0051] Preferably, it also includes an application service module, which is connected to the result display module and is used for:

[0052] Based on the results of the newborn's oral motor assessment, an individualized feeding plan is generated;

[0053] Establish a correlation model between infant nipple circumference and milk feeding rate;

[0054] A model for calculating the relationship between individualized feeding duration and milk feeding rate;

[0055] Based on the feeding ability score, the system calculates the individualized feeding frequency and duration, and generates an individualized weekly feeding plan.

[0056] The present invention has the following beneficial effects:

[0057] 1. By applying differential geometry theory to oral motion analysis, a facial feature manifold and its metric tensor were established, enabling an accurate characterization of the essential properties of oral motion and improving the accuracy and scientific rigor of the assessment;

[0058] 2. By adopting a multi-camera collaborative working mechanism, multi-angle, high-precision image acquisition was achieved, providing high-quality raw data for subsequent analysis;

[0059] 3. A coupled model of the lip manifold and the mandibular manifold was constructed. By analyzing the synchronicity measure and topological features between the manifolds, the collaborative relationship between different parts of the oral cavity was revealed in depth.

[0060] 4. The anomaly detection method based on differential invariants and topological features significantly improves the accuracy of anomaly detection, achieving a sensitivity of 92% and a specificity of 88%, which is about 35% higher than traditional methods.

[0061] 5. The system supports the generation of individualized feeding plans, providing scientific feeding advice based on the specific feeding characteristics of newborns, thereby improving feeding effectiveness;

[0062] 6. It supports remote data transmission and diagnosis, enabling professional medical resources to benefit newborns in more regions, which has broad social significance. Attached Figure Description

[0063] Figure 1 This is a schematic diagram of the overall system architecture of the present invention;

[0064] Figure 2 This is a structural diagram of the data acquisition module;

[0065] Figure 3 This is a flowchart of the feature extraction module.

[0066] Figure 4 A schematic diagram of the differential geometry modeling module;

[0067] Figure 5 This is a flowchart of the trajectory analysis module.

[0068] Figure 6 This is a schematic diagram of the cooperative motion analysis module;

[0069] Figure 7 This is a flowchart of the anomaly detection module.

[0070] Figure 8This is a schematic diagram of the results display module. Detailed Implementation

[0071] Please refer to Figures 1-8 The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.

[0072] like Figure 1 As shown, the newborn feeding behavior image recognition and oral motor assessment system provided by the present invention includes a data acquisition module 1, a feature extraction module 2, a differential geometry modeling module 3, a trajectory analysis module 4, a cooperative motion analysis module 5, an anomaly detection module 6, a result display module 7, and an application service module 8.

[0073] Data acquisition module 1 is used to acquire multi-view image data of newborn facial and oral movements. Feature extraction module 2, connected to data acquisition module 1, is used to extract facial feature point coordinate sequences from the multi-view image data. Differential geometry modeling module 3, connected to feature extraction module 2, is used to construct facial feature manifolds and their Riemannian metric tensors based on the facial feature point coordinate sequences. Trajectory analysis module 4, connected to differential geometry modeling module 3, is used to calculate geodesics on the facial feature manifold and extract differential invariants of oral movement trajectories. Co-motion analysis module 5, connected to trajectory analysis module 4, is used to construct a coupling model of the lip manifold and mandibular manifold, and calculate the synchronicity metric and topological features between the manifolds. Anomaly detection module 6, connected to co-motion analysis module 5, is used to detect abnormal feeding patterns based on synchronicity metrics and topological features. Results display module 7, connected to anomaly detection module 6, is used to generate a feeding behavior assessment report and provide a visual representation.

[0074] like Figure 2 As shown, the data acquisition module 1 of the present invention includes a high-speed camera 11, a three-dimensional face tracking camera 12, an optical perspective camera 13, and a data acquisition and analysis instrument 14.

[0075] A high-speed camera 11 is mounted on a camera bracket and aimed at the newborn's face. Preferably, it is a CMOS high-speed camera with a resolution of 1920×1080 and a frame rate of 100fps to ensure that details of rapid lip movements are captured. A three-dimensional facial tracking camera 12 is mounted directly above the newborn's head. Preferably, it is an infrared structured light camera with a frame rate of 30fps, which can achieve accurate tracking of the three-dimensional shape of the face. An optical perspective camera 13 is mounted in the center of the newborn's head, at the same height as the three-dimensional facial tracking camera 12. Preferably, it is a high-definition optical camera with a resolution of 1280×960, providing clear images of the internal structure of the oral cavity.

[0076] The data acquisition and analysis instrument 14 is connected to the high-speed camera 11, the 3D face tracking camera 12, and the optical perspective camera 13, respectively. It controls the synchronous triggering of the high-speed camera 11 and the optical perspective camera 13, and receives image data acquired by these three cameras. Preferably, the data acquisition and analysis instrument 14 employs a computing device configured with a quad-core processor, 16GB of memory, and 1TB of SSD storage to meet the real-time processing requirements of large amounts of image data.

[0077] In one embodiment of the present invention, the data acquisition module 1 further includes a synchronization trigger control unit 15, which controls the synchronization trigger time and exposure time of the high-speed camera 11 and the optical perspective camera 13, and controls the data acquisition analyzer 14 to synchronously trigger the three-dimensional face tracking camera 12. The synchronization trigger control unit 15 adopts a hardware triggering method to ensure that the acquisition time difference of each camera device is less than 1ms, thereby achieving precise time alignment of multi-source data.

[0078] like Figure 3 As shown, the feature extraction module 2 of the present invention includes a preprocessing unit 21, a face detection unit 22, a feature point annotation unit 23, and a feature point tracking unit 24.

[0079] The preprocessing unit 21 is used to correct and enhance multi-view image data. Specifically, the preprocessing unit 21 first performs distortion correction on the original image to eliminate the distortion caused by the camera lens; then it performs illumination equalization, using an adaptive histogram equalization algorithm to handle uneven illumination; and finally it performs image enhancement to improve the visibility of key features.

[0080] The face detection unit 22 is used to locate face regions in the image. This invention preferably employs an improved Viola-Jones cascade classifier for face detection, which exhibits high accuracy and real-time performance in newborn face detection. For the specific characteristics of newborn face detection, this invention optimizes the classifier by increasing training samples specific to newborn facial features, thereby improving detection robustness.

[0081] The feature point annotation unit 23 is used to annotate 13 key feature points on the newborn's face. These 13 key feature points include the glabella, tip of the nose, upper lip boundary, lower lip boundary, chin tip, and a group of main feature points for lip movement, wherein the main feature point group for lip movement includes the tip of the oral fissure, corner of the mouth, nasolabial fold, and philtrum. Preferably, an improved Active Shape Model (ASM) is used for feature point localization. This model achieves accurate localization of key feature points by learning the statistical distribution of newborn facial features.

[0082] The feature point tracking unit 24 tracks 13 key feature points over time to generate a facial feature point coordinate sequence. Preferably, the feature point tracking is performed using the pyramid Lucas-Kanade optical flow method combined with Kalman filtering, which is robust to changes in illumination and rapid movement. To further improve tracking accuracy, this invention introduces temporal consistency constraints, filtering out abnormal tracking results by analyzing the continuity and smoothness of feature point motion.

[0083] Through the above processing, the facial feature point coordinate sequence output by feature extraction module 2 is in the form of P= ,in Indicates time The set of three-dimensional coordinates of all feature points at time.

[0084] like Figure 4 As shown, the differential geometry modeling module 3 of the present invention includes a triangulation unit 31, a parameterization mapping unit 32, a metric tensor unit 33, and a coordinate system establishment unit 34.

[0085] Triangulation unit 31 is used to perform Delaunay triangulation on the facial feature point coordinate sequence to generate a triangular mesh. Delaunay triangulation can maximize the minimum angle of the triangle, avoid generating elongated triangles, and facilitate subsequent parameterization mapping. Preferably, the data structure of the triangular mesh is designed as T= , ,..., Each tᵢ contains the vertex index and normal vector information of the triangle.

[0086] The parameterization mapping unit 32 is used to establish a mapping function from the two-dimensional parameter domain to the three-dimensional feature manifold. This invention employs a parameterization method based on Least Squares Conformal Mapping (LSCM), which maximizes the preservation of angle invariance and reduces deformation. Specifically, for each triangular facet, a local parametric coordinate system is established. Construct mapping function The expression :R²→R³ maps a two-dimensional parameter domain to a three-dimensional space. Mapping function. It can be represented as:

[0087] ,

[0088] Where u and v are parameter domain coordinates, both taking values ​​in the range [0,1], representing two-dimensional coordinate points on the parameterized plane; , , These are the component functions of the three-dimensional spatial coordinates, representing the x, y, and z coordinates in the mapped three-dimensional space, respectively.

[0089] To ensure a smooth transition in the parameter overlap region, this invention employs an interpolation method based on radial basis functions (RBF):

[0090] ,

[0091] Where w is the weighting coefficient, a 3×1 vector that determines the degree of influence of the control point on each component of the three-dimensional coordinates; is a radial basis function that maps distances to weights; (u,v) are points in the parameter domain; For each control point, there is a 2×1 vector representing the position of the control point in the parameter domain; represents the Euclidean distance; n is the number of control points, usually taken as 2-3 times the number of feature points to ensure sufficient interpolation accuracy.

[0092] Preferably, the present invention uses a multiquadric function as the radial basis function:

[0093] ,

[0094] Where r is the distance, i.e. The shape parameter controls the smoothness of the function, with a preferred value of 0.5. This parameter value was determined through extensive experimentation, achieving a good balance between smoothness and preservation of local features.

[0095] Metric tensor unit 33 is used to construct the Riemann metric tensor of the facial feature manifold, where the lip region is assigned a higher weight coefficient than other facial regions. The Riemann metric tensor G defines the distance and angle metrics at any point on the manifold, for points in the parameter domain. Its metric tensor It can be represented as:

[0096] ,

[0097] in, It is a 2×2 positive definite symmetric matrix, representing the midpoint of the parameter domain. The Riemannian metric tensor at that location; For mapping At point The Jacobian matrix at point is a 3×2 matrix; express The transpose of is a 2×3 matrix; · denotes matrix multiplication. Jacobian matrix. Defined as:

[0098] ,

[0099] in, express The partial derivatives with respect to u, and similarly for other terms, represent the rate of change of the three-dimensional spatial coordinates as the coordinates in the parameter domain change.

[0100] To enhance the system's sensitivity to lip movements, this invention performs weighted processing on the metric tensor:

[0101] ,

[0102] in, It is the weighted Riemannian metric tensor; β represents the weighting coefficient for the lip region, and β represents the weighting coefficient for other regions; both are positive real numbers. · indicates scalar multiplication of the matrix, meaning each element in the matrix is ​​multiplied by this coefficient. In practical applications, it is preferable to use β = β + β. =2.5, =1.0, this parameter setting showed the best sensitivity for detecting lip movements in clinical validation. Regions with higher weights (lip region) are brought closer together in the geodesic calculation, making the system more sensitive to changes in that region.

[0103] The coordinate system establishment unit 34 is used to establish a manifold coordinate system with the center of the lips as the origin and coordinate transformation rules. Specifically, the center point of the lips (usually the midpoint of the upper and lower lip boundaries) is first determined as the coordinate origin. Then, the coordinate axis directions are defined according to facial anatomical features: the X-axis points to the left and right (horizontal direction), the Y-axis points to the up and down (vertical direction), and the Z-axis is perpendicular to the facial plane (front and back direction). To ensure compatibility between the coordinate system and the metric tensor, this invention defines metric-compatible coordinate transformation rules to ensure the covariance of the metric tensor under coordinate transformation.

[0104] Through the above processing, the results output by the differential geometry modeling module 3 include the parameterized facial feature manifold M, the Riemannian metric tensor field G', and the manifold coordinate system C, which provide a mathematical basis for subsequent trajectory analysis.

[0105] like Figure 5 As shown, the trajectory analysis module 4 of the present invention includes a geodesic calculation unit 41, a curvature feature calculation unit 42, a shape operator construction unit 43, and a differential invariant extraction unit 44.

[0106] The geodesic calculation unit 41 is used to calculate the geodesic between key point pairs in the lip region on the facial feature manifold. The geodesic is the shortest path between two points on the manifold, reflecting the true distance considering the curvature of the surface. This invention employs an improved Fast Marching Method (FMM) to solve the geodesic equation:

[0107] ,

[0108] Where y(t) represents a parameterized geodesic, which is a curve in three-dimensional space. These are the parameters of the curve; The covariant second derivative of a geodesic describes the "acceleration" of the curve on a manifold; The Christoffel symbol is used to describe the local geometric properties of a manifold. and These represent geodesic parameters. The first derivative of the first derivative The and the first Each component describes the "velocity" of the curve; , , It is an indicator, with a value of 1 or 2 (corresponding to the two dimensions of the parameter domain). and .

[0109] Christopher Symbol From metric tensor Its derivative is calculated to obtain:

[0110] ,

[0111] in, For measuring tensors The amount is The first of the matrix Line number Column elements; The components of its inverse matrix are The first of the matrix Line number Column elements; express right The partial derivatives of , and similar terms for other terms, and Representing parameters respectively and The above formula uses Einstein's summation convention to sum the repeated index l (l takes the value of 1 or 2).

[0112] To improve computational efficiency, this invention first uses a straight line in Euclidean space as the initial value for the geodesic, and then solves the geodesic equation through iterative optimization. Preferably, 25 points are sampled for each geodesic to form a discrete representation. This sampling density can fully capture the details of lip movements in practical applications while maintaining computational efficiency.

[0113] The curvature characteristic calculation unit 42 is used to calculate the Gaussian curvature, mean curvature, and principal curvature of the geodesic. The Gaussian curvature K and mean curvature H are important geometric quantities describing the local bending characteristics of a surface, defined as:

[0114] ,

[0115] ,

[0116] in, Gaussian curvature is a measure of the intrinsic geometry of a surface, which does not change as the surface is curved into three-dimensional space. The average curvature reflects the average degree of curvature of the surface. and These are the first and second basic forms, respectively, both of which are 2×2 matrices; det(·) represents the determinant of the matrix; tr(·) represents the trace of the matrix, i.e., the sum of the elements on the main diagonal; I-1 represents the inverse of matrix I; · represents matrix multiplication.

[0117] The first fundamental form I is equivalent to the metric tensor G', and the second fundamental form II can be calculated by the shape operator W: II = I·W. Principal curvature and These are the eigenvalues ​​of the shape operator W, satisfying:

[0118] ,

[0119] ,

[0120] in, and Principal curvatures represent the maximum and minimum curvature of a surface in two mutually perpendicular directions. This is generally agreed upon... ≥ .

[0121] To eliminate the influence of noise, this invention applies a Gaussian filter to smooth the calculated curvature features, with a filter kernel size of 15 frames and a standard deviation σ = 0.8. This parameter setting demonstrates optimal noise suppression in practical applications while preserving the details of the motion features.

[0122] Shape operator construction unit 43 is used to construct the Weingarten shape operator describing the local bending characteristics of the lip movement trajectory. The Weingarten shape operator W is a 2×2 matrix, defined as:

[0123] ,

[0124] in, The Weingarten shape operator describes the local bending properties of a surface; The first fundamental form, namely the metric tensor ; It is the second basic form; express The inverse matrix; · denotes matrix multiplication; the negative sign indicates the agreed direction selection.

[0125] The eigenvalues ​​of W are the principal curvatures. and The eigenvectors indicate the direction of the principal curvature. By analyzing the shape operator, the local curvature characteristics of the lip movement trajectory, such as concavity / convexity and anisotropy, can be extracted. These characteristics directly reflect the pattern and intensity of the lip movement.

[0126] The differential invariant extraction unit 44 is used to calculate curvature-based differential invariants, which include the shape index and curvature amplitude. The shape index (SI) and curvature amplitude (CM) are invariants based on the principal curvature, invariant to rigid body transformations, and are defined as follows:

[0127] ,

[0128] ,

[0129] in, The shape index, with a value range of [-1, 1], describes the shape type of the surface. Different values ​​correspond to different shapes (such as grooves, saddle points, convexities, etc.). The curvature amplitude, with a value range of [0, +∞), describes the degree of curvature of the surface; and π is the principal curvature; arctan is the arctangent function; π is pi, approximately equal to 3.14159.

[0130] In neonatal feeding behavior analysis, normal sucking is typically characterized by a shape index within the range of [0.65, 0.85], while abnormal sucking usually has a shape index less than 0.5 or greater than 0.9. Curvature amplitude reflects sucking intensity; normal sucking typically has a curvature amplitude within the range of [0.2, 0.6]. These thresholds were determined through analysis of extensive clinical data and have high discriminative power.

[0131] Through the above processing, the results output by trajectory analysis module 4 include geodesic set Γ, curvature feature set C, shape operator set W, and differential invariant set I, providing basic data for subsequent cooperative motion analysis.

[0132] like Figure 6As shown, the cooperative motion analysis module 5 of the present invention includes a multi-manifold coupling unit 51, a synchronization calculation unit 52, a topology feature extraction unit 53, and a feature space construction unit 54.

[0133] Multi-manifold coupling element 51 is used to construct the lip manifold Mandibular mandibular and facial feature manifold A coupled model is proposed, defining mapping functions between manifolds. This coupled model achieves a unified representation of the coordinated motion of various parts of the oral cavity by defining the correspondences and mapping functions between manifolds. Specifically, the mapping functions are defined. This describes the mapping relationship between the lip manifold and the mandibular manifold to the facial feature manifold.

[0134] In practical implementation, a nonlinear mapping method based on thin plate spline (TPS) is adopted, which can effectively handle non-rigid deformation. The TPS mapping can be expressed as:

[0135] ,

[0136] in, It is a mapping function that maps points on the lip manifold and the mandibular manifold. Mapped onto the facial feature manifold; These are the coordinates of a point on the source manifold; It is a constant vector representing a global translation; It is an affine transformation matrix; It is a weight vector; These are basis functions, usually taken as . ; It is a control point; It refers to the number of control points.

[0137] Synchronization calculation unit 52 is used to calculate the temporal correlation between the lip trajectory and the mandibular trajectory, generating a synchronization metric matrix. Synchronization metric The synchronicity measure quantifies the degree of spatiotemporal coordination between lip and jaw movements, defined as:

[0138] ,

[0139] in, This is a measure of synchronicity, with a value range of [value range missing]. A larger value indicates better synchronization. This represents the number of sampling time points; For time indexing; and They represent time respectively The state of the mandibular and mandibular mandibles at that time; For time points The relevant functions at that location.

[0140] The relevant function C is defined as follows:

[0141] ,

[0142] in, For geodesic distance, it represents the distance between two manifold states; This is a scale parameter that controls the range of distance influence. Let be the phase difference, representing the phase difference between the motions of two manifolds; exp is an exponential function; cos is a cosine function. Preferably, =0.5, this parameter shows the best synchronous detection sensitivity in practical applications.

[0143] Synchronization metric matrix It is A matrix, where n is the number of sampling time points, and the matrix elements are... Indicates a point in time and The degree of synchronization between them. By analyzing this matrix, periodic patterns and anomalous changes in motion can be identified.

[0144] The topological feature extraction unit 53 is used to calculate the persistent homology features of the multi-manifold coupling model and extract the Betti number sequence. Persistent homology is a topological data analysis method that can extract the topological structural features of a dataset. This invention extracts the Betti number sequence by analyzing the persistent homology of point cloud data on a manifold. ,in Indicates the number of connected components. Indicates the number of rings, Indicates the number of holes.

[0145] In persistent homology analysis, a series of simple complexes at different scales ε are constructed, and the birth and death of topological features (such as connected components, loops, and holes) are tracked. Persistent features are considered to be the intrinsic topological structure of the data, while transient features may be noise.

[0146] During neonatal feeding, normal feeding typically exhibits a stable Betty number sequence, while abnormal feeding manifests as abrupt or unstable changes in the Betty number. Preferably, a duration threshold ε = 0.3 is set to filter transient topological changes. This threshold, determined through clinical validation, effectively distinguishes between genuine topological changes and noise.

[0147] Feature space construction unit 54 is used to construct a high-dimensional feature space and establish normal region boundaries. This invention combines extracted differential invariants, synchronicity measures, and topological features to form a high-dimensional feature vector, and then performs dimensionality reduction using the Multidimensional Scaling (MDS) method to preserve the data's topological structure. MDS maps data to a low-dimensional space by maintaining the distance relationships between point pairs in the high-dimensional space.

[0148] In the dimensionality-reduced feature space, based on a large number of normal feeding samples, the Support Vector Domain Description (SVDD) method is used to establish the boundaries of the normal regions. The SVDD method models the normal regions as a hypersphere in the feature space, with the interior of the sphere being the normal region and the exterior being the abnormal region.

[0149] Preferably, the target dimension of the feature space is 5, which effectively reduces computational complexity while maintaining the data structure. The selection of the target dimension is based on the cumulative explained variance ratio (usually the smallest dimension with a cumulative explained variance of 95% or more is selected).

[0150] Through the above processing, the results output by the cooperative motion analysis module 5 include the coupled manifold model M, the synchronization metric matrix S, the topological eigenvector β, and the feature space mapping function. This provides a multidimensional feature representation for anomaly detection.

[0151] like Figure 7 As shown, the anomaly detection module 6 of the present invention includes a feature mapping unit 61, a distance calculation unit 62, an anomaly type identification unit 63, and a severity assessment unit 64.

[0152] The feature mapping unit 61 is used to map the currently observed feature vector to the feature space. Specifically, for the currently observed feature vector f, the feature space mapping function is used... Map it to the reduced feature space:

[0153] ,

[0154] in, The mapped feature vector has a dimension of 5; The original feature vector contains differential invariants, synchronicity measures, and topological features; It is the feature space mapping function, determined by the feature space construction unit 54.

[0155] The distance calculation unit 62 is used to calculate the distance from the observed feature to the boundary of the normal region. This invention uses Mahalanobis distance to measure the deviation of the observed point from the normal region.

[0156] ,

[0157] in, For observation point The distance to the boundary B of the normal region; The mean vector of normal samples, with dimensions equal to... Same; Σ is the covariance matrix of the normal samples, with a dimension of 5×5; Represents vector difference; Represents the transpose of the vector difference; Denotes the inverse of the covariance matrix; It represents the square root.

[0158] Larger distance values ​​indicate a greater degree of deviation of the observation point from the normal area, potentially indicating anomalies. In practical applications, distance values ​​are typically standardized to the [0,1] interval to facilitate the setting of uniform thresholds.

[0159] The abnormality type identification unit 63 is used to identify the type of feeding abnormality, which includes synchronicity abnormalities and topological abnormalities. Synchronicity abnormalities are mainly manifested as poor coordination between lip and jaw movements, and are usually associated with swallowing difficulties; topological abnormalities are mainly manifested as abnormalities in the topological structure of oral motor patterns, and are usually associated with insufficient or excessive sucking force. This invention identifies abnormality types by analyzing abnormal directions in the feature space.

[0160] ,

[0161] in, and These represent the components of the feature vector in the dimensions of synchronicity and topological features, respectively; and This corresponds to the normal mean. and For threshold; Represents absolute value. Preferably, , These thresholds demonstrated the best accuracy in abnormality type identification during clinical validation.

[0162] The severity assessment unit 64 is used to assess the severity of the anomaly based on distance indicators and anomaly duration. This invention classifies anomaly severity into three levels: mild, moderate, and severe, with the assessment criteria as follows:

[0163] ,

[0164] The normalized distance is achieved by using Mahalanobis distance. Mapping to the [0,1] interval yields the normalized distance using the following method: ,in This is the scaling factor, preferably 3.0. The closer the normalized distance is to 1, the more normal it is; the closer it is to 0, the more abnormal it is.

[0165] Furthermore, the duration of the abnormality is also an important factor in assessing its severity; the longer the duration, the higher the severity. Preferably, when the duration of the abnormality exceeds 30 seconds, the severity level is increased by one grade. This threshold is determined based on clinical observation, considering that occasional brief abnormalities in normal newborns are normal, while persistent abnormalities may indicate a problem.

[0166] Through the above processing, the results output by the anomaly detection module 6 include the anomaly detection result D={is_abnormal,abnormal_type,severity,confidence}, which provides a scientific basis for clinical intervention.

[0167] like Figure 8 As shown, the result display module 7 of the present invention includes a data aggregation unit 71, an indicator calculation unit 72, a visualization generation unit 73, a report generation unit 74, and a remote transmission unit 75.

[0168] The data aggregation unit 71 is used to integrate the analysis results of various functional modules. Specifically, the data aggregation unit 71 collects the output results of the differential geometry modeling module 3, trajectory analysis module 4, cooperative motion analysis module 5, and anomaly detection module 6, and manages and organizes them in a unified manner.

[0169] The indicator calculation unit 72 is used to generate comprehensive assessment indicators such as lip movement intensity, movement frequency, lip and facial movement synchronicity, cleft lip slope, lip opening, occlusal index, and respiratory-swallowing coordination. These indicators are calculated based on the analysis results of the aforementioned modules, providing quantitative standards for feeding behavior assessment. Preferably, lip movement intensity is calculated based on curvature amplitude CM, movement frequency is based on periodic analysis of movement trajectory, synchronicity is based on synchronicity measure S, cleft lip slope and lip opening are based on the geometric relationship of lip feature points, occlusal index is based on mandibular movement characteristics, and respiratory-swallowing coordination is based on the temporal relationship of breathing and swallowing actions.

[0170] The visualization generation unit 73 is used to create graphical representations of oral motion trajectories, synchronicity curves, and abnormal detection results. Preferably, three-dimensional interactive visualization technology is used to intuitively display the oral motion trajectory; a time-series diagram is used to represent synchronicity changes; and a heatmap is used to represent abnormal detection results. These visualizations enable doctors to intuitively understand the analysis results and improve diagnostic efficiency.

[0171] The report generation unit 74 is used to generate a structured assessment report. The assessment report includes basic information, quantitative indicators, abnormal test results, expert recommendations, etc., providing comprehensive support for clinical decision-making. Preferably, the report is in PDF format and supports mixed text and graphics and interactive content.

[0172] The remote transmission unit 75 securely transmits the assessment results to a remote server via a wireless communication protocol for the doctor to view. This invention uses encrypted HTTPS protocol for data transmission to ensure the security and privacy of patient data. Furthermore, the system supports role-based access control to ensure that only authorized personnel can access the assessment results.

[0173] Through the above processing, the results display module 7 provides an intuitive and comprehensive display of feeding behavior assessment results, making it easy for doctors to understand and apply.

[0174] like Figure 1 As shown, the present invention also includes an application service module 8, which is connected to the result display module 7 and is used to generate an individualized feeding plan based on the oral motor assessment results of the newborn.

[0175] Specifically, application service module 8 first establishes a correlation model between the infant's nipple circumference and milk feeding rate based on the newborn's oral motor assessment results. This model is constructed based on a large amount of clinical data and machine learning algorithms, and takes the following form:

[0176] ,

[0177] in, The milk feed rate is expressed in milliliters per minute (ml / min). The nipple circumference is measured in centimeters (cm). This refers to the time spent eating, expressed in minutes (min). This is a proportionality coefficient, with units of ml / (min·cmb). The exponent of the nipple circumference is dimensionless. is the time decay coefficient, in units of 1 / min; e is the base of the natural logarithm, approximately equal to 2.71828. Preferably, =0.8, =1.5, =0.05, these parameter values ​​showed the best predictive accuracy in clinical validation.

[0178] Secondly, application service module 8 calculates a model relating individualized feeding duration and milk delivery rate. This model considers factors such as the infant's sucking ability and oral motor patterns, and takes the following form:

[0179] ,

[0180] in, The duration of eating is expressed in minutes (min). V represents the total milk volume, expressed in milliliters (ml); V represents the milk feed rate, expressed in milliliters per minute (ml / min). It is a synchronicity index, with a value range of [0,1], and is dimensionless; The adjustment coefficient is dimensionless. Preferably, =2.0, this parameter value can reasonably reflect the impact of synchronicity on feeding time. Synchronicity Index The closer a value is to 1, the more coordinated the movements of the lips and jaw, and the higher the efficiency of eating. The closer a value is to 0, the worse the coordination, and the longer the eating time required.

[0181] Next, application service module 8 calculates individualized feeding frequency and duration based on the feeding ability score. The feeding ability score is calculated based on oral motor assessment results and ranges from 0-100. Feeding frequency and feeding duration The calculation formula is:

[0182] ,

[0183] ,

[0184] in, This refers to the actual number of feedings, expressed as feedings per day. The standard feeding frequency (usually 8 times / day) is expressed in times / day. The actual feeding time is in minutes (min); T is the theoretical feeding time calculated according to the previous formula, in minutes (min). Feeding ability is scored, with values ​​ranging from [0, 100], dimensionless; α and β are adjustment coefficients, dimensionless. Preferably, =0.5, =0.3, these parameter values ​​allow for reasonable adjustment of feeding frequency and duration. Feeding ability score The higher the value, the stronger the feeding ability, and the closer the required feeding frequency and duration are to the standard value; The lower the value, the weaker the feeding ability, requiring more frequent feedings and longer feeding times.

[0185] Finally, application service module 8 generates an individualized weekly feeding plan, including the number of feedings per day, the amount of food per feeding, the duration of each feeding, and the recommended nipple type. This plan combines the newborn's growth and development status, feeding ability score, and the calculation results of the aforementioned model to provide scientific feeding guidance for newborns.

[0186] Through the above processing, application service module 8 realizes the transformation from assessment results to practical application, providing personalized feeding support for newborns.

[0187] The workflow of the neonatal feeding behavior image recognition and oral motor assessment system of the present invention is as follows:

[0188] First, multi-view image data of the newborn's facial and oral movements are acquired through data acquisition module 1. Specifically, high-speed camera 11 captures a frontal image of the newborn's face, 3D facial tracking camera 12 captures the 3D morphology of the face, and optical perspective camera 13 provides images of the internal structure of the oral cavity. Data acquisition and analysis instrument 14 controls the synchronous triggering of each camera device to ensure time alignment of multi-source data.

[0189] Secondly, the feature extraction module 2 extracts the facial feature point coordinate sequence from the multi-view image data. The preprocessing unit 21 corrects and enhances the original image, the face detection unit 22 locates the face region, the feature point annotation unit 23 annotates 13 key feature points, and the feature point tracking unit 24 tracks the temporal changes of these feature points to generate the facial feature point coordinate sequence.

[0190] Next, the differential geometry modeling module 3 constructs a facial feature manifold and its Riemannian metric tensor based on the coordinate sequence of facial feature points. The triangulation unit 31 generates a triangular mesh, the parameterization mapping unit 32 establishes a parameterization mapping, the metric tensor unit 33 constructs a Riemannian metric tensor, and the coordinate system establishment unit 34 establishes a manifold coordinate system.

[0191] Then, trajectory analysis module 4 calculates geodesics on the facial feature manifold and extracts differential invariants of the oral cavity movement trajectory. Geodesic calculation unit 41 solves the geodesic equation, curvature feature calculation unit 42 calculates curvature features, shape operator construction unit 43 constructs the Weingarten shape operator, and differential invariant extraction unit 44 calculates differential invariants such as shape index and curvature amplitude.

[0192] Next, the cooperative motion analysis module 5 constructs a coupling model of the lip manifold and the mandibular manifold, and calculates the synchronicity measure and topological features between the manifolds. The multi-manifold coupling unit 51 constructs the coupling model, the synchronicity calculation unit 52 calculates the synchronicity measure matrix, the topological feature extraction unit 53 extracts the Betti number sequence, and the feature space construction unit 54 constructs the feature space.

[0193] Subsequently, the anomaly detection module 6 detects anomaly patterns based on synchronization metrics and topological features. The feature mapping unit 61 maps feature vectors to the feature space, the distance calculation unit 62 calculates the distance to the normal region, the anomaly type identification unit 63 identifies the anomaly type, and the severity assessment unit 64 assesses the severity of the anomaly.

[0194] Finally, the results display module 7 generates a feeding behavior assessment report and displays it visually. The data aggregation unit 71 integrates and analyzes the results, the indicator calculation unit 72 calculates the comprehensive assessment indicators, the visualization generation unit 73 creates graphical representations, the report generation unit 74 generates the assessment report, and the remote transmission unit 75 transmits the results to a remote server.

[0195] In addition, the application service module 8 generates individualized feeding plans based on the evaluation results, including establishing a correlation model between nipple circumference and milk delivery speed, calculating individualized feeding duration and feeding frequency, and generating a weekly feeding plan.

[0196] Through the above process, the neonatal feeding behavior image recognition and oral motor assessment system of the present invention realizes accurate assessment and personalized guidance of neonatal feeding behavior, providing a scientific tool for early detection and intervention of feeding problems.

[0197] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A neonatal feeding behavior image recognition and oral motor assessment system, characterized in that, include: The data acquisition module is used to acquire multi-view image data of the newborn's facial and oral movements; The feature extraction module, connected to the data acquisition module, is used to extract the facial feature point coordinate sequence from the multi-view image data; The differential geometry modeling module, connected to the feature extraction module, is used to construct the facial feature manifold and its Riemannian metric tensor based on the facial feature point coordinate sequence. The trajectory analysis module, connected to the differential geometry modeling module, is used to calculate the geodesics on the facial feature manifold and extract the differential invariants of the oral cavity movement trajectory; The trajectory analysis module includes: A geodesic calculation unit is used to calculate geodesics between key point pairs in the lip region on the facial feature manifold. The curvature feature calculation unit is used to calculate the Gaussian curvature, average curvature, and principal curvature of the geodesic; Shape operator construction unit, used to construct Weingarten shape operator to describe the local bending characteristics of the lip movement trajectory; And a differential invariant extraction unit, used to calculate curvature-based differential invariants, the differential invariants including shape exponent and curvature amplitude; A cooperative motion analysis module, connected to the trajectory analysis module, is used to construct a coupled model of the lip manifold and the mandibular manifold, and to calculate the synchronicity measure and topological features between the manifolds; the cooperative motion analysis module includes: Multi-manifold coupling element is used to construct a coupling model of the lip manifold, mandibular manifold, and facial feature manifold. It defines the mapping function from the lip manifold and mandibular manifold to the facial feature manifold and uses a nonlinear mapping method based on thin plate splines to handle the non-rigid deformation between manifolds. The synchronization calculation unit is used to calculate the temporal correlation between the lip trajectory and the mandibular trajectory based on geodesic distance and phase difference, and generate a synchronization metric matrix; The topological feature extraction unit is used to construct simple complexes at different scales and track the birth and death of topological features to calculate the continuous homology features of the multi-manifold coupling model and extract the Betti number sequence. And a feature space construction unit, which combines differential invariants, synchronicity measures and topological features to form a high-dimensional feature vector, reduces the dimensionality through a multi-dimensional scaling method, constructs the feature space and establishes the normal region boundary using the support vector domain description method; An anomaly detection module, connected to the cooperative motion analysis module, is used to detect abnormal feeding patterns based on the synchronicity metric and topological features. The system also includes a results display module, which is connected to the anomaly detection module and is used to generate a feeding behavior assessment report and display it visually.

2. The system of claim 1, wherein, The data acquisition module includes: A high-speed camera, mounted on a camera bracket and aimed at the newborn's face; A 3D facial tracking camera is installed directly above the newborn's head; An optical perspective camera is installed in the center of the newborn's head, at the same height as the three-dimensional facial tracking camera; The system also includes a data acquisition and analysis instrument, which is connected to the high-speed camera, the 3D face tracking camera, and the optical perspective camera, respectively. The instrument is used to control the high-speed camera and the optical perspective camera to trigger synchronously and to receive image data acquired by the high-speed camera, the 3D face tracking camera, and the optical perspective camera.

3. The system of claim 2, wherein, The data acquisition module also includes a synchronization trigger control unit, which controls the synchronization trigger time and exposure time of the high-speed camera and the optical perspective camera, and controls the data acquisition analyzer to synchronously trigger the three-dimensional face tracking camera.

4. The system of claim 1, wherein, The feature extraction module includes: The preprocessing unit is used to correct and enhance the multi-view image data; The face detection unit is used to locate face regions in an image; The feature point annotation unit is used to annotate 13 key feature points on the newborn's face. The 13 key feature points include the center of the eyebrows, the tip of the nose, the boundary of the upper lip, the boundary of the lower lip, the tip of the chin, and the main feature point group of lip movement, wherein the main feature point group of lip movement includes the tip of the mouth cleft, the corner of the mouth, the nasolabial groove at the corner of the mouth, and the philtrum. And a feature point tracking unit, used to track the 13 key feature points over time to generate a facial feature point coordinate sequence.

5. The system according to claim 1, characterized in that, The differential geometry modeling module includes: Triangulation unit, used to perform Delaunay triangulation on the facial feature point coordinate sequence to generate triangular mesh; The parameterized mapping unit is used to establish a mapping function from a two-dimensional parameter domain to a three-dimensional feature manifold; A metric tensor unit is used to construct the Riemann metric tensor of the facial feature manifold, wherein the lip region is assigned a higher weight coefficient than other facial regions. And a coordinate system establishment unit, used to establish a manifold coordinate system with the center of the lips as the origin and coordinate transformation rules.

6. The system according to claim 1, characterized in that, The anomaly detection module includes: The feature mapping unit is used to map the currently observed feature vector to the feature space; The distance calculation unit is used to calculate the distance from the observed feature to the boundary of the normal region; An anomaly type identification unit is used to identify the type of feeding anomaly, which includes synchronization anomalies and topological anomalies. It also includes a severity assessment unit for evaluating the severity of anomalies based on distance metrics and anomaly duration.

7. The system according to claim 1, characterized in that, The results display module includes: The data aggregation unit is used to integrate the analysis results of various functional modules; The index calculation unit is used to generate comprehensive assessment indicators for oral and lip movement intensity, movement frequency, synchronicity of oral and facial movements, cleft lip slope, lip opening, occlusal index, and respiratory-swallowing coordination. A visualization generation unit is used to create graphical representations of oral motion trajectories, synchronization curves, and abnormality detection results; The report generation unit is used to generate structured assessment reports; It also includes a remote transmission unit for securely transmitting assessment results to a remote server via a wireless communication protocol for doctors to view.

8. The system according to claim 1, characterized in that, It also includes an application service module, which is connected to the result display module and is used for: Based on the results of the newborn's oral motor assessment, an individualized feeding plan is generated; Establish a correlation model between infant nipple circumference and milk feeding rate; A model for calculating the relationship between individualized feeding duration and milk feeding rate; Based on the feeding ability score, the system calculates the individualized feeding frequency and duration, and generates an individualized weekly feeding plan.

Images