Multi-mode cooperative intelligent infant tending control system and method

Abstract

The invention discloses a multi-mode cooperative intelligent infant tending control system and method. The system comprises a sensor module, a data fusion module, a path planning module, a trajectory prediction module, a parameter optimization module and a trajectory generation module. According to the invention, data are collected through the sensor module on the RGB-D camera, the laser radar and the six-dimensional force sensor arranged on the mechanical arm, an environment sensing model is constructed, and a feeding bottle is positioned; the mechanical arm plans a path according to the sensing information and completes the actions of grabbing a feeding bottle, measuring milk powder and receiving water; when the two arms cooperatively work, the distance is monitored in real time to prevent collision, the grabbing and scraping actions are accurately controlled by virtue of feedback of a force sensor, and meanwhile, the rotating and shaking actions are dynamically adjusted by virtue of IMU (Inertial Measurement Unit) attitude compensation and vibration monitoring; the milk frothing device can efficiently and safely realize milk frothing.

Description

Technical Field

[0001] The present invention relates to a control system and method, and in particular to a multimodal collaborative intelligent infant and child care control system and method. Background Art

[0002] In the fast-paced world of modern life, smart home appliances are increasingly becoming ubiquitous. Smart milk makers are particularly effective tools for new parents, particularly in the area of ​​childcare. While traditional milk maker methods are simple, they lack precise control and intelligent sensing, leading to numerous inconveniences. For example, precise control of the amount of powdered milk and water temperature is crucial to ensure accurate nutritional intake. Furthermore, constant monitoring of the baby's condition is crucial, adding to the burden on parents.

[0003] However, most existing milk-making equipment is single-function and lacks intelligence, making it difficult to meet diverse needs. Ordinary automatic milk-making machines can only perform simple brewing operations and lack environmental perception and adaptive adjustment capabilities. This makes them particularly difficult to operate properly when faced with changes in the bottle's position or surrounding obstacles. Although robotic arm technology has been widely used in the industrial field, its application in home childcare scenarios is still in the exploratory stage, facing challenges such as multimodal data fusion, path planning in complex environments, and collaborative operation between robotic arms. Research on intelligent milk-making robotic systems can achieve controlled care for infants and young children, and therefore has important practical significance. Summary of the Invention

[0004] Purpose of the invention: The purpose of the present invention is to provide an infant care control system to achieve milk foaming efficiently and safely, and on the other hand, to provide an infant care control method.

[0005] Technical solution: The raising control system of the present invention comprises:

[0006] The sensor module is used to collect visual, point cloud, and torque data, including an RGB-D camera, lidar, and six-dimensional force sensor configured on the robotic arm;

[0007] A data fusion module, used to fuse the federated Kalman filter algorithm with the data collected by the sensor module to build a dynamic environment perception model;

[0008] The path planning module is used to introduce the SPD-Conv module into the YOLOv9 object detection algorithm to detect the position of the bottle and plan the path for the robot arm to grasp the bottle by combining the adaptive step size RRT-Connect algorithm;

[0009] The trajectory prediction module is used to fuse TCN and KAN to build a hybrid model, process multimodal data, introduce adversarial domain adaptation (ADA) into the hybrid model, align the spatial features of the trajectory prediction results and point cloud data, and obtain an improved ADA-TCN-KAN model to predict obstacle trajectories;

[0010] A parameter optimization module is used to dynamically adjust the selection probability according to the population diversity to improve the PRGO algorithm, optimize the parameters of the ADA-TCN-KAN model, and obtain the IPRGO-ADA-TCN-KAN model, which can avoid obstacles when encountering them;

[0011] The trajectory generation module is used to generate motion trajectories through B-spline trajectory smoothing technology, enabling the robotic arm to grab the bottle, measure the milk powder, and scrape the milk powder to complete the milk foaming task.

[0012] Preferably, the RGB-D camera is configured at the end of the robotic arm to obtain information about the bottle and the environment and to identify close-range targets; the lidar is configured on a fixed frame shared by the dual robotic arms to assist in positioning and obstacle avoidance; the six-dimensional force sensor is configured at the contact surface between the end effector of the robotic arm and the bottle to measure the contact force and torque between the end of the robotic arm and the environment.

[0013] Preferably, the RGB-D camera measures z cam =[u,v,d] T , u and v represent the pixel coordinates of the object in the image plane, and d represents the depth value;

[0014]

[0015] Among them, z cam ~N(0,R cam ) represents the noise in the camera measurement process;

[0016] R cam =diag(2 2 ,2 2 ,0.03 2 ), f x 、f y Represents the focal length of the camera in the x and y directions, c x 、c x Represents the components of the principal point coordinates of the camera image plane in the x and y directions; Represents the coordinate components of a 3D space point in the camera coordinate system.

[0017] Preferably, the laser radar measures z lidar =[ρ,α,β] T , ρ represents the straight-line distance from the target to the radar, α represents the azimuth angle, and β represents the elevation angle;

[0018]

[0019] Among them, z lidar ~N(0,R lidar ) represents the noise in the lidar measurement process;

[0020] R lidar =diag(0.02 2 ,0.01 2 ,0.01 2 ), p x 、p y 、p z represents the coordinate component of the target point in the Cartesian coordinate system, and ρ represents the distance from the target point to the lidar.

[0021] Preferably, the six-dimensional force sensor measures z force =[F x ,F y ,F z ,T x ,T y ,T z ] T , F x 、F y 、F z represents the three-axis components of the end effector contact force, T x 、T y 、T z Represents the three-axis components of the terminal torque;

[0022]

[0023] Among them, z force ~N(0,R force ) represents the noise in the measurement process of the six-dimensional force sensor;

[0024] R force =diag(0.5 2 ,0.5 2 ,0.5 2 ,0.1 2 ,0.1 2 ,0.1 2 ), J(x) represents the Jacobian matrix of the end of the robot arm, g represents the acceleration of gravity, m represents the load mass, I F represents the inertia tensor, ω represents the angular velocity, a x 、a y 、a z They represent the acceleration components of the end of the robotic arm in the three-dimensional space directions respectively.

[0025] Preferably, the dynamic environment perception model is as follows:

[0026]

[0027] in, represents the inverse of the fused global estimated covariance matrix; represents the inverse of the i-th local estimated covariance matrix, represents the global state estimation after fusion, The i-th local state estimate, The dynamic weight of the i-th sensor or local estimate, The inverse of the noise covariance matrix of the i-th sensor observation.

[0028] Preferably, the path planning module includes:

[0029] Insert the SPD-Conv module at each downsampling position of CSPDarknet to replace the original stride convolution;

[0030] An SPD feature fusion module is added to the horizontal connection of PANet, dynamic feature compensation is used to calculate channel weights, and the occlusion area is dynamically adjusted by combining occlusion data enhancement;

[0031] In the first 80% of the path, the default step size s = 0.1m is used to reduce the calculation time; in the last 20% of the path, the step size is adjusted according to the local obstacle density ρ;

[0032] Construct the configuration space of the dual-arm working area, divide the free space and obstacle area, and initialize the parameters;

[0033] The master arm determines the root node of the starting tree through forward kinematics FK and the root node of the target tree through inverse kinematics IK. Similarly, the slave arm determines the root node of the starting tree and the root node of the target tree to ensure the coordination of the two arms.

[0034] The discrete path points are converted into continuous trajectories by minimizing the energy function to obtain the control points, adding acceleration constraints, and solving the feasible trajectory through quadratic programming QP.

[0035] Preferably, the ADA-TCN-KAN model improvement process is:

[0036] Input the state vector, historical trajectory, lidar point cloud, and RGB-D visual data into the hybrid model that integrates TCN and KAN;

[0037] The state vector is converted into a feature vector through a fully connected layer mapping; the historical trajectory uses a 4-layer dilated causal convolution to extract temporal features; the lidar point cloud is processed by farthest point sampling (FPS) combined with a multi-layer perceptron (MLP); the RGB-D visual data is subjected to feature extraction using a convolutional neural network;

[0038] The extracted features are fused through a dynamic gating mechanism, the gating network is used to calculate the weights, and then weighted fusion is performed to output the fused features;

[0039] In order to align the feature distribution, adversarial domain adaptation (ADA) is introduced. The discriminator D is used to optimize the adversarial loss so that the distribution of fused features is consistent with that of real features, and the fused features are optimized.

[0040] The improved time series modeling layer processes the data and then outputs the prediction results through the linear layer, representing the three-dimensional coordinates of the obstacle at each future moment.

[0041] Preferably, the parameter optimization module includes:

[0042] The hyperparameters of the ADA-TCN-KAN model are encoded as fiber root position vectors and the parameter ranges are set;

[0043] Randomly generate N fiber root vectors to form the initial population, and use the prediction mean square error (MSE) as the fitness;

[0044] Dynamically adjust selection probability based on population diversity;

[0045] During the population update phase, each fiber root is updated in different modes based on dynamic probability to balance local development and global exploration:

[0046] The taproot system mode, i.e., local development, uses the current optimal solution information to conduct a local fine search and generate a new solution using the following formula; the fibrous root system mode, i.e., global exploration enhancement, uses Lévy flight to achieve long-distance exploration and form a new solution;

[0047] Input training data and configure the network structure according to the hyperparameters; use the specified learning rate for backpropagation optimization; calculate the MSE on the validation set and update the fitness; until the maximum number of iterations is reached or the fitness converges;

[0048] After the iteration, the fiber root vector with the best fitness is selected, and its corresponding hyperparameter is the optimization result, and the IPGRO-KAN-ADATCN model is obtained.

[0049] The tending control method of the present invention comprises the following steps:

[0050] S1, perceives the environment through the RGB-D camera, lidar and six-dimensional force sensor configured on the robotic arm;

[0051] S2, the main arm, locates the bottle by introducing the SPD-Conv module in YOLOv9 and combines it with the adaptive step-size RRT-Connect algorithm for path planning to determine a collision-free motion path;

[0052] S3. The slave arm moves synchronously to the milk powder can and locates the opening position through visual guidance. The master arm replaces the end effector with a milk powder spoon. The laser radar is used to monitor the height of the milk powder in the milk powder spoon. The force of the measuring and scraping action is achieved in combination with force control feedback.

[0053] S4: The master arm moves the bottle to the top of the water cup, visually locates the water receiving position, and the slave arm synchronously rotates to complete the milk foaming;

[0054] S5. If the laser radar finds an obstacle on the path, the updated environmental status of the fusion system is integrated, and the B-spline smoothing technology is combined with the QP optimization path to adjust the trajectory of the robotic arm to achieve obstacle avoidance.

[0055] Beneficial effects: Compared with the existing technology, the present invention has the following significant advantages: 1. The path planning module enables the robotic arm to quickly plan the optimal path to avoid obstacles, thereby efficiently grabbing the bottle and measuring the milk powder, reducing unnecessary movements and shortening the milk frothing time. The operation accuracy is much higher than manual or simple automated milk frothing methods, ensuring that each milk frothing action is accurate and correct; 2. Through the real-time monitoring of the distance between the two robotic arms and the collaboration of the collision prediction algorithm, combined with the B-spline trajectory smoothing technology, non-interference collaborative operation is achieved. During the milk frothing process, the two arms work closely together, which not only improves efficiency but also effectively avoids collision accidents. The operation is safe and reliable, providing users with a safe milk frothing experience; 3. 1. The six-dimensional force sensor in the sensor module monitors the grasping force in real time, and the PID force control technology is used for precise adjustment, so that the robotic arm can grasp the bottle and scrape the milk powder stably, avoiding the bottle from falling or inaccurate amount of milk powder due to improper force, ensuring the accurate amount of milk powder for each milk making, and thus ensuring the quality of the milk making, so that the baby can drink nutritionally balanced milk; 4. By adopting multiple sensor fusion technologies, RGB-D cameras and lidars work together to accurately perceive the milk making environment. Even if the bottle is partially blocked, its position can be accurately obtained; 5. By constructing a dynamic environmental perception model, the system can grasp environmental changes in real time, provide reliable basic information for milk making, and improve operational stability. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 Schematic diagram of the process of the present invention;

[0057] Figure 2 Schematic diagram of the process of introducing SPD-Conv module into YOLOv9 in this invention;

[0058] Figure 3Schematic diagram of the process of optimizing the parameters of the ADA-TCN-KAN model using the improved PRGO algorithm of the present invention;

[0059] Figure 4 This is a schematic diagram of the process of implementing the milk-making task with two arms of the present invention. DETAILED DESCRIPTION

[0060] The technical solution of the present invention is described in detail below with reference to the accompanying drawings.

[0061] Multimodal collaborative intelligent infant care control system, including:

[0062] The sensor module is equipped with an RRGB-D camera, a solid-state lidar, and a six-dimensional force sensor.

[0063] An RRGB-D camera, mounted at the end of the robotic arm, dynamically captures information about the bottle and its surroundings. The viewing angle is adjusted by moving the end-effector to enable close-range target recognition. Solid-state LiDARs, installed on a fixed frame shared by the dual robotic arms and deployed on the top and sides of the system, cover the entire robotic arm workspace and provide submillimeter point cloud data for assisted positioning and obstacle avoidance. A six-dimensional force sensor, mounted at the interface between the end-effector and the bottle, directly measures the contact force and torque between the end and the environment. Integrated directly into the gripper or suction cup, it monitors the contact force / torque during grasping in real time to ensure smooth control.

[0064] Define the system state:

[0065] State vector Among them, p x 、p y 、p z Indicates the position of the object in the global coordinate system, v x 、v y 、v z represents the linear velocity of the object, θ and ψ represent the roll angle, pitch angle, and yaw angle.

[0066] Observation equation z = h(x) + v;

[0067] RGB-D camera observation equation, observation function and observation matrix:

[0068] RGB-D camera measurement z cam =[u,v,d] T , u and v represent the pixel coordinates of the object in the image plane, and d represents the depth value;

[0069]

[0070] Among them, z cam ~N(0,R cam) represents the noise in the camera measurement process;

[0071] R cam =diag(2 2 ,2 2 ,0.03 2 ), f x 、f y Represents the focal length of the camera in the x and y directions, c x 、c x Represents the components of the principal point coordinates of the camera image plane in the x and y directions; Represents the coordinate components of a 3D space point in the camera coordinate system.

[0072] Solid-state lidar observation equation, observation function and observation matrix:

[0073] Solid-state lidar measurement lidar =[ρ,α,β] T , ρ represents the straight-line distance from the target to the radar, α represents the azimuth angle, and β represents the elevation angle;

[0074]

[0075]

[0076] Among them, z lidar ~N(0,R lidar ) represents the noise in the lidar measurement process;

[0077] R lidar =diag(0.02 2 ,0.01 2 ,0.01 2 ), p x 、p y 、p z represents the coordinate component of the target point in the Cartesian coordinate system, and ρ represents the distance from the target point to the lidar.

[0078] Six-dimensional force sensor observation equation, observation function and observation matrix:

[0079] Six-axis force sensor measurement measurement z force =[F x ,F y ,F z ,T x ,T y ,T z ] T , F x 、F y 、F z represents the three-axis components of the end effector contact force, T x、T y 、T z Represents the three-axis components of the terminal torque;

[0080]

[0081] Among them, z force ~N(0,R force ) represents the noise in the measurement process of the six-dimensional force sensor;

[0082] R force =diag(0.5 2 ,0.5 2 ,0.5 2 ,0.1 2 ,0.1 2 ,0.1 2 ), J(x) represents the Jacobian matrix of the end of the robot arm, g represents the acceleration of gravity, m represents the load mass, I F represents the inertia tensor, ω represents the angular velocity, a x 、a y 、a z They represent the acceleration components of the end of the robotic arm in the three-dimensional space directions respectively.

[0083] IMU observation equation, observation function and observation matrix:

[0084] IMU measures z imm =[a x ,a y ,a z ,ω x ,ω y ,ω z ] T , a x 、a y 、a z represents the acceleration component, ω x 、ω y 、ω z represents the angular velocity component;

[0085]

[0086] Among them, z imm ~N(0,R imm ) represents the measurement noise; R represents the attitude rotation matrix (projecting gravity g into the sensor coordinate system); C represents the attitude angular velocity conversion matrix (associating joint angular velocity and the angular velocity measured by IMU), I represents the unit matrix;

[0087] R imm =diag(0.1 2 ,0.12 ,0.1 2 ,0.01 2 ,0.01 2 ,0.01 2 ), J(x) represents the Jacobian matrix of the end of the robot arm, g represents the acceleration of gravity, m represents the load mass, I F represents the inertia tensor, ω represents the angular velocity, a x 、a y 、a z They represent the acceleration components of the end of the robotic arm in the three-dimensional space directions respectively.

[0088] Establish the discrete state equation based on Newtonian kinematics:

[0089] x k|k-1 =F k x k-1 +G k w k ;

[0090] Among them, x k|k-1 represents the system state predicted at time k based on the information at time k-1, x k-1 represents the state of the system at time k-1;

[0091]

[0092] F k represents the state transfer matrix, Δt represents the sampling period, G k represents the noise driving matrix, w k represents the process noise, θ and ψ represent the roll angle, pitch angle, and yaw angle, respectively, and R represents the attitude rotation matrix, which projects the gravity g into the sensor coordinate system.

[0093] The data fusion module is used to use the federated Kalman filter algorithm to fuse the visual (RGB-D camera acquisition), point cloud (lidar acquisition) and torque (six-dimensional force sensor acquisition) data collected by the sensor module to build a dynamic environment perception model.

[0094] (1) Local filter update, Kalman filtering is performed independently on each sensor:

[0095] Prediction formula:

[0096]

[0097] Update formula:

[0098]

[0099] in, represents the prior state estimate at time k; represents the state estimate of the local filter (camera, radar, force sensor); P k and Represent the posterior estimated covariance and the prior estimated covariance respectively; K k represents the Kalman gain; H represents the observation matrix; Q represents the process noise covariance matrix; R represents the observation noise covariance matrix, represents the state estimate of the local filter (camera, radar, force sensor); P k and Represent the posterior estimated covariance and the prior estimated covariance respectively; K k represents the Kalman gain, Q represents the process noise covariance matrix, and R represents the observation noise covariance matrix.

[0100] Global information fusion:

[0101] Covariance weighted fusion strategy:

[0102]

[0103] Dynamic weight adjustment:

[0104]

[0105] represents the inverse of the fused global estimated covariance matrix; represents the inverse of the i-th local estimated covariance matrix, represents the global state estimation after fusion, The i-th local state estimate, The dynamic weight of the i-th sensor or local estimate, The inverse of the noise covariance matrix of the i-th sensor observation.

[0106] When the LiDAR is in an open environment, the lidar , when RGB-D improves ω in complex scenes cam .

[0107] The path planning module integrates the SPD-Conv module into the YOLOv9 target detection algorithm to enhance the ability to extract small target features and address the problem of partial occlusion of the bottle. It also detects the bottle's position and, combined with the adaptive step-size RRT-Connect algorithm, improves operational accuracy and plans the path for the robotic arm to grasp the bottle.

[0108] (1) An SPD-Conv module is inserted at each downsampling position of CSPDarknet to replace the original stride convolution; the original structure Conv (k = 3, s = 2) is converted to the new structure: SPD → Conv → Conv (k = 1, s = ) → C.

[0109] (2) Add the SPD feature fusion module to the horizontal connection of PANet, and the formula is as follows:

[0110] P out =SPDConv(Concat(P high ,UpSample(P low )));

[0111] Among them, P high Represents high-level semantic features, P low Represents the underlying detail features, P out Represents the features output after being processed by the SPD feature fusion module.

[0112] (3) Using dynamic feature compensation, through w c =σ(MLP(AvgPool(F SPD )))Calculate the channel weights and generate:

[0113] F final =w c ⊙F SPD +(1-w c )⊙F shallow ;

[0114] Among them, w c represents the channel weight vector, σ represents the sigmoid activation function, F final The final generated feature, F SPD represents the features processed by the SPD feature fusion module, F shallow Represents shallow features.

[0115] Combined with occlusion data enhancement, through the formula:

[0116]

[0117] Among them, A occ Indicates the parameter value after occlusion data enhancement, A base Represents the basic parameters, which are used to calculate A occ The starting value epoch represents the training round; the occlusion area is adjusted dynamically, and the initial A base It started at 20% and eventually reached 50%.

[0118] (4) In the first 80% of the path, the default step size s = 0.1m is used to reduce the calculation time. In the last 20% of the path, the step size is adjusted according to the local obstacle density ρ, and the formula is:

[0119]

[0120] Where ρ represents the local obstacle density, s max =0.15m,s min =0.02m, λ=0.3, k=2.5, ρ0=50, ρ th =30.

[0121] (5) Construct the configuration space of the dual-arm working area and divide the free space C free With the obstacle area, the initialization parameters are: the main arm determines the starting tree T through forward kinematics FK s The root node q s-start =FK -1 (Θ s-current ), determine the target tree T through inverse kinematics IK g The root node q s-gool =IK(T s-target ); Determine q from the arm in the same way s-start With q s-gool , ensuring the coordination of the two arms' goals.

[0122] Random sampling: In free space C free At the same time, a random node q is generated in the main arm s-rand With random node q from arm b-rand ;

[0123] Nearest neighbor search: In the current tree T s or T g Find the same as q s-rand The nearest node q s-near ; Find q from the arm by analogy b-near ;

[0124] Step length calculation: According to q s-near The obstacle density ρ at , the expansion step size s is calculated using the adaptive step size formula;

[0125] New node generation:

[0126] Main arm: along q s-near to q s-rand Generate new nodes in the direction:

[0127]

[0128] From the arm: Generate q in the same way b-new .

[0129] Collision detection: Check the main arm qs-near to q s-rand , slave arm q b-near to q b-rand Check whether the paths collide and the distance between the two arms at the same time:

[0130]

[0131] If d s-b <d s-b (safety threshold) or hits an obstacle, the expansion is abandoned; if there is no collision, an edge connection is added to the tree. Repeat the double tree expansion until T s With T g The nodes are successfully connected to form a discrete path node sequence from the start to the target, which serves as the preliminary motion path of the robotic arm.

[0132] (6) Convert discrete path points into continuous trajectories:

[0133]

[0134] By minimizing the energy function:

[0135]

[0136] Get the control point P i ;

[0137] Among them, N i,p (t) represents the B-spline basis function, i represents the basis function index, p represents the spline order (p=3), P i represents the trajectory control point;

[0138] Dynamic constraint processing: Add acceleration constraint:

[0139]

[0140] The feasible trajectory is solved by quadratic programming (QP).

[0141] The trajectory prediction module is used to fuse TCN and KAN to build a hybrid model, process multimodal data, introduce adversarial domain adaptation (ADA) into the hybrid model, align the trajectory prediction results with the spatial features of solid-state lidar point clouds and RGB-D camera visual data, and obtain an improved ADA-TCN-KAN model to predict obstacle trajectories.

[0142] Model input: state vector Historical Track X traj ∈R T×3 , T = 20 frames, each frame contains 3D coordinates. Solid-state LiDAR point cloud 3D coordinates of N point clouds;

[0143] RGB-D camera vision Contains image size H×W and channels C.

[0144] Multimodal feature extraction: The state vector is mapped to F through a fully connected layer state =FC(x), converted to feature vector

[0145] The historical trajectory uses 4 layers of dilated causal convolution to extract temporal features and output

[0146] The lidar point cloud is processed by the Farthest Point Sampling (FPS) combined with the Multi-Layer Perceptron (MLP):

[0147]

[0148] RGB-D camera data is used to extract features through convolutional neural networks:

[0149]

[0150] Feature fusion processing: The extracted features are fused, first splicing F state 、F traj 、F lidar 、F rgb ,form Then they are fused through a dynamic gating mechanism;

[0151] Use the gating network to calculate the weight g = sigmoid (W g F concat +b g ), then perform weighted fusion: F fusion =g⊙F traj +(1-g)⊙F' concat , F' concat Represents the features after dimension adaptation, and outputs the fusion feature F fusion ∈R T×512 .

[0152] Adversarial Domain Adaptation Optimization: To align feature distributions, we introduce Adversarial Domain Adaptation (ADA) and optimize the adversarial loss through the discriminator D:

[0153]

[0154] in, represents the mathematical expectation of the fusion features, Represents the mathematical expectation of the true feature.

[0155] This process makes the fusion feature F fusion With the true feature F realThe distribution of is more consistent, improving the compatibility of multimodal features and the generalization ability of the model.

[0156] Obstacle trajectory prediction:

[0157] Obstacle trajectory prediction is performed on the optimized fusion features:

[0158] First, through the time series modeling layer F pred =ReLU(W pred F fusion +b pred ) processing, and then output the prediction result through the linear layer The final output Represents the three-dimensional coordinates of the obstacle at each future moment.

[0159] The parameter optimization module optimizes the parameters of the ADA-TCN-KAN model through the improved PRGO algorithm to obtain the IPRGO-ADA-TCN-KAN model, which can avoid obstacles when encountering them.

[0160] (1) Encode the hyperparameters of the ADA-TCN-KAN model (learning rate lr, TCN dilation, KAN head number n_heads) into the fiber root position vector X i = [lr, dilation, n_heads], and set the parameter range; randomly generate N fiber root vectors to form the initial population {X1, X2, ..., X N}, use the model's prediction mean square error (MSE) on the validation set as fitness:

[0161]

[0162] Among them, y t represents the real trajectory coordinates, Represents the predicted coordinates of the model, and N represents the total number of samples.

[0163] (2) Traditional PRGO fixes the probability of taproot / fibrous root system (50% each), which may lead to insufficient exploration or inadequate development. Dynamically adjust the selection probability according to population diversity to enhance the global exploration ability. The dynamic probability calculation formula is:

[0164]

[0165] in, represents the average fitness of the current population, represents the current optimal fitness, and ò represents the smoothing parameter.

[0166] (3) In the population update phase, the algorithm is based on the dynamic probability p t , for each fiber root Adopt different update modes to balance local development and global exploration.

[0167] Taproot system mode (local development): This mode focuses on using the current optimal solution information to perform local fine search and generate a new solution using the following formula:

[0168]

[0169] Among them, X best represents the current global optimal solution, represents a randomly selected fiber root, and a1∈(-0.5,1.5) represents a random coefficient, which is used to control the moving step size toward the optimal solution.

[0170] To break through the local optimal limit, the root system mode uses Lévy flight to achieve long-distance exploration. The new solution is generated by the following formula:

[0171]

[0172] Among them, the Lévy flight step length calculation formula is:

[0173]

[0174] s follows the Lévy distribution,

[0175] Its heavy-tail characteristic gives the algorithm the opportunity to make long-distance jumps. u represents the scaling factor, which is used to adjust the range of Lévy flight. a4 controls the search step size of the root system pattern, and combined with the Lévy flight mechanism, it promotes the algorithm to explore a wider solution space.

[0176] If you interpret Seed in a new way j If the range is exceeded, the boundary is forced to be pulled back. If the new solution is calculated to adapt to the Use Seed j replace

[0177] (4) Input training data and configure the network structure according to the hyperparameters; use the specified learning rate lr for backpropagation optimization; calculate the MSE on the validation set, update the fitness, and repeat steps 2 and 3 until the maximum number of iterations is reached or the fitness converges.

[0178] After the iteration, the fiber root vector X with the best fitness is selected. best , and its corresponding hyperparameters are the optimization results, and the IPGRO-KAN-ADATCN model is obtained.

[0179] The trajectory generation module is used to generate motion trajectories through B-spline trajectory smoothing technology, enabling the robotic arm to grab the bottle, scrape the milk powder, and complete the milk foaming task.

[0180] (1) The multimodal sensor perceives the environment in real time. The main arm uses the improved YOLOv9-SPD to locate the milk bottle and combines the adaptive step size RRT-Connect algorithm to plan a collision-free path. The IMU unit monitors the posture of the main arm in real time and fuses it with the six-dimensional force sensor data to ensure stable grasping of the milk bottle. The slave arm moves synchronously to the milk powder can and locates the opening position through visual guidance. The IMU assists in monitoring the posture of the end of the slave arm and lightly touches the can body to assist in stability. The main arm replaces the end effector with a milk powder spoon and uses lidar to monitor the height of the milk powder in the spoon. Combined with force control feedback, accurate measurement is achieved. The force of the scraping action is maintained within a safe range through PID control. At the same time, the IMU provides real-time feedback on the acceleration of the robot arm to compensate for motion jitter.

[0181] (2) The main arm moves the bottle to the top of the water cup, visually locates the water receiving position, and the IMU monitors the main arm's motion posture and acceleration in real time, assisting with dynamic correction of visual positioning. The slave arm performs a synchronous rotation, and the paths of the two arms are smoothed by the B-spline algorithm to meet dynamic constraints. The distance between the two arms is monitored in real time, and the local path optimization is triggered by combining the movement trend of the manipulator predicted by the IMU. At the same time, the IMU monitors the vibration data during the rotation process and dynamically fine-tunes the rotation smoothness to ensure a smooth shaking effect while avoiding overflow.

[0182] (3) In path planning, after the first 80% of detection is completed in the initialization phase, the environment perception is maintained by relying on the lidar and IMU, and YOLOv9 detection is no longer enabled at a high frequency. The global pre-planning is based on the lidar map and the fixed-step RRT-Connect algorithm is used to generate the initial obstacle avoidance path.

[0183] During dynamic adjustment, the IMU monitors the movement of the robotic arm. If the lidar detects a new obstacle on the path, the fusion system updates the environmental status, optimizes the local path through B-spline smoothing combined with QP, and adjusts the trajectory of the robotic arm to achieve dynamic obstacle avoidance and complete milk foaming.

[0184] A multimodal collaborative intelligent infant care control method includes the following steps:

[0185] S1. Perceive the environment through the RGB-D camera, lidar, and six-dimensional force sensor configured on the robotic arm.

[0186] S2 and the main arm locate the bottle by introducing the SPD-Conv module in YOLOv9 and perform path planning in combination with the adaptive step-size RRT-Connect algorithm to determine a collision-free motion path.

[0187] S3. The slave arm moves synchronously to the milk powder can, locates the opening position through visual guidance, replaces the end effector of the main arm with a milk powder spoon, uses the laser radar to monitor the height of the milk powder in the milk powder spoon, and combines force control feedback to achieve the strength of the measuring and scraping action.

[0188] S4: The main arm moves the bottle to the top of the water cup, visually locates the water receiving position, and the slave arm synchronously rotates to complete the milk foaming.

[0189] S5. If the laser radar finds an obstacle on the path, the updated environmental status of the fusion system is integrated, and the B-spline smoothing technology is combined with the QP optimization path to adjust the trajectory of the robotic arm to achieve obstacle avoidance.

Claims

1. A multimodal collaborative intelligent infant care control system, characterized in that: include: The sensor module is used to collect visual, point cloud, and torque data, including an RGB-D camera, lidar, and six-dimensional force sensor configured on the robotic arm; A data fusion module, used to fuse the federated Kalman filter algorithm with the data collected by the sensor module to build a dynamic environment perception model; The path planning module is used to introduce the SPD-Conv module into the YOLOv9 object detection algorithm to detect the position of the bottle and plan the path for the robot arm to grasp the bottle by combining the adaptive step size RRT-Connect algorithm; The trajectory prediction module is used to fuse TCN and KAN to build a hybrid model, process multimodal data, introduce adversarial domain adaptation (ADA) into the hybrid model, align the spatial features of the trajectory prediction results and point cloud data, and obtain an improved ADA-TCN-KAN model to predict obstacle trajectories; A parameter optimization module is used to dynamically adjust the selection probability according to the population diversity to improve the PRGO algorithm, optimize the parameters of the ADA-TCN-KAN model, and obtain the IPRGO-ADA-TCN-KAN model, which can avoid obstacles when encountering them; The trajectory generation module is used to generate motion trajectories through B-spline trajectory smoothing technology, enabling the robotic arm to grab the bottle, measure the milk powder, and scrape the milk powder to complete the milk foaming task.

2. The tending control system according to claim 1, characterized in that: The RGB-D camera is configured at the end of the robotic arm to obtain information about the bottle and the environment and identify close-range targets; the laser radar is configured on a fixed frame shared by the two robotic arms to assist in positioning and obstacle avoidance; the six-dimensional force sensor is configured at the contact surface between the end effector of the robotic arm and the bottle to measure the contact force and torque between the end of the robotic arm and the environment.

3. The tending control system according to claim 1, characterized in that: The RGB-D camera measures z cam =[u,v,d] T , u and v represent the pixel coordinates of the object in the image plane, and d represents the depth value; Among them, z cam ~N(0,R cam ) represents the noise in the camera measurement process; R cam =diag(2 2 ,2 2 ,0.03 2 ), f x 、f y Represents the focal length of the camera in the x and y directions, c x 、c y Represents the components of the principal point coordinates of the camera image plane in the x and y directions; Represents the coordinate components of a 3D space point in the camera coordinate system.

4. The tending control system according to claim 1, characterized in that: The lidar measures z lidar =[ρ,α,β] T , ρ represents the straight-line distance from the target to the radar, α represents the azimuth angle, and β represents the elevation angle; Among them, z lidar ~N(0,R lidar ) represents the noise in the lidar measurement process; R lidar =diag(0.02 2 ,0.01 2 ,0.01 2 ), p x 、p y 、p z represents the coordinate component of the target point in the Cartesian coordinate system, and ρ represents the distance from the target point to the lidar.

5. The tending control system according to claim 1, characterized in that: The six-dimensional force sensor measures z force =[F x ,F y ,F z ,T x ,T y ,T z ] T , F x 、F y 、F z represents the three-axis components of the end effector contact force, T x 、T y 、T z Represents the three-axis components of the terminal torque; Among them, v force ~N(0,R force ) represents the noise in the measurement process of the six-dimensional force sensor; R force =diag(0.5 2 ,0.5 2 ,0.5 2 ,0.1 2 ,0.1 2 ,0.1 2 ), J(x) represents the Jacobian matrix of the end of the robot arm, g represents the acceleration of gravity, m represents the load mass, I F represents the inertia tensor, ω represents the angular velocity, a x 、a y 、a z They represent the acceleration components of the end of the robotic arm in the three-dimensional space directions respectively.

6. The tending control system according to claim 1, characterized in that: The dynamic environment perception model is as follows: in, represents the inverse of the fused global estimated covariance matrix; represents the inverse of the i-th local estimated covariance matrix, represents the global state estimate after fusion, The i-th local state estimate, The dynamic weight of the i-th sensor or local estimate, The inverse of the noise covariance matrix of the i-th sensor observation.

7. The tending control system according to claim 1, characterized in that: The path planning module includes: Insert the SPD-Conv module at each downsampling position of CSPDarknet to replace the original stride convolution; An SPD feature fusion module is added to the horizontal connection of PANet, dynamic feature compensation is used to calculate channel weights, and the occlusion area is dynamically adjusted by combining occlusion data enhancement; In the first 80% of the path, the default step size s = 0.1m is used to reduce the calculation time; in the last 20% of the path, the step size is adjusted according to the local obstacle density ρ; Construct the configuration space of the dual-arm working area, divide the free space and obstacle area, and initialize the parameters; The master arm determines the root node of the starting tree through forward kinematics FK and the root node of the target tree through inverse kinematics IK. Similarly, the slave arm determines the root node of the starting tree and the root node of the target tree to ensure the coordination of the two arms. The discrete path points are converted into continuous trajectories by minimizing the energy function to obtain the control points, adding acceleration constraints, and solving the feasible trajectory through quadratic programming QP.

8. The tending control system according to claim 1, characterized in that: The ADA-TCN-KAN model improvement process is as follows: Input the state vector, historical trajectory, lidar point cloud, and RGB-D visual data into the hybrid model that integrates TCN and KAN; The state vector is converted into a feature vector through a fully connected layer mapping; the historical trajectory uses a 4-layer dilated causal convolution to extract temporal features; the lidar point cloud is processed by farthest point sampling (FPS) combined with a multi-layer perceptron (MLP); the RGB-D visual data is subjected to feature extraction using a convolutional neural network; The extracted features are fused through a dynamic gating mechanism, the gating network is used to calculate the weights, and then weighted fusion is performed to output the fused features; In order to align the feature distribution, adversarial domain adaptation (ADA) is introduced. The discriminator D is used to optimize the adversarial loss so that the distribution of the fused features is consistent with that of the real features, and the fused features are optimized. The improved time series modeling layer processes the data and then outputs the prediction results through the linear layer, representing the three-dimensional coordinates of the obstacle at each future moment.

9. The tending control system according to claim 1, characterized in that: The parameter optimization module includes: The hyperparameters of the ADA-TCN-KAN model are encoded as fiber root position vectors and the parameter ranges are set; Randomly generate N fiber root vectors to form the initial population, and use the prediction mean square error (MSE) as the fitness; Dynamically adjust selection probability based on population diversity; During the population update phase, each fiber root is updated in different modes based on dynamic probability to balance local development and global exploration: The taproot system mode, i.e., local development, uses the current optimal solution information to conduct a local fine search and generate a new solution using the following formula; the fibrous root system mode, i.e., global exploration enhancement, uses Lévy flight to achieve long-distance exploration and form a new solution; Input training data and configure the network structure according to the hyperparameters; use the specified learning rate for backpropagation optimization; calculate the MSE on the validation set and update the fitness; until the maximum number of iterations is reached or the fitness converges; After the iteration, the fiber root vector with the best fitness is selected, and its corresponding hyperparameter is the optimization result, and the IPGRO-KAN-ADATCN model is obtained.

10. A multimodal collaborative intelligent infant care control method, characterized in that: The following steps are involved: S1, perceives the environment through the RGB-D camera, lidar and six-dimensional force sensor configured on the robotic arm; S2, the main arm, locates the bottle by introducing the SPD-Conv module in YOLOv9 and combines it with the adaptive step-size RRT-Connect algorithm for path planning to determine a collision-free motion path; S3. The slave arm moves synchronously to the milk powder can and locates the opening position through visual guidance. The master arm replaces the end effector with a milk powder spoon. The laser radar is used to monitor the height of the milk powder in the milk powder spoon. The force of the measuring and scraping action is achieved in combination with force control feedback. S4: The master arm moves the bottle to the top of the water cup, visually locates the water receiving position, and the slave arm synchronously rotates to complete the milk foaming; S5. If the laser radar finds an obstacle on the path, the updated environmental status of the fusion system is integrated, and the B-spline smoothing technology is combined with the QP optimization path to adjust the trajectory of the robotic arm to achieve obstacle avoidance.

Images